[announce] c64_tools (DSP loader and IPC)

[Askarus]

You're writing so much lines because of 4 MB of RAM.

It's nothing compared to what we have available and what we might gain from the DSP.

Just use reserve it for DSP and don't worry about it.

 

[_jr_]

As notaz said: a usable interface to the DSP was available all along and nobody used it. Why should it change now? The lowerlatency is nice, but 4MB wasted is still 4MB too much. I think it is only fair to wait for the appearance of more than a test app. bsp's work wouldn't be wasted (or underappreciated) since it is already quite possible to develop for his system and a large percentage of pandora users is quite capable of making the neccessary changes to their start scripts to use it.

 

[milkshake]

chicken and egg though... if its in their by default more developers will consider it since it should reach most of the install base.

 

[Linux-SWAT]

With some little working examples, i'm sure that more pple will get attracted.

Until now, even with the "old" DSP module enabled, it still sounded like mystical stuff.

 

[milkshake]

Yes but the default boot doesn't have dsp enabled by default... See where I'm going with this?...

 

[bsp]

@_jr_: have you actually worked with the DSP on the Pandora before ? The first thing I tried was DspLink (downloaded straight from TI). As usual (sorry, always had bad experiences with this software and I am not the only one), it did not build and install out of the box (had to spend quite some time to examine its build system, add/edit paths, manually copy header files after it finally build, ..). Then, it caused kernel crashes during the very first API calls (long before a DSP image was loaded). When you are working with it in a professional environment and you have direct support from TI, well, then it works OK (apart from being slow). But I do this in my spare time so I decided that instead of wasting time with that software again, I'll write my own. Besides, while developing the c64_tools I noticed a hardware bug in my Pandora (maybe other units are affected by this, too), that would have prevented DspLink or DspBridge from working anyway. Last but not least: The c64_tools is a ready-to-use package that needs very little setup time. Much more hobby-developer friendly if you ask me. The improved performance (the next update will introduce a new feature that will outperform DspLink by factor 1,000,000, not kidding), is just a nice side effect. Lastly, about the 4 MBytes debate: Take a look at how much memory is wasted by ARM-side processes (e.g. just the xfce power manager application uses a bit more than 4MBytes, the panel >8 Mbytes, ..). I don't see any discussions about that (and I am not complaining, just saying). If you were truly concerned about waste, you would see that there is plenty of room for memory-optimization elsewhere. The DSP just needs a couple of MBytes at a fixed memory address, no way around that as it seems.

@Linux-SWAT: what kind of example are you missing ? are the existing ones not enough to get started ?

 

[rohezal]

I would love to see the dsp used . Sound emulation (like in Wine + qemu passthrough) or some calls can be handled by the DSP. Maybe even the x86 float stuff, collect 1000 float unconnected operations, copy them to the dsp, do the rest via cpu and let the cpu and dsp wait for each other. 1000 independent floating point operation sounds a lot, but you have to do 16 float operations per vertex for transformations, which was done on the cpu in the old time.

 

[Linux-SWAT]

Well, i didn't went far, i just also tried to compile DSP stuff from Ti some time ago, and failed. I didn't even look at the examples.

What kind of examples did they ship ?

 

[_jr_]

Would you be surprised if I told you that I use xterm (without window manager) as the main gui


And I didn't want to imply that your work is not an improvement. Only that with the pandora user base inclusion in the default firmware is not neccessary for adoption. Please excuse me if it sounded disrespectful.


edit: but I don't have a problem having it enabled by default, since I can just disable it util I need it

 

[milkshake]

Not needed but would help if users didn't have to enable it.

 

[rohezal]

Not needed but would help if users didn't have to enable it.

Well I would enable it by default and let users disable it when they don't want it. Like the zram feature.

 

[milkshake]

Yeah that sounds like a good option

 

[Askarus]

I'd kick out that zram stuff and get DSP instead.

 

[milkshake]

Zram gives you more ram though?

 

[notaz]

Users that have their own boot scripts hopefully know what they are doing and would need to add the required options themselves. Since they've already figured out how to use custom boot scripts in the first place, that should not be an issue (it's a very simple modification, after all).

Not really, as they usually use SD creator .pnd from the repo.

These 4 MBytes would not be lost after all: If the DSP is not used, the memory could still be used for applications, although there would be some special handling required (an extra mmap() call in the app., I could add that to the kernel module so the app. does not require root privileges).

We really don't want that, this would add unwanted dependencies to your module and perhaps firmware version and create unnecessary obstacle for backward compatibility, should we want to do it on some future device.

After reading that you might have seen something that would allow for such a fixed-address mapping, I took another look at "dma-mapping.h" and "dma-coherent.c". There is a function called "dma_declare_coherent_memory()" which looks like it could be used for this purpose. I tried it out, but it always results in a kernel oops (17) / "kernel NULL pointer dereference at virtual address 00000114".

I've looked at it too and it seems to be possible with some board support code changes in kernel itself. Basically board code declares CMA region at fixed memory address during early boot and your module can then request it by passing proper dev pointer when it loads. Your module would need to be rewritten as platform driver though, this is for dev pointer magic to work.
If you are not building your own kernels I can do board code portion if you like and build it for you.

 

[notaz]

Yes but the default boot doesn't have dsp enabled by default... See where I'm going with this?...

FYI dspbridge is available and enabled by default for over a year now..

 

[bsp]

@Linux-SWAT: yep, that's what I mean. Not easy to use. The examples that come with DspLink are quite simple, though (which is a good thing because that's what examples are meant to be)

@_jr_: no offence taken. I hope I did not sound angry, but there were good reasons why I wrote this (i.e. not just because of the "not-invented-here" coder syndrome / to re-invent the wheel ) Heh, I am also not surprised that you prefer xterm. I often find myself using the terminal rather than the UI, bash aliases "ftw"

The reason why I advocate a 4 MB memory 'gap' is that this should be enough even for rather complex DSP applications. There are not any right now, but in case there will be, no further boot option adjustments will be necessary to run specific applications (-> user friendly, no need to even know about these kind of details, power users can disable the gap if they really do not need it. I guess (not only) milkshape understands what I mean).

Right now the c64_tools example DSP image uses ~24 KBytes for code (including DSPBIOS), and ~4 KBytes for data. We could settle for a 2 or even 1 MByte gap but this may lead to problems later on.

So, there's no immediate need for this, I do not even know when the Pandora team is planning to release a new firmware, could be months until then.
This is just something that should be considered before a FW update is rolled out.

What's really needed is some kind of "killer application" for this. E.g. what rohezal said sounds reasonable (QEMU float emulation). Personally, I am thinking about moving the SID emulation of a C64 emulator to the DSP, or the SPC emulation of a SNES emulator but I will have to take a closer look at those emulators, first (I hope the sources are available?! different topic, though)

(EDIT: C64/SNES are maybe not the best targets since these emulators hardly use more than 50% CPU on a Ghz Pandora, just had a look via 'top')

@Notaz: (almost posted the message then saw your reply): You are more proficient at Linux kernel internals than I am so if you could do the necessary changes to the kernel and the c64 module and post a patch, I will gladly try it out / adapt the rest of the software when it works (I have everything set up to compile new kernels).
 

 

[rohezal]

How much does the overhead "cost" to execute an operation on the dsp? A floating point operation costs around 300 lines of C code in qemu (not sure how many instructions this means). Is it ok to let the dsp execute one floating point instruction? Or do we need to catch lets say 1000 instructions and give a pointer to the data struct (opcode param1 param2) to the dsp?

struct floatingPointOp

{

 int opcode;

 float param1;

 float param2;

} flop;

 

[Exophase]

How much does the overhead "cost" to execute an operation on the dsp? A floating point operation costs around 300 lines of C code in qemu (not sure how many instructions this means). Is it ok to let the dsp execute one floating point instruction? Or do we need to catch lets say 1000 instructions and give a pointer to the data struct (opcode param1 param2) to the dsp?

struct floatingPointOp

{

 int opcode;

 float param1;

 float param2;

} flop;

Before going further on with this I think you should understand that the DSP on Pandora doesn't even do floating point, while the CPU at least has a slow scalar FPU and a fast vector one (but not totally IEEE-754 compliant)

I don't know what 300 lines of C code translates to in terms of instructions and cycles on Cortex-A8 but regardless it sounds like it's being done really inefficiently.

You're never going to be able to use the DSP to meaningfully accelerate something as lowly as a floating point instruction, and you're not going to be able to capture lots of them before you must stall due to the possibility of a dependency. DSP is only going to be useful for emulation if you have really big or mostly autonomous tasks, not things that rely on quick round-trip communication.

What's really needed is some kind of "killer application" for this. E.g. what rohezal said sounds reasonable (QEMU float emulation). Personally, I am thinking about moving the SID emulation of a C64 emulator to the DSP, or the SPC emulation of a SNES emulator but I will have to take a closer look at those emulators, first (I hope the sources are available?! different topic, though)

(EDIT: C64/SNES are maybe not the best targets since these emulators hardly use more than 50% CPU on a Ghz Pandora, just had a look via 'top')

It's like you say, those emulators are fast enough as is on Pandora and not really worth the complexity of moving tasks to the DSP for. Even from a power consumption point of view I'm skeptical that you'll use less power moving a little bit of the runtime to the DSP - it may result in saving a little bit of clock and voltage on the CPU end but just having to keep the DSP on at all probably comes with a big chunk of static leakage power.

Maybe SPU or even GPU emulation of PS1 could benefit, even though that's "fast enough" in most cases the high-resolution mode still pushes it over sometimes. I guess the biggest killer app would be GPU emulation for DS. In theory I could see a highly optimized renderer being faster on the DSP than on the CPU, at least if you had a fast enough solution for generating reciprocals. But it'd be very hard to optimize it to that extent.

 

[milkshake]

Any examples on where it could be used to improve any software or emulators specifically?