[announce] c64_tools (DSP loader and IPC)

[bsp]

Hi all,

 as promised here it is.. the first version of my DspLink replacement, the "c64_tools" package.

 It does not have as many bells and whistles as DspLink and rather tries to be "lean and mean".

 That means that it does not have message queues, RingIO, and what not.

 It *does* have some advantages though:

  1) It actually works (DspLink just gave me a kernel crash in PROC_attach(), could be fixed, maybe, I'm not interested.)

  2) It is a lot faster.

 It should support multiple clients/processes (implemented but not tested so far).

[EDIT 23Sep2013: this has become quite a lengthy thread. if you are looking for the latest release of c64_tools, point your browser here: http://tkscript.de/c64_tools]


 The package contains the following main components:

   c64_load -- A COFF2 loader that loads .out files (DSP images) created with CodeComposerStudio and boots the DSP (GPL).

   libc64 -- A GPP side utility library. Currently it is mainly used by 'c64_load' (LGPL).

   c64.ko -- A kernel module that handles messaging between the ARM and the C64+ DSP (GPL)
              (tested with 3.2.45, i.e. the SuperZaxxon 1.55 standard kernel)

   go64.sh -- A utility script that loads the kernel module (optionally with debug output), and loads/boots the DSP image.

   libc64_dsp -- A DSP side utility library that mainly handles the DSP-side messaging part (MIT). It also defines
                  a simplistic remote procedure call (RPC) interface.

   tests/ -- ARM-side testcases. Currently just "test_hello" which can send a single 4 byte message (when called w/o args),
              perform a single RPC (when called w/ two args), or run an RPC benchmark (when called w/ three args) (takes a while!).


 Architecture-wise, the messaging interface boils down to simple remote procedure calls:

 The DSP side registers a number of "components".

 Each component has an init() (optional), exec() (required), and exit() (not used ATM) function.

 Each component can also have an arbitrary number of "commands".

 Each command takes two 32bit input parameters and can return up to two 32bit output values.

 I included an example component that is basically a simple calculator that can add/sub/mul/div two signed integers.


 Trust me, this thing was a nightmare to code because the "mailbox" subsystem in my Pandora did not behave like the TI docs said it should.

 For example, even after all incoming mailbox messages are read and the right bits in the IRQSTATUS_0/1 register were set in the IRQ handlers, that thing refuses to trigger any more "newmsg" interrupts. Sometimes it generates a flood of interrupts and spurious mailbox messages.

 Figuring all that out (without a debugger/JTAG emulator!), and implementing workarounds for that, really took quite some time (I spent almost every evening during the past week working on this! and I'm talking long evenings! and entire weekends. gaah!!)

 The next issue was cache coherency between the ARM and the C64+.

 I knew there was a problem, either on the GPP or the DSP side. Tried a million things, failed a million-1 one times.

 It turned out that when the L2 of the DSP cache is enabled, the DSPBIOS BCACHE_wb*() calls do not work as their documentation says they should (cache lines are sometimes not written back).

 The only solution was to disable the L2 cache.

 Well, I think we can live with that (for now) -- the DSP has 32k 1st level data + 32k code cache so hopefully that will not have a large performance impact (given that each "number-crunching" loop fits into the L1P cache and EDMA transfers are used to stream working buffers in and out of the L1DSRAM (48k available, IIRC). (the L1DSRAM offers single cycle access, in case you did not know).


 Pheww.. I'm quite exhausted and the weekend is almost over. What the hack


 In case I get hit by a meteoroid tomorrow, please find attached the current sources (and precompiled binaries).

 There's still some work to be done:

   - add utility functions to allocate physically contiguous shared memory (planned that for today but everything took longer than expected)

   - improve kernel module so that it supports select() to wait for reply messages.

   - figure out whether the L2 cache issue is a HW problem or a bug in DSPBIOS, and maybe find a solution for that (any help is appreciated!)


 Last but not least: Here are the benchmark results (1,000,000 RPCs ("add" command of the "demo_calc" component)):


go64.sh: ARGS="" DSPIMAGE="./c64p_simple_dm3730.out"
[...] loading DSP image from "./c64p_simple_dm3730.out"
[dbg] coff_open: file size=212637 (0x00033e9d)
[...] reading 52 section headers....................................................
[...] initializing DSP memory and copying sections..
[...] text=0x00006080 data=0x000012df bss=0x00704368
[...] DSP image loaded successfully. starting DSP..
[dbg] waiting for DSP startup..
[...] DSP is up and running!
[ 686.469329] c64: init ok. module build date: Sep 15 2013
pandora:/home/bsp# ./test_hello ; dmesg -c
[dbg] DSP numRead=4, debug=0x07000002
[...] write testmsg: 4 bytes written.
[dbg] DSP numRead=4, debug=0x37010007
pandora:/home/bsp# ./test_hello q; dmesg -c
[dbg] DSP numRead=4, debug=0x37010007
[dbg] DSP numRead=4, debug=0x77005607
[...] log_rpc() OK: result=86 (expected 86)
[dbg] DSP numRead=4, debug=0x77005607
pandora:/home/bsp# ./test_hello q l; dmesg -c
[dbg] DSP numRead=4, debug=0x77005607
[...] starting calibration
[...] starting benchmark
[...] benchmark finished. => 1000000 iterations in 73312 millisecs.
pandora:/home/bsp#


Configuration:

[ ] VID2 hack (uses VID scaler regs as msgbuf)
[ ] flush_cache_all()
[x] DSP L1D
[ ] DSP L2D


==> 0.073312 millisec messaging overhead per roundtrip/RPC.
==> 13640.33 roundtrips/RPCs per second.

==> ~46.7863 times faster than DspLink (based on M-HT's measurement)

Last but not least, this could be improved by a 1000 times if it were possible to access the DSP's L1DSRAM on the GPP side. I tried (I used that method once before on a similar TI board running QNX), but on the Pandora that memory is not accessible (maybe that could be fixed? A linux issue ? anyone?).

Greetings,

Bastian
 

p.s.: In case you are wondering what the "VID2 hack" is: Out of desperation when trying to fix the cache coherency issue, I used the VID2 layer scaler registers of the DSS for IPC / as a message buffer. That did not fix it (interesting, I thought peripheral I/O is always uncached..). So don't worry, that hack is not needed.

p.p.s.: In order to compile this, you must set the following vars: "CROSS_COMPILE=arm-none-linux-gnueabi-" and "CROSS_ROOT=<root of pandora fs>" (must contain usr/src/pandora-kernel/)

p.p.p.s: precompiled binaries can be found in the "bin-15Sep2013/" directory, if you just want to try it out (please do and let me know whether this works on your HW!)

p.p.p.p.s: I almost forgot: put the attached "autoboot.txt" in the root folder of your left SD card (changes some kernel boot params to reserve memory for the DSP)

 (if you have a "classic" 256 MBytes Pandora, use the CLASSIC__autoboot.txt and rename it autoboot.txt, of course!)

p.p.p.p.p.s: by "MH-T's measurement" I am referring to this thread: http://boards.openpandora.org/index.php/topic/10788-running-scalers-on-dsp/ (I have previously worked with DspLink and the overhead stated in that thread sounds reasonable)

c64_tools-15Sep2013_dist.tar.gz

autoboot.txt

CLASSIC__autoboot.txt

c64_tools-23Sep2013_dist.tar.gz

 

[notaz]

Interesting stuff.

I think MH-T was using dspbridge, which is different than dsplink.

If I understand you right, you were able to send ~13640 "add" commands per second, which includes retrieving the result? If so, that's quite nice.

I think MH-T had image transfer times added into his calculations, and that involves cache flushing/invalidation on ARM (and perhaps DSP?) side, and at least on ARM it's slow as cache operations involve walking the whole virtual address range (that the image takes) in cacheline steps and issuing multiple slow MCR instructions on each step.

I've had some ideas like using write-through cache on ARM side, so that cache cleaning isn't even needed, or cleaning the whole cache instead of walking the virtual address range, but never got around doing actual work on this.

 

[milkshake]

What the he'll does all this mean? That the pandora finally can use the dsp? And could it actually be useful for increased performance of any of the pandora software? Video decoding? Etc

 

[notaz]

BTW, current pandora kernel has CMA (contiguous memory allocator) patches merged, so you should be able to allocate large continuous regions through the kernel dma_alloc() API (it will relocate memory pages in case there is fragmentation). This way no special autoboot.txt and memory wasting (while DSP is unused) should be needed, you'd just need to coordinate the allocated physical address range with the DSP.

There a MMU on DSP side, so supposedly you could even map normal Linux memory pages on DSP address space (like the GPU already does).

 

[bsp]

@milkshake: Yep, I hope that's what it all means

@notaz: sending the "add" commands includes retrieving the result, of course. In my past experience DspLink usually took >2ms for an ARM<>C64 message roundtrip (i.e. sending a message and receiving the reply). The Nokia folks (who are behind DSPBridge, AFAIK), also implemented a dynamic loader for C64 code, that's sth my loader does not support but I don't think it's necessary for the Pandora. Thanks for the hint about the dma_alloc() API, I'll look into this (I'm not a kernel guru so help on such topics is really appreciated!). On the other hand, while working on this project, I have noticed that sometimes mmap() in both user and kernel land (ioremap_nocache()) returns addresses that cause crashes on access, that's why I'm leaning towards reserving a larger chunk of memory early on. I know that the DSP has an MMU and I should have included that in the "work to be done" part. Unfortunately it just supports 16 entries but that should be enough to protect the GPP OS from misbehaving DSP code. What takes up a lot of memory (->waste) is the data/pool area. This could be configured dynamically -- what matters to "c64_load" and the "c64.ko" kernel module is the reset_vector address and the DSP code/data/bss area (currently ~8 MBytes although that could be less, say 4 MBytes (the example DSP image requires only a few dozen KBytes)). Maybe we should agree on this ? In previous projects, the DSP image was almost 8 MBytes in size but that included complex renderers, voice recognition software, .. i.e. maybe we should reserve at least 4 MBytes for the DSP and have different memory configs (selectable via uboot or autoboot.txt) for applications that need a lot of shared memory ?

p.s.: using dynamic memory allocation for the pool/data area could also lead to situations where enough (virtual) RAM is free on the GPP-side but not enough physically contiguous pages are available. (e.g. when the system is under load)

 

[Ian J]

What the he'll does all this mean? That the pandora finally can use the dsp? And could it actually be useful for increased performance of any of the pandora software? Video decoding? Etc

+1

This is all gobbleaduke to me.

Damn Ninja'd h34r:

 

[notaz]

I still don't understand why you want boot-time memory reservation when you can just alloc during init of your kernel module using the CMA. Are there some limitations for DSP reset vector placement or are the DSP binaries non-relocatable or something like that? Note that dspbridge can work (and run dsp BIOS) without any boot-time memory reservation at all.

mmap() shouldn't return you bad addresses unless you're running out of memory. mmap() is used all over the place in the OS and various emulators, so many things would go bad if it didn't work. Note that it's not supposed to give you continuous physical memory (unless you map /dev/mem), you have to use dma_alloc() in kernel for that.

If you have specific kernel related issues or questions, feel free to report and I can try to help to debug or advice.

 

[Linux-SWAT]

Far above me, but sounds like a breakthrough thanks to a lot of work !

 

[bsp]

@notaz: regarding the mmap() issue: in my COFF loader I originally mmap()'d each section separately. Occasionally that caused a kernel crash (resp. page fault) when writing to that memory. My workaround was to map the entire DSP code/data area early on, instead of many (52) individual mmap() calls. To answer your questions: the DSP binary is not relocatable (CCS creates an absolute memory map). There is no limitation on the reset_vector placement -- I just chose a config that I hoped would match the "boot with memory cut for DSP" config (which does not work in SZ1.55, 1Ghz edition, btw (black screen of death)). But hey, for now I'm just glad that the IPC works -- memory allocation is a different topic and I will try the dynamic allocation approach you suggested (if it works that would be great, I'm not fond of waste, either).

@Linux-SWAT: thanks, that rly was a lot of work but I hope that this will finally encourage devs to use the 2nd processor (did you know that there also is a 3rd processor, an ARM926, in the Pandora? TI just did not document that one)

p.s.: in case anyone is eager to start hacking: you can use the libc64 dsp_mmap() function to allocate shared memory but for now you need to run your app as root. The pool currently is statically located @0x87000000, size=0x01000000 (16 MBytes). Remember to send physical addresses to the DSP if you use pointers.

 

[Linux-SWAT]

Wow, the OP is a triple core !

What is its original purpose ? How is it accessible ?

 

[bsp]

I have no idea - go ask TI   could this be the equivalent to the 1541 floppy drive coprocessor in the C=64 world ? who knows (the 6510 maniacs actually used that one to speed up e.g. vector/matrix math in demos -- d'oh!)

 

[Linux-SWAT]

http://processors.wiki.ti.com/index.php/Feature_Comparison:_ARM_926,_1136_and_Cortex-A8

 

[Askarus]

Might this help for emulators as well?

I'm thinking of PSP and N 64.

 

[Exophase]

Might this help for emulators as well?

I'm thinking of PSP and N 64.

I doubt it'll help with PSP or N64 very much. Maybe a little if they move audio emulation to it.

Could help more for DS if the 3D emulation is handled on it, but that's a ton of work (plus, with DraStic right now the multithreaded 3D is kind of glitchy)

 

[M-HT]

==> ~46.7863 times faster than DspLink (based on M-HT's measurement)

I used dspbridge in my testing and not DspLink.
In your calculation you are using 3.43 ms as my overhead of one RPC call. This number includes flushing/invalidating caches. Later in the thread I commented out all cache invalidating/flushing and the result was 0.7 ms (I don't remember if I commented it out only on the ARM side or on both ARM and DSP sides). I was also using DSP for Dummies framework, which might have increased the time compared to what you're using.

If your way of calling the DSP is faster, that's good, but I think that the main problem with the DSP is finding some functionality to run on the DSP. Either existing functionality that can be offloaded to the DPS to free the main processor, or some new functionality that can be added to programs at low overhead cost.

 

[ibisum]

What this means: we can use the OP to make a really, really nice synthesizer!  YAY!!

And that 3rd Processor, the ARM926: WTF?!!   How can we put it to use?  

(Following this thread with *great* interest .. I have a few DSP synths I could port if this is working smoothly..)

 

[EvilDragon]

Hm, still wondering... could the DSP be used to create some CRT-Like effects for the emulators and games?

 

[notaz]

IIRC ARM926 only has access to DSP's RAM and not the main memory, so it would be extremely difficult to use it for anything (it's original purpose is for controlling TI's super-secret fixed-function video decoding blocks, it's kind of like ARM940 on GP2X but much more limited).

 

[Binky]

Could the ARM926 be used to handle battery charging while the main processor sleeps?

 

[bsp]

@M-HT:

Ok then.. sounds like DspLink has improved a bit over the years (it's been 5 years since I last had to use it on Linux and back then it was more in the >2ms per roundtrip range, IIRC. On QNX the messaging overhead was definitely quite high (in lock-step with the 1000Hz OS timer).

10 times faster is still good, I guess.

Last but not least, sending RPCs/triggering interrupts is not the only possible way to communicate with the DSP.

A program could use spinlocks for processor-heavy but very fast synchronization (e.g. to offload smaller functions to the DSP).
Probably the best way to do that would be to use parts of the L1DSRAM to avoid cache flushes.

As I said before, I do not know whether it is a HW limitation of the OMAP that the ARM cannot access the DSP SRAM, maybe Notaz can comment on that?
(update: while still writing this mail I opened the c64p ref. guide and found out that there are memory protection registers which can be used to grant the CPU access to the L1D -- will have to investigate!)

Speaking of caches: One big issue, depending on the kind of application, could be that the memory shared with the DSP is always uncached on the ARM side.

This makes it virtually impossible to effectively operate on the same dataset with both processors (not the same pages/cachelines, of course,
although that would even be possible if both ARM and DSP merely read from certain areas).

(as a practical example: picture an application that wants to render graphics with both the DSP and the GPP. Both processors would need read-access to maybe even the same textures, or need write-access to the framebuffer(s) (in turn, of course).)

Maybe it would make sense to offer an ioctl() in the driver that invalidates or writes back the ARM data cache for a selected area.
Some of the kernel functions in arch/asm/include/asm/cacheflush.h look they could be used for this.
On the DSP side DSPBIOS has the BCACHE API which applications can use for that purpose. The DSP also has the MAR registers which control which memory regions are cacheable. Unfortunately the granularity of these is 32 MBytes.

Well, just an idea.

About finding applications for the DSP: anything involving heavy calculations on streaming data with as little branching as possible would fit the bill.
E.g. vector/matrix multiplications, graphics blitters, video decoding, audio processing.

IMHO, the biggest "problem" with the DSP is that it is a) quite difficult to program (if you want to get good performance) and B) it is very hard to debug. Maybe (or especially??) also the afore-mentioned c) achieving effective ARM/DSP parallelization.

Using TI's cycle-exact C64 simulator for developing DSP software is probably the sanest thing to do (unless you can afford a JTAG emulator, these things are expensive!)

But -- like the "DSP for Dummies" page also stated -- the DSP is currently not used at all so even running plain C code with zero or little DSP-specific optimizations would already be a step forward.




@torpor: +1! looking forward to your synths! do you have any sound samples available ?


@ED: the text above should answer your question. So yes, it could be used for that. E.g. an emulator could, maybe even in collaboration with the DSP, render the graphics for one frame, then writeback the ARM caches for the framebuffer area, call the DSP and let it apply a post processing effect while the ARM calculates the next soundbuffer fragment(s).

Alternatively, depending on the emulator, the entire sound emulation could be offloaded to the DSP but as far as my (limited) knowledge of emulator goes, this wouldn't be practical unless there is little interaction with the main (emulated) CPU.

Maybe the SNES sound hardware would be a candidate for that -- AFAIK most games just upload data to the SPC and let it do "its thing". I read that there are some games who are constantly updating the sample data, that would require extra syncs.


@notaz:

I took a look at the CMA today. It sounds very promising and if I understood correctly, it should be able to re-arrange pages to form contiguous regions when the memory is fragmented.

Today I rewrote the c64 kernel module so that it uses the "modern" way of creating devices (via udev). That worked fine (no more mknod, yea!).

However, I then tried out the CMA API, in particular dma_alloc_coherent() and dma_free_coherent().

There seems to be an issue with the alloc function: Calling it results in an "Oops - undefined instruction 0 [#1]".

I then tried this http://article.gmane.org/gmane.linux.kernel/1263136 simple CMA test module (the misc device failed to register but adding ".minor = MISC_DYNAMIC_MINOR" to the miscdevice struct initialization fixed that).

That example also crashes with the same kernel Oops.

Would it be possible for you to take a look at this issue ?

(EDIT: I've attached the kernel crash log, in case that helps. I also tried adding "cma=16m" to the bootargs, which resulted in crash log "b")
 

EDIT: I just took a look at TI's CMEM source and saw that it now supports CMA. I also saw that CMEM can call dma_alloc_coherent() with NULL as a device parameter, in which case the "global area" (according to a source comment - is that OK ?) is used for allocation. Tried that, seems to work (at least it returns an address and does not crash). So, hurrah!

Seems that the CMA example code I used for reference (see link above) is not valid anymore.

The CMEM source is GPL btw, and I see no reason why this could not be used for memory allocation, along with my "c64_tools".

I will still add a minimal set of API functions for shared memory allocation and cache management to the kernel module and libc64.

Anyways, that's it for today.

kernel_crash_log-16Sep2013.txt

kernel_crash_log-16Sep2013b.txt