[announce] c64_tools (DSP loader and IPC)


Ok, that was a tough one.

But good news everyone: The overlay loader is working very nicely. It was more work than initially expected but so be it.

What really kept me busy the past 6h was that I noticed that the fastcall RPCs did not work as expected under certain conditions (DSP/ARM not seeing the same data sometimes).

My solution was to move the "todsp" and "togpp" message 'registers' (in L1DSRAM) to two separate cachelines.

"todsp" may only be written (and read) by the GPP, "togpp" only by the DSP.

This also meant a rewrite of the fastcall protocol, of course. Fortunately, this did not slow things down (0.002166 millisec / RPC).

Seems like the L1DSRAM is not as tightly coupled as I initially thought (or is this a hardware bug??) it's not a documented feature, after all.

But well, this works so far (did a lot of stresstests). All DSP example components have been moved into separate .out files. Up to 5 DSP image .out files can now be loaded/used simultaneously, yay!

And Notaz will be pleased to hear that I went the extra mile to add an "emergency unload" feature to the kernel module which, when fops release is called, unregisters all components registered by the overlay images loaded by the app, in case an application crashes w/o doing that the proper way. So, no need for ugly/mandatory application parachutes / signal handlers -- yay again.

Damn, at times this was a real nightmare to code but I guess we're close to "achievement unlocked -- the DSP!" :)

I'll edit this message later today and create a new release. Just too tired now (4:40am..). But kindof happy ;)
 

EDIT: New release available at http://tkscript.de/c64_tools/

Code:
** Changes

 05-Oct-2013
   - added: DSP auto suspend / resume (in suspend, DSP consumes no power).
             The DSP is automatically suspended when the last application using it quits.
             Vice versa, it resumes execution / is restarted when the next app. connects.

   - added: support for dynamically loaded overlay images (4*256k + 1*1024k).
             Overlays are unregistered when dsp_close() is called and their refcount becomes 0.
             In case an application crashes, an "emergency unload" mechanism in the kernel module
              takes care of this to prevent DSP code areas from getting lost.

             The DSP build system automatically creates 4 .out file variants for the 256k areas
              (linked to different base addresses).

             Which of these 4 variants is loaded when dsp_component_load() is called depends on
              what area is available / not already in use by other applications.
             "demo_calc", "demo_checksum", "demo_calc_fastcall" are examples for "area2" (256k) overlays.
             "demo_fractal" is an example for an "area3" (1024k) overlay.

   - added: DSP-side global syscalls table. Currently includes some c64_tools utility functions,
             cache management utilities, and mem/string handling functions.

   - changed: DSP components no longer require DSP/BIOS (=> considerable decrease in code size)
               For DSP component development, the following (free as in beer) TI packages are required:
                $HOME/ti/c64xplus-iqmath_2_01_04_00
                $HOME/ti/fastRTS_c62xc64x_1_42
                $HOME/ti/TI_CGT_C6000_7.4.5

   - changed: cleaned up DSP build system and moved common .mk code into include files

   - changed: cleaned up "core" tconf and split it into multiple .tcis (see dsp/tci/ folder).
               "config.tci" is the central memory configuration file.
               It is included by the new gen_link_areas.tks script, which generates the DSP linker
                command files (.cmd) and the "overlay_sections.c" source, which is also included
                by the GPP-side build.
               Unless you want to change the memory layout, you do not need to worry about this, though.

   - fixed: added mtx_clients mutex to kernel module to fix some (potential) multiprocess concurrency issues

   - changed: partially rewrote fastcall protocol to use two separate ARM/C64+ cachelines.
               A more complex handshake protocol is used now but fortunately that
               did not have a negative effect on the performance
 
Last edited by a moderator:
This is one of the threads on the boards I am currently most excited about. Not necessarily because there is a burning purpose I'd like to see this used for, just because it is very cool, plus I appreciate the amount of time going into it. Looking forward to seeing the first apps/games using the DSP! :)
 
*bump*

and here's the release (see edited post on top of this page).

I had to make a last-minute change: I noticed that running e.g. the RPC stresstest (./c64_tc 3) in parallel multiple times had the unfortunate effect that once the first stresstest finished, it unloaded the overlays which made the other stresstest processes fail. d'oh! So I just added a (DSP-side) reference counter mechanism which should prevent this situation. At least the tests run fine now (tested w/ 8 in parallel).

This will not be the last update to the c64_tools but I hope that I can focus on DSP library / component development now.

The next update will add EDMA3 functions to the syscalls but this may take some time.

@ekianjo: very good idea. I'll do that now :)

@Steven Craft: Yes, me too. There should not be any reasons to _not_ use the DSP now. At least it probably will not get any easier than this. :)
 
Hey!  I thought you've just recovered from a nasty cold.  Well, I guess this awesome DSP work of yours has warmed you up :)

I hope devs like Exophase and Notaz can use DSP to add some cool filters to PCSX-ReArmed :)
 
Last edited by a moderator:
And Notaz will be pleased to hear that I went the extra mile to add an "emergency unload" feature to the kernel module which, when fops release is called, unregisters all components registered by the overlay images loaded by the app, in case an application crashes w/o doing that the proper way. So, no need for ugly/mandatory application parachutes / signal handlers -- yay again.
\o/

- added: support for dynamically loaded overlay images (4*256k + 1*1024k).
...
The DSP build system automatically creates 4 .out file variants for the 256k areas
(linked to different base addresses).

Which of these 4 variants is loaded when dsp_component_load() is called depends on
what area is available / not already in use by other applications.
Hmm so any program wanting it's own DSP side code will have to ship 4 versions of compiled binaries from the same DSP code?


Not a huge problem but makes me wish relocation would exist..

BTW you keep mentioning the blitter engine you did for your work that you can't reuse.. Did it actually work out well and reached the project goals? I.e. was the DSP good enough for the work?
 
Hmm so any program wanting it's own DSP side code will have to ship 4 versions of compiled binaries from the same DSP code?
not necessarily. applications like games or emulators should use "area3", i.e. only one .out image needed for that.


(unless you want to run multiple games/emus simultaneously, each with their own DSP code, which is very unlikely)


"area2" is meant as a development / staging area for shared DSP code like video decoders, audio engines etc.


these things should be moved to the (core) "area1" as soon as they are stable.

Not a huge problem but makes me wish relocation would exist..
technically, the TI tool chain can produce ELF files (a relatively new feature, though) but debugging the DSP is hard enough as it is and being able to look at .map files helps a lot. E.g. I hope to implement proper DSP-side crash handling somewhen in the future and ppl can then look at the reported addresses to see what function / instruction caused a crash. Would be much harder to implement with dynamic relocation.

BTW you keep mentioning the blitter engine you did for your work that you can't reuse.. Did it actually work out well and reached the project goals? I.e. was the DSP good enough for the work?
oh yes, it was. Actually the engine is used by many of the major automotive companies from Germany, the USA, and South Korea.


It is much more than just a blitter engine, though. It implements an OGLES 1.x compliant (well, nearly) rasterizer and is used for both map rendering and HMIs.


Nowadays, these companies use hardware graphics accelerators to do the job in their future products but back in 2008 the DSP was a good choice (for products released in 2010/2011).


Besides, the DSP was not only used for graphics rendering but also for e.g. voice recognition.


I am also just being cautious here -- even if _I_ were to write some optimized blitloops for the Pandora, IMHO, I strongly doubt that it would hurt the business of my employer. But to be on the safe side, I'll just write some standard C blitloops and leave the task of implementing "clean room", assembly optimized versions of these, to other people. (you do not even have to use assembly, intrinsics will do fine)
 
Last edited by a moderator:
Very nice work, should we make a seperate thread for wanted library functions which could be included in the default dsp image?

Like:

blitting

video decoding

path finding (like fast marching)
 
A few pages back, I suggested to add a "DSP corner" subforum to the development section.

Threads about DSP coding in general, ideas/requests, specific components (blitters, audio/video decoders, ..) should go there, IMHO.

(and this thread should be moved there, too)
 
Hi,

I see 3 useage, in my point of view, and with increasing difficulty of implementation, for the DSP:

1. Code an audio Plugin for mupen64plus v2. The audio is not a big workload, but if it can be offloaded from main CPU, it will certainly be good. Also, the modular side of mupen64 make it a good canditate to have a DSP Plugin to test, while maintaining compatibility with "not DSP-able system".

2. Export the PPSSPP sound codeto DSP. It's a bit more complicated, and it should be done without Atrac+ at first, and than with it on a second time.

3. Port most of the sound transformation of OPenAL-Soft to DSP. OPneAL can be quite CPU intensive in some game (take JK2 ou 3, and test with OpenAL or without to see what I mean). This is clearly more complicated to port, but the benefit can be very important, OpenAL can consume more than 10%~15% of CPU on some scene (quick "TOP" unrelialble benchmark).
+1 to all these excellent ideas !

Now that mupen64plus v2 is out, do you have any plan regarding your 1st point ?

btw, thanks for all your hard work on this emulator :)

Bye, Magic Sam
 
Hi,

I see 3 useage, in my point of view, and with increasing difficulty of implementation, for the DSP:

1. Code an audio Plugin for mupen64plus v2. The audio is not a big workload, but if it can be offloaded from main CPU, it will certainly be good. Also, the modular side of mupen64 make it a good canditate to have a DSP Plugin to test, while maintaining compatibility with "not DSP-able system".

2. Export the PPSSPP sound codeto DSP. It's a bit more complicated, and it should be done without Atrac+ at first, and than with it on a second time.

3. Port most of the sound transformation of OPenAL-Soft to DSP. OPneAL can be quite CPU intensive in some game (take JK2 ou 3, and test with OpenAL or without to see what I mean). This is clearly more complicated to port, but the benefit can be very important, OpenAL can consume more than 10%~15% of CPU on some scene (quick "TOP" unrelialble benchmark).
+1 to all these excellent ideas !

Now that mupen64plus v2 is out, do you have any plan regarding your 1st point ?

btw, thanks for all your hard work on this emulator :)

Bye, Magic Sam
Thanks.

I a few project that were on hold while I worked on mupen that I what to complete (or at least progress a bit, like Arx), and than I take a look at this DSP business.
 
There needs to be a firmware update anyways.
Yes, I'd like to have that firmware update so I don't have to mess with kernel files by hand, but I guess there is chicken and egg problem here maybe. Firmware update wait for a software using it, software update wait for a firmware update to ease things...
 
There needs to be a firmware update anyways.
Yes, I'd like to have that firmware update so I don't have to mess with kernel files by hand, but I guess there is chicken and egg problem here maybe. Firmware update wait for a software using it, software update wait for a firmware update to ease things...
Firmware shouldnt wait for software to come first. an OS s role is to enable you to do things.
 
I think notaz said the next firmware update will include the dsp tools.
 
Notaz's worry isn't unwarranted. We've had DSPLink and Bridge for quite a while now and no one has used them. The only difference now is that there's some extra hype around the new library, and bsp as the author is willing to sit and help people. That's a huge benefit but by no means a guarantee it'll get used. Adding something more to the firmware that isn't going to be used is just another thing to take up space and maintenance time so I understand his hesitation to just add it.
 
It is more easy to be used now and there will be a set of basic functions in the standard image. pretty neat. And it is a bit faster then the dspLink I think.
 
Last edited by a moderator:
Back
Top