[announce] c64_tools (DSP loader and IPC)

[levi]

Heh, I've never heard interrupt jump tables referred to as trampolines before, but I guess it makes sense

 

[bsp]

that's because on the C64+, the interrupt tables do not just contain interrupt vectors (like e.g. on the Amiga) but actual code. There are 32 bytes available for each interrupt (8 instructions / 1 fetch packet). Usually you just place a branch instruction there and NOP the rest out -- voila, you've created a trampoline

p.s.: but yeah, I won't implement the syscalls like that. I will use a simple function pointer array (resp. structure with fxn ptrs, from a C programmer's point of view), located at a fixed address.

 

[bsp]

..implemented a useful feature today: DSP auto suspend / resume.

i.e. the DSP is powered down automatically when there are no applications left using it, and vice versa it automotically resumes program execution when the next client app. connects (all handled by libc64 and c64.ko, no scripts).

with C64_DEBUG=1, the output looks like this (8 RPC benchmarks started in parallel):

[8] + Done                       ./c64_tc 3
[7] + Done                       ./c64_tc 3
[6] + Done                       ./c64_tc 3
[5] + Done                       ./c64_tc 3
[4] + Done                       ./c64_tc 3
[3] + Done                       ./c64_tc 3
[2] + Done                       ./c64_tc 3
[1] + Done                       ./c64_tc 3
pandora:/home/bsp# dmesg -c
[ 4865.566894] c64::dev:pen: client connected (filep=0xd4241c00).
[ 4865.566955] c64::dsp_resume: resuming DSP program execution @0x8604bca0
[ 4865.567016] c64::dsp_start: dsp_config.ram.phys_addr=0x86001000 size=0x003ff000
[ 4865.567016] c64::dsp_start: entryPoint=0x8604bca0
[ 4865.894439] c64::dev:pen: client connected (filep=0xd426ed00).
[ 4866.180908] c64::dev:pen: client connected (filep=0xd67d7800).
[ 4866.414825] c64::dev:pen: client connected (filep=0xd426ef00).
[ 4866.631011] c64::dev:pen: client connected (filep=0xd4146100).
[ 4866.869110] c64::dev:pen: client connected (filep=0xd426e500).
[ 4867.061126] c64::dev:pen: client connected (filep=0xd64f1c00).
[ 4867.310821] c64::dev:pen: client connected (filep=0xd67ab180).
[ 4892.433135] c64::dev::release: client disconnected (filep=0xd4241c00).
[ 4896.670989] c64::dev::release: client disconnected (filep=0xd426ef00).
[ 4896.836914] c64::dev::release: client disconnected (filep=0xd426ed00).
[ 4897.057891] c64::dev::release: client disconnected (filep=0xd67d7800).
[ 4897.220733] c64::dev::release: client disconnected (filep=0xd67ab180).
[ 4897.421173] c64::dev::release: client disconnected (filep=0xd426e500).
[ 4897.474731] c64::dev::release: client disconnected (filep=0xd64f1c00).
[ 4897.572631] c64::dev::release: client disconnected (filep=0xd4146100).
[ 4897.572662] c64::dev::release: powering down DSP after last client disconnected.

(The baseline DSP power consumption, while a client app. is connected, is ~23mA, in case you missed that on the previous pages.)

 

[WizardStan]

So is there any real reason it cannot be loaded at boot time and just used without thought?

 

[bsp]

you can do that now, if you want to. with the auto suspend/resume, it does not consume any power unless it is actually used. however, if you load it at boot time, it will reserve 4 MBytes of memory and some people were really concerned about that I'd suggest to make it a feature that can be toggled, like WiFi, LEDs, .. and let the user decide. If it is not enabled at boot time, the first DSP app. would show a dialog and prompt for the root passwd, then load the kernel module and boot the "core" DSP image.

p.s.: TBH, I expect this to work fine everywhere but so far it has been tested on only a few of all the Pandoras out there (~4500-5000?). Just to be on the safe side, this should not be enabled by default until more field testing has been done. I am currently working on the dynamic (overlay) loader. Once that is done, I'll write a few more (simple) demos, and package the whole thing into an installer PND, and some demo application PNDs.

 

[_wb_]

Very nice work, bsp! Keep going!

 

[Hồng Thất Công]

How does this help the average Joe like me?

 

[ekianjo]

How does this help the average Joe like me?

you dont have more useful comments to make?

 

[sebt3]

How does this help the average Joe like me?

It doesnt help you directly. But once bsp have finished his works, all the others dev will be able to use his works results. Giving you more performances in the app/emu that will use that. Ultimatly it could mean more games playable.

 

[Hồng Thất Công]

How does this help the average Joe like me?

you dont have more useful comments to make?

What I meant was does this help speed up existing apps? Just install the DSP pnd and things will automatically get sped up?  I know bsp's work in DSP here is very valuable and by no means I'm trying to make stupid comments.  I was just asking an innocent question....

How does this help the average Joe like me?

It doesnt help you directly. But once bsp have finished his works, all the others dev will be able to use his works results. Giving you more performances in the app/emu that will use that. Ultimatly it could mean more games playable.

Thanks!  This is the answer I'm looking for.  Thanks again!

 

[rohezal]

Do we have a list of functions we would like to have in the standard DSP image?

My ideas:

Fast Marching Wayfinding (not so easy to implement without recurssion but possible i think)

Matrix4x4 * Vertex4 multiply

Matrix3x3 * Vertex3 multiply

Blitting (with alpha channel)

Weighted absolute or squared distance of a block of pixels (like 4x4 pixels in image1 minus 4x4 pixels in image2)

Edge dection (Sobel or Roberts)

Video Decoding (minitube could realy use some speed up)

 

[_wb_]

Image, video and audio codecs would be nice - it would be really cool if I could view and resize images and view and transcode videos from my camera, and those should be things the DSP is really good at. Imagine transcoding a movie in the background while the CPU is still available to do other things...

 

[WizardStan]

What I meant was does this help speed up existing apps? Just install the DSP pnd and things will automatically get sped up?

No, it becomes another tool that developers can use. It doesn't automatically make things faster, the devs need to be able to take advantage of the tool, so programs will need to be rewritten.

 

[bsp]

a small update from my side: working hard on the overlay loader feature.

the DSP-side code for that is practically finished. Good news is that building custom "overlay components" does not require the dreaded "tconf" anymore (remember, that tool which uses a Javascript interpeter written in Java. It takes ages to start up..I wrote a simple script that generates the actual linker scripts. Good enough for this usecase.).


So building the components is very fast now (~4 sec for all components on my old Linux notebook).

It is also no longer necessary to install DSP/BIOS (~290MB install!), the C/C++ compiler ("cgtools") is all that's needed for component development (for convenience purposes, the default build system references the iqmath and fastRTS libs but these are rather small and very useful).

This also means that the overlay images are a lot smaller (~4k code), .out size ~19k instead of ~206k (w/ DSPBIOS).

The DSP build system also became even more simple to use (moved common .mk code to include files).

E.g. just by including "build_area2.mk" the build system will build 4 area2 (256k) overlay image variants (each with a different base address) for you

vice versa, including "build_area3.mk" will create an image suitable for area3 (1MB).

It is still possible to access at least some of the DSP/BIOS functions in components since I simply added them to the global (c64_tools) "syscalls" table/struct (e.g. cache functions, mem/string functions, soon DMA utility functions, ..)

I have just added new OVERLAY_ALLOC and OVERLAY_FREE ioctls to the kernel module (which should also be more thread/process safe now, forgot to use mutices in the fops). These are, as you may have guessed, used to allocate/free one of the 5 dynamic loader overlay sections (don't worry, these are libc64 internals, they do not have to be used directly).

Some thought was needed for safe addition/removal of components (to avoid race conditions between different threads/processes): The basic procedure is:
 - DSP initiates fastcall sequence (no other tasks/thread can use the DSP during that time, i.e. it is locked down),

 - then the GPP loads the new overlay sections,

 - sends an "invalidate your instruction and data caches and register new component"  RPC to the DSP,

 - then terminates the fastcall sequence.

There's also another thing which just caused me to wrote a couple of lines of comments in the kernel mod.: If an app. loads an overlay/registers a new component, and forgets to unregister the component and free the overlay area(s), that area(s) will stay allocated until the next DSP reset (go64.sh re-run).

This is both a bug and a feature:

A feature since you can write an app that simply installs a new component, then exits.

A bug since if an app. crashes and did not install a crash handler that frees the area, the area will be lost for other apps (again, until the next manual DSP reset).

Maybe I'll just install a "parachute" on the GPP side to do the clean up, although the best way would be if the kernel module would track component registrations and automatically do the clean up when a client exits. This would mean quite some work, though, and I'll probably will not implement that soon.
(can't wait to actually write some useful DSP code libraries, guess that's more important to users, too)

If all that sounds like techno babble to you: It is. But once this is done, creating fast audio/video (and what not) routines for the Pandora DSP should be a breeze (it is already relatively straight forward but ATM (last release) there can only be one DSP image, support for multiple ones will be the major update in the next release).

 

[notaz]

Maybe I'll just install a "parachute" on the GPP side to do the clean up, although the best way would be if the kernel module would track component registrations and automatically do the clean up when a client exits. This would mean quite some work, though, and I'll probably will not implement that soon.
(can't wait to actually write some useful DSP code libraries, guess that's more important to users, too)

The kernel-side cleaning should be pretty straightforward, I don't know why you keep saying it's lots of work. You can use struct file you get through fops to track if the user program is still alive or not. Whenever user program exits, crashes or whatever, fops->release will be called, and you can use struct file as a key to your own lists or resources to figure out what needs to be cleaned. This is the only way to do it reliably IMO, I would hate if some library installs crash handlers behind my back (like SDL sometimes does, but it can be disabled at least), I need the segfault handler for my own things, and some other programs need it too.

We still need to do fw integration BTW..

 

[bsp]

The problem is that the fops release fxn would have to use the other fops (read/write/poll) to interact with the DSP to unregister whatever components the client app. might have installed (and possibly did not clean up).

I am not saying that this is impossible but a whole lot more libc64 functionality would have to be moved into the kernel module (and compared to the last release, a lot already did).

Since this is not a big issue, in my honest opinion, but only a minor / cornercase one if an app using the DSP is coded sloppily, I will go for the parachute solution for now. (This may still be changed later on, though, but I'd rather dedicate my (spare!) time to more important/practical tasks in the near future). Let's not forget that this only affects "staging" DSP components, not the firmware/resident ones.

I would hate if some library installs crash handlers behind my back

understandable. the practical solution to that would be to just call dsp_close() in your own crash handler (and forbid libc64 to install its own crash handler, I'll implement that tomorrow)

I don't know why you keep saying it's lots of work

that's because it *is* a lot of work.
 

We still need to do fw integration BTW..

yes we do and I am working hard on making the next release a proper one.

what does your FW release roadmap look like ? (PM if you don't want this to be public)

 

[notaz]

There are not many plans for the next fw, ED wanted to tune some scripts (when he can find time) and I still need to tune build so that "correct" SGX driver comes with 1GHz by default. So next release kind of belongs to c64_tools.

 

[ekianjo]

that "correct" SGX driver comes with 1GHz by default.

Which driver do you consider "correct"  ?

 

[notaz]

The one that has less glitches but breaks old mupen (and that is supposed to be superseded soon).

 

[ekianjo]

The one that has less glitches but breaks old mupen (and that is supposed to be superseded soon).

I heard from Ptitseb that more recent drivers may break SDL2 compatibility, though. RIckyD does not work anymore with the non-default driver and that's because it's SDL2 incompatible, apparently.