[announce] c64_tools (DSP loader and IPC)

[bsp]

I've noticed running make rebuilds libc64.so even if nothing changed. You should really just use $(TARGET_SHARED) and $(TARGET_STATIC) as targets instead of bin_shared and bin_static that you currently have.

yea, the makefile dependencies probably need some cleaning up. Already did that for the DSP-side makefiles.

libc64/ ? What for? I don't see any headers there

sry, my bad.

There is also this in inc_libc64.h:

#include "../dsp/libc64_dsp/include/com.h"

could it maybe be moved to include/ and renamed to dsp_com.h or something?

It's not that easy. This particular suggestion could be done but there's more header files needed from the dsp folder, e.g. the ones in dsp/components (and its subfolders). Just keep the entire dsp/ folder for now (by that I mean all .h files in it).

btw, here are the changes I made today:

+ change license of c64_dev.h to LGPL and remove it from inc_libc64.h


+ move internal dsp.h fxns to dsp_priv.h


+ adapt c64_tc to USE_DSP_PRIV


+ remove DSP_WRITE_REGS define


o build versioned libc64.so.1 (and symlink libc64.so)


+ rename c64_msg_t to dsp_msg_t and remove old (unused) dsp_msg_t

+ _msgFrom argument to dsp_rpc_recv() and dsp_rpc() may be NULL now

+ rename c64_kmod/ to kmod/


+ added dsp_cache_t enum (uncached, writeback, writethrough) and argument to dsp_shm_init()


+ update kmod dev.c to support mem types

x change prot flags of fshm to L_PTE_MT_UNCACHED ("strongly ordered") (--> does not work correctly)

+ remove dsp_shm_get()  (mem is now returned by dsp_shm_init())


+ rename dsp_shm_init() to dsp_shm_alloc()


+ rename dsp_shm_exit() to dsp_shm_free()


+ pass dsp_mem_region_t to dsp_shm_free()

+ adapt c64_tc and c64_fractal to new dsp_shm_*() semantics

+ release http://tkscript.de/c64_tools/c64_tools-15Oct2013_snapshot.tar.gz


especially the dsp_shm_alloc()/free() change should be more future proof now.

if the need arises, the implementation can be changed in the future to support multiple allocations per cache type without changing the API.

the snapshot is not for integration, of course, just in case you (or anyone else) wants to take a look.

 

[crow_riot]

Thanks for continuously working on this!

In the other thread, we're talking about sound mixing. What comes to my mind is: would it be somehow to get the hardware mixers address like we have the framebuffer address into the DSP API, so that we could directly output sound from the DSP ?

 

[Rockthesmurf]

For what it is worth (probably not a lot to be fair), I feel the user should just have to #include <dsp.h>. I get the reasoning behind having various different include files, but I personally feel just #include <dsp.h> should work and handle including everything required. On top of this, if you wanted to provide another version where users are more 'dependency aware' that is fine (although I honestly doubt anyone will want to use the more complicated version). I personally feel #include <dsp.h> looks like a neat professional; a package ready for external consumption. Having to do any 'special' includes to make an API work correct doesn't (to me) come across as well.

A very minor thing in the grand scheme, but I figure I'll throw my opinion out there anyway..! Looking forward to the release,

 

[bsp]

here's the changelog for today:

+ add c64_tc testcases TC_SHM_CACHE_NONE, TC_SHM_CACHE_RW, TC_SHM_CACHE_R
   -> CACHE_NONE is definitely working (slooow)
   -> CACHE_RW and CACHE_R result in same performance (no slow/uncached writes with CACHE_R?!)
+ fixed: forgot to set "type" field of shm_alloc_req in dsp_shm_alloc() (was always 0 => DSP_CACHE_NONE)
+ use pgprot_noncached, pgprot_writethrough macros in kmod/dev.c (instead of __pgprot())
+ use pgprot_stronglyordered macro for fshm mapping in kmod/dev.c (instead of pgprot_uncached) (same thing, though)
+ some temporary hardcoded klog output (lo freq) ("xxx c64::dev::mmap")
+ fixed: round up to pagesize already in dsp_shm_alloc(), not only in kmod SHM_ALLOC ioctl
? is the shared memory returned by dma_alloc_coherent() always write-through ?

   it does not seem to make any difference whether dsp_cache_wb() is called or not
? why is TC_SHM_CACHE_RW contiguous->contiguous memcpy performance much lower than paged->paged
+ change C64_IMAGE_SITE_PATH to "/lib/dsp/"
+ release http://tkscript.de/c64_tools/c64_tools-16Oct2013_snapshot.tar.gz
 

testcase output:

pandora:/home/bsp# cat test_shm.sh
dmesg -c

./go64.sh; dmesg -c

./c64_tc 11; dmesg -c

./c64_tc 12; dmesg -c

./c64_tc 13; dmesg -c

pandora:/home/bsp# . ./test_shm.sh
 
go64.sh: ARGS="" DSPIMAGE="./core.out"
[...] loading DSP image from "./core.out"
[...] reading 68 section headers....................................................................
[...] dsp_autoresume_set: autoresume is now disabled.
[...] initializing DSP memory and copying sections..
[...]  text=0x0000b140  data=0x00001607  bss=0x0007fd6c
[...] DSP image loaded successfully. starting DSP..
[dbg] waiting for DSP startup..
[...] DSP is up and running!
[35227.288421] c64::dsp_poweroff: IVA2_PRM_PWSTST=0x00000000 => DSP is OFF
[35227.290039] c64: exit OK.
[35227.349700] c64: init ok. module build date: Oct 16 2013  (**4MB DSP image size**)
[35227.617614] c64::dev::ioctl: starting DSP @0x86050ea0
[35227.891937] c64::dsp_poweroff: IVA2_PRM_PWSTST=0x00000000 => DSP is OFF
[...] selected testcase 11 ("TC_SHM_CACHE_NONE")
[...] coff_load_overlay: text=0x000005a0  data=0x0000006c  bss=0x000010dc
[...] coff_load_overlay: text=0x000004c0  data=0x00000064  bss=0x000010dc
[...] coff_load_overlay: text=0x00000600  data=0x00000074  bss=0x000010dc
[...] loc_tc_shm_speedcheck: fill paged mem.......................done: 100 * 8192 kbytes in 564 ms => 1418.440 MBytes/sec
[...] loc_tc_shm_speedcheck: fill contiguous mem..................done: 100 * 8192 kbytes in 21178 ms => 37.775 MBytes/sec
[...] loc_tc_shm_speedcheck: copy paged to paged mem..............done: 100 * 8192 kbytes in 554 ms => 1444.043 MBytes/sec
[...] loc_tc_shm_speedcheck: copy contiguous to contiguous mem....done: 100 * 8192 kbytes in 25801 ms => 31.007 MBytes/sec
[...] loc_tc_shm_speedcheck: copy contiguous to paged mem.........done: 100 * 8192 kbytes in 16317 ms => 49.029 MBytes/sec
[...] loc_tc_shm_speedcheck: copy paged to contiguous mem.........done: 100 * 8192 kbytes in 11886 ms => 67.306 MBytes/sec
[35227.926818] xxx c64::dev::ioctl: SHM_ALLOC req.type=0
[35227.948730] xxx c64::dev::mmap: shm OLD vm_page_prot=0x0000030f
[35227.948760] xxx c64::dev::mmap: shm DSP_CACHE_NONE
[35227.948760] xxx c64::dev::mmap: shm NEW vm_page_prot=0x00000303
[35304.311126] c64::dsp_poweroff: IVA2_PRM_PWSTST=0x00000000 => DSP is OFF
[...] selected testcase 12 ("TC_SHM_CACHE_RW")
[...] coff_load_overlay: text=0x000005a0  data=0x0000006c  bss=0x000010dc
[...] coff_load_overlay: text=0x000004c0  data=0x00000064  bss=0x000010dc
[...] coff_load_overlay: text=0x00000600  data=0x00000074  bss=0x000010dc
[...] loc_tc_shm_speedcheck: fill paged mem.......................done: 100 * 8192 kbytes in 557 ms => 1436.266 MBytes/sec
[...] loc_tc_shm_speedcheck: fill contiguous mem..................done: 100 * 8192 kbytes in 547 ms => 1462.523 MBytes/sec
[...] loc_tc_shm_speedcheck: copy paged to paged mem..............done: 100 * 8192 kbytes in 554 ms => 1444.043 MBytes/sec
[...] loc_tc_shm_speedcheck: copy contiguous to contiguous mem....done: 100 * 8192 kbytes in 2238 ms => 357.462 MBytes/sec
[...] loc_tc_shm_speedcheck: copy contiguous to paged mem.........done: 100 * 8192 kbytes in 1939 ms => 412.584 MBytes/sec
[...] loc_tc_shm_speedcheck: copy paged to contiguous mem.........done: 100 * 8192 kbytes in 2518 ms => 317.712 MBytes/sec
[35304.344238] xxx c64::dev::ioctl: SHM_ALLOC req.type=1
[35304.365447] xxx c64::dev::mmap: shm OLD vm_page_prot=0x0000030f
[35304.365478] xxx c64::dev::mmap: shm DSP_CACHE_RW
[35304.365478] xxx c64::dev::mmap: shm NEW vm_page_prot=0x0000030f
[35312.783691] c64::dsp_poweroff: IVA2_PRM_PWSTST=0x00000000 => DSP is OFF
[...] selected testcase 13 ("TC_SHM_CACHE_R")
[...] coff_load_overlay: text=0x000005a0  data=0x0000006c  bss=0x000010dc
[...] coff_load_overlay: text=0x000004c0  data=0x00000064  bss=0x000010dc
[...] coff_load_overlay: text=0x00000600  data=0x00000074  bss=0x000010dc
[...] loc_tc_shm_speedcheck: fill paged mem.......................done: 100 * 8192 kbytes in 553 ms => 1446.655 MBytes/sec
[...] loc_tc_shm_speedcheck: fill contiguous mem..................done: 100 * 8192 kbytes in 546 ms => 1465.201 MBytes/sec
[...] loc_tc_shm_speedcheck: copy paged to paged mem..............done: 100 * 8192 kbytes in 557 ms => 1436.266 MBytes/sec
[...] loc_tc_shm_speedcheck: copy contiguous to contiguous mem....done: 100 * 8192 kbytes in 2233 ms => 358.262 MBytes/sec
[...] loc_tc_shm_speedcheck: copy contiguous to paged mem.........done: 100 * 8192 kbytes in 1932 ms => 414.079 MBytes/sec
[...] loc_tc_shm_speedcheck: copy paged to contiguous mem.........done: 100 * 8192 kbytes in 2528 ms => 316.456 MBytes/sec
[35312.816131] xxx c64::dev::ioctl: SHM_ALLOC req.type=2
[35312.837341] xxx c64::dev::mmap: shm OLD vm_page_prot=0x0000030f
[35312.837371] xxx c64::dev::mmap: shm DSP_CACHE_R
[35312.837371] xxx c64::dev::mmap: shm NEW vm_page_prot=0x0000030b
[35321.252258] c64::dsp_poweroff: IVA2_PRM_PWSTST=0x00000000 => DSP is OFF
pandora:/home/bsp#



@Steven Craft: You do realize that this discussion is about whether you write


#include <dsp.h>
instead of


#include <inc_libc64.h>


I'll stick with the latter since that's my way of naming header files that include other header files.

Everyone's free to copy "inc_libc64.h" to his or her project dir, remove unwanted includes and just write


#include "dsp.h"
I do not see the problem.
 

 

[notaz]

+ add c64_tc testcases TC_SHM_CACHE_NONE, TC_SHM_CACHE_RW, TC_SHM_CACHE_R


   -> CACHE_NONE is definitely working (slooow)


   -> CACHE_RW and CACHE_R result in same performance (no slow/uncached writes with CACHE_R?!)

You can't disable cache just for writes, it's either cached for both r/w or not cached at all. The only thing you can change is if the writes go to cache only (when you write to location that's in cache, otherwise it just goes to memory) or to cache+memory (write-through). Note that there is also a write buffer, so the CPU doesn't have to wait for writes to finish (you need to issue special barrier instructions to flush the writes before you start DMAs and such).

+ use pgprot_stronglyordered macro for fshm mapping in kmod/dev.c (instead of pgprot_uncached) (same thing, though)

I'm not sure about Cortex-A8, but in general it's not necessarily true, uncached may allow to combine/reorder writes, but stronglyordered not (so stronglyordered is intended for hardware registers where the order of accesses is important).

? is the shared memory returned by dma_alloc_coherent() always write-through ?


   it does not seem to make any difference whether dsp_cache_wb() is called or not

It should not be as I saw code in the kernel for cleaning the caches for those buffers. For proper test you should do reads before writes on GPP and some data may be stuck in caches then (run some checksum on DSP maybe?).


 

? why is TC_SHM_CACHE_RW contiguous->contiguous memcpy performance much lower than paged->paged

Did you touch the memory before benchmarking? Keep in mind that Linux allocates memory only when you first touch it (and fault), so first accesses are always very slow.

 

[bsp]

+ add c64_tc testcases TC_SHM_CACHE_NONE, TC_SHM_CACHE_RW, TC_SHM_CACHE_R


   -> CACHE_NONE is definitely working (slooow)


   -> CACHE_RW and CACHE_R result in same performance (no slow/uncached writes with CACHE_R?!)

You can't disable cache just for writes, it's either cached for both r/w or not cached at all. The only thing you can change is if the writes go to cache only (when you write to location that's in cache, otherwise it just goes to memory) or to cache+memory (write-through). Note that there is also a write buffer, so the CPU doesn't have to wait for writes to finish (you need to issue special barrier instructions to flush the writes before you start DMAs and such).

hmm my understanding of write-through was that writes go to both cache and memory ASAP. Is there anything extra that needs to be done before the DSP can see the "actual" data ? What barrier instructions are we talking about ?

+ use pgprot_stronglyordered macro for fshm mapping in kmod/dev.c (instead of pgprot_uncached) (same thing, though)

I'm not sure about Cortex-A8, but in general it's not necessarily true, uncached may allow to combine/reorder writes, but stronglyordered not (so stronglyordered is intended for hardware registers where the order of accesses is important).

Of course that is not true in general but the Pandora kernel implementation for pgprot_noncached() is exactly the same as pgprot_stronglyordered().

You remember I mentioned that I had to resort to using 2 different cachelines for GPP/DSP IPC in L1DSRAM ? My guess is that this may be related to write re-ordering but pgprot_stronglyordered() does not seem to be implemented correctly. Well, we have a workaround.

? why is TC_SHM_CACHE_RW contiguous->contiguous memcpy performance much lower than paged->paged

Did you touch the memory before benchmarking? Keep in mind that Linux allocates memory only when you first touch it (and fault), so first accesses are always very slow.

yep, take a look at the source (c64_tc.c)

another thing that has been boggling my mind the past days is that I have a very simple benchmark that just allocates two 8MB areas via malloc(), copies these via memcpy() a 100 times, and measures the time it takes. For some reason that I have not figured out, yet, the throughput with that is just 266 MBytes/sec on the Pandora (similar results for Raspberry Pi and iMX6).

The results of today's benchmark are much more in the ballpark that I would have expected (1.4GB/s).

I also do not quite understand these results:

[...] selected testcase 12 ("TC_SHM_CACHE_RW")


[...] loc_tc_shm_speedcheck: copy paged to paged mem..............done: 100 * 8192 kbytes in 554 ms => 1444.043 MBytes/sec


[...] loc_tc_shm_speedcheck: copy contiguous to contiguous mem....done: 100 * 8192 kbytes in 2238 ms => 357.462 MBytes/sec

why is the second result ~4 times lower than the paged-to-paged memcpy !? (in the kmod mmap I do not change the vm_prot flags if DSP_CACHE_RW is used. Is there any flag that needs to be added to the default flags in order to get the full speed ?)

 

[notaz]

hmm my understanding of write-through was that writes go to both cache and memory ASAP. Is there anything extra that needs to be done before the DSP can see the "actual" data ? What barrier instructions are we talking about ?

dsb and dmb. I think dsb is the one needed here, as dmb is for SMP core synchronization (I think?).

another thing that has been boggling my mind the past days is that I have a very simple benchmark that just allocates two 8MB areas via malloc(), copies these via memcpy() a 100 times, and measures the time it takes. For some reason that I have not figured out, yet, the throughput with that is just 266 MBytes/sec on the Pandora (similar results for Raspberry Pi and iMX6).

The results of today's benchmark are much more in the ballpark that I would have expected (1.4GB/s).

I also do not quite understand these results:

[...] selected testcase 12 ("TC_SHM_CACHE_RW")

[...] loc_tc_shm_speedcheck: copy paged to paged mem..............done: 100 * 8192 kbytes in 554 ms => 1444.043 MBytes/sec

[...] loc_tc_shm_speedcheck: copy contiguous to contiguous mem....done: 100 * 8192 kbytes in 2238 ms => 357.462 MBytes/sec

why is the second result ~4 times lower than the paged-to-paged memcpy !? (in the kmod mmap I do not change the vm_prot flags if DSP_CACHE_RW is used. Is there any flag that needs to be added to the default flags in order to get the full speed ?)

It's hard to say, needs to be investigated. Maybe some other memory attributes, there are things like inner (L1 cache) and outer (L2) cacheability, we did some tests on this with Exophase long ago.. I'll see if I can find some time at the weekend to attempt to look at this.

 

[Farox]

This thread is so full of technical details (that i barely understand..) but is tremendously fascinating  

 

[pmprog]

This thread is so full of technical details (that i barely understand..) but is tremendously fascinating

I'd like to echo this sentiment. I used to think I knew a fair bit of the technical side to computers, but through a variety of reasons, I feel like I never left the 90's in terms of my understanding of technology.

 

[sparkymark79]

it is all very exciting, boggles how a few of the pandorians brains are wired to understand such low level stuff!. Remember that film Daryll?

 

[Rockthesmurf]

I'll stick with the latter since that's my way of naming header files that include other header files.


Everyone's free to copy "inc_libc64.h" to his or her project dir, remove unwanted includes and just write


#include "dsp.h"
I do not see the problem.

There is definitely no problem! You are the author, there was no desire to say you are wrong. From what I can recall, the 20-30 libraries I am currently using in active projects all use a similar include name, and none of them are "inc_<proj>.h" - that isn't to say anyone is right or wrong, just an observation from the libraries I interact with. It is possible lots of libraries I do not use do exactly as you do. Just from my perspective the most obvious include for a project is #include <projName> it is what I consider 'standard' (although my consideration of standard maybe completely different to lots of other people). Of course I could fork your project, change the include name, and create my own fork, but it would be beyond silly, which is why I mentioned here instead.

This is all very minor in the grand scheme of things of course..

 

[bsp]

This thread is so full of technical details (that i barely understand..) but is tremendously fascinating

hey Farox. M-HT and me did optimize the fractal benchmark a short while ago and it turned out that the Cortex is faster, after all (~7.5s ARM, ~10s DSP, ~5s both). This is what I had initially expected (lots of branches that the DSP is not so good at) although I was still surprised how well the DSP performed that task (especially in regards to power consumption: 1.25W/h ARM, ~0.75W/h DSP, ~1.4W/h both, or, when factoring in the execution time and calculating the amps: ( ((Wh / 360) / 4V) * sec * 1000 ): ~6.5mA ARM, ~5.2mA DSP, ~4.8mA both). Thanks for the video, anyway -- even if the benchmark results have changed, it still shows that the DSP is very suitable for offloading tasks.

From what I can recall, the 20-30 libraries I am currently using in active projects all use a similar include name, and none of them are "inc_<proj>.h" - that isn't to say anyone is right or wrong

very true, I don't think I have seen that anywhere else before, either

hmm my understanding of write-through was that writes go to both cache and memory ASAP. Is there anything extra that needs to be done before the DSP can see the "actual" data ? What barrier instructions are we talking about ?

dsb and dmb. I think dsb is the one needed here, as dmb is for SMP core synchronization (I think?).

dsb/dmb are correct, see http://infocenter.arm.com/help/topic/com.arm.doc.dui0489c/CIHGHHIE.html and here http://infocenter.arm.com/help/topic/com.arm.doc.faqs/ka14041.html

This is certainly something to keep in mind, thanks for the tip!

I also read something about the Linux wmb() and smp_wmb() macros today but if dsb/dmb are available in userspace then that's a more efficient solution.

about the shared memory vs. malloc() speed: I dug through quite some kernel programming threads today and it seems that this is somehow a known (ARM Linux?) problem with no solution to it. I found threads that mentioned that kmalloc()'d memory is said to be slower than vmalloc()'d mem, others were seeing similar performance issues when using dma_alloc_coherent(). Someone mentioned that when using dma_map_single(), the speed issue does not occur (although that limits the "shared memory" size to 4096 bytes, of course). Last but not least, some said that this does not occur when statically reserving memory and using ioremap() but we don't want to go back there, right.. that's why I tried the DMA_ATTR_NO_KERNEL_MAPPING thing today, btw (see below).

I think I'll try the ioremap() / statically reserved mem speedcheck tomorrow and if this solves the issue we could look at how to convince dma_alloc_attrs() to just reserve the physical memory and not create any kernel mapping.

minor update today (wasted too much time trying to figure out the CACHE_RW shm speed issue):

+ cleaned up the new c64_tc CACHE* testcases (loc_speedcheck_exec()) and added copy32 and fill32 variants


+ try: in case of CACHE_RW, use dma_alloc_attrs() w/ WEAK_ORDERING+WRITE_COMBINE+NO_KERNEL_MAPPING,


        then ioremap() the physical address. did not work, NO_KERNEL_MAPPING is probably not implemented,


        at least ioremap() failed with BADADDRESS. (see dev.c__DMA_ATTRS_TRY)


+ release http://tkscript.de/c64_tools/c64_tools-17Oct2013_snapshot.tar.gz

 

[bsp]

..and another update (this one is more interesting):

+ move some config related defines from dsp_common.h to dsp_config.h
+ add DSP_STATICMEM_CACHE_*_SIZE defines to dsp_config.h

   (default=8MB uncached, 2*10MB write-through cached)
+ add support for cached/uncached shared memory in kmod/dev.c

   (when using boot-time statically allocate mem (deprecated))
+ reduced TC_SHM_SPEECHECK_SIZE from 8MB to 5MB to work with staticmem config
+ run statically allocated shared memory speed (c64_tc 12) -> same speed as CMA mem
+ rename dsp_fshm_get() to dsp_fshm_scratchbuffer_get()
+ add DSP_FSHM_SCRATCH_SIZE define to dsp_config.h
+ add FSHM_ALLOC and FSHM_FREE ioctls
+ add DSP_FSHM_ALLOC_GRANULARITY (+SHIFT) defines to dsp_config.h
+ add fshm memory allocator to kmod/dev.c (simplistic but not just a pool,
   finds best match free block range and joins adjacent super blocks upon free())
+ add dsp_fshm_alloc() and dsp_fshm_free() (dynamically allocate and free persistent L1DSRAM)
+ add TC_FSHM_ALLOCATOR testcase to c64_tc
+ add FSHM_DEBUG ioctl (--> allocator testcase works OK)
+ release http://tkscript.de/c64_tools/c64_tools-18Oct2013_snapshot.tar.gz
 

This update answers a question no-one has asked (yet): How is the DSP L1DSRAM shared among GPP-side processes ?

The basic idea regarding SRAM partitioning is:

• 24k (half of it) is assigned to the DSP (can use the c6x linker to allocate / assign regions)
• 16k are used as a volatile scratchbuffer that is accessible by both DSP and GPP (volatile= not persistent across remote procedure calls). The function dsp_fshm_scratchbuffer_get() is used to query the virt/phys addresses and size of that buffer.
• 7.875k can be dynamically allocated and freed by GPP apps. The default allocation granularity is 256 bytes. The allocated memory is persistent and can be used for e.g. virtual soundchip registers (or whatever someone comes up with). The API functions for that are dsp_fshm_alloc() and dsp_fshm_free()
• 0.125k (128 bytes, two cachelines) are used by c64_tools internally to implement "fastcalls"
  

 

[notaz]

So what's left until it's ready for firmware inclusion? The only thing I miss is the load script, but I can hack go64.sh for that I guess.

It might be good to make a test build of firmware with preliminary integration. I also have an idea of small demo .pnd to test drive this, but I'd like to have integration done before I start with it.

 

[bsp]

It's almost done, here's what's still missing:

• Doug Lea memory allocator integration (planned to do that today)
  
• eDMA3 syscalls (essential for high performance stream processing in combination with SRAM)
  
• examples / testcases for these
  
• API documentation clean up / HTML docs (probably via doxygen)
  
• op_dsp_c64.sh script
  
• testing on as many Pandoras as possible
  

If you want to speed things up, you could create the op_dsp_c64.sh script (based on go64.sh), create a (private) test-firmware, and start working on your demo .pnd while I take care of the other tasks.

The 'testing' task could be done by the community, once a test-firmware is publically available (so far I've heard that the DSP examples do work on a classic Pandora).

A nice-to-have feature in the new firmware would be DSP overclocking support, e.g. up to 1Ghz (works fine on my Pandora, I posted the howto and results a few pages back).
I.e. add "System->DSP-Speed" and "System->DSP stress test" utilities.

 

[Rockthesmurf]

@bsp I wonder if you considered using static asserts for things like:

if(COFF_FILE_HEADER_SIZE == sizeof(coff_file_header_t))

It is the sort of thing I typically do more like:

#define PiStaticAssert(COND,MSG) typedef char static_assertion_##MSG[(COND)?1:-1]
PiStaticAssert( COFF_FILE_HEADER_SIZE == sizeof(coff_file_header_t), expected_COFF_FILE_HEADER_SIZE_to_match_sizeof_coff_file_header_t );

With the philosophy I prefer to catch as much as possible at compile time, instead of waiting until runtime.

 

[bsp]

^^ That's a good idea. (http://stackoverflow.com/a/3385536 ?). I'll look into that and add that in the next update. Thanks for the review!

todays' update:

+ integrate Doug Lea's "dl_malloc" engine
+ add dsp_mspace_create() API call
+ add dsp_mspace_malloc() API call
+ add dsp_mspace_memalign() API call
+ add dsp_mspace_free() API call
+ add dsp_mspace_destroy() API call
+ add TC_SHM_MSPACE (15) testcase to c64_tc
+ increase granularity of FSHM allocator to 64 bytes (one cache line)
+ add dsp_fastcall_rpc_send() and dsp_fastcall_rpc_recv()
+ fix cornercase DSP overlay load multiprocess synchronization issue
+ terminate fastcall sequence in kmod.c::fops__release in case client forgot to
  (this fixes a sporadic 20+ second system freeze issue when canceling e.g. the
   TC_RPC_ADD_FASTCALL_BENCHMARK with control-c)
+ added TC_CTRLC (16) (always worked fine) and TC_FASTCALL_CTRLC (17) testcases
+ release http://tkscript.de/c64_tools/c64_tools-19Oct2013_snapshot.tar.gz
 

The new mspace feature can not only be used for shared memory but for all kinds of memory, of course.

It is still optional, though. Applications may choose to manage the memory any way they like, I just figured that most devs probably appreciate the convenience of a readily available malloc() / free() style allocator. Doug Lea's "dl_malloc" is well tested and exists since 1992.

If, for some reason, libc64 is statically linked in a GPL or LGPL application, the linker will discard the dl_malloc code if it is not used.

It is highly recommended to _not_ statically link it, though.

The new dsp_fastcall_rpc_send() and dsp_fastcall_rpc_recv() functions work very similar to dsp_rpc_send() and dsp_rpc_recv(). They are used for asynchronous DSP communication, e.g. call rpc_send(), do sth. else on the GPP, then call rpc_recv(). When using fastcalls, the application _must_ call rpc_recv() even if there are no return values.

Last but not least, the kernel module now takes care of terminating an ongoing fastcall sequence, e.g. when the app. has crashed or was killed via ctrl-c.

 

[notaz]

ok I have the OS package which can at least load core image and run fractal demo. There is still that problem with libc64.so.1, so I added temporary hack for that. I also miss my inc_libc64_mini.h

It's nice that power management is fully working (it's often major PITA to do on OMAP3), I can keep c64.ko loaded and fully suspend the system and resume without problems (what can't be always said about dspbridge). Plus just having c64.ko loaded doesn't seem to use any power.

There does seem to be a kernel crash if c64.ko is loaded after unloading dspbridge (bridgedriver.ko) though, here is a log:

root@notaz-openpandora2:~# modprobe bridgedriver
[   58.486816] bridgedriver: module is from the staging directory, the quality is unknown, you have been warned.
[   58.534362] DSP Bridge driver loaded
root@notaz-openpandora2:~# rmmod bridgedriver
[   67.050567] drv_get_first_dev_extension: Failed to retrieve the object handle
[   67.058349] drv_get_first_dev_extension: Failed to retrieve the object handle
[   67.066040] drv_destroy: Failed to store DRV object
[   67.071289] mgr_destroy: Failed to store MGR object
root@notaz-openpandora2:~# 
root@notaz-openpandora2:~# /usr/pandora/scripts/op_dsp_c64.sh 
go64.sh: ARGS="" DSPIMAGE="/lib/dsp/core.out"
[...] loading DSP image from "/lib/dsp/core.out"
[...] reading 68 section headers....................................................................
[...] dsp_autoresume_set: autoresume is now disabled.
[   91.196258] c64: init ok. module build date: Oct 20 2013  (**4MB DSP image size**)
[   91.484832] Unhandled fault: external abort on non-linefetch (0x1008) at 0xe0b98044
[   91.493072] Internal error: : 1008 [#1]
[   91.497192] Modules linked in: c64(O) mailbox bluetooth g_cdc omaplfb(O) pvrsrvkm(O) snd_pcm_oss snd_mixer_oss ipv6 fuse ads7846 hwmon [last unloaded: bridgedriver]
[   91.514221] CPU: 0    Tainted: G         C O  (3.2.51 #903)
[   91.520172] PC is at reg32_bit_clear+0x1c/0x34 [c64]
[   91.525482] LR is at 0xc057a040
[   91.528961] pc : [<bf12c0e0>]    lr : [<c057a040>]    psr: a0000013
[   91.528961] sp : d61abeb8  ip : beffffff  fp : 00011e08
[   91.541290] r10: 0000ef05  r9 : d61aa000  r8 : d617e180
[   91.546875] r7 : 00000005  r6 : bf134070  r5 : 00000000  r4 : 00000001
[   91.553924] r3 : e0b98044  r2 : 00000012  r1 : 00000012  r0 : e0b98044
[   91.560852] Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   91.568542] Control: 10c5387d  Table: 96230019  DAC: 55555555
[   91.574645] Process c64_load (pid: 2015, stack limit = 0xd61aa238)
[   91.581329] Stack: (0xd61abeb8 to 0xd61ac000)
[   91.585998] bea0:                                                       bf132ed4 bf12c9d0
[   91.594757] bec0: bf134070 bf132edc d620d780 bf12ea94 d610c804 00000000 d610c7e8 c00a4ce8
[   91.603546] bee0: d610cb00 d62191c0 db3e9aa4 c00b0b38 d610cb00 c00b0e2c d610c808 40849000
[   91.612304] bf00: d62191c0 00005452 00000003 00000003 00000005 d617e180 d61aa000 00000000
[   91.621002] bf20: 00011e08 c00dfaf8 00000000 00000003 c05b61d0 d620d680 00000001 db3e99c0
[   91.629760] bf40: 00000003 d610c804 d610c808 80000013 d620d780 40849000 d620d680 d61aa000
[   91.638519] bf60: 00400000 00000000 d620d780 00000003 0000ef05 00000005 c0013e28 00000000
[   91.647216] bf80: 00011e08 c00dffa4 00000000 00000000 00048000 00011de8 00000044 00012e38
[   91.655975] bfa0: 00000036 c0013c80 00011de8 00000044 00000005 0000ef05 00000003 4012e268
[   91.664764] bfc0: 00011de8 00000044 00012e38 00000036 00011e14 00011ddc 00011dc0 00011e08
[   91.673522] bfe0: 00000000 befdec08 4011cc2c 401ea63c 60000010 00000005 00015b3c 0005eb1c
[   91.682220] [<bf12c0e0>] (reg32_bit_clear+0x1c/0x34 [c64]) from [<bf12c9d0>] (dsp_poweron_and_reset+0x8c/0x10c [c64])
[   91.693572] [<bf12c9d0>] (dsp_poweron_and_reset+0x8c/0x10c [c64]) from [<bf12ea94>] (fops__unlocked_ioctl+0x6f8/0x154c [c64])
[   91.705657] [<bf12ea94>] (fops__unlocked_ioctl+0x6f8/0x154c [c64]) from [<c00dfaf8>] (do_vfs_ioctl+0x80/0x4f4)
[   91.716339] [<c00dfaf8>] (do_vfs_ioctl+0x80/0x4f4) from [<c00dffa4>] (sys_ioctl+0x38/0x60)
[   91.725219] [<c00dffa4>] (sys_ioctl+0x38/0x60) from [<c0013c80>] (ret_fast_syscall+0x0/0x30)
[   91.734283] Code: e3a01004 eb3c1cf8 e2503000 08bd8010 (e5932000) 
[   91.740783] ------------[ cut here ]------------
[   91.745819] WARNING: at arch/arm/mach-omap2/omap_l3_smx.c:186 omap3_l3_app_irq+0x12c/0x134()

This could be dspbridge's fault, but maybe you can do something about it.

 

[bsp]

There does seem to be a kernel crash if c64.ko is loaded after unloading dspbridge (bridgedriver.ko) though

Yea I noticed that when I tried dspbridge last week. Looks like a power management issue, i.e. the DSP probably cannot be powered on after dspbridge has been used and that's why the registers are not available and an access to them causes the non-linefetch error.


[64126.819885] c64::dsp_poweroff: IVA2_PRM_PWSTST=0x00000000 => DSP is OFF
[64126.821472] c64: exit OK.
[64150.421020] bridgedriver: module is from the staging directory, the quality is unknown, you have been warned.
[64150.507934] proc_load: Processor Loaded /lib/dsp/baseimage.dof
[64150.510955] Unhandled fault: external abort on non-linefetch (0x1028) at 0xfb038034
[64150.511077] Internal error: : 1028 [#1]
[64150.511138] Modules linked in: bridgedriver(C+) mailbox_mach mailbox wl1251_sdio wl1251 aufs squashfs xz_dec aes_generic bluetooth mac80211 cfg80211 ecb omaplfb(O) pvrsrvkm(O) zram(C) snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss ipv6 fuse ads7846 hwmon [last unloaded: c64]
[64150.513183] CPU: 0 Tainted: G C O (3.2.45 #895)
[64150.513336] PC is at omap_dm_timer_start+0x48/0xb8
[64150.513427] LR is at 0xfb038034
[64150.513488] pc : [<c0036368>] lr : [<fb038034>] psr: 20000013
[64150.513488] sp : d7bcdad8 ip : 00000001 fp : db92db80
[64150.513702] r10: d7ac3000 r9 : 00000001 r8 : 00000001
[64150.513854] r7 : 00000001 r6 : 00000001 r5 : 00000003 r4 : db8f4600
[64150.513916] r3 : c05872fc r2 : 00000000 r1 : 00000000 r0 : 00000000
[64150.514068] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[64150.514160] Control: 10c5387d Table: 97b1c019 DAC: 55555555
[64150.514282] Process modprobe (pid: 2974, stack limit = 0xd7bcc238)
[64150.514434] Stack: (0xd7bcdad8 to 0xd7bce000)
[64150.514495] dac0: 00000005 bf266a40
[64150.514648] dae0: 00000002 bf24b000 00000000 00000090 daecac00 00000001 bf262458 bf249d94
[64150.514739] db00: 00000090 daecaf64 daecac00 00000000 d7bcdb44 bf249378 00000000 d7bcdb3c
[64150.514892] db20: 00000001 00000001 daecaf80 dc8f0000 bf266cdc dc8f0004 00000001 00000000
[64150.515014] db40: c00cebf0 00000000 00000000 00000011 00003843 00000090 00000010 303d4449
[64150.515167] db60: c0592400 11000000 00000000 00000000 d0c43e80 00000001 00000000 db92db80
[64150.515258] db80: 00003843 0000001c 00000032 bf258b94 d7bcdbb5 d0d70300 20255c00 da7ce000
[64150.515380] dba0: d0c43e80 d7bcdbb5 d0d70300 bf2594bc 000080d0 696c2f00 73642f62 61622f70
[64150.515533] dbc0: 6d696573 2e656761 00666f64 00000000 00000000 00000000 00000000 00000000
[64150.515625] dbe0: 00000000 80080008 00000000 00000000 00000000 00000000 00000000 db802300
[64150.515777] dc00: d7bcdc60 c00c7d04 c05d8dc4 d955ee00 d955ee00 bf24b760 d7bcdc40 bf250c10
[64150.515899] dc20: d955ee00 bf24b7a8 bf24b760 d955ee64 bf24b7f0 bf24ba08 00000032 d955ee54
[64150.516052] dc40: bf24b7e8 bf24b7e8 00000000 00000000 00000000 c007c801 00000000 00000000
[64150.516143] dc60: 00000000 00000000 00000000 bf24b6d8 bf24b760 bf24b7a8 bf24b828 bf24b7f0
[64150.516296] dc80: dc400000 d0db0840 00000032 c00c7d04 bf24b7f0 00000000 d0db0840 d7bcdcd8
[64150.516418] dca0: bf266808 bf251cd4 c05d8dc4 d955ee00 d0db0400 d7bcdbb5 00000000 000046b8
[64150.516571] dcc0: 00000000 d0d70300 bf266c64 d0db0440 bf266808 bf24f3f4 da7ce808 00d70300
[64150.516662] dce0: d7bcde10 d7bcde10 00000000 bf25adbc 4f495400 3150414d 00303135 00000000
[64150.516784] dd00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[64150.516937] dd20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[64150.517028] dd40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[64150.517181] dd60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[64150.517303] dd80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[64150.517456] dda0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[64150.517547] ddc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[64150.517669] dde0: 00000000 00000000 00000000 00000000 da7ce808 d0db0480 d0db0400 d0c42488
[64150.517822] de00: d0db0440 bf2620f4 00000000 db92f808 db92f810 00000000 db92f808 c05e1f3c
[64150.517974] de20: db92f83c bf266838 bf26d000 c027eadc c027eac4 c027d6f0 db92f808 bf266838
[64150.518035] de40: db92f83c 00000000 bf26d000 c027d88c bf266838 c027d800 00000000 c027c7d8
[64150.518188] de60: db805938 db92de30 bf266838 c05a6b10 d0c43900 c027d0c0 bf264c6c d0d48000
[64150.518341] de80: d7ac3540 bf266838 bf2668d4 d7bcc000 00000000 bf26d000 00003843 0000001c
[64150.518432] dea0: 00000032 c027ddac 00000000 c05b5180 bf2668d4 d7bcc000 00000000 bf26d000
[64150.518554] dec0: 0000001c c000866c d7b7bf40 00000000 0000003b d7b7bf40 00000000 bf26691c
[64150.518707] dee0: bf26691c bf2668d4 00000001 d0db0140 00000001 0000001c 00000032 c006d55c
[64150.518859] df00: bf2668e0 0003b000 00000000 bf266a08 d7bcc000 e0e3cb8c e0e3cac4 00000003
[64150.518951] df20: d0c43680 e0e1a000 0003a31b e0e408a0 e0e4068c e0e50ad8 d7b30800 00023ce8
[64150.519073] df40: 00026a78 00000000 00000000 00000030 00000031 00000025 00000029 0000001f
[64150.519226] df60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 c050c944
[64150.519378] df80: 51e15080 00009720 00000000 00000000 00000080 c0013e28 d7bcc000 00000000
[64150.519439] dfa0: 00000000 c0013c80 00009720 00000000 40207000 0003a31b 00016100 00016100
[64150.519592] dfc0: 00009720 00000000 00000000 00000080 00000000 000160a4 00016098 00000000
[64150.519744] dfe0: 000160f0 bef1d964 0000b180 401a3264 60000010 40207000 ebfff17b e5951000
[64150.519927] [<c0036368>] (omap_dm_timer_start+0x48/0xb8) from [<bf24b000>] (dsp_clk_enable+0xb8/0x120 [bridgedriver])
[64150.520172] [<bf24b000>] (dsp_clk_enable+0xb8/0x120 [bridgedriver]) from [<bf249d94>] (dsp_peripheral_clk_ctrl+0xf4/0x168 [bridgedriver])
[64150.520355] [<bf249d94>] (dsp_peripheral_clk_ctrl+0xf4/0x168 [bridgedriver]) from [<bf249378>] (bridge_brd_start+0x2cc/0x4e8 [bridgedriver])
[64150.520629] [<bf249378>] (bridge_brd_start+0x2cc/0x4e8 [bridgedriver]) from [<bf258b94>] (proc_start+0x90/0x118 [bridgedriver])
[64150.520843] [<bf258b94>] (proc_start+0x90/0x118 [bridgedriver]) from [<bf2594bc>] (proc_auto_start+0x1c0/0x204 [bridgedriver])
[64150.521057] [<bf2594bc>] (proc_auto_start+0x1c0/0x204 [bridgedriver]) from [<bf24f3f4>] (api_init_complete2+0x8c/0xa8 [bridgedriver])
[64150.521270] [<bf24f3f4>] (api_init_complete2+0x8c/0xa8 [bridgedriver]) from [<bf25adbc>] (dsp_init+0xec/0x100 [bridgedriver])
[64150.521484] [<bf25adbc>] (dsp_init+0xec/0x100 [bridgedriver]) from [<bf2620f4>] (omap34_xx_bridge_probe+0x194/0x2f4 [bridgedriver])
[64150.521667] [<bf2620f4>] (omap34_xx_bridge_probe+0x194/0x2f4 [bridgedriver]) from [<c027eadc>] (platform_drv_probe+0x18/0x1c)
[64150.521820] [<c027eadc>] (platform_drv_probe+0x18/0x1c) from [<c027d6f0>] (driver_probe_device+0x9c/0x1ac)
[64150.521972] [<c027d6f0>] (driver_probe_device+0x9c/0x1ac) from [<c027d88c>] (__driver_attach+0x8c/0x90)
[64150.522064] [<c027d88c>] (__driver_attach+0x8c/0x90) from [<c027c7d8>] (bus_for_each_dev+0x64/0x88)
[64150.522216] [<c027c7d8>] (bus_for_each_dev+0x64/0x88) from [<c027d0c0>] (bus_add_driver+0x1ec/0x2ac)
[64150.522369] [<c027d0c0>] (bus_add_driver+0x1ec/0x2ac) from [<c027ddac>] (driver_register+0x78/0x17c)
[64150.522491] [<c027ddac>] (driver_register+0x78/0x17c) from [<c000866c>] (do_one_initcall+0x34/0x174)
[64150.522644] [<c000866c>] (do_one_initcall+0x34/0x174) from [<c006d55c>] (sys_init_module+0xe0/0x1b80)
[64150.522796] [<c006d55c>] (sys_init_module+0xe0/0x1b80) from [<c0013c80>] (ret_fast_syscall+0x0/0x30)
[64150.522888] Code: e5845044 e3a00000 e8bd8038 e594e028 (e59ec000)
[64150.523040] ------------[ cut here ]------------
[64150.523132] WARNING: at arch/arm/mach-omap2/omap_l3_smx.c:161 omap3_l3_app_irq+0xe8/0x134()
[64150.523284] In-band Error seen by MPU at address 0
[64150.523345] Modules linked in: bridgedriver(C+) mailbox_mach mailbox wl1251_sdio wl1251 aufs squashfs xz_dec aes_generic bluetooth mac80211 cfg80211 ecb omaplfb(O) pvrsrvkm(O) zram(C) snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss ipv6 fuse ads7846 hwmon [last unloaded: c64]
[64150.525390] [<c001a6a0>] (unwind_backtrace+0x0/0xf8) from [<c003f21c>] (warn_slowpath_common+0x54/0x64)
[64150.525543] [<c003f21c>] (warn_slowpath_common+0x54/0x64) from [<c003f2c0>] (warn_slowpath_fmt+0x30/0x40)
[64150.525634] [<c003f2c0>] (warn_slowpath_fmt+0x30/0x40) from [<c002eb70>] (omap3_l3_app_irq+0xe8/0x134)
[64150.525787] [<c002eb70>] (omap3_l3_app_irq+0xe8/0x134) from [<c007b6c0>] (handle_irq_event_percpu+0x54/0x194)
[64150.525939] [<c007b6c0>] (handle_irq_event_percpu+0x54/0x194) from [<c007b828>] (handle_irq_event+0x28/0x38)
[64150.526092] [<c007b828>] (handle_irq_event+0x28/0x38) from [<c007d544>] (handle_level_irq+0x84/0x108)
[64150.526245] [<c007d544>] (handle_level_irq+0x84/0x108) from [<c007afc4>] (generic_handle_irq+0x30/0x40)
[64150.526397] [<c007afc4>] (generic_handle_irq+0x30/0x40) from [<c0014b54>] (handle_IRQ+0x38/0x84)
[64150.526489] [<c0014b54>] (handle_IRQ+0x38/0x84) from [<c0013834>] (__irq_svc+0x34/0x78)
[64150.526611] [<c0013834>] (__irq_svc+0x34/0x78) from [<c003f2d0>] (oops_exit+0x0/0x18)
[64150.526763] [<c003f2d0>] (oops_exit+0x0/0x18) from [<0000000b>] (0xb)
[64150.526855] ---[ end trace 2cd474fbe5489dfb ]---
[64150.527069] ---[ end trace 2cd474fbe5489dfc ]---


As you can see, dspbridge crashes in a similar way when it is started after c64.ko has been unloaded (external abort on non-linefetch (0x1028) at 0xfb038034), see the log above. No idea why it's doing that and I have never seen a crash like that with c64_tools alone ((un-)loaded the module dozens of times w/o rebooting)

ok I have the OS package which can at least load core image and run fractal demo. There is still that problem with libc64.so.1, so I added temporary hack for that. I also miss my inc_libc64_mini.h

Nice. I'll add a workaround today for the seemingly broken linker on my dev host, then you can build a properly versioned .so on yours. And you'll get your "mini" include helper

It's nice that power management is fully working (it's often major PITA to do on OMAP3), I can keep c64.ko loaded and fully suspend the system and resume without problems (what can't be always said about dspbridge). Plus just having c64.ko loaded doesn't seem to use any power.

Implementing the PM / power off was not as easy as I hoped, indeed. The HW only shuts down if all dependencies are met and the TRM did not mention the details. Well, it's working for two weeks now and I have not seen any issues with it.

 

[notaz]

OK I wrote my first DSP component today, and doing it felt a bit.. uncomfortable.

First thing, your build system doesn't allow out-of-tree (out of c64_tool folder) components, as the linker scripts (pre.cmd, link_area*.cmd, post.cmd) are supplied with relative path to cl6x, and I'm certainly going to need custom DSP code, as well as some existing projects perhaps.

Next this black magic

#pragma DATA_SECTION(component_test_dsp, ".sec_com");

..could be be hidden away in DSP_COMPONENT_MAIN by passing component structure to the macro like DSP_COMPONENT_MAIN(component_test_dsp). Or maybe it's not even needed, as everything seems to work without that pragma?

dsp_component_t could also move component_id to the bottom of the structure so that /* component id, assigned in mlb_component_register() */ doesn't have to be specified.