[announce] c64_tools (DSP loader and IPC)

[notaz]

Here's a new release of c64_tools.

Built and tested it today, seems to be good. Will push OS update after ED's python stuff is tested.

Would be nice if you did something like this in install.mk

ifdef CROSS_ROOT
CFLAGS= -I$(CROSS_ROOT)/usr/include
endif

intstead of -I'ing CROSS_ROOT unconditionally. The toolchain I use can find it's headers and libs itself, so I have no such thing as CROSS_ROOT, and end up with this:

cc1: error: include location "/usr/include" is unsafe for cross-compilation


... because CROSS_ROOT is empty. Have to redirect it to /tmp or somewhere each time. And no I don't want to edit install.mk because it'll be overwritten on each update.

You should also make "DSP started @0x86055d60" KERN_DEBUG, like "DSP is OFF" is. It's printing the same thing over and over again and doesn't deserve to be KERN_INFO IMO.

 

[bsp]

ok, thanks for testing + integration.

Here's another (the final?) update (exactly three months after the initial announcement, yay),


** Changes

15-Dec-2013
- changed: the CROSS_ROOT env.var. is optional now
(if it is not set, CROSS_KERNEL _must_ be set)
- added support for CROSS_KERNEL env.var. to force specific kernel source directory
- changed some debug messages to use KERN_DEBUG instead of KERN_INFO
(cosmetic change, only relevant for real (non-X11) consoles)
- fixed: dsp_power_notify() was not called if 'pwrlog' kmod option was set to 0 (false)
- changed: decreased udelay()s in kmod/dsp_c64.c
- added proper DSP startup sync (wait for signal/flag from DSP instead of udelay())
- added 'c64_pwrbench' testcase
(benchmarks DSP suspend/resume latency (~1.37 millisec @800Mhz, ~2.84 millisec @200Mhz))

Download: http://tkscript.de/c64_tools/c64_tools-15Dec2013_dist.tar.gz

 

[Askarus]

(exactly three months after the initial announcement, yay),

That's one month to much.

 

[levi]

Still, it's shorter than Two Months

 

[Askarus]

Still, it's shorter than Two Months

That's true.

That's so true.

 

[bsp]

Final(?) stats:

 3348 GPP .h
20428 GPP .c
 1573 DSP .h
 6167 DSP .c
  600 GPP+DSP makefiles
-------------------------
32116 LOC

10753 GPP/DSP gfx_tests .c
  900 GPP/DSP gfx_tests .h
  132 GPP/DSP gfx_tests makefiles
-------------------------
11521 LOC


32116 LOC + 11521 LOC = 43637 LOC

319 hours total (30Aug2013 - 15Dec2013)

=> 136.793 LOC/h

'Man year' =~ 2000 hours

=> (319/2000) * 12 = 1.914 'Man months'

With careful application of the waterfall model, proper requirement/design/configuration/project/change/test/deployment/integration management, a few additional devs, and, last but not least, several dozens of meetings and report sheets, it could have probably been done in a year, maybe two. All for the low, low price of 494352 US$ (always remember to outsource the devs!). yay!

 

[Hồng Thất Công]

Sad but true!

 

[sparkymark79]

Many Thanks for your effort on this. It all sounded very technical and confusing but I know it was a lot of work.

 

[M-HT]

I've been looking at the sources and reading OMAP documentation and I have some questions about caching.

L1DSRAM:

According to documentation, it's not cached on DSP side. What about the GPP side? I think it shouldn't be cached, because fastcall uses it, but asking doesn't hurt.

L2SRAM:

It's cached on the DSP side, but cache coherency is maintained for data accesses from the DSP CPU and through the IVA slave port (cache coherency for instructions is not maintained). Are accesses from GPP side going through the IVA slave port? And is L2SRAM cached on the GPP side?

Shared memory (allocated with dsp_shm_alloc):

On GPP side it's cached based on parameter to dsp_shm_alloc. How is it cached on the DSP side?

 

[notaz]

I can answer the GPP part - being cached or not is not a property of the memory but ARM itself, ARM just has 4GB address space and you can define which areas of it are cached (to ARM's L1, L2 or both) by setting attributes in ARM's page tables.

Or did you mean something like accessing main RAM though the DSP's caches somehow?

 

[M-HT]

I can answer the GPP part - being cached or not is not a property of the memory but ARM itself, ARM just has 4GB address space and you can define which areas of it are cached (to ARM's L1, L2 or both) by setting attributes in ARM's page tables.

I'm aware of that, the question is whether the memory areas (L1DSRAM and L2SRAM) are cached (and how) or not. Similarly for the DSP side.

Or did you mean something like accessing main RAM though the DSP's caches somehow?

I'm essentially asking that when I'm passing data through the various shared memory areas (to and from the DSP), on which of the areas should I call cache invalidation/writeback (on GPP side and on DSP side) ?

 

[M-HT]

Since nobody answered, I did some testing (and read more source codes). The testing consisted of writing data on the GPP side and reading it on DSP side and vice versa.

On the GPP side:

L1DSRAM and L2SRAM are not cached. Shared memory is cached based on parameter to dsp_shm_alloc.

I found this in the source codes and confirmed with testing.

Only exception is shared memory with DSP_CACHE_HUGETLB. Caching is not explicitly set in the source code (at least I didn't find it). Documentation states "GPP must call dsp_cache_wb() before DSP can see data", but I didn't need it in my testing. Maybe my testing was not good enough in this case, also the behaviour may change in the future, so it's safer to call dsp_cache_wb according to documentation.

On the DSP side:

I didn't use cache_inv, cache_wb or other functions in my testing and everything worked.

Only when testing L2SRAM, two times I had wrong results, but I couldn't reproduce it later. Before the first test I did run some bad code on the DSP, and during the second test my CPU was overclocked to 800 MHz - maybe these were the causes of the wrong results, maybe it was something different.

Now the less fun part.

During testing, two times I lost XFCE menu and taskbar, so the only solution was HW reset. The first time, one SD card and the NAND became corrupted. I had to reflash because my Pandora was unbootable.

I don't know if the cause was the DSP or something else - I did the testing through ssh connection.

Also one time when I was resetting the system, it didn't shutdown properly - again I needed HW reset.

 

[Magic Sam]

Hi,

@ M-HT, regarding your "less fun part", I reported a similar issue here.

At least I'm not alone anymore

Bye, Magic Sam

 

[notaz]

What pandoras do you both have?
 

Only exception is shared memory with DSP_CACHE_HUGETLB.

I've hacked hugetlb to always be write-through cached in the kernel for my own greedy purposes (still haven't released what I wanted to use it for, oh well). It'll probably stay that way but yeah, it's a good idea to call dsp_cache_wb() anyway.
 

On the DSP side:
I didn't use cache_inv, cache_wb or other functions in my testing and everything worked.
Only when testing L2SRAM, two times I had wrong results, but I couldn't reproduce it later. Before the first test I did run some bad code on the DSP, and during the second test my CPU was overclocked to 800 MHz - maybe these were the causes of the wrong results, maybe it was something different.

I did similar test on "normal" shared memory and I'm quite sure it was cached on DSP side. Are you reading the value before updating it? If you only write in a loop, the DSP might not have that line cached and it probably doesn't write-allocate cachelines (ARM also doesn't).
 

During testing, two times I lost XFCE menu and taskbar, so the only solution was HW reset. The first time, one SD card and the NAND became corrupted. I had to reflash because my Pandora was unbootable.

I did quite a lot of messing around, including bad DSP code and all was fine (had some crashes but no corruption). Of course it could just be luck, there could be some stale cache in DSP's path, timing violations or something that cause rare errors. I know individual chips have different tolerance for these things, for example when first pandoras started shipping, a few users reported that audio was playing white-noise like sounds instead of what it should. It turned out McBSP clock edge was incorrectly programmed, but it only affected like 1 every 20 units, all other were fine with the wrong setting. So you might have unique opportunity to debug and fix that issue B)

 

[M-HT]

What pandoras do you both have?

I have an original CC Pandora.

Only exception is shared memory with DSP_CACHE_HUGETLB.

I've hacked hugetlb to always be write-through cached in the kernel for my own greedy purposes (still haven't released what I wanted to use it for, oh well). It'll probably stay that way but yeah, it's a good idea to call dsp_cache_wb() anyway.

Or maybe in the future, hugetlb memory will be allocated in kernel module and not in userspace.

On the DSP side:

I didn't use cache_inv, cache_wb or other functions in my testing and everything worked.

Only when testing L2SRAM, two times I had wrong results, but I couldn't reproduce it later. Before the first test I did run some bad code on the DSP, and during the second test my CPU was overclocked to 800 MHz - maybe these were the causes of the wrong results, maybe it was something different.

I did similar test on "normal" shared memory and I'm quite sure it was cached on DSP side. Are you reading the value before updating it? If you only write in a loop, the DSP might not have that line cached and it probably doesn't write-allocate cachelines (ARM also doesn't).

Not in the same function call. I'll have to test it.I also didn't test fastcall, only normal call. I'll also have to test that.

During testing, two times I lost XFCE menu and taskbar, so the only solution was HW reset. The first time, one SD card and the NAND became corrupted. I had to reflash because my Pandora was unbootable.

I did quite a lot of messing around, including bad DSP code and all was fine (had some crashes but no corruption). Of course it could just be luck, there could be some stale cache in DSP's path, timing violations or something that cause rare errors. I know individual chips have different tolerance for these things, for example when first pandoras started shipping, a few users reported that audio was playing white-noise like sounds instead of what it should. It turned out McBSP clock edge was incorrectly programmed, but it only affected like 1 every 20 units, all other were fine with the wrong setting. So you might have unique opportunity to debug and fix that issue B)

I have no sure way of reproducing the issue and I have no idea how I would debug it (and not that much free time) so I'll pass.

 

[Magic Sam]

Hi,

@ notaz: I have a Rebirth Pandora.

Bye, Magic Sam

 

[bsp]

@M-HT: regarding your cache questions:

L1/L2 SRAM:

 The part of L1 SRAM that is used for caching main memory data (L1DSRAM) is read-allocate/write-back, i.e. only reads will allocate a cacheline, writes will be cached if an allocated cacheline exists, write-misses will go through a 4x64bit write buffer straight to main memory.

 You have to use the cache_*() functions to writeback any data that has not been comitted to main memory, yet.

 L2D is not used at all since I couldn't get it to work (neither could DSP/BIOS..). That's why all of L2 SRAM is configured as flat/general purpose memory (~60% as fast as L1, i.e. ~1.83 GB/sec read or write).

 The part of L1/L2 SRAM that is not used for caches is not associated with any main memory so no cache_*() calls are required.

 According to my tests (and yours), the L1/L2 SRAM is not cached on the GPP side. In fact, reads/writes from/to that memory from the GPP side are very(!) slow (~15 MB/sec vs. ~3 GB/sec on DSP side). It's still fast enough for quickly passing small amounts of data (e.g. 'fastcall' messages), though.

 As I mentioned a while ago, I found out that there seems to be some kind of GPP<>DSP access issue when both processors simultaneously write to the same DSP cache line (the system crashed/reset).
 That's why I placed the TO_DSP and TO_GPP 'fastcall' messaging 'register' sets in different cache lines.
 Just something to keep in mind, although it's not a common DSP usecase, of course.



Shared (main) memory:

 This is always cached on the DSP side (cacheability on the DSP side can only be configured in 32MB blocks).

 On the GPP side there is up to one memory block for each cache type (and process). The dsp_shm_alloc() parameter is used to select/allocate the memory block (the idea is that apps. should then use the dsp_mspace_*() calls for memory management).

 The HUGETLB type is a special case, though: The userspace API ensures that the application can only do one allocation (which may span multiple pages, though), and libc64 will create a lookup table so e.g. dsp_virt_to_phys() can quickly map virtual to physical addresses.
 However, the kernel module API does not have this 'one-allocation-only' limitation, i.e. you could write a utility function that allocates a hugetlb page and simply uses the kernel module to determine the physical memory address via the C64_IOCTL_HUGETLB_V2P ioctl.

 Like I mentioned in the source code, allocating those HUGETLB pages in userland is not perfect but then again this is just an experimental hack, more or less done exclusively on behalf of notaz.



Regarding missing icons/NAND/SD corruption:

 Like I said, I for one haven't seen this issue even once on any of my two Pandoras (both DM3730, though) and other users in this thread said the DSP examples worked fine on their OMAP3530 (classic/rebirth) Pandoras.
 
 The DSP code and GPP testcases don't go anywhere near the NAND or SD card controllers, of course.

 To be honest, I have no clue what was causing the issue on magic_sam's and M-HT's units. Maybe it's a coincidence and not related to the DSP at all, maybe it's faulty hardware, maybe it's related to overclocking, ..

 Setting up a couple (~10..20) of CC and Rebirth Pandoras with no history of filesystem issues and letting one half of them run DSP stresstests and the other half run CPU stresstests for a day or so could give an indication whether DSP usage really is increasing the risk of catastrophic failure . Unless ED gets bored and does this, it's not going to happen, though, I guess :/
 

 

[M-HT]

I did more testing of the DSP side. Unless mentioned otherwise, I had the same results for all three types of memory (L1DSRAM, L2SRAM and main memory).

To reiterate - when I tested only reads or only writes I didn't need to call cache_inv()/cache_wb().

Notaz mentioned testing writing after reading - in this test I needed to call cache_wb().

Then I did all tests using fastcall (which really is much faster than standard call) - this time I needed to call both cache_inv() and cache_wb().

When testing L2SRAM, a few times I got wrong results (which I couldn't reproduce) even when I called cache_inv() and cache_wb() - either there's some bug or L2SRAM might not be safe for passing data between DSP and GPP.

___

I found a minor bug, although I'm not sure it affects anything.

In file dsp/install.mk, in line:


LDFLAGS= $(WFLAGS) -z --warn_sections -mv64+
the parameter -mv64+ should be before parameter -z. When it's after, it means writing map file "v64+".___

Unfortunately I managed to corrupt the NAND again and had to reflash.

This time I didn't lose the whole menu, but when I chose "Log out" in the menu nothing happened so I did HW reset and Pandora didn't boot again.

___

Some additional questions:

Function dsp_physgpp_to_physdsp() translates physical GPP address to physical DSP address - where would you get physical GPP address ? Because dsp_mem_region_t contains virtual GPP address and physical DSP address.

Can I use IDMA and how ?

I tried it by writing to registers IDMA1_SOURCE, IDMA1_DEST and IDMA1_COUNT, but it didn't seem to do anything.

 

[bsp]

There's a good reason why you did not need to call invalidate on the DSP side when using the regular RPCs: The DSP mailbox interrupt handler (mlb.c:mlb_irq_handler()) already calls BCACHE_wbInvAll() ! (unfortunately I have no good explanation _why_ this call is needed (it shouldn't be))

This should explain your observations.

In case of fastcalls, there are no 'implicit' writeback/invalidate calls, that's why you had to do them yourself.


Regarding L2SRAM:

I just added four new testcases to the c64_tc application:

• 46: DSP writes random data to L2SRAM and builds checksum, GPP also builds checksum and compares it to DSP checksum; uses regular RPCs; DSP calls cache_wb() after data has been written
• 47: GPP writes random data to L2SRAM and builds checksum, DSP also builds checksum and compares it to GPP checksum, uses regular RPCs; DSP calls cache_inv() before reading data
• 48: DSP writes random data to L2SRAM and builds checksum, GPP also builds checksum and compares it to DSP checksum; uses fastcall RPCs
• 49: GPP writes random data to L2SRAM and builds checksum, DSP also builds checksum and compares it to GPP checksum; uses fastcall RPCs
  
  
  

Each test executes 1000 iterations. In each iteration, 64 KBytes of pseudo-random data is being generated.

I've run these tests several times and the checksums always matched, so I would say that it _is_ safe to use L2SRAM for passing data (at least on my units).

Could you please run these tests, too ? I've attached the source code and binaries.


Regarding the (supposed) linker command line bug in dsp/install.mk:

The command line is fine, I think you mistook '-z' for '-m'. '-z' just tells the compiler/linker frontend that the following options are linker options. I.e. similar to GCC -Wl except that the linker options use a 'normal' syntax instead of being separated by commas.


Regarding your (continuing) NAND troubles:

First of all, it really s*cks that you had to reflash _again_

Since the DSP is only active/powered on while a DSP app. is running, and most likely it was powered down by the time you tried to log out, I'd suspect that for some unknown reason, some kernel memory (filesystem buffers?) got corrupted earlier on. On the other hand, it can't be just a software issue since it only seems to affect certain units.

Q: Did you still overclock your unit ? Did you run any EDMA tests ? Maybe you could try to avoid EDMA for a while to see if it has anything to do with this issue. Since you just reflashed, maybe you could do a fresh reboot and run some non-DMA DSP tests (in a loop) overnight ?!


Regarding dsp_physgpp_to_physdsp():

The documentation is a bit misleading here, the dsp_mem_region_t.phys_addr field stores the GPP physical address. Most of the time this is also the DSP physical address, the only exception being the DSP SRAM memory areas.


Regarding IDMA:

I have never used that. You are probably aware that it's not a general purpose DMA. It is limited to DSP-local memory/SRAM (channel 1) resp. external registers (NOT memory) (channel 0).

According to the docs (spru871k), the src/dst addresses have to be aligned to 32 bytes and the IDMA registers must be written to in a specific order (src addr, dst addr, count). Writing to 'count' should trigger the transfer.

It's probably not worth the trouble to use IDMA, unless you intend to write DSP-side device drivers where register writes become a performance bottleneck.

c64tools_l2sram_testcase_update-09Jan2014.tar.gz

 

[M-HT]

L2SRAM tests:

I ran your tests and they ran correctly. But my tests also ran correctly. Only few times they didn't. And when they didn't and I ran the test again, the same test ran correctly.

The bug in linker command line:

When '-mv64+' is after '-z' it means:


-m,--map_file=file Produce a listing of all input and output sections
into <file>.

When '-mv64+' is before '-z' it means:

  -mv,--silicon_version=id 
                          Target processor version (when not specified,
                           compiler defaults to --silicon_version=6200)

NAND troubles:

I didn't overclock my Pandora (when the corruption occurred). And I didn't test the EDMA yet.

dsp_physgpp_to_physdsp():

I tried dsp_l1sram_scratchbuffer_get(), dsp_l1sram_alloc(), dsp_l2sram_alloc() and dsp_shm_alloc() and I compared dsp_mem_region_t.phys_addr with dsp_virt_to_phys(dsp_mem_region_t.virt_addr) - it's the same value for all four functions.

So either dsp_mem_region_t.phys_addr field is the DSP physical address or the DSP physical address is the same as GPP physical address.

IDMA:

I know it's limited, but it's supposed to be faster than EDMA. It can copy memory, but it can also fill memory with a value - that might be useful sometimes.

Addresses have to be aligned to 32 bytes on channel 0. On channel 1 they must be aligned to 4 bytes.

I tried writing to IDMA registers in specific order, but it didn't seem to do anything, that's why I'm asking if someone already tried it (successfully).

NAND troubles - part 2:

After running you tests a few times (and only your tests) I tried unmounting my SD card (I closed the lid in between, if it makes a difference) and my Pandora froze -> HW reset -> unbootable Pandora -> reflash.