[announce] c64_tools (DSP loader and IPC)

[rohezal]

This is epic . Fast blitting via dsp. I would love see other code with. Maybe a sound chip for the N64. Or a few percentage (like 20%) rendered by the gpu the other half by the dsp.

 

[bsp]

I think this is quite good, indeed.

the only reason why there was no src update yesterday is that I was struggling a bit with getting double buffering via /dev/fb right (got it now) and I also have to fix some minor memory arbitration issue (DMA seems to have prio over the LCD at the moment -- the whole screen flickers every now and then -- I've seen this before and I think I know how to fix this).

..and before someone starts pondering why the heck anyone would want to render 1500 32x32 ARGB32 sprites with srcover blending @63 FPS on the DSP: It's a benchmark -- just render one tenth of that and enjoy your extended battery life!

p.s.: This reminds me: After DSP hell, now I really have to code a bullet hell shoot-em-up. Always wanted to do that

 

[rohezal]

p.s.: This reminds me: After DSP hell, now I really have to code a bullet hell shoot-em-up. Always wanted to do that

This is terrible you can now have 1500 32x32 bullets flying around you. Must be a very hard game .

 

[Hồng Thất Công]

p.s.: This reminds me: After DSP hell, now I really have to code a bullet hell shoot-em-up. Always wanted to do that

This is terrible you can now have 1500 32x32 bullets flying around you. Must be a very hard game .

Let's recreate Sega Saturn's Radiant Silvergun on the Pandora!

 

[bsp]

This is terrible you can now have 1500 32x32 bullets flying around you. Must be a very hard game .

you should see my previous games. That one game (3d wipeout-style racer) ..10% of the users absolutely loved it, 90% struggled with the first level. In my book that's a win, better than 50% liking it "somehow / in a way".

..have to port that some day..

Let's recreate Sega Saturn's Radiant Silvergun on the Pandora!

heh, if you take care of the graphics. I am thinking of sth along the lines of TUMIKI fighters (lots of 'particles').

 

[Cloudef]

I really do not know about GLES on the Pandora, still did not get around to trying it. I also did not find any reliable fill rate and power consumption figures about the SGX530 on the web so I really cannot do any comparisons, yet.

I can say that the overdraw is the worst bottlenecks I've found on pandora's SGX.

(with transparency of course, hidden surface removal deals with it quite well for opaque geometry)

But it's pretty easy to deal with it using convex shapes and such (reducing overdraw).

You won't really hit bandwidth, if you optimize vertexdata to use less memory (drawcall batching helps as well I guess).

Of course texture switching is quite bad as well, but can be easily avoided with atlas textures.

Neverthless, your sprite system sounds interesting and wondering if it could be used with SGX (on framebuffer context) without flickering for onscreen sprites / text.

p.s.: This reminds me: After DSP hell, now I really have to code a bullet hell shoot-em-up. Always wanted to do that

Also great!

 

[Rockthesmurf]

Basically agree with Cloudef above, the Pandora isn't alone here, but fill rate is typically the enemy. Feels like we are edging closer and closer to having the c64_tools rolled out and seeing the first cool real world application using it..!

 

[bsp]

^^ It will take quite some time to write that game, don't expect to play that anytime soon (but you know how much work a game means)


Today I played around with the dsprite engine. Fixed a bug in the premultiplied blend fxn which now uses saturated adds to deal with overflows.

The c64_dsprite testcase also has quite a number of command line options now, I also added an 8x8 test image (the (default) logo image is 32x32).

For bullet hell, run it like this:
 (on a CC/Rebirth Pandora you may want to reduce the number of sprites for 60Hz vsync)

pandora:/home/bsp# ./c64_dsprite 1 4000 1 100 1.0 1 1
[...]         mode: MODE_ALPHATEST
[...]     #sprites: 4000
[...]   clear_mode: GPP
[...]  #iterations: 100
[...]        slomo: 1.000000
[...]        vsync: YES
[...]       img_nr: 8x8
[...] omapfb_plane_enable: ok, plane 0 is now disabled.
[...] omapfb_plane_setup: ok, plane 1 is now disabled.
[...] omapfb_plane_setup: OK, plane 1 is configured size=(800; 480) geo=(0; 0; 800; 480) virt=(816; 992) bpp=32.
[...] omapfb_plane_setup_dsp_mem: plane 1 virt_addr=0x40827000, phys_addr=0x9c900000 size=0x00317000.
[...] omapfb_init: OK.
[...] coff_load_overlay: text=0x00000ca0  data=0x00000068  bss=0x00001110
[dbg] fshm_img: phys_addr=0x5cf0e000, virt_addr=0x4009a000
[dbg] fshm_img: DSP phys_addr=0x10f0e000
[dbg] loc_init: loc_sprite_init() ok.
[...] 100 iterations in 1670 millisecs.
[...]  ==> avg FPS=59.880
[...] omapfb_plane_enable: ok, plane 1 is now disabled.
[dbg] omapfb_exit: [1] unmapped 3239936 bytes @virt=0x40827000 (phys=0x9c900000).
[...] omapfb_plane_enable: ok, plane 0 is now enabled.
pandora:/home/bsp#  
The full options are:


pandora:/home/bsp# ./c64_dsprite -h
Usage: ./c64_dsprite [mode] [num_sprites] [clear_mode] [num_iterations] [speed] [vsync] [img_nr]

             mode: 0=copy, 1=alphatest, 2=premul srcover + saturate, 3=srcover [default is 2]
      num_sprites: 1..4096 [default is 256]
       clear_mode: 0=none, 1=GPP (memcpy), 2=DSP (EDMA) [default is 1]
   num_iterations: 1..60*60*60 (~1h at 60Hz) [default is 60*10]
            speed: sprite movement speed scale (float value) [default is 1.0]
            vsync: 1=on, 0=off [default is 1]
           img_nr: 0=logo (32x32), 1=bullet (8x8) [default is 0]




And here's the c64_tools update (the changelog is the shortened diff to the last 'dist' release (05Oct2013)):
 
Download: http://tkscript.de/c64_tools/c64_tools-25Oct2013_dist.tar.gz

EDIT: BBS URL bug, correct d/l addr is


http://tkscript.de/c64_tools/c64_tools-25Oct2013_dist.tar.gz

 

 25-Oct-2013
   - fixed: DSP build dependencies
   - changed: DSP overlays can now be compiled w/o installing DSP/BIOS
   - changed: build versioned libc64.so.1
   - changed: renamed c64_msg_t to dsp_msg_t
   - changed: renamed dsp_shm_init()/exit() to dsp_shm_alloc()/free()
   - added: dsp_shm_alloc() memory attributes DSP_CACHE_NONE, DSP_CACHE_R (write-through), DSP_CACHE_RW
   - added: dsp_fshm_alloc() and free() (and associated ioctls). Used for L1DSRAM mem. allocation in userspace.
             The allocation granularity is 64 bytes (one cacheline).
   - added: malloc()/free() style memory allocator for e.g. shared memory. Uses Doug Lea's "dl_malloc".
             New API functions: dsp_mspace_create()/malloc()/memalign()/free()/destroy()
   - fixed: terminate fastcall-sequence in kmod/dev.c in case client forgot to
   - fixed: multiprocess overlay load race condition
   - added: dsp_virt_to_phys() and dsp_phys_to_virt() API calls
   - added: dsp_cache_wbinvall() API call
   - added: DSP ringbuffer for printf(). New DSP syscalls puts()/printf()/vprintf(). New API call dsp_logbuf_print()
   - added: new DSP syscalls qdma_init()/wait()/copy1d()/copy2d()
   - added: new DSP example components test_logbuf, test_qdma, dsprite
   - added: new GPP examples "c64_minimal" and "c64_dsprite"
   - added: new "tests/c64_tc.c" testcases:
                11: TC_SHM_CACHE_NONE
                12: TC_SHM_CACHE_RW
                13: TC_SHM_CACHE_R
                14: TC_FSHM_ALLOCATOR
                15: TC_SHM_MSPACE
                16: TC_CTRLC
                17: TC_FASTCALL_CTRLC
                18: TC_LOGBUF
                19: TC_LOGBUF_MANY
                20: TC_LOGBUF_SPLITLINE
                21: TC_LOGBUF_NO_LF
                22: TC_QDMA_COPY1D
                23: TC_QDMA_COPY2D

@notaz: Today's release should be good enough for firmware integration (the incremental update we talked about)

 

[notaz]

@notaz: Today's release should be good enough for firmware integration (the incremental update we talked about)

Good, I'll try to roll it out over the weekend (I'm slow, I know..)

 

[bsp]

Good, I'll try to roll it out over the weekend (I'm slow, I know..)

thanks! (and man, you gotta work on your reaction time.. 15 minutes..c'mon.. )


Now that we have some real-world graphics benchmark results, I was curious what the power consumption looks like.


Being a big fan of _wb_'s sysinfo, I ran the 'bullet hell' benchmark (see above) and used sysinfo to generate this plot:

(I wonder what it would look like if this were done with NEON or the GPU. (I really have no idea!))

p.s.: this is with WiFi and USB-host off and running on battery power only

 

[Farox]

@bsp incorrect link posted on your post ....correct link to download is http://tkscript.de/c64_tools/c64_tools-25Oct2013_dist.tar.gz

Maybe time for new video about it ?

 

[bsp]

@bsp incorrect link posted on your post ....correct link to download is http://tkscript.de/c..._dist.tar.gz

whoopsie daisy.. fixed it, thanks for noticing!

(EDIT: the URL was correct, seems like the BBS fsckd it up, had to quote it)

(EDIT#2: maybe both, I had to delete the message line, save, then repost the correct URL in order for it to show)

Maybe time for new video about it ?

I don't know, maybe wait for further feedback ? My guts tell me this is good but I have not compared it to GPU/NEON, yet.

 

[Hồng Thất Công]

@bsp incorrect link posted on your post ....correct link to download is http://tkscript.de/c..._dist.tar.gz

whoopsie daisy.. fixed it, thanks for noticing!

Maybe time for new video about it ?

I don't know, maybe wait for further feedback ? My guts tell me this is good but I have not compared it to GPU/NEON, yet.

NEON vs DSP?  Epic fight!  Fight of the Pandora Millenium

 

[bsp]

NEON vs DSP?  Epic fight!  Fight of the Pandora Millenium

relax, this is not a GPP vs DSP fight, this is about whether it is advantageous, in terms of power efficiency, to use the DSP in addition to the GPP.

 

[Hồng Thất Công]

NEON vs DSP?  Epic fight!  Fight of the Pandora Millenium

relax, this is not a GPP vs DSP fight, this is about whether it is advantageous, in terms of power efficiency, to use the DSP in addition to the GPP.

I was just joking...  Sorry  

Of course DSP can coexist with GPP and give the Panda a set of wings

 

[notaz]

I've been looking at cache clean/invalidate code, and I can tell you why it doesn't work. You first allocate those CMA buffers to kernel virtual address space with dma_alloc_attrs(), and then remap that mapping to userspace (different VA) with remap_pfn_range() in fops__mmap(). Now the program accesses memory through it's user mapping, but you do cache operations on the kernel mapping (different addresses). As Cortex-A8 has VIPT (virtually indexed physically tagged) caches, the caches are associated with virtual address, so your operations had no effect.

I'm not sure why you get crashes though, maybe remap_pfn_range() actually removes the kernel mapping and you end up invalidating kernels other data that gets allocated there? I've also read that multiple mappings of the same physical space with different mem attributes is bad thing for ARM, but I can't find it in TRM now. I'll try to backport DMA_ATTR_NO_KERNEL_MAPPING code from newer kernels just in case, so you can uncomment that flag.

In either case you should only use dma_alloc_attrs() return as a handle for dma_free_attrs() and nothing else.

I can also say outer_inv_range() does nothing because it's meant for external L2 controllers (A8 has internal one), so you can remove that code.

DMA_BIDIRECTIONAL is also unimplemented, use dmac_flush_range() for C64_CACHE_AC_WBINV instead.

 

[bsp]

I've been looking at cache clean/invalidate code, and I can tell you why it doesn't work. You first allocate those CMA buffers to kernel virtual address space with dma_alloc_attrs(), and then remap that mapping to userspace (different VA) with remap_pfn_range() in fops__mmap(). Now the program accesses memory through it's user mapping, but you do cache operations on the kernel mapping (different addresses). As Cortex-A8 has VIPT (virtually indexed physically tagged) caches, the caches are associated with virtual address, so your operations had no effect.

ok that makes sense. As you can read in the source comments and the release notes, the cache inv/wb code is borrowed from TI's CMEM kernel module, so I hoped it would just work. Since it did not, the code is not compiled in (#if 0 block). I also was not aware of the VIPT synonym issue.


Does this mean that the userspace virtual addresses have to be passed to the kernel dmac_*() functions ?

I'm not sure why you get crashes though, maybe remap_pfn_range() actually removes the kernel mapping and you end up invalidating kernels other data that gets allocated there?

I do not know, either. Cache invalidates/writebacks should be harmless resp. just cause unnecessary cache refills which might slow things down a little bit (depending on which memory region is invalidated). Why they would crash is beyond me, and it is hard to debug since the system freezes so there's no way to look at the klog.

For the not so technically inclined reader: do not worry, the code is not compiled in because of that so there are no crash issue with the current c64_tools release.

In either case you should only use dma_alloc_attrs() return as a handle for dma_free_attrs() and nothing else.

I have not seen any general issue with remapping the address range to userspace. cmem/dspbridge/.. do the same thing, otherwise there would be no way to access the shared memory in userspace.


However, there's no reason to have a kvirtaddr for the shared memory but as we both have learnt, there's currently no way to avoid the Linux kernel mapping, i.e. not until DMA_ATTR_NO_KERNEL_MAPPING is implemented.

I can also say outer_inv_range() does nothing because it's meant for external L2 controllers (A8 has internal one), so you can remove that code.

Unfortunately these kind of Linux kernel functions are not documented and I had to assume that the naming of these functions is in accordance with ARM's terminology where the term "inner cache" refers to L1 and "outer cache" to L2 (see Cortex-A8 TRM).


If this really is NOP, the call can be removed, although it is not compiled in right now, anyway.

DMA_BIDIRECTIONAL is also unimplemented, use dmac_flush_range() for C64_CACHE_AC_WBINV instead.

If the call is NOP it can be removed, too, of course. Just like outer_inv_range(), it is currently not compiled in, anyway (#if 0 block)


About the cache functions in general: There currently is no need to use them, anyway.

Since the memory is either uncached (DSP_CACHE_NONE) or write-through (DSP_CACHE_R), applications do not have to writeback caches explicitely.

Reads are always uncached so there's no need to explicitely invalidate caches, either (DSP_CACHE_R was intended to give you write-through, read-cached memory but due to kernel limitations the reads are currently uncached).


A few pages back I also mentioned that DSP_CACHE_RW is currently the same as DSP_CACHE_R since the L_PTE_MT_BUFFERABLE kernel flag does not seem to have any effect.


I too read a forum comment about ARM Linux having issues with multiple virtual memory mappings for one and the same physical address range (the comment just said that, no further explanations), that's why I tried DMA_ATTR_NO_KERNEL_MAPPING which turned out to be not implemented.

It would definitely be nice if you would backport / implement it, like you said.


The cache functions will be needed once DSP_CACHE_R and DSP_CACHE_RW work as intended (aforementioned kernel issue) and it would be nice if this issue could be addressed for the official SZ1.56 release since that will increase shared memory read speed on the GPP-side by approximately factor 3.


As for now - how's the progress on the incremental update ?

 

[notaz]

Does this mean that the userspace virtual addresses have to be passed to the kernel dmac_*() functions ?

Yes.

I'm not sure why you get crashes though, maybe remap_pfn_range() actually removes the kernel mapping and you end up invalidating kernels other data that gets allocated there?

I do not know, either. Cache invalidates/writebacks should be harmless resp. just cause unnecessary cache refills which might slow things down a little bit (depending on which memory region is invalidated). Why they would crash is beyond me, and it is hard to debug since the system freezes so there's no way to look at the klog.

Invalidate without clean will destroy data (C64_CACHE_AC_INV case in your code). Let's say that kernel mapping is freed on remap to userspace, so it's reused by the kernel, and some driver puts some pointers there. It reads these pointers and updates them, but as writeback is default policy, they'll go to L1 or L2 cache line and not go back to memory. Out of the blue your code shows up and invalidates all the range, so data in cache is thrown away. A bit later driver needs to do some work, then ends up fetching old invalid pointers from RAM and crashes.

However, there's no reason to have a kvirtaddr for the shared memory but as we both have learnt, there's currently no way to avoid the Linux kernel mapping, i.e. not until DMA_ATTR_NO_KERNEL_MAPPING is implemented.

It turns out in later kernels DMA_ATTR_NO_KERNEL_MAPPING is implemented for some iommu mappings only, whatever that is.

About the cache functions in general: There currently is no need to use them, anyway.


Since the memory is either uncached (DSP_CACHE_NONE) or write-through (DSP_CACHE_R), applications do not have to writeback caches explicitely.


Reads are always uncached so there's no need to explicitely invalidate caches, either (DSP_CACHE_R was intended to give you write-through, read-cached memory but due to kernel limitations the reads are currently uncached).

That's not true, reads are cached for both DSP_CACHE_R and DSP_CACHE_RW on user mappings, vm_page_prot defines this. Not changing vm_page_prot give you writeback, pgprot_writethrough gives WT and both of them have the reads cached. Kernel mappings have it uncached, like you say.

A few pages back I also mentioned that DSP_CACHE_RW is currently the same as DSP_CACHE_R since the L_PTE_MT_BUFFERABLE kernel flag does not seem to have any effect.

Yes because the default is cached+buffered already.

I too read a forum comment about ARM Linux having issues with multiple virtual memory mappings for one and the same physical address range (the comment just said that, no further explanations), that's why I tried DMA_ATTR_NO_KERNEL_MAPPING which turned out to be not implemented.


It would definitely be nice if you would backport / implement it, like you said.

Like said newer kernels only use the flag in some iommu subsystem, maybe I should try porting it to normal dma_* functions, hmm..

As for now - how's the progress on the incremental update ?

Mostly done, still messing with the kernel a bit..

Edit:

maybe I should try porting it to normal dma_* functions, hmm..

I've looked at it and it doesn't look like it can done without some pretty bad hacks, which would likely break things and would be problematic for kernel updates in the future.. I've hacked those kernel mappings to always be writethrough (for testing) and tested with your remap with also writethrough attrs and no changes were noticed and the performance discrepancy in "./c64_tc 13" benchmark was still there.

 

[bsp]

Does this mean that the userspace virtual addresses have to be passed to the kernel dmac_*() functions ?

Yes.

Ok I will give it a try.

I do not know, either. Cache invalidates/writebacks should be harmless resp. just cause unnecessary cache refills which might slow things down a little bit (depending on which memory region is invalidated). Why they would crash is beyond me, and it is hard to debug since the system freezes so there's no way to look at the klog.

Invalidate without clean will destroy data (C64_CACHE_AC_INV case in your code). Let's say that kernel mapping is freed on remap to userspace, so it's reused by the kernel, and some driver puts some pointers there. It reads these pointers and updates them, but as writeback is default policy, they'll go to L1 or L2 cache line and not go back to memory. Out of the blue your code shows up and invalidates all the range, so data in cache is thrown away. A bit later driver needs to do some work, then ends up fetching old invalid pointers from RAM and crashes.

Hmm ok, on second thought that's true. However, any resulting crashes would still be the after effect of a kernel issue since the c64 driver only invalidates its own memory areas, of course.


Maybe your aforementioned guess is the reason for that ("maybe remap_pfn_range() actually removes the kernel mapping and you end up invalidating kernels other data").


Although, if this were true, the dma_free_attrs() function should be checked, too, in order to make sure that it does not try to unmap the wrong kernel memory (have not seen any issues so far, though.)


I just took a quick look at that function: It effectively calls arch/arm/mm/dma-mapping.c:arm_dma_free(), which uses pfn_to_page() to determine struct page*, then calls __dma_free_buffer() which in turn calls __free_page() for each shared memory page.


Would not this mean that dma_free_attrs() would potentially free random kernel memory if remap_fn_range() actually was buggy and unmapped the kernel memory ? It seems rather unlikely to me since that would quickly result in crashes (?!).

However, there's no reason to have a kvirtaddr for the shared memory but as we both have learnt, there's currently no way to avoid the Linux kernel mapping, i.e. not until DMA_ATTR_NO_KERNEL_MAPPING is implemented.

It turns out in later kernels DMA_ATTR_NO_KERNEL_MAPPING is implemented for some iommu mappings only, whatever that is.

hmm that's a bummer. I was quite surprised to see that the kernel does not implement that feature since to me it seems "the easiest thing" to do -- simply NOT map the memory.

Reads are always uncached so there's no need to explicitely invalidate caches, either (DSP_CACHE_R was intended to give you write-through, read-cached memory but due to kernel limitations the reads are currently uncached).

That's not true, reads are cached for both DSP_CACHE_R and DSP_CACHE_RW on user mappings, vm_page_prot defines this. Not changing vm_page_prot give you writeback, pgprot_writethrough gives WT and both of them have the reads cached. Kernel mappings have it uncached, like you say.

hmm, I just re-ran the mem benchmarks and yes, I did remember that incorrectly.

With DSP_CACHE_NONE, "copy contiguous to paged mem" crawls at ~16 MBytes/sec (=> definitely uncached)


With DSP_CACHE_R the performance is better: "copy contiguous to paged mem" runs at ~330 MBytes/sec.


However, "copy paged to paged" performs at ~990 MBytes/sec so something "fishy" is going on there -- the shared memory (contiguous mem) reads should not be 3 times slower than this.

It would definitely be nice if you would backport / implement it, like you said.

Like said newer kernels only use the flag in some iommu subsystem, maybe I should try porting it to normal dma_* functions, hmm..

The unfortunate thing about this is that we do not even know whether this will fix e.g. the shared memory read-performance issue, or whether not having that flag would cause random kernel memory to be unmapped (which seems, for aforementioned reasons, unlikely to me).

As for now - how's the progress on the incremental update ?

Mostly done, still messing with the kernel a bit..

goodie. thanks for all your efforts!

 

[notaz]

Would not this mean that dma_free_attrs() would potentially free random kernel memory if remap_fn_range() actually was buggy and unmapped the kernel memory ? It seems rather unlikely to me since that would quickly result in crashes (?!).

Yeah it doesn't look like remap_pfn_range() touches the kernel mapping, so your crashes remain a mystery.

However, "copy paged to paged" performs at ~990 MBytes/sec so something "fishy" is going on there -- the shared memory (contiguous mem) reads should not be 3 times slower than this.

I have a theory about this: pandora's RAM module actually has 2 256M RAM chips on 2 different chip selects. Paged memory most likely ends up on first chip (PA < 0x90000000), and if you look at dmesg, CMA tends to reserve space at the end of physical address space, so ends op on the second chip. For whatever reason second may be slower than the first.

It would definitely be nice if you would backport / implement it, like you said.

Like said newer kernels only use the flag in some iommu subsystem, maybe I should try porting it to normal dma_* functions, hmm..

The unfortunate thing about this is that we do not even know whether this will fix e.g. the shared memory read-performance issue, or whether not having that flag would cause random kernel memory to be unmapped (which seems, for aforementioned reasons, unlikely to me).

yeah it's most likely not it (I edited my previous post at the same time you posted about this).