[announce] c64_tools (DSP loader and IPC)

[crow_riot]

CC pandora: 3850 to get about 60hz with small orange sprites and 1350 for the pandora logo test

 

[bsp]

@magic_sam / crow_riot: These results are w/o overclocking ? I expected the difference to be bigger, this looks like the tests are much more RAM limited than I had thought. Very nice. As you know, the DSP on the 1Ghz Pandora is almost twice as fast (800Mhz vs. 430Mhz).

EDIT: indeed. I just downclocked the DSP to 400Mhz and the results are the same, so it is really just the RAM

 

[Farox]

on my CC

3600 to obtain 60.386 FPS @600Mhz

and

1270 the other test 60.096 FPS @600Mhz

 

[bsp]

This all quite cool because this means that the GPU (which I still did not test) is most likely not faster than this.

The DSP module can read and write the entire 800x480 screen 9.17 (RGB565) resp. 4.58 times (ARGB32) at 60Hz!

The srcover codepaths are still heavily unoptimized, that's why they currently only run at half speed. Once optimized, that's when the increased clock rate of the Ghz edition will probably make a bigger difference.

Can anyone please run the 32x32 test with mode=2 (premultiplied srcover) ?

i.e.


./c64_dsprite 2 795 1 100 1.0 0 0
(=> 60.096 FPS on 1Ghz)

 

[Magic Sam]

Here you go:

Spoiler

bash-3.2$ ./c64_dsprite 2 795 1 100 1.0 0 0
[...]         mode: MODE_PREMUL_SRCOVER
[...]     #sprites: 795
[...]   clear_mode: GPP
[...]  #iterations: 100
[...]        slomo: 1.000000
[...]        vsync: NO
[...]       img_nr: 32x32
[...] omapfb_plane_enable: ok, plane 0 is now disabled.
[...] omapfb_plane_setup: ok, plane 1 is now disabled.
[...] omapfb_plane_setup: OK, plane 1 is configured size=(800; 480) geo=(0; 0; 800; 480) virt=(864; 1088) bpp=32.
[...] omapfb_plane_setup_dsp_mem: plane 1 virt_addr=0x4083e000, phys_addr=0x9c900000 size=0x00396000.
[...] omapfb_init: OK.
[...] coff_load_overlay: text=0x00000ca0  data=0x00000068  bss=0x00001110
[dbg] fshm_img: phys_addr=0x5cf0e000, virt_addr=0x4027d000
[dbg] fshm_img: DSP phys_addr=0x10f0e000
[dbg] loc_init: loc_sprite_init() ok.
[...] 100 iterations in 2088 millisecs.
[...]  ==> avg FPS=47.893
[...] omapfb_plane_enable: ok, plane 1 is now disabled.
[dbg] omapfb_exit: [1] unmapped 3760128 bytes @virt=0x4083e000 (phys=0x9c900000).
[...] omapfb_plane_enable: ok, plane 0 is now enabled.

 

[Hồng Thất Công]

This all quite cool because this means that the GPU (which I still did not test) is most likely not faster than this.

The DSP module can read and write the entire 800x480 screen 9.17 (RGB565) resp. 4.58 times (ARGB32) at 60Hz!

The srcover codepaths are still heavily unoptimized, that's why they currently only run at half speed. Once optimized, that's when the increased clock rate of the Ghz edition will probably make a bigger difference.

Can anyone please run the 32x32 test with mode=2 (premultiplied srcover) ?

i.e.


./c64_dsprite 2 795 1 100 1.0 0 0
(=> 60.096 FPS on 1Ghz)

Here you go on Rebirtth at 600Mz:

Spoiler

bash-3.2$ ./c64_dsprite 2 795 1 100 1.0 0 0

[...]         mode: MODE_PREMUL_SRCOVER

[...]     #sprites: 795

[...]   clear_mode: GPP

[...]  #iterations: 100

[...]        slomo: 1.000000

[...]        vsync: NO

[...]       img_nr: 32x32

[...] omapfb_plane_enable: ok, plane 0 is now disabled.

[...] omapfb_plane_setup: ok, plane 1 is now disabled.

[...] omapfb_plane_setup: OK, plane 1 is configured size=(800; 480) geo=(0; 0; 800; 480) virt=(864; 1088) bpp=32.

[...] omapfb_plane_setup_dsp_mem: plane 1 virt_addr=0x4080d000, phys_addr=0x9c900000 size=0x00396000.

[...] omapfb_init: OK.

[...] coff_load_overlay: text=0x00000ca0  data=0x00000068  bss=0x00001110

[dbg] fshm_img: phys_addr=0x5cf0e000, virt_addr=0x400b5000

[dbg] fshm_img: DSP phys_addr=0x10f0e000

[dbg] loc_init: loc_sprite_init() ok.

[...] 100 iterations in 2076 millisecs.

[...]  ==> avg FPS=48.170

[...] omapfb_plane_enable: ok, plane 1 is now disabled.

[dbg] omapfb_exit: [1] unmapped 3760128 bytes @virt=0x4080d000 (phys=0x9c900000).

[...] omapfb_plane_enable: ok, plane 0 is now enabled.

bash-3.2$ 

 

[bsp]

thanks.

For some reason the DSP over/under clocking currently does not work (it did before, I posted the results a few pages back) -- Have to find out why.

The CC/Rebirth results are also a bit surprising, it's still the RAM difference (75% of 1Ghz throughput).

I hope the DSP/BIOS programs the DSP clocks correctly and does not overclock the DSP on CC/Rebirth units (which is unlikely, though). I don't touch the clock setup, have to use the reg_write utility for that.

Well, time to install Notaz' new kernel sources so I can rebuild c64.ko, I guess..

 

[Hồng Thất Công]

Not sure if you need this but here's another run on my Rebirth at 600MHz:

bash-3.2$ ./c64_dsprite 2 615 1 100 1.0 0 0

[...]         mode: MODE_PREMUL_SRCOVER

[...]     #sprites: 615

[...]   clear_mode: GPP

[...]  #iterations: 100

[...]        slomo: 1.000000

[...]        vsync: NO

[...]       img_nr: 32x32

[...] omapfb_plane_enable: ok, plane 0 is now disabled.

[...] omapfb_plane_setup: ok, plane 1 is now disabled.

[...] omapfb_plane_setup: OK, plane 1 is configured size=(800; 480) geo=(0; 0; 800; 480) virt=(864; 1088) bpp=32.

[...] omapfb_plane_setup_dsp_mem: plane 1 virt_addr=0x40782000, phys_addr=0x9c900000 size=0x00396000.

[...] omapfb_init: OK.

[...] coff_load_overlay: text=0x00000ca0  data=0x00000068  bss=0x00001110

[dbg] fshm_img: phys_addr=0x5cf0e000, virt_addr=0x4001b000

[dbg] fshm_img: DSP phys_addr=0x10f0e000

[dbg] loc_init: loc_sprite_init() ok.

[...] 100 iterations in 1666 millisecs.

[...]  ==> avg FPS=60.024

[...] omapfb_plane_enable: ok, plane 1 is now disabled.

[dbg] omapfb_exit: [1] unmapped 3760128 bytes @virt=0x40782000 (phys=0x9c900000).

[...] omapfb_plane_enable: ok, plane 0 is now enabled.

bash-3.2$ 

 

[bsp]

Ok, built the new kernel and recompiled c64.ko with dsp_poweroff() commented out (for testing purposes).

Now changing the DSP clock rate (register 48004040) actually has an effect, although the same DPLL multiplier (default=20) results in a lower performance
(e.g. dsprite test with 795 32x32 sprites runs at 45 FPS instead of 60)

Good thing I kept the clock setup logs from a few weeks ago..

The difference seems to be register 48004004 (CM_CLKEN_PLL_IVA2) which reads 0x15 with dsp_poweroff() enabled (cannot change it with reg_write, always jumps back to 0x15), and 0x17 with dsp_poweroff() disabled) (the 0x15 is written by dsp_poweroff()).

The meaning of these bits is described on p. 430 of the TRM:
  0x15 = 0b10101 => EN_IVA2_DPLL=5 (DPLL in low power bypass mode), IVA2_DPLL_FREQSEL=1  (TRM says that's the resetval (1) and I just verified that it is)
  0x17 = 0b10111 => EN_IVA2_DPLL=7 (DPLL in lock mode),             IVA2_DPLL_FREQSEL=1

This means that DSP over/underclocking only works when the DPLL lock mode is selected.

I am a bit confused now about whether the DSP uses the correct clock rate.

Can someone please run the "c64_fractal_dsponly" test on a CC/Rebirth ?
(it's not RAM limited so that should at least show whether the DSP speed is the same on Ghz+CC. For reference, it takes ~10sec on my Ghz Pandora.)
 

 

[rohezal]

DSP ONLY: ca. 13 secs when I just count the display time. With starting up it were around 17 secs.

 

[Farox]

Here is my CC results:

bash-3.2$ ./c64_fractal_dsponly
[...] omapfb_plane_enable: ok, plane 0 is now disabled.
[...] omapfb_plane_setup: ok, plane 1 is now disabled.
[...] omapfb_plane_setup: OK, plane 1 is configured size=(128; 128) geo=(0; 0; 800; 480) virt=(128; 128) bpp=32.
[...] omapfb_plane_setup_dsp_mem: plane 1 virt_addr=0x4086d000, phys_addr=0x8c050000 size=0x00010000.
[...] omapfb_init: OK.
[...] coff_load_overlay: text=0x00000980  data=0x000005e8  bss=0x00001278
[...] 3600 iterations in 12827 millisecs.
[...] omapfb_plane_enable: ok, plane 1 is now disabled.
[dbg] omapfb_exit: [1] unmapped 65536 bytes @virt=0x4086d000 (phys=0x8c050000).
[...] omapfb_plane_enable: ok, plane 0 is now enabled.
bash-3.2$

 

[bsp]

Thanks, rohezal and Farox, that confirms my assumption.

In the meantime I just wrote a DSP "nop benchmark".

Executing 800,000,000 NOPs takes 2160 milliseconds here which suggests that the DSP is currently clocked at 370Mhz!

EDIT: The test executes 8000000 * 100 NOPs, so the error / overhead due to branching might be so high that it's actually 430Mhz (which is the CC/Rebirth DSP clock rate). Ergo: wrong DSP clock on 1Ghz Pandoras (..should be 800Mhz).

 

[PokeParadox]

Just a quick note that everything worked fine after a reflash - note sure why it tripped up for me before...


Very nice seeing those icons moving around!

 

[rohezal]

Oh, this is bad. But otherwise: The performance on the GHZ will be way better when they are correctly clocked .

Or: if they cant be correctly clocked, ED can get the money for 3000 Pandoras back (or at least a part of it).

 

[bsp]

Nah, don't worry.

I already have a patch that clocks the DSP at the correct rate (800 million NOPs in ~1sec), and tested it with the demos. Works fine without crashes so far.

EDIT: in numbers (on 1Ghz Pandora):

• 1850 32x32 ARGB32 sprites with alphatesting @60Hz  => 113 mpixel/sec
  
  
• 1395 32x32 ARGB32 sprites with premultiplied alpha blending @60Hz  (remember: unoptimized loop, should be 1850)
  
  
• 6150 8x8 ARGB32 sprites with alphatesting @60Hz

c64_fractal_dsponly takes 8386 milliseconds instead of 10880.

c64_fractal_gppdsp takes 4426 milliseconds instead of 5054 (probably hitting the RAM limit)

 

[notaz]

Probably nobody changed the clock setup code for DSP since it was added for OMAP3 (I think it's only done in x-load/u-boot, the kernel doesn't care).

Note that the DSP sits on the same voltage rail as ARM. If the user sets his max OPP lower (/proc/pandora/cpu_opp_max), it will be unsafe to clock the DSP at 800MHz. Also note that currently OPP is automatically reduced when the user underclocks ARM using /proc/pandora/cpu_mhz_max .

I guess the current kernel OPP code needs to be made DSP aware and something like /proc/pandora/dsp_mhz_max needs to be added. Does your code have to reprogram DPLL to bring the DSP up? I wonder if it has to be coordinated with you, or the kernel can handle it by itself.

It kind of complicates things though, like when DSP is not used (c64_tools are not loaded), but dsp_mhz_max is left set with high value, and ARM underclocked, how the code should know it can reduce OPP or not..

BTW TI recommended clocks for DSP/ARM are (DM3730):

OPP1/OPP50: 260/300

OPP2/OPP100: 520/600

OPP3/OPP130: 660/800

OPP4/OPP1G: 800/1000

(OMAP3):

OPP1: 90/125

OPP2: 180/250

OPP3: 360/500

OPP4: 400/550

OPP5: 430/600

 

[bsp]

My initial expectation was that the boot/kernel code would setup the default (OPP4) DSP clocks (~430Mhz for OMAP3530, ~800MHz for DM3730).

I yet have to fully understand the "low power bypass mode" (see http://boards.openpandora.org/topic/14334-announce-c64-tools-dsp-loader-and-ipc/page-21#entry285588) and why it effectively limits the DSP to ~400Mhz speed and completely disregards the DPLL multiplier.

My current patch sets the IVA2 DPLL to "lock mode" during resume, then the DPLL multiplier is set to 0x40 to get to ~800Mhz via reg_write (cmdline) (in "bypass mode" the multiplier has no effect).

The bypass mode has to be set in order for the DSP to power off correctly, so (partially) yes, that part of the DPLL (CM_CLKEN_PLL) has to be done in c64.ko.

"I wonder if it has to be coordinated with you, or the kernel can handle it by itself." -- I would say that it should be the kernel that handles this (multiplier setting), I would not want to configure the DPLL multiplier in c64_tools since lower OPPs won't provide enough power for the full 800Mhz clock rate.

Long story short: Set the DPLL multiplier in the kernel, c64_tools will just toggle between lock and bypass mode.

We need a proper over/underclocking interface, anyway, and this should look very similar to the GPP interface.

 

[crow_riot]

@bsp - i dont remember what exactly, but i'm pretty sure i've overclocked my CC to 800 when I've made the dsprite tests

 

[notaz]

My initial expectation was that the boot/kernel code would setup the default (OPP4) DSP clocks (~430Mhz for OMAP3530, ~800MHz for DM3730).

Actually TI considers those clocks "overdrive" with reduced life expectency, proper defaults according TI are (MPU/DSP) 500/360 for OMAP3 and 600/520 for DM3730.

I still don't know how to implement this opp/clock control thing.. Currently OPP and ARM clock is controlled by cpufreq driver, and /proc/pandora/cpu_*_max files are hacks to constrain how low or high cpufreq can go with OPP and CPU clock. Perhaps I should use the user selected DSP clock to further constrain how low cpufreq can go with OPPs, but then I need to know when DSP is turned on/off to not apply the constrain needlessly.

 

[bsp]

proper defaults according TI are (MPU/DSP) 500/360 for OMAP3 and 600/520 for DM3730.

hmm, I was not aware that a stock "Ghz" Pandora already used overclocking, good to know. On the other hand, the overclocking probably only becomes an issue when the system is under constant load (vs. spending most of the time in deep sleep) ? Or is it already the increased voltage that shortens the life span ?

I still don't know how to implement this opp/clock control thing.. Currently OPP and ARM clock is controlled by cpufreq driver, and /proc/pandora/cpu_*_max files are hacks to constrain how low or high cpufreq can go with OPP and CPU clock. Perhaps I should use the user selected DSP clock to further constrain how low cpufreq can go with OPPs, but then I need to know when DSP is turned on/off to not apply the constrain needlessly.

From what I understand, the problem is very similar to what is already done with the GPP.


E.g. it is probably not safe to run the GPP at 1Ghz with the OPP1 voltage levels, either. Neither does it make sense to use OPP5 voltage to run it at e.g. 100Mhz.


You mentioned that you automatically change the OPP when the (GPP) clock is changed, probably by looking at the table you posted above, then picking the best matching OPP/voltage level.

(and vice versa, when the OPP is changed directly (if that is possible), you inc- or decrease the GPP clock rate so it fits..?)


I would suggest that you do the same with the DSP clock -- determine the minimum OPP required for the desired GPP and DSP clocks and set that one.


You do not have to know whether the DSP is currently powered on/running, you can write to CM_CLKSEL1_PLL_IVA at any time.

I guess the current kernel OPP code needs to be made DSP aware and something like /proc/pandora/dsp_mhz_max needs to be added

Yes, it's the logical thing to do.


I tried to find out about the low power bypass mode today. For one thing, it exists for the GPP as well (you may have noticed that CM_CLKEN_PLL_MPU and CM_CLKEN_PLL_IVA2 look _very_ similar). I found a message in the TI E2E forum but unfortunately it was one of those where someone asks sth. interesting and nobody bothers to answer :/


The question already contained some info, though, e.g. it mentioned that in LPBM, the DPLLs (which exactly..?) are somehow bypassed, saving 3ma per DPLL (does not seem much..how many DPLLs are there anyway..), and that it is meant for low-power usecases like MP3 replay.


I still have no idea how that DPLL bypass is done, respectively where the GPP/DSP clocks are derived from in that case.


Last night I did some power measurement tests and did not find any measurable difference between LPBM and "lock mode". We can do some more tests later on to see if it's worth the effort to care for LPBM (e.g. for decoding an MP3 while the CPU is in deep sleep).



For now, just make sure to initialize the DSP frequency during boot (right next to the code that sets up the GPP freq.), and I'll simply set the DSP DPLL to "locked mode" before the DSP is booted so it uses the previously programmed frequency.


I can PM you the patch but it's really just a single line: in "kmod/dsp_c64.c:dsp_poweron_and_reset()" you have to add the line


reg32_write(IVA2_CM_CLKEN_PLL, (1u << 4) | 7u); /* Allow over/underclocking */
after the RSTCTRL reg. write (around line 79).

@crow_riot: thanks for the info. Are you sure you did not increase the system bus speed as well, or disabled screen clears ? I would not have expected GPP overclocking to make any difference in this testcase.