[announce] c64_tools (DSP loader and IPC)


No it doesn't. According to spruf98x.pdf section 14.4.2, after reset it starts executing instructions in the ICTM sequencer memory (which must be filled before the reset by DSP or GPP)
No, as far as the 'must be filled' part of your statement is concerned.
Well, it must be filled if you want to execute meaningful code.
The section you mentioned states that

1. The DSP initializes the ITCM sequencer memory with sequencer code.

2. The DSP initializes the DTCM sequencer memory with sequencer data.

3. The DSP sets the clock divider for the sequencer module in the IVA.VIDEOSYSC_CLKDIV register.

4. The DSP releases the sequencer from reset by clearing the PRCM.RM_RSTCTRL_IVA2[2] RST3_IVA2 bit.

[..]

The DSP and the sequencer processor communicate through a message box; two interrupts are provided for this purpose.
If releasing the DSP from reset causes the video sequencer to immediately execute (arbitrary?) code, how is this all going to work when the DSP is supposed to be the one programming that 'sequencer' in the first place ?
The boot sequence is (can be) like this:First IVA2.2 is released from reset (bit 1 of PRCM.RM_RSTCTRL_IVA2), then you can upload code/data to DSP memories, then DSP is release from reset (bit 0 of PRCM.RM_RSTCTRL_IVA2), then you can upload code/data to sequencer, then sequencer is released from reset (bit 2 of PRCM.RM_RSTCTRL_IVA2).

At least according to section 14.2.1.2.1 which states (among other information):

After chip power on, the IVA2.2 subsystem is kept under reset and only the RET_RST is released from

reset. Then, only the WUGEN is released from reset as part of the core domain. The IVA2.2 remains

under reset until the microprocessor unit (MPU) clears the PRCM.RM_RSTCTRL_IVA2[1] RST2_IVA2 bit.

On this action, the PRCM switches on the IVA2.2 power domain, sets the clocks back, and releases the

power-on reset. When the IVA2.2 power-on sequence completes (hardware handshake), the PRCM

releases the IVA2_RST2 reset. At this stage, the DSP megamodule is kept under reset (unless the MPU

also cleared the PRCM.RM_RSTCTRL_IVA2[0] RST1_IVA2 bit); the MPU can upload some code and

data in the C64x+ memory. When the MPU has uploaded the code in the C64x+ memory, the MPU clears

the RST1_IVA2 bit, releasing the DSP megamodule from reset. At this point, the sequencer is kept under

reset (unless the PRCM.RM_RSTCTRL_IVA2[2] RST3_IVA2 bit was cleared); the DSP can upload some

code and data in the sequencer memory. When the DSP has uploaded the coded in the sequencer

memory, the DSP clears the RST3_IVA2 bit, releasing the sequencer from reset.

Besides, you don't want to seriously call this 'fairly good documentation' ? It's just a few lines of text.
I wasn't calling 'fairly good documentation' just this one section. Information about the sequencer are also in other sections.
There's no info how that 'message box' communication between DSP and video sequencer actually works (one can only guess that the instruction/data memories are copied that way).
It works through two interrupts mentioned in sections 14.3.4.2 and 14.3.4.3.Data and instructions can be copied from the DSP using EDMA or using EFI (Extended Function Interface). From the CPU the sequencer memories are accessible directly.

There's also no memory map of the sequencer / no info what kind of registers it can access. Just having an ARM9 w/o any I/O would be pretty pointless but the docs only point to a generic ARM968E-S technical reference for further reference.
Sequencer memory mapping is described in sections 14.3.4.8 and 14.3.8.EDMA view of sequencer is described in section 14.3.2.2.

Other views are described elsewhere (for instance in sections 2.4.5 and 2.4.6).

Last but not least, I also wonder how that video sequencer is supposed to be disabled, since it's not / can't be used (because of missing docs). Do we have to assume that it's just sitting there in an infinite loop, draining the battery? The docs don't say anything about that.
The sequencer is kept in reset until you release it (already mentioned section 14.2.1.2.1).
____

I suggest reading the whole section 14 (or at least skim through it) to see what information is available there.

It seems like you didn't read it, or you already forgot what's written in there.

(Reading ARM968E-S technical reference doesn't hurt either.)

That's why I asked what information is missing, because to me it seems there is enough information about the sequencer to use it if you wanted to.
 
The boot sequence is (can be) like this:
First IVA2.2 is released from reset (bit 1 of PRCM.RM_RSTCTRL_IVA2), then you can upload code/data to DSP memories, then DSP is release from reset (bit 0 of PRCM.RM_RSTCTRL_IVA2), then you can upload code/data to sequencer, then sequencer is released from reset (bit 2 of PRCM.RM_RSTCTRL_IVA2).
Except that DSP won't boot if you don't release the sequencer reset (RST3) too after releasing RST1, so there is no way to upload code to SEQ before starting it, at least with the way bsp is booting the whole thing.
 
Here is how I think it should (can) work (in regard to PRCM.RM_RSTCTRL_IVA2, ignoring other registers) according to the manual - please correct me if I'm wrong.
At the beginning bits 0-2 are 1 - everything is kept in reset.
Then bit 1 should be cleared to start IVA2.2 (DSP and sequencer are kept in reset).
Then DSP memories can be filled.
Then bit 0 should be cleared to start DSP (sequencer is kept in reset).
Then sequencer memories can be filled.
Then bit 2 should/can be cleared to start sequencer.



Looking at the sources, all I see is writing value 1 to the register. This clears DSP and sequencer and keeps IVA2.2 in reset ?
Edit: My mistake. It clears IVA2.2 and sequencer and keeps DPS in reset. And in another procedure bit 0 is cleared, which starts the DSP.
 
Last edited by a moderator:
Looking at the sources, all I see is writing value 1 to the register. This clears DSP and sequencer and keeps IVA2.2 in reset ?
Yeah, if you just clear bit1/write 5, DSP code doesn't start after clearing bit0. If that write is left 1, then it starts fine.
Actually I've found how to start it without clearing RST3 (RSTCTRL bit2), but then power off no longer works, it goes into INACTIVE state instead of OFF and leaves all caches ON no matter what:

Code:
--- /tmp/dsp_stresstest-14Jan2014b/src/dsp_c64.c        2014-01-14 21:28:07.000000000 +0200
+++ dsp_c64.c   2014-01-16 16:56:01.102315571 +0200
@@ -95,7 +95,11 @@
#endif
      
       /* Reset DSP (RST1 bit) (do NOT reset MMU or the GPP OS will freeze!!) */
+#if 0
       reg32_write(IVA2_PRM_RSTCTRL, 1);
+#else
+      reg32_bit_clear(IVA2_PRM_RSTCTRL, 1);
+#endif

       /* Allow over/underclocking. */
#ifndef USE_FORCED_LOWPOWER_BYPASS
@@ -147,10 +151,10 @@
          }
#endif
       }
-    
-    
-      /* Enable automatic transition */
-      reg32_write(IVA2_CM_CLKSTCTRL, 3);
+
+      /* Use software-supervised wakeup */
+      reg32_write(IVA2_PRM_PWSTCTRL, reg32_read(IVA2_PRM_PWSTCTRL) | (3u << 0));
+      reg32_write(IVA2_CM_CLKSTCTRL, 2);

       /* Wait for 'on' transition */
       {
@@ -160,13 +164,16 @@
          {
             timeout++;
            
-            if(timeout >= 0x20000000u)
+            if(timeout >= 0x200000u)
             {
                break;
             }
          }
       }

+      /* back to automatic transitioning */
+      reg32_write(IVA2_CM_CLKSTCTRL, 3);
+
       /* Read current power state */
       {
          sU32 val = reg32_read(IVA2_PRM_PWSTST);
(BTW that 0x20000000u timeout takes forever to expire (at least 5minutes) and should be reduced).
 
Last edited by a moderator:
This seems to work, including poweroff:

Code:
--- /tmp/dsp_stresstest-14Jan2014b/src/dsp_c64.c        2014-01-14 21:28:07.000000000 +0200
+++ dsp_c64.c   2014-01-16 17:54:24.207534646 +0200
@@ -94,8 +94,8 @@
       dev_iva2_clk_set_enable(S_FALSE);
 #endif
       
-      /* Reset DSP (RST1 bit) (do NOT reset MMU or the GPP OS will freeze!!) */
-      reg32_write(IVA2_PRM_RSTCTRL, 1);
+      /* Bring IVA core and MMU out of reset */
+      reg32_bit_clear(IVA2_PRM_RSTCTRL, 1);
 
       /* Allow over/underclocking. */
 #ifndef USE_FORCED_LOWPOWER_BYPASS
@@ -148,7 +148,9 @@
 #endif
       }
      
-     
+      /* Request ON state */
+      reg32_write(IVA2_PRM_PWSTCTRL, reg32_read(IVA2_PRM_PWSTCTRL) | (3u << 0));
+
       /* Enable automatic transition */
       reg32_write(IVA2_CM_CLKSTCTRL, 3);
 
@@ -160,13 +162,16 @@
          {
             timeout++;
             
-            if(timeout >= 0x20000000u)
+            if(timeout >= 0x200000u)
             {
                break;
             }
          }
       }
 
+      /* not sure why we need this? */
+      reg32_write(IVA2_PRM_PWSTCTRL, reg32_read(IVA2_PRM_PWSTCTRL) & ~3);
+
       /* Read current power state */
       {
          sU32 val = reg32_read(IVA2_PRM_PWSTST);
@@ -540,7 +545,7 @@
          {
             timeout++;
             
-            if(timeout >= 0x20000000u)
+            if(timeout >= 0x200000u)
             {
                break;
             }
Will test stability later.
 
Last edited by a moderator:
@M-HT: Looks like you're right. I just spent two hours reading about the video sequencer (and taking notes). Seems like I missed that part of the TRM, although so far I've mainly been looking for info about the DSP, not the videoseq.  The docs might be enough to actually execute some code on that ARM926, maybe I'll give it a try somewhen. TBH, the iLF and iME documentation are rather sparse, IMHO. Maybe they make sense to someone who's an expert in the field of video decoders.


On topic: The ARM in the video sequencer block can only access the last 32k of the DSP L2SRAM, the EDMA3, its internal memories (DMEM 4k and IMEM 8k), the motion estimation (iME) and loop filter (iLF) engines, plus a handful of related config registers.

It cannot write to RAM. The only way to do that would be to use EDMA3 and the programming sequence required for that does not just happen by accident.

This rules out the idea that the ARM can accidently cause any NAND (or other RAM) writes.

Since the video sequencer data/instruction memories are indeed accessible via the CortexA8 (or the DSP via EFI, or via DSP EDMA3, maybe also by accessing the L3 addresses on the DSP-side), it should be enough to either wipe (zerofill) the instruction memory before booting DSP/SEQ (the ARM will probably simply crash and burn once it hits the end of its instruction memory), or place an infinite loop at the first instruction so the ARM can later be programmed properly and the loop can then be overwritten by a NOP.

Anyway, yesterdays tests have already shown that the videosequencer has nothing to do with the system lockups / NAND issues seen on (some of?) the OMAP3530 based units.

notaz confirmed this by running stripped-down versions of the DSP image and reporting that this worked w/o any issues.

@notaz: regarding the patches you posted. first of all, could you please post the entire source file for readabilities sake? second, the reg32_bit_clear(IVA2_PRM_RSTCTRL, 1) probably does not exactly do what you think it does: Your comment says "Bring IVA core and MMU out of reset". According to the TRM, clearing that bit means "MMU reset and Video Sequencer hardware accelerator reset are cleared". Isn't that what you wanted to avoid in the first place, i.e. rather keep the videoseq in reset when the DSP is booted, instead of releasing it from reset?
 
I've tested stability of the last patch (full file attached) and it's the same (randomly hangs the system).

I don't know why the manual documents both RST2_IVA2 and RST3_IVA2 bits (bit1 and bit2) as "Video sequencer reset", but RST2_IVA2 is absolutely required to use the DSP, both TI sources and TRM guides clear it. Clearing RST3_IVA2 bit is what boots the SEQ/ARM9, and this is what I'm trying to avoid with my patch.

dsp_c64.c.gz
 

Attachments

  • dsp_c64.c.gz
    4.1 KB · Views: 256
It's not really surprising that this didn't solve the issue. Your findings might come in handy when/if trying to do sth useful with the video accelerator, though.

You were right about the RST bit: there are actually two videoseq related bits, one for resetting the accelerators (probably iLF and iME), and another one for the ARM9. The one you were clearing was for the accelerators and the MMU.

I hope you agree that the tests we did yesterday indicated that the issue is not related to the GPP-side DSP startup code but rather to something that happens on the DSP-side (during init, but apparently not in the DSP/BIOS startup code).

I am going to post some new test DSP images now. Please load them with 'c64_load_notestmsg', then run TC_IDLE (your hacked version with decreased or no timeout should be fine for that purpose). If possible, test each image after a fresh reboot. I'll do the same before posting an image, of course.
 

EDIT#1: added core.out__2 and 3

EDIT#2: added core.out__4

EDIT#3: added core.out__5

EDIT#4: added core.out__6

EDIT#5: added core.out__7

EDIT#6: added core.out__8

EDIT#7: added core.out__9 and 10
 

Attachments

  • core.out__1.gz
    48.8 KB · Views: 238
  • core.out__2.gz
    76.2 KB · Views: 236
  • core.out__3.gz
    48.8 KB · Views: 236
  • core.out__4.gz
    51.5 KB · Views: 239
  • core.out__5.gz
    78.5 KB · Views: 236
  • core.out__6.gz
    79.3 KB · Views: 238
  • core.out__7.gz
    78.1 KB · Views: 248
  • core.out__8.gz
    48.7 KB · Views: 225
  • core.out__9.gz
    46.9 KB · Views: 244
  • core.out__10.gz
    47 KB · Views: 234
Last edited by a moderator:
core.out__1.gz - hangs, although survives a lot longer than default core.out

core.out__2.gz - couldn't make it hang, but lost the NAND after tests finished

core.out__3.gz - hangs

core.out__4.gz - hangs

core.out__5.gz - hangs

core.out__6.gz - hangs

core.out__7.gz - hangs

core.out__8.gz - hangs (but survives longer), corrupts NAND

core.out__9.gz - no hang, but NAND breakage

core.out__10.gz - no hang, NAND state unknown

I hope you agree that the tests we did yesterday indicated that the issue is not related to the GPP-side DSP startup code but rather to something that happens on the DSP-side (during init, but apparently not in the DSP/BIOS startup code).
Sure, but knowing we have ARM9 executing some potentially uninitialized memories made me feel uneasy, so I thought I'd do something about it.
 
Last edited by a moderator:
I did more testing and I now believe the hang and NAND breakage are separate problems. Out of interest I ran the test with core.out__DUMMY_BIOS_5_42_01_09 from yesterday for several minutes and NAND unexpectedly died. Then rebooted and tried core.out__DUMMY_IDLE_BIOS_5_42_01_09 and that one killed NAND too. Then rebooted and tried core.out__DUMMY_BIOS_5_42_01_09 again and it was fine.

So the hang shows up rather consistently with looped TC_IDLE test and my reports should be correct.

NAND breakage is random and hard to reproduce, so some of my previous "good" reports might actually still have it.

I've also played a bit with all the latest files on DM3730 and couldn't reproduce any problems.

Edit: yet more testing on core.out__DUMMY_BIOS_5_42_01_09, could not reproduce anything bad no matter what. Then loaded core.out__2.gz, and bang, NAND corrupted totally, reflashing.. Can some badness be surviving reboots?
 
Last edited by a moderator:
Sure, but knowing we have ARM9 executing some potentially uninitialized memories made me feel uneasy, so I thought I'd do something about it.
I just wrote a small utility that dumps the seq. instruction memory. It indeed is random (had to turn off the unit for 10sec or so to randomize the memory).

I also tried your version of dsp_c64.c.

While the DSP was powered on (TC_IDLE with 30sec delay) I could see that the ARM9 indeed was still in reset (by peeking 48306050).

However, my unit froze the second that the DSP was powered off. I rebooted+retried 3 times to be sure. I reverted the source and the freeze during power off disappeared.

Btw, are you really sure that counting to 0x20000000 on the GPP + polling the PWST reg. takes 5min ?

I use the same timeout for other things and it only takes a couple of seconds. For the power on/off wait the decreased timeout is probably enough, though.

As for the test results: core.out__1 does little more than the dummy test. It writes to L1DMPAA (main difference), plus it also does some L1DSRAM debug writes. No HWI init, no RPC init or dispatch.

So, are you really sure that yesterday's DUMMY image IDLE test never hangs ?

Could you please retest?

It may be something the BIOS startup does, after all.

I also just had an idea what else we could try.. we don't need any DSP image, after all: The kmod already generates a tiny bit of DSP code (for jumping to the actual entry point). I could modify that to create an infinite loop, that should work as the most minimalistic 'DSP image' imaginable :)

EDIT: and here it is (the "micro DSP image") (again, use c64_load_notestmsg + TC_IDLE test)

p.s. I also just added the modified src file for core.out__1, in case you want to take a look.

micro_dsp_image-17Jan2014.tar.gz
 

Attachments

  • micro_dsp_image-17Jan2014.tar.gz
    29.6 KB · Views: 232
  • mlb.c__1.txt
    41.6 KB · Views: 649
Last edited by a moderator:
Today's testing is real painful, I'm reflashing the 5th time already..

I also tried your version of dsp_c64.c.

While the DSP was powered on (TC_IDLE with 30sec delay) I could see that the ARM9 indeed was still in reset (by peeking 48306050).

However, my unit froze the second that the DSP was powered off. I rebooted+retried 3 times to be sure. I reverted the source and the freeze during power off disappeared.
I'll admit it wasn't tested on 3730, so I tried too and yeah, In-band Error there :( . Works fine on 3530 though.
Btw, are you really sure that counting to 0x20000000 on the GPP + polling the PWST reg. takes 5min ?
Quite sure, it seems reading IVA2_PRM_PWSTST is real slow on 3530.
So, are you really sure that yesterday's DUMMY image IDLE test never hangs ?

Could you please retest?
Yes, I did multiple tests today. What I consider "hang" (no response from xfce and serial console) never happened. The other thing is NAND hw crash, it sometimes makes xfce unresponsive and spits lots of errors to dmesg, but serial console still works, as long as you don't use rootfs. That I had twice (still talking about DUMMY/IDLE here), but couldn't reproduce later.
EDIT: and here it is (the "micro DSP image") (again, use c64_load_notestmsg + TC_IDLE test)
Hangs right after starting looping TC_IDLE test, and even blanks the screen somehow!? Retried 3 times and got the same.If I only do single iteration, the test passes fine.

EDIT: some more tests confirm that my RST3-prevention hack doesn't solve hangs, nor NAND hw crash.
 
Last edited by a moderator:
Today's testing is real painful, I'm reflashing the 5th time already..
Amen to that. I lost count of how many times I rebooted today. Admittedly, having to reflash is far worse, though..

I'll admit it wasn't tested on 3730, so I tried too and yeah, In-band Error there :( . Works fine on 3530 though.
Ok, thanks for confirming this. Couldn't see the klog message myself (no serial console here).


I think we've tried hard enough. This is definitely a case for TI support (which we don't have).


If even the micro DSP image is not working right, I'm really out of ideas now.


From what I can tell, the implementation is 'by the book'. There's nothing in the TI docs that indicates that booting (resp. powering off) the DSP might result in NAND failure.


Maybe we really should go over to 'plan B' and simply keep the DSP enabled on 3530 units once it has been booted once.


The most important thing is to protect the users from catastrophic (NAND) failure. A slight increase in power consumption seems like a fair trade-off.


To everyone who bought an OMAP3530 ('Rebirth', 'CC') unit: Bad luck, seems like the DSP on yours doesn't like being powered off (and on again). Battery consumption will increase by ~100 mW once some application requests/boots the DSP. Not ideal but then again, not _that_ bad.


To everyone who bought a DM3730 ('Ghz') unit: Good for you -- the DSP, even if power cycled thousands of times per second, works fine.
 
Last edited by a moderator:
To everyone who bought an OMAP3530 ('Rebirth', 'CC') unit: Bad luck, seems like the DSP on yours doesn't like being powered off (and on again). Battery consumption will increase by ~100 mW once some application requests/boots the DSP. Not ideal but then again, not _that_ bad.
It's a fair trade-off. There's no need to have a perfect solution at this stage anyway - the most important is to have a working DSP that can be used in applications, in a safe way. The battery impact is minimal.   
 
I still suspect the DSP is very picky about the state it's in when it's being powered off, if it's not perfect it goes insane and does something bad. They probably fixed that on 3730.

With that in mind, I've added this code to dsp_poweroff():

// reset the DSP (set RST1_IVA2)
reg32_bit_set(IVA2_PRM_RSTCTRL, 0);

// change BOOTMOD, pulse RST2_IVA2 so that new mode is latched (idea from dspbridge)
reg32_write(IVA2_SCM_GENERAL_BOOTMOD, 1);
reg32_bit_set(IVA2_PRM_RSTCTRL, 1);
udelay(100);
reg32_bit_clear(IVA2_PRM_RSTCTRL, 1);
udelay(100);

// reset to IDLE boot mode
reg32_bit_clear(IVA2_PRM_RSTCTRL, 0);
udelay(100);
The hanging tests still hang with this, but the hang only manifests on looped test so it might be separate issue to NAND problem. I'm now testing this with core.out__9.gz , which used to break NAND but works so far. I'll test this more after I get some sleep.
EDIT: Scratch that, it doesn't help anything.
 
Last edited by a moderator:
I still suspect the DSP is very picky about the state it's in when it's being powered off, if it's not perfect it goes insane and does something bad. They probably fixed that on 3730.
It's definitely something that has been fixed on 3730, although so far I couldn't find an errata that mentions this.

Regarding last night's 'micro DSP image': I made a stupid mistake there. There's a (new) define 'USE_DSP_INFINITE_LOOP' in dsp_c64.c (right at the top). When determining 'entryAddr', I used 'ifdef' instead of 'ifndef' so the infinite loop was never used (d'oh!).


Please find attached a source update that corrects this.


I also did some other small changes in dsp_c64.c:

  • increase udelay() at the start of dsp_poweroff() to 4*250 uS  (there's is sth fishy going on here: the udelay(10) in previous versions was definitely required on 3730 when powercycling the DSP as fast as possible. Maybe using a longer udelay() on 3530 has a positive effect?)
  • disable automatic transitions at the beginning of dsp_poweroff()
  • disable IVA2 clock before (instead of after) enabling automatic transitions during dsp_poweroff()
  • insert udelay(10) call after each regwrite in dsp_poweroff() (yeah, this is utterly desparate, I know)

May I bug you with another request-for-test ?

micro_dsp_image-17Jan2014b.tar.gz
 

Attachments

  • micro_dsp_image-17Jan2014b.tar.gz
    38.4 KB · Views: 247
I think I have some good news (for a change).

I just transcribed the power on/off functions from SysLink (omap3530pwr.c) to c64_tools.

This resulted in the following improvements:

  • udelay() call at the begining of dsp_poweroff() is no longer needed, even if the DSP is powercycled as fast as possible (in previous releases, removing that call resulted in sporadic 'in-band' errors in the klog)
  • considerable decrease in power consumption during c64_pwrbench (powercycle-as-fast-as-possible)  (now ~1.6W instead of ~2.7W)
  • considerable decrease in IDLE power consumption (barely measurable now, maybe ~10mW. In previous releases this was ~80mW, IIRC).
Could you please try this on your OMAP3530 unit?

Even if it hangs eventually, removing the dsp_poweroff() call on 3530 should not make much of a difference now in terms of power consumption.

p.s.: All tests seem to work fine so far. Overclocking works, too.

dsp_syslinkpwronoff-17Jan2014.tar.gz
 

Attachments

  • dsp_syslinkpwronoff-17Jan2014.tar.gz
    31.5 KB · Views: 236
Last edited by a moderator:
Back
Top