[announce] c64_tools (DSP loader and IPC)

[notaz]

Since you already found out that removing the dsp_close() call seems to improve the situation, could you please comment out line 933 in kmod/dev.c (dsp_poweroff()) and re-run the stresstest on your 3530 ?

Yup that helps. I then uncommented it again to confirm it's not something else and it locked up the system on the very first iteration just after printing 'start test_cycle_idx=0, testcase=1 ("TC_TESTMSG")' and before printing "OK.".
Edit: ugh I just noticed you ioremap before every register read/write, any reason for that? That doesn't look like a smart thing to do for performance, as it has to modify page tables each time and flush TLB caches. Although ioremap() might be giving you some static mapping, I'm not sure..

Edit2: I've added lots of prints in dsp_poweroff() (almost after every line), and it seems the lockup happens after the function returns, I guess it makes the SoC unstable and it dies somewhere else, perhaps on next poweron?

Edit3: OK found it, I don't know line number since I messed up the file, but it's this line in dsp_c64.c after which the system dies:

reg32_bit_clear(IVA2_MMU2_CNTL, 1);
.. and I don't really understand what you're doing here, clearing some reserved bit? [edit, edit... the arg is bit number, so it's fine, I see it now].It looks like you should avoid writing to this register if it doesn't have that bit set as it seems to be dangerous for whatever reason. The verbose comments around it also suggest you had trouble with it before.

Edit4: It looked like commenting IVA2_MMU2_CNTL line out helped and things worked, but as soon as I removed debug prints it hanged later in dsp_poweron_and_reset(), somewhere before this function returns. So it looks like some timing issue (something that needs to be waited for is not waited enough, I guess OMAP3 takes longer that DM3730 to finish something before you can talk to it again).

Edit5: You should take spinlock and disable interrupts during these critical register programming code sections, otherwise Linux can context switch to another program while you are in the middle of it, and it might for example access file system (NAND) which will hit OMAP while it's not ready. You already use mutex, but that only protects from 2 simultaneous c64_tools users..

static DEFINE_SPINLOCK(my_lock);
...

  unsigned long   flags;

  spin_lock_irqsave(&my_lock, flags);
  // some critical register programming here..
  spin_unlock_irqrestore(&my_lock, flags);

 

[bsp]

Yup that helps. I then uncommented it again to confirm it's not something else and it locked up the system on the very first iteration

Ok, at least now we know that it really is related to power on/off and not some sporadic runtime memory write gone wrong.

Edit: ugh I just noticed you ioremap before every register read/write

Not before _every_ register access, only the regs used during startup/shutdown.


Performance-critical registers (e.g. the mailbox registers) are mapped only once.

OK found it, I don't know line number since I messed up the file, but it's this line in dsp_c64.c after which the system dies:

reg32_bit_clear(IVA2_MMU2_CNTL, 1);

Edit4: It looked like commenting IVA2_MMU2_CNTL line out helped and things worked

You can't simply comment this line out since it, as the comment above that line implies, disables the DSP MMU.

(the trouble I had with this before was to figure out how to disable the MMU in the first place, hence the debug prints before and after)


Without this line, the DSP will only be able to access its local memories. Actually, I just removed/commented out that line and my unit locked up without it.


The call/regwrite also can't be moved (just tested that), it needs to be done after FCLKEN and before the 'on' transition (CLKSTCTRL regwrite), otherwise the whole system will freeze.

It looks like you should avoid writing to this register if it doesn't have that bit set as it seems to be dangerous for whatever reason

Is this a guess or an observation, i.e. if you test the bit and only set it if it's still cleared, does this prevent the system lockup ? That would be weird but if it works..

You should take spinlock and disable interrupts during these critical register programming code sections

On the one hand, the code only writes to DSP related registers which the rest of the kernel does not touch -- otherwise the stresstest would have failed on 3730 as well, sooner or later.

On the other hand, using a spinlock can't hurt so I'll add it.


I guess I'll have to take another look at the TRM later today, maybe some kind of waiting needs to be done after disabling the MMU.


On p.945 it says that there is a way to soft-reset that thing, and a resetdone bit that can be polled.


We could also try to leave the MMU enabled and set it up to have access to RAM (except for kernel area) and DSP regs, but not the NAND area. This will probably be the best course of action, I guess.

 

[notaz]

Well like I said it looked like removing IVA2_MMU2_CNTL access helped, but it could be just that just resulted longer print time when debug prints before it and after it combined, so whatever OMAP was busy with had more time to complete before next register write. However I printed IVA2_MMU2_CNTL value and it was always 0 on 3530, perhaps it was disabled the first time module was loaded and writes are not needed on subsequent DSP restarts? Or is the MMU active even when the register states it isn't?

 

[bsp]

Apparently the MMU really is deactivated by default. The IVA2_MMU2_CNTL regwrite is still critical, without it my 3730 unit locks up.

What works, though, is replacing the regwrite with a regread (!). Replacing the read or write with a udelay() or printk() does _not_ work, so it's not just a timing issue.

Please find attached the updated source. I included the binaries, some utility scripts and a readme.txt, in case someone else wants to run the stresstest.

Source changes:

• remove IVA2_MMU_CNTL regwrite and replace it with a regread (prints errmsg if reg. does not contain expected value)
• add wait-loop (w/ timeout) after 'on' and 'off' transitions (polls IVA2_PRM_PWSTST register)
• print error messages to klog if DSP fails to power on resp. off
• postpone L2SRAM power-on until DSP 'on' transition is asserted
• use spinlock for critical reg.access during power on/off and boot

dsp_stresstest-14Jan2014.tar.gz

 

[notaz]

Apparently the MMU really is deactivated by default. The IVA2_MMU2_CNTL regwrite is still critical, without it my 3730 unit locks up.


What works, though, is replacing the regwrite with a regread (!). Replacing the read or write with a udelay() or printk() does _not_ work, so it's not just a timing issue.

I'm starting to think it's actually missing read after a write to some register, which is required to flush the write on OMAPs. You said you are aware of it, but don't seem to be using it much. The clk_*() API does it after all clock enables at least.

Source changes:

Hmm still not using clk_ API.. I've found it also waits for CM_IDLEST_IVA2 to indicate the module is not idle after enabling IVA clock.

I'll do the actual testing later in the evening.

 

[notaz]

I ran the test and it works, but NAND died shortly from start. TC_TESTMSG also runs longer than usual before dying, so it's a move to right direction, I guess..

 

[bsp]

Well, I didn't really expect this to fix the issue. This kind of defect looks just like a HW failure, not much that can be done about that (..but maybe you are able to find a workaround?)

I've added the clk_*() calls in the meantime (see attachment), you could give these another go. I would be surprised if this changed anything, though.

Tests have been running fine so far for a couple of hours on my 3730.

dsp_stresstest-14Jan2014b.tar.gz

 

[notaz]

You're right, it doesn't help at all. I've also added readbacks to reg32_write() and it doesn't help either. I don't know what else to try, at least it now survives longer than the version we started from.

 

[notaz]

Section 3.5.1.9 of DM3730 manual has reset sequences explained, with graphs and stuff, but I failed to get anything working by following that.

Edit:

Adding a 10ms sleep at the start of dsp_poweroff() seems to help TC_TESTMSG stability yet more, so maybe the DSP sometimes is still running code while it's power is cut off, which causes it to misbehave, like throwing a random write somewhere? Although the NAND breakage is so consistent, I dunno, it's probably something else..

Maybe you could just keep the DSP on at all times on 3530, it's not hard to detect in code by calling cpu_is_omap3630() (3630 is different 3730 marketing name), and it doesn't use much power while idle, does it?

 

[notaz]

I've tried maybe 40+ of different things last night with no success (and a reboot each time, ugh).

I've verified that it's enough to just do dsp_open()/dsp_close() in a loop to get problems, so I'm working with that testcase.

I've found one thing that helps for sure - setting IVA2_SCM_GENERAL_BOOTMOD to 1. This doesn't boot your core.out but some ROM code instead, that is documented as "configures the PDCCMD and then executes the IDLE instruction". With that I could run the open/close test without problems (also verified c64.ko still power cycles everything). So I guess something core.out does (or doesn't do) is causing the problem.

I remember you were using some TI's stock BIOS image during development and before rolling out your own core.out, maybe we could try that instead. No functionality is needed, just powering up/down.

 

[bsp]

Yea, the TRM is quite vague in regards to the power up/down procedures.

I mainly looked at TI reference code (CCS scripts and DspLink) to see how it's done in practice
(without it I probably would never have figured out how to power that thing off!)

The delay() at the beginning of dsp_poweroff() is a bit worrying, indeed. You've probably seen the udelay(10) call there, and my comment.


I really don't understand why that makes a difference -- an almost arbitrary length of time has already passed when that function is called (the udelay only makes a difference if the DSP is powered on/off at the rate of a thousand times per second, though).

About the DSP still running code when it's powered off: This is highly unlikely since there are no background tasks running on the DSP.


The RPCs are "serialized", i.e. a request is sent to the DSP, and the GPP always expects/waits for a reply (the only exception here is the TESTMSG but the stresstest deliberately skips this test). After the DSP has sent its reply to the GPP it immediately (a few cycles later) goes back to sleep (by executing the IDLE instruction). Because of this, you should probably use something like the ADD test for testing (TESTMSG will not cause the DSP to send a reply).

Leaving the DSP on all the time should be the last resort. The 'idle' power consumption will increase by ~100mW, IIRC.


I would like you to try another test: I would like to know whether the issue has anything to do with actually running any DSP code. I just wrote a small testcase that simply powers on the DSP and keeps it in reset, then powers it off again. You can configure the number of iterations, and the number of milliseconds to wait between power on/off.

Could you please run that test ('c64_reset') and see if this eventually locks up your unit (or even corrupts the NAND) ?

It is of course crucial that you run this after a fresh reboot and you must not use op_dsp_c64.sh or go64.sh to insert the kernel module (this would boot the DSP). Instead, just do 'insmod c64.ko', then run the test.

It's normal that there will be an error message at the end of the test complaining that the DSP could not be resumed (there is no valid start address since no DSP image was loaded).
 

p.s.: I wrote this message while you were writing yours. So you are saying that you suspect the DSP image to be culprit. Could you still run the c64_reset test, just to be sure ? The result will probably be similar to your IDLE bootmode test but please test it anyway.

The core.out DSP image is (still) using DSP/BIOS (closed source blackbox, unfortunately). The reason for that is that the BIOS contains a handful of useful utility functions to manage interrupt handlers and caches. These things cannot be written in C (and C64 assembly is hell!).

c64_tools-snapshot-15Jan2014.tar.gz

 

[bsp]

I just installed some older DSP/BIOS versions and compiled some alternative core.out images.

EDIT#2: core.out__BIOS_5_33_06: Does not work on my unit, system freezes immediately

core.out__BIOS_5_40_03_23: Does not work on my unit, system freezes immediately

EDIT: core.out__BIOS_5_41_04_18: Works

core.out__BIOS_5_41_10_36: Works

(the original core.out uses 5_42_01_09, btw)

To load the image, gunzip it, then do "insmod c64.ko", then "c64_load <core.out__xyz>".

 

[notaz]

Could you please run that test ('c64_reset') and see if this eventually locks up your unit (or even corrupts the NAND) ?

As expected it works fine, even with delay of 0. Sometimes starts flooding dmesg with '[---] dsp_power_on_and_reset: ioctl() failed. errno=16 ("Device or resource busy").' though, but no lockups or NAND problems.
BTW I've noticed the manual states that as soon as RST3 is cleared (bit2 of IVA2_PRM_RSTCTRL), SEQ (ARM9) starts booting. I suppose that's not a very good thing as it might be executing trash from it's memories. I've tried not to clear RST3 and of course DSP no longer worked.

I'll test those BIOSes a bit later.

 

[bsp]

of course it didn't.

nah, srsly, one can only hope that the ARM9 executes some safe ROM code. Otherwise they should've documented that d*mn thing.

And yes, please run those test images.

I've prepared another set of tests resp. images. I stripped down the DSP image and removed everything non-vital for booting. Please see attachment. It also contains the images I posted earlier, the source for the 'dummy' image, plus a readme.txt.
 

 

[M-HT]

nah, srsly, one can only hope that the ARM9 executes some safe ROM code. Otherwise they should've documented that d*mn thing.

No it doesn't. According to spruf98x.pdf section 14.4.2, after reset it starts executing instructions in the ICTM sequencer memory (which must be filled before the reset by DSP or GPP).
Also I find it fairly good documented (in spruf98x.pdf), but I'm no expert. What information are you missing?

 

[bsp]

No it doesn't. According to spruf98x.pdf section 14.4.2, after reset it starts executing instructions in the ICTM sequencer memory (which must be filled before the reset by DSP or GPP)

No, as far as the 'must be filled' part of your statement is concerned. The section you mentioned states that

1. The DSP initializes the ITCM sequencer memory with sequencer code.


2. The DSP initializes the DTCM sequencer memory with sequencer data.


3. The DSP sets the clock divider for the sequencer module in the IVA.VIDEOSYSC_CLKDIV register.


4. The DSP releases the sequencer from reset by clearing the PRCM.RM_RSTCTRL_IVA2[2] RST3_IVA2 bit.

[..]

The DSP and the sequencer processor communicate through a message box; two interrupts are provided for this purpose.

If releasing the DSP from reset causes the video sequencer to immediately execute (arbitrary?) code, how is this all going to work when the DSP is supposed to be the one programming that 'sequencer' in the first place ?


Besides, you don't want to seriously call this 'fairly good documentation' ? It's just a few lines of text.

There's no info how that 'message box' communication between DSP and video sequencer actually works (one can only guess that the instruction/data memories are copied that way).

There's also no memory map of the sequencer / no info what kind of registers it can access. Just having an ARM9 w/o any I/O would be pretty pointless but the docs only point to a generic ARM968E-S technical reference for further reference.


I also haven't seen any of the other DSP drivers set up that video sequencer.


Also, just because the docs say it can be programmed by the DSP does not mean its memories haven't been initialized with some safe ROM code after reset.


Last but not least, I also wonder how that video sequencer is supposed to be disabled, since it's not / can't be used (because of missing docs). Do we have to assume that it's just sitting there in an infinite loop, draining the battery? The docs don't say anything about that.

 

[notaz]

'c64_tc 10' is extremely long test; it sometimes takes 50+ power cycles for things to break, so I still tested with TC_RPC_ADD (./c64_tc 2 2 0) when possible, and hacked test 10 (TC_IDLE) with removed sleep for dummy images.

core.out__BIOS_5_33_06 - In-band Error seen by IVA_SS at address 0, FAILED to start DSP!

core.out__BIOS_5_40_03_23 - In-band Error (infinite loop print)

core.out__BIOS_5_41_04_18 - starts, TC_RPC_ADD hangs randomly, hacked_TC_IDLE not tested

core.out__BIOS_5_41_10_36 - starts, TC_RPC_ADD and hacked_TC_IDLE hangs randomly

core.out__DUMMY_BIOS_5_41_04_18 - starts, hacked_TC_IDLE is ok

core.out__DUMMY_BIOS_5_42_01_09 - starts, hacked_TC_IDLE is ok

core.out__DUMMY_IDLE_BIOS_5_41_04_18 - starts, hacked_TC_IDLE is ok

core.out__DUMMY_IDLE_BIOS_5_42_01_09 - no need to test this one, I guess

 

[bsp]

Interesting, so the first two tests, which cause my 3730 to freeze, simply cause klog messages on your 3530.

Did you boot the DUMMY images with c64_load or c64_load_notestmsg ?

Could you please boot core.out__BIOS_5_41_10_36 w/ c64_load_notestmsg and repeat TC_IDLE ?

 

[notaz]

Well the system is not usable during those In-band Errors, but it manages to send endless stream of them to serial. Although on first test it recovered eventually.

I did all tests with c64_load_notestmsg, it seems TC_RPC_ADD worked anyway (well, until it hanged).

 

[bsp]

Yep, I also noticed that having c64_load send that testmsg seems to be superfluous.

Anyway, according to your test reports, TC_IDLE randomly hangs even though not a single message has been exchanged between GPP/DSP, and the test works with the dummy image, which eliminates potential BIOS startup issues.

That limits the possible causes of error to quite a few lines of code.

EDIT: To be precise, something that happens during DSP startup (until the DSP calls IDLE) must be causing the issue. The interrupt handlers (GPP or DSP) are not being called at all.

I'll prepare some new test images later today.