System Lockup With Render-To-Texture


puppeh

Still Fresh
Joined
Mar 20, 2011
Messages
7
Hi,

I'm working on porting an old, half-finished project of mine over to GLES2 on the Pandora, but I've hit what appears to be a driver or kernel bug which results in a full-system lockup when rendering to a texture (using a framebuffer object). I'm running Hotfix 5 & using a cross-compiler toolchain from (IIRC) http://blogs.distant-earth.com/wp/?p=109.

I've tried to put together a reasonably self-contained testcase (which isn't actually meant to do anything in particular, apart from exhibit the crash):

http://panic.cs-bristol.org.uk/~jules/rtt-crash.tar.gz

TBH I have no idea where to begin tracking the problem down. Here are a few observations which may or may not be helpful:

1. A few frames are rendered (to the texture), either correctly or not (I'm not sure). An FPS counter writing to stdout usually writes one value before the system crashes, corresponding to ~1 seconds' worth of frames at 60fps.

2. Commenting out the call to glBindFramebuffer() which redirects rendering to the FBO stops the code from crashing (search for "NO_CRASH_IF_THIS_IS_COMMENTED_OUT").

3. Reducing the value of "unsigned int numpolys = 100" in render_shadow to, say, 10 makes the program run for longer, suggesting some kind of resource depletion issue (something not being freed per-frame?). Using a value of 50 makes the program run for ~2 seconds before locking up the system.

4. The render-to-texture example from the PVR SDK (Training Course "09_RenderToTexture") works fine for me. I can't figure out what the important difference is from my code.

5. GDB doesn't seem very helpful. A "crashed" binary seems to be spending a lot of time in ioctl.

The test case should compile with just "./compile.sh", then copy rtt-crash, shadowcast.vtx, shadowcast.frag to the target & run. (Bits of this code are stolen from all over the place without attribution, so sorry about that :). I've also cut it down quite quickly & dirtily from the larger project, so it probably doesn't make much sense any more).

So: has anyone played with this stuff before, and can confirm if it works? Can anyone see anything blatantly stupid I've done? Has anyone perhaps got a system running a later kernel (with SGX drivers enabled) who can test it for me? (I didn't have any luck building the last myself).

I'd welcome any comments or ideas. TIA!

Cheers,

Jules
 
It looks like gcc's optimiser could be causing it, it works fine at -O0, I let it run for about one minute.
I've not seen anything obvious in the generated asm that'd cause it yet.

Edit:
Adding -mfloat-abi=softfp to the gcc options fixes it. You should be using this anyway otherwise gcc won't generate fp instructions (it'll do it all through function calls).

I don't know why this is causing the crash though.
 
Thanks for the reply (wow that was quick!). The -mfloat-abi=softfp option unfortunately doesn't fix anything in the main project, but gives a really interesting data point. I have about half a theory about what might be happening now I think: the binary SGX driver probably contains VFP/NEON instructions. For code compiled without -mfloat-abi=softfp, the likelihood is high that the first NEON instruction encountered will be in the SGX driver, rather than in "user" code (though I'm not sure why the distinction is important, unless the NEON code is in the actual kernel).

Now this is important because IIUC the kernel enables NEON lazily per-process so as not to slow down context switches for integer-only processes. The first NEON instruction executed by the process causes a trap handler to be invoked, which enables the NEON unit then re-executes the trapping instruction. But 2.6.27 is kind of old, and its NEON support is probably immature (though I'd hoped that whatever patches are applied on top of vanilla 2.6.27 for the Pandora kernel might have fixed that).

I'll investigate more later. (I uploaded a new version of rtt-crash.tar.gz with a "deliberate" bug fixed).
 
I've now managed to get my original project, and the cut-down testcase, both working flawlessly (err, or at least not crashing) on an IGEPv2 board, using (I think) the same version of the SGX drivers, but with a later kernel (*). This to me points very strongly towards a kernel issue. I guess I'll have to try getting a later kernel running on the Pandora some time...

(*) 2.6.34-rc2, using Debian, following the very nice instructions and git repository from http://jozz.no-ip.org/wiki/igep/debian.
 
After rather a lot of time, I've managed to get a 2.6.37 kernel (from the pandora-37 git branch) running on my Pandora. Interestingly, and frustratingly, this doesn't seem to have fixed my original issue though: my code still manages to lock up the SGX.

With the following recipe, approximately:

(1) Kernel pandora-37, with no (or only trivial) changes to omap3-pandora-defconfig

(2) Debian-on-SD card recipe from http://pandorawiki.org/index.php?title=Debian_On_SD

(3) SGX binary libraries and kernel module from TI's "Graphics_SDK_setuplinux_4_03_00_02.bin", but NOT the X server.

(4) Xorg driver changed to "fbdev" (to avoid immediate segfault on server start, thanks DJWillis & notaz for hints).

My code runs for a while, then dmesg shows goes into a loop, apparently trying to resurrect a locked-up SGX:

Mar 28 13:26:56 pandora kernel: [ 250.064208] PVR_K: HWRecoveryResetSGX: SGX Hardware Recovery triggered
Mar 28 13:26:56 pandora kernel: [ 250.064270] PVR_K: SGX debug (1.6.16.3977)
Mar 28 13:26:56 pandora kernel: [ 250.064270] PVR_K: (P0) EUR_CR_EVENT_STATUS: 00000000
Mar 28 13:26:56 pandora kernel: [ 250.064300] PVR_K: (P0) EUR_CR_EVENT_STATUS2: 00000000
Mar 28 13:26:56 pandora kernel: [ 250.064331] PVR_K: (P0) EUR_CR_BIF_CTRL: 00000000
Mar 28 13:26:56 pandora kernel: [ 250.064361] PVR_K: (P0) EUR_CR_BIF_INT_STAT: 00000000
Mar 28 13:26:56 pandora kernel: [ 250.064361] PVR_K: (P0) EUR_CR_BIF_FAULT: 00000000
Mar 28 13:26:56 pandora kernel: [ 250.064392] PVR_K: (P0) EUR_CR_BIF_MEM_REQ_STAT: 00000000
Mar 28 13:26:56 pandora kernel: [ 250.064422] PVR_K: (P0) EUR_CR_CLKGATECTL: 00212120
Mar 28 13:26:56 pandora kernel: [ 250.064422] PVR_K: (P0) EUR_CR_PDS_PC_BASE: 002033E0
Mar 28 13:26:56 pandora kernel: [ 250.064453] PVR_K: Flip Command Complete Data 0 for display device 1:
Mar 28 13:26:56 pandora kernel: [ 250.064483] PVR_K: SRC 0: (Not in use)
Mar 28 13:26:56 pandora kernel: [ 250.064514] PVR_K: SRC 1: (Not in use)
Mar 28 13:26:56 pandora kernel: [ 250.064514] PVR_K: SGX Host control:
Mar 28 13:26:56 pandora kernel: [ 250.064544] PVR_K: (HC-0) 0x00000001 0x00000000 0x00000000 0x00000000
Mar 28 13:26:56 pandora kernel: [ 250.064575] PVR_K: (HC-10) 0x00000001 0x0000000A 0x0001B04A 0x00000002
Mar 28 13:26:56 pandora kernel: [ 250.064605] PVR_K: (HC-20) 0x00000000 0x00000001 0x00000000 0x00000080
Mar 28 13:26:56 pandora kernel: [ 250.064636] PVR_K: (HC-30) 0x0000AB58 0xFD061020 0x00000000 0x00000000
Mar 28 13:26:56 pandora kernel: [ 250.064636] PVR_K: SGX TA/3D control:
Mar 28 13:26:56 pandora kernel: [ 250.064666] PVR_K: (T3C-0) 0x0F003000 0x0F003120 0x0F002000 0x0F0A5900
Mar 28 13:26:56 pandora kernel: [ 250.064697] PVR_K: (T3C-10) 0x00000001 0x00000002 0x00000001 0x0F007F40
Mar 28 13:26:56 pandora kernel: [ 250.064727] PVR_K: (T3C-20) 0x00000000 0x00000000 0x00000000 0x00000000
Mar 28 13:26:56 pandora kernel: [ 250.064758] PVR_K: (T3C-30) 0x00000000 0x00000000 0x00000000 0x00000000
Mar 28 13:26:56 pandora kernel: [ 250.064758] PVR_K: (T3C-40) 0x00000000 0x00000000 0x00000000 0x00000000
Mar 28 13:26:56 pandora kernel: [ 250.064788] PVR_K: (T3C-50) 0x0F007F40 0x00000000 0x00000000 0x0F007F40
Mar 28 13:26:56 pandora kernel: [ 250.064819] PVR_K: (T3C-60) 0x00000000 0x00000000 0x00000000 0x00000000
Mar 28 13:26:56 pandora kernel: [ 250.064849] PVR_K: (T3C-70) 0x00000000 0x00000000 0x00000000 0x00000000
Mar 28 13:26:56 pandora kernel: [ 250.064880] PVR_K: (T3C-80) 0x00000000 0x00000000 0x0F000000 0x8B84B000
Mar 28 13:26:56 pandora kernel: [ 250.064910] PVR_K: (T3C-90) 0x0F08F6C0 0x00000000 0x0F0883A0 0x0F007F40
Mar 28 13:26:56 pandora kernel: [ 250.064941] PVR_K: (T3C-A0) 0x00000000 0x0F0883A0 0x00000000 0x0F0BC400
Mar 28 13:26:56 pandora kernel: [ 250.064971] PVR_K: (T3C-B0) 0x00000001 0x00000000 0x00000000 0x00000000
Mar 28 13:26:56 pandora kernel: [ 250.064971] PVR_K: (T3C-C0) 0x9062A7D2 0x00000000 0x001004A3 0x00000000
Mar 28 13:26:56 pandora kernel: [ 250.065002] PVR_K: (T3C-D0) 0x00000000 0x00000000 0x00000000 0x00000164
Mar 28 13:26:56 pandora kernel: [ 250.065032] PVR_K: (T3C-E0) 0x00000164 0x0F000000 0x80008000 0x80048000
Mar 28 13:26:56 pandora kernel: [ 250.065063] PVR_K: (T3C-F0) 0x0F004000 0x0F007C20 0x0F002020 0x00000000
Mar 28 13:26:56 pandora kernel: [ 250.065093] PVR_K: (T3C-100) 0x00000000 0x00000000 0x00000000 0x00000000
Mar 28 13:26:56 pandora kernel: [ 250.065124] PVR_K: SGX Kernel CCB WO:0x62 RO:0x5B
Mar 28 13:26:57 pandora kernel: [ 251.259521] PVR_K: HWRecoveryResetSGX: SGX Hardware Recovery triggered
Mar 28 13:26:57 pandora kernel: [ 251.259582] PVR_K: SGX debug (1.6.16.3977)
Mar 28 13:26:57 pandora kernel: [ 251.259582] PVR_K: (P0) EUR_CR_EVENT_STATUS: 00000000
Mar 28 13:26:57 pandora kernel: [ 251.259613] PVR_K: (P0) EUR_CR_EVENT_STATUS2: 00000000
...

after ctrl-C'ing my binary, watchdog/0 seems to be using lots of CPU:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6 root RT 0 0 0 0 S 91.2 0.0 5:09.94 watchdog/0

or periodically twl4030-irq:

6 root RT 0 0 0 0 S 45.6 0.0 5:54.12 watchdog/0
181 root -51 0 0 0 0 R 45.6 0.0 0:02.26 twl4030-irq

Maybe I'm seeing a horrible power-management issue?
 
It also happens on .27, I think it's driver's fault, see here:
http://www.gp32x.de/board/index.php?/topic/59061-latest-sgx-driver/

I was trying to report this to TI but they insist for a testcase. The code you last posted is rather large, could you try reducing it?
 
Last edited by a moderator:
Thanks for the reply! I've reduced the test case it a little, to a single source file at least. It could probably go a little smaller by hardwiring matrices etc., but I think that'll just obscure things unnecessarily.

New download in the same location as before, i.e. http://panic.cs-bristol.org.uk/~jules/rtt-crash.tar.gz. It'd be awesome if we can get TI to take notice of this issue!
 
Maybe interesting extra information, maybe not: the older "OMAP35x_Graphics_SDK_setuplinux_3_01_00_02.bin" release doesn't work any better with my Debian setup, though the crashing behaviour is slightly different (happening sooner or later). This is also using a pandora-37 kernel, and applying a few poorly-understood hacks to get the PVR kernel modules compiling. (I don't see any syslog output when the machine crashes with the earlier version.)
 
Hi,

I've been continuing to look at this problem on and off (mostly off :)) over the last months, and I think I've finally got a solution. A combination of the following ingredients seems to allow render-to-texture to work without locking up the system:

  1. Running a 2.6.37 kernel. Mine identifies as "Linux pandora 2.6.37.6-00038-g7819ce5" (I built it a while ago). I have a Debian userland for experimenting with, running off SD card -- I haven't reproduced a working solution with the official firmware, so far.
  2. Installing TI's 4.04.00.02 release (awkward: I did it semi-manually), and making the following change to GFX_Linux_KM/services4/srvkm/hwdefs/sgxerrata.h:

    Code:
    --- ../hwdefs.old/sgxerrata.h   2011-10-09 00:01:20.000000000 +0100
    +++ sgxerrata.h 2011-10-08 23:59:36.000000000 +0100
    @@ -61,6 +61,7 @@
            #endif
             #if SGX_CORE_REV == 103
                    #define FIX_HW_BRN_22934
    +              #define FIX_HW_BRN_28889
             #else
    
            #if SGX_CORE_REV == 110

    This workaround doesn't exist in older driver versions, and is not enabled for revision 1.0.3 by default -- but appears to be crucial.
  3. Build SGX kernel modules with (e.g.):

    Code:
    TARGET_KERNEL=/mnt/newdisk/jules/pandora/kernel/pandora-kernel
    TARGET_VERSION=2.6.37.6-00038-g7819ce5
    
    ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- make \
      KERNEL_PATH=$TARGET_KERNEL \
      KERNEL_SRC=$TARGET_KERNEL \
      KDIR=$TARGET_KERNEL \
      KERNELDIR=$TARGET_KERNEL \
      SUPPORT_XORG=0 \
      V=1 \
      KERNEL_VERSION=$TARGET_VERSION CC=arm-linux-gnueabi-gcc  LD=arm-linux-gnueabi-ld  \
      AR=arm-linux-gnueabi-ar  BUILD=release TI_PLATFORM=omap3 OMAPES=2.x CROSS_COMPILE=arm-linux-gnueabi-

    Specifying the SGX version (OMAPES=2.x) is important, as is the SUPPORT_XORG=0 setting.
  4. Use the framebuffer, not X, for running the GLES2 app. I can't get X working with this setup -- TI/imgtec have implemented DRI/DRM support in their own X server (bundled with the driver), but it apparently doesn't work on the Pandora's 1.0.3-revision SGX chip. If you build with SUPPORT_XORG=1, the pvrsrvkm device fails to show up in /proc/devices, and consequently none of the userspace tools can find the GPU. If you use a different X server (e.g. normal Xorg from Debian), you get errors about missing or broken DRI2 support.
  5. (Non-step) the setting "DisableHWTextureUpload=1" in /etc/powervr.ini (introduced in a fairly recent driver version) doesn't appear to make any difference, for me.

So, the next steps might be to:

  • Try to backport the FIX_HW_BRN_28889 workaround to the older kernel module currently used by official firmware, and hope that will make things work properly with the older userspace binaries as well.
  • Update official firmware to the newer TI SGX support blobs, and figure out how to make X+GLES work, with or without the buggy DRI/DRM-enabled X server.

Thoughts, anyone?
 
Back
Top