SGX 3D Freeze on CC model

[ptitSeb]

I open this thread to try find a fix / workaround on the hang that happens sometimes on 3D driver of the CC model only.

I have witnessed it in a few game, like Torus_trooper (both GLES port by M-HT and my glshim port), Sable (Loic2003 port using glshim), nprQuake with NPR rendring (glshim, but it's ok with default normal renderer) and Tasty Static (glshim by lunixbochs).

I was assuming it was some bugs with Line drawing, but it may not.

M-HT also hinted that it triggers when rendering off screen.

I worked with Torus-trooper to try find a workaround. On a CC model, Torus runs, the intro screen run fine as long as it is empty, but in game, quickly, it freeze (but the music continue). 

• Try to define a simple glScissor(0,0,800,480); and enabling Scissor. => No effect it seems.
• Try to define a simple glScissor(1,1,799,479); and enbling Scissor, just in case the 0,0 is a special case => No effect.
• Render offscreen in an FBO. Created a 512*256 small FBO (it's GLES1.1, so Texture have to be power of 2 sized), render game in 640x480=>512x256 in a texture, then show the texture at buffer swap => Quite stable. I've been able to play until level 4 (and loose), but that it crash at intro screen.
• Render offscreen if an FBO. Created a large 1024*512 FBO, render game in 640x480, and use a centred glViewport(192,16,640,480) and show part of the texture at buffer swap => Immediate crash when starting the game (especially with 4.03.00.02 driver)

Strange...

I have launched the game with a debugger and Interupted inside the freeze.

Here is the BackTrace:


#0  0x403a45ac in ioctl () from /lib/libc.so.6
#1  0x41430bd0 in PVRSRVBridgeCall (hServices=<value optimized out>, ui32FunctionID=3223086915, pvParamIn=<value optimized out>, ui32InBufferSize=<value optimized out>,
    pvParamOut=0xbefff3a8, ui32OutBufferSize=8)
    at /home1/prathap/fresh/imgtiddk16_netrakernel/ti_references/sources/GFX_Linux_DDK/src/eurasia/services4/srvclient/env/linux/common/pvr_bridge_u.c:242
#2  0x41430508 in PVRSRVEventObjectWait (psConnection=<value optimized out>, hOSEvent=0xf)
    at /home1/prathap/fresh/imgtiddk16_netrakernel/ti_references/sources/GFX_Linux_DDK/src/eurasia/services4/srvclient/env/linux/common/osfunc_um.c:457
#3  0x41c87514 in CBUF_GetBufferSpace (apsBuffers=<value optimized out>, ui32DWordsRequired=15204, ui32BufferType=1, bTerminate=IMG_FALSE)
    at /home1/prathap/fresh/imgtiddk16_netrakernel/ti_references/sources/GFX_Linux_DDK/src/eurasia/eurasiacon/common/buffers.c:534
#4  0x41c56d8c in GetVertexIndexBufferSpace (gc=0x234620, ui32NumIndices=<value optimized out>, ui32NumVertices=<value optimized out>, ui32SizePerIndex=<value optimized out>)
    at drawvarray.c:641
#5  0x41c577bc in DrawBatchOnVBuffer (gc=0x234620, eMode=1, ui32First=0, ui32CountIn=1103462188, ui32NumIndices=5828, eType=0, elements=0x0, ui32VertexStart=0, ui32VertexCount=5828)
    at drawvarray.c:1272
#6  0x41c5973c in glDrawArrays (mode=1, first=0, count=5828) at drawvarray.c:2277
#7  0x400f7458 in glDrawArrays () from /mnt/utmp/torus_trooper/lib/libGL.so.1
#8  0x00000000 in ?? ()


In the backtrace, we see the freeze is inside a glDrawArrays with mode=GL_LINES (=1) with a vertex count quite high (probably the wireframe part of the tunnel that I draw in a single draw call).

Also, I continued and break again, the backtrace is just the same, nothing changed.

In case someone wants to try, here a version with the large FBO, crashing immediatly (in game) on a CC (should works fine on Rebirth/Gigahertz): http://ptitseb.openpandora.org/torus_trooper.pnd

Any hint someone?

 

[notaz]

From what I've seen with similar hangs it's SGX hardware lockup of some sorts, perhaps some microcode problem or something. If something can be done it's probably indirect only, like maybe throwing in some glFlush() / glFinish() to split the load on the chip into smaller parts.

 

[ptitSeb]

From what I've seen with similar hangs it's SGX hardware lockup of some sorts, perhaps some microcode problem or something. If something can be done it's probably indirect only, like maybe throwing in some glFlush() / glFinish() to split the load on the chip into smaller parts.

Ok, I'll try to put some glFlush inside the torus drawing, see if that change something.

 

[JonnyH]

It looks like it's ran out of vertex buffer space, and for some reason failing to flush the current buffer to get more space.

Can you try changing the "DefaultVertexBufferSize" apphint in powervr.ini? (The default value is "204800" (200k) - try increasing that to 800k or something).

5000 vertices is quite a lot to be passing into a user-array DrawArrays - can this be moved to a buffer object? Failing that, a work-around may be to dispatch them into 2 (or more) DrawArrays() calls

Do you have unneeded vertex state enabled (front/back colours, textures enabled) - each of those would cause more space to be needed in the vertex buffer

Is there any chance you can get an apitrace (https://github.com/apitrace/apitrace)? If that's impossible, can you just list all the previous DrawArrays() (or DrawElements()) arguments.

As an aside - which driver version are you using? I'm guessing 1.6 from the paths in the debug info (Which I find surprising that it hasn't been stripped)

 

[JonnyH]

From what I've seen with similar hangs it's SGX hardware lockup of some sorts, perhaps some microcode problem or something. If something can be done it's probably indirect only, like maybe throwing in some glFlush() / glFinish() to split the load on the chip into smaller parts.

By default glFlush/glFinish won't actually do what you want here, as they don't actually cause the hw to do anything (by default -that can be modified by setting the "FlushBehaviour" apphint to '1' (Flush and wait for vertex processing to complete) or '2' (Flush and wait for all rasterisation to complete).

 

[ptitSeb]

It looks like it's ran out of vertex buffer space, and for some reason failing to flush the current buffer to get more space.

Can you try changing the "DefaultVertexBufferSize" apphint in powervr.ini? (The default value is "204800" (200k) - try increasing that to 800k or something).

5000 vertices is quite a lot to be passing into a user-array DrawArrays - can this be moved to a buffer object? Failing that, a work-around may be to dispatch them into 2 (or more) DrawArrays() calls

Do you have unneeded vertex state enabled (front/back colours, textures enabled) - each of those would cause more space to be needed in the vertex buffer

Is there any chance you can get an apitrace (https://github.com/apitrace/apitrace)? If that's impossible, can you just list all the previous DrawArrays() (or DrawElements()) arguments.

As an aside - which driver version are you using? I'm guessing 1.6 from the paths in the debug info (Which I find surprising that it hasn't been stripped)

Ok, I can try that. Do you have a document describing the structure of the powervr.ini file ? In which section of the file should I put DefaultVertexBufferSize=800000, just in [default] ?

apitrace I can try, but it's not easy to get.

The draw call is big in that case here, with more than 5000 vertex, but I don't think it need to be the case to have the freeze. Here, it's a "simple" line, with blend activated, but no texture. Only a single simple color.

 

[JonnyH]

I'm not aware of any documentation of the powervr.ini file, but the general layout is:

[process_name]

key=value

with a special [default] section that is loaded for all processes.

For example, if I want to set 'Option1' to 'foo' for all processes, except 'mygame' which you want 'bar':

[default]

Option1=foo

[mygame]

Option1=bar

I am also not aware of any list of supported options/values anywhere online, but you can get a sampling by googling for 'powervr.ini' - as a fair number of published systems have example files with various options.

 

[ptitSeb]

I have try to use an update powervr.ini file, but with no changed result, still crashing as soon as game start. I'll try the glFlush with updated powervr.ini later...

My file contains

[default]
WindowSystem=libpvrPVR2D_FLIPWSEGL.so
DefaultVertexBufferSize=800000

 

[Farox]

I don't know maybe this could help on sintax of a powervr.ini file ?

http://processors.wiki.ti.com/index.php/SGXDbg

maybe we need to change the parameter ParamBufferSize.

from the linked page :

This parameter buffer size can be increased, when complex 3D scenes are used in the application. For simple 2D applications, changing this parameter makes no effect. For eg to set 32MB as ParamBufferSize, edit /etc/powervr.ini to add below line-
ParamBufferSize=33554432

 

[ptitSeb]

Thanks @Farox, interesting page. I'll try this weekend.

 

[fantomid]

Hello there !!

I sent mails to linuxbochs some times ago to help him about the Tasty Static crash on CC model.

I didn't manage to find the problem. Not an expert in graphic layers. Sorry linuxbochs  .

But during last FOSDEM, I assisted to a conference about piglit (link1, link2) that could be used to test the gles driver of the Pandoras.

If someone got time, maybe it could be great to have the results for the Pandoras. The results I managed to get was not very useful.

Then, we could add more tests to the test suite.

 

[JonnyH]

Sorry, I can't really help without a proper reproduction case (I'm still looking for a pandora). An apitrace would probably be good enough, then I can run it on another omap platform. Getting an arm build and running it with LD_PRELOAD=/path/to/egltrace.so ./application should output a trace file into the current directory (might be quite large!).

I'm a bit confused:

According to http://pandorawiki.org/SGX_driversthe 'CC' model has a SGX revision 1.0.3, but we /don't/ support that in 1.6 - can someone paste the contents of '/proc/pvr/version' of a device that shows the crash? The exact driver / sgx revision will help me match it up to our internal codebase.

 

[ptitSeb]

Sorry, I can't really help without a proper reproduction case (I'm still looking for a pandora). An apitrace would probably be good enough, then I can run it on another omap platform. Getting an arm build and running it with LD_PRELOAD=/path/to/egltrace.so ./application should output a trace file into the current directory (might be quite large!).

I'm a bit confused:

According to http://pandorawiki.org/SGX_driversthe 'CC' model has a SGX revision 1.0.3, but we /don't/ support that in 1.6 - can someone paste the contents of '/proc/pvr/version' of a device that shows the crash? The exact driver / sgx revision will help me match it up to our internal codebase.

I'll try to provide everythong toonight. I get a trace on a Gigahertz model, so a working trace, because on the CC, it freeze and render everything difficult.

About the content of the file, I'll do it tonight (no Pandora here), but I cat say that the PowerVR revision is "2" for CC model, "3" for Rebirth and 5 for Gigahertz.

 

[JonnyH]

Sorry, I can't really help without a proper reproduction case (I'm still looking for a pandora). An apitrace would probably be good enough, then I can run it on another omap platform. Getting an arm build and running it with LD_PRELOAD=/path/to/egltrace.so ./application should output a trace file into the current directory (might be quite large!).

I'm a bit confused:

According to http://pandorawiki.org/SGX_driversthe 'CC' model has a SGX revision 1.0.3, but we /don't/ support that in 1.6 - can someone paste the contents of '/proc/pvr/version' of a device that shows the crash? The exact driver / sgx revision will help me match it up to our internal codebase.

I'll try to provide everythong toonight. I get a trace on a Gigahertz model, so a working trace, because on the CC, it freeze and render everything difficult.

About the content of the file, I'll do it tonight (no Pandora here), but I cat say that the PowerVR revision is "2" for CC model, "3" for Rebirth and 5 for Gigahertz.

That '2', '3' and '5' doesn't look like one of our core revisions - I was hoping for a line like:

System Version String: SGX revision = 1.2.5

That would mean the core revision was 1.2.5. They're always 3 dot-separated values, (the first always being '1')

 

[ptitSeb]

@JonnyH: that is the content of /etc/powervr-esrev that contains 2,3 or 5.

I'll do a cat tonight of the /proc/prv/revision tonight on all my Pandora model (I don't have any of them with me for now).

 

[notaz]

According to http://pandorawiki.org/SGX_driversthe 'CC' model has a SGX revision 1.0.3, but we /don't/ support that in 1.6

TI managed to build v1.6 blobs for 1.0.3 somehow (and were shipping them), but they're only partially working. IIRC the disassembly of those 1.0.3 blobs is nearly identical to 1.2.1 on v1.6, while on v1.4 they differ substantially. Probably the person building them has set old defines for 1.0.3, but the codebase had support already removed. Somehow the result still "works" (for some games only).

And I'm 100% sure /proc/prv/revision contains 1.0.3 on CC, like the wiki says.

 

[JonnyH]

@JonnyH: that is the content of /etc/powervr-esrev that contains 2,3 or 5.

I'll do a cat tonight of the /proc/prv/revision tonight on all my Pandora model (I don't have any of them with me for now).

http://processors.wiki.ti.com/index.php/AM35x-OMAP35x_Graphics_SDK_Getting_Started_Guidesuggests that ES2.x is 1.0.3 - which is not supported by us on 1.6.

I would then be interested if you're currently running 1.4 (TI's version 3.01.x or 4.00.x) or 1.3 (TI's version 3.x) - again that should be in /proc/pvr/version.

As an aside, it might also be running out of index buffer space - can you increase the 'DefaultIndexBufferSize' apphint in the same way? (To 800k?)

 

[notaz]

BTW here is my old bug report to TI:

http://e2e.ti.com/support/dsp/omap_applications_processors/f/447/t/95950

in short their answer was "1.0.3 is not supported".

 

[JonnyH]

According to http://pandorawiki.org/SGX_driversthe 'CC' model has a SGX revision 1.0.3, but we /don't/ support that in 1.6

TI managed to build v1.6 blobs for 1.0.3 somehow (and were shipping them), but they're only partially working. IIRC the disassembly of those 1.0.3 blobs is nearly identical to 1.2.1 on v1.6, while on v1.4 they differ substantially. Probably the person building them has set old defines for 1.0.3, but the codebase had support already removed. Somehow the result still "works" (for some games only).

And I'm 100% sure /proc/prv/revision contains 1.0.3 on CC, like the wiki says.

If you bypass the revision check, it will likely mostly 'work', as most of the revision-specific code is working around small quirks/bugs/changes, so it's probably not surprising that it would 'mostly' work. I wouldn't touch it with a 10ft barge pole, however, as some of those differences /will/ break things.

 

[M-HT]

On my CC Pandora /proc/pvr/version contains:

Version 1.4.14.2514 (release)

System Version String: SGX revision = 1.0.3