Optimizing a 3D renderer for NEON is not for the faint of heart. Definitely a lot of work from the ground up if you want to get something generally worth using.
I touched a little bit on the issue you bring up about having to mask in my earlier post. You're right, you no longer have the ability to directly have an early exit on pixels. This isn't quite as bad as it seems, since at least for some polygons that early test won't be good on branch prediction, but it can hurt sometimes. There are still some things you can do, though:
- Check if all of the pixels in the polygon fail and abort the whole thing. This doesn't take much overhead. Let's say you calculate visibility for 16 pixels at a time using a vectors of 16x8-bit, so 0xFF means visible and 0x00 means not visible. If you keep a 16x8 accumulator register and subtract the masks from it at each iteration, then after you're done parallel add the results together, you will get a number of how many pixels are visible. If that number is 0 it means you don't have to render the polygon. If that number is the total pixels in the polygons it means you can use a function that doesn't check the masks. If you're rendering to tiles instead of the entire screen, large polygons will often intersect against the edges of the tile, which will give you more chances to have polygon segments that are entirely obscured.
- Do some kind of hierarchical or partial Z checking where you don't render pixels in some tile that all fail the Z test. Can be done with a hierarchical Z comparison as well. A lot more complex and maybe not conducive to CPU operations or worth the overhead. I haven't ever tried this.
- Run-line compress the pixel stream to remove obscured pixels. I haven't tried this either.
Compression would work something like this: let's say your polygon is two rows and visibility masks that look like this:
0xFF 0xFF 0xFF 0xFF 0xFF 0x00 0x00 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0x00 0x00 0x00 0xFF 0x00 0x00
0xFF 0xFF 0xFF 0xFF 0x00 0x00 0x00 0x00 0xFF 0xFF 0xFF 0xFF 0x00 0x00 0x00 0xFF
First you can convert this to a bitmap per row, so it looks like this:
1111100111111000100b
1111000011110001b
(where the most significant bit is the leftmost one)
This can be done fairly efficiently with NEON.
Then you iterate through those bitmaps to create a list of spans. You load up a 32-bit value, use clz (__builtin_clz in C, there's also a NEON version) to see how many 0 bits to skip, then another clz on the inverted value (or maybe clo, count leading ones) to see how large the span is. Shift off the spans you accounted for as you go. Repeat until the bitmap is zero. So you'd have a list that looks like:
5, 2, 6, 3, 1
4, 4, 4, 3, 1
Which says how many pixels to render then how many to skip.
Then you can compress the pixel data you have so the ones to skipped are skipped, and resume processing them with SIMD functions on the compact version. Whether or not this is worth it really depends on how much work you're doing on the pixels after the depth test. It's probably not worth doing if it has tests that need the texel loaded, like alpha test. Pattern test you can handle by ANDing the visibility bitmap with the pattern.
Then when it comes time to write out the pixel and update depth you use the list that says how many to draw and how many to skip to determine where to write it.
I'm not doing anything too extreme for hidden surface removal in DS emulation because the system requires that opaque polygons are rendered from the top of the screen to the bottom (and if you don't some games break, like missing text and things like that). And because higher things on the screen tend to be further away this works against early Z. But in your case, games can apply some level of sorting (for example, on a per-object basis) to try to render objects nearest to the camera first, to maximize early Z. This could be beneficial even with the early Z exit you have now.