notaz said:
I've just ran some tests on this, and it looks like this might be indeed one of major bottlenecks. When I write words and post-increment address by 1kB, my test runs more than 8 (!) times slower than with post-increment of 32 bytes or less. It seems write buffer won't even work then, there is no difference if it's enabled or not. I thought this could have something to do with TLB cache misses so I set up single 1MB section in MMU, ran the test there and got mostly the same results. I can stuff some nops between those accesses without affecting measurements, so it must be memory latency issue.
It's not the MMU/TLB misses, it's the DDR interface. 1KB is a typical size of DDR rows. If you keep accessing within the same rows then you just need to supply a column address, and this can overlap with read/write cycles (pipelined) allowing you to sustain full bandwidth. If you need to move to a different row then you need to do row address strobing AND row precharge and possibly another step that I'm forgetting.
My guess is that Wiz is performing exactly one full precharge/row/column cycle on every non-cached memory load, cacheline load, and cacheline evict, with only the write buffer being able to detect same row access between consecutive accesses. You probably also get a ton more efficiency using ldm on non-cached regions than several ldrs, because of this. I think the GP2X is probably the same way, but the relative damage is much worse here because the memory clock to core clock ratio is 1:4 instead of 1:2, and the DDR aspect of the RAM only applies to data transfers, not all of the clocks spent on address/precharge.
Like the GP2X it seems like the values selected for the timings of these operations are way too high - you should look into lowering them and trying the test again, if you have the databook. You can play around with these registers:
Code:
typedef enum
{
MEM_CFG = 0x14800,
MEM_TIME0 = 0x14802,
MEM_TIME1 = 0x14804,
MEM_REFRESH = 0x14808,
MEM_CONTROL = 0x1480A
} wiz_memory_regs_enum;
My tests were much more sequential than 1KB so I didn't see much difference, didn't bother to do wider ones (should have).
notaz said:
Nah, there is nothing like that in documentation.
Yeah, I didn't expect it.
notaz said:
EDIT: Well, after increasing buffer size I can see TLB misses hit pretty hard, as expected. Although hitting different page on every access is extremely unlikely, that scenario slows things down at least 5 times compared to exact same code with no TLB misses (which can be achieved thanks to 1MB section setting in MMU).
It's not just hitting a different page on every access, it'd be causing a TLB miss on every access. The TLB should be pretty effective almost all the time for most programs, MUCH more effective than icache/dcache, and a TLB miss isn't that expensive since it's hardware TLB. Modifying Linux to use 1MB page regions is probably out, so you'd have to use the upper memory/mmuhack to get this. Maybe someone can try putting their huge arrays here but I doubt it'll make a big difference.
