Exophase
Nothing good will ever come of Exophase.
I felt like doing some more test numbers (everything is in 32bit quantities)
Write to uncached, unbuffered area: 15-16, at 5 words out it was 15 per, the 6th one cost 16. This could have to do with cycle alignment with the bus clocks (it should be 1/4th CPU clocks)
Write to uncached, buffered area: 4 cycles, overlappable. I am not joking. I have no idea how it pulls this off, but apparently it is capable of sustaining full bus bandwidth for fills (533MB/sec) while draining off the write buffer, and each of these cycles can be overlapped with a non-external memory access instruction, so hiding the str entirely is not hard at all in usual cases. I have no idea how it pulls this off, my only guess is that somehow the commands are all allowed to overlap each other or something. I have no idea why the write buffer can hit memory so much more rapidly than the CPU can, but this means that as far as writes are concerned, the small burst width does not even really matter. Incidentally though, there is a cycle cost + 1 for when it starts (I have no idea when "starting" is exactly, could be first non-store instruction or something?), so it might be 5 cycles with an overlap somewhere.
Load from uncached cacheable memory (cache miss): 72 cycles. I do not know where the constituents go, but this is carrying 6 cycles of loop overhead, so I wanted to see if anything was eaten by critical word first. Thus...
Two loads from uncached cacheable memory: 72 cycles. There is the critical word first access overlapping the additional loads until the next access, so it could be as much as 78 cycles raw - so how much is it exactly? Can find out by padding nops until it starts going up. It got kind of weird here - with 3 nops it is 78 for the whole iteration, with 4-6 79, with 7-8 80... two loads from consecutive addresses was 73 instead of 72. Second load from address 28 bytes away is 81 cycles - I think this is closer to the real cache miss speed.
This should probably be investigated more.
Load from uncacheable memory: 72-73 cycles. Yes, it is that bad, that much worse than store to uncacheable memory. It is even worse than a cache miss. Why? I have no idea. Incidentally, two took about 145, one about 73.. seems like there could be a cycle overlap.
Load from cacheable uncached memory followed by store to same region (should be cache miss + dirty cache line writeback): 148 cycles. This is what a lot of your cache misses are going to look like. It is not pretty and I think it could be one of the primary bottlenecks of the system (hence why we are seeing not that amazing performance). You would think the writeback would be as fast as the writebuffer drain but apparently not, or something else is limiting it somehow.
Remarks: Writes to write buffered memory are really fast. Unfortunately the only critical area for this is writing to the framebuffer (with MMUhack on) - MAYBE if the connection to the GTE is fast enough you could quickly stream things to it to work as a coprocessor. But only if it does not need to spit memory back out to you because uncached reads are all kinds of slow. I wish there was some other way to optimize towards this but there is not really, unless you can somehow push an algorithm towards more eager writing instead of reading.
Cache line retiring is slow. If you can manage to organize your data so read-only/write-seldom variables are bunched together in the same cache lines you might mitigate this some, and this would be some improvement for any platform. I do not know how much of a difference it would really make though, and it is damn hard to engineer code like this (in ASM, in anything higher, forget it)
So overall, you can reach full 533MB/sec bus bandwidth writing things (if you do an optimized fractal generator it would probably be super fast) - you can even shove other instructions in 3 of those 4 cycle slots. But reading things you would get nowhere near that. The best I could get is 78 cycles spanning a cache line read, no amount of unrolling would improve that, and 72-73 cycles for a single word read from uncached (so cache is definitely the way to go, even if you thrash it to bits - the spatial locality bonus the line reading gives is huge). That means 9.75 cycles per word, which is over twice as bad as the write bandwidth, or 218.667MB/sec, while thrashing up your cache (and in the process probably causing a bunch of retirement writebacks slowing you down a lot more unless you are reading a ton). Pretty unfortunate.
I would love to have explanations for where the cycles are actually going. If any kind of expert wants to give some theories then by all means.
Write to uncached, unbuffered area: 15-16, at 5 words out it was 15 per, the 6th one cost 16. This could have to do with cycle alignment with the bus clocks (it should be 1/4th CPU clocks)
Write to uncached, buffered area: 4 cycles, overlappable. I am not joking. I have no idea how it pulls this off, but apparently it is capable of sustaining full bus bandwidth for fills (533MB/sec) while draining off the write buffer, and each of these cycles can be overlapped with a non-external memory access instruction, so hiding the str entirely is not hard at all in usual cases. I have no idea how it pulls this off, my only guess is that somehow the commands are all allowed to overlap each other or something. I have no idea why the write buffer can hit memory so much more rapidly than the CPU can, but this means that as far as writes are concerned, the small burst width does not even really matter. Incidentally though, there is a cycle cost + 1 for when it starts (I have no idea when "starting" is exactly, could be first non-store instruction or something?), so it might be 5 cycles with an overlap somewhere.
Load from uncached cacheable memory (cache miss): 72 cycles. I do not know where the constituents go, but this is carrying 6 cycles of loop overhead, so I wanted to see if anything was eaten by critical word first. Thus...
Two loads from uncached cacheable memory: 72 cycles. There is the critical word first access overlapping the additional loads until the next access, so it could be as much as 78 cycles raw - so how much is it exactly? Can find out by padding nops until it starts going up. It got kind of weird here - with 3 nops it is 78 for the whole iteration, with 4-6 79, with 7-8 80... two loads from consecutive addresses was 73 instead of 72. Second load from address 28 bytes away is 81 cycles - I think this is closer to the real cache miss speed.
This should probably be investigated more.
Load from uncacheable memory: 72-73 cycles. Yes, it is that bad, that much worse than store to uncacheable memory. It is even worse than a cache miss. Why? I have no idea. Incidentally, two took about 145, one about 73.. seems like there could be a cycle overlap.
Load from cacheable uncached memory followed by store to same region (should be cache miss + dirty cache line writeback): 148 cycles. This is what a lot of your cache misses are going to look like. It is not pretty and I think it could be one of the primary bottlenecks of the system (hence why we are seeing not that amazing performance). You would think the writeback would be as fast as the writebuffer drain but apparently not, or something else is limiting it somehow.
Remarks: Writes to write buffered memory are really fast. Unfortunately the only critical area for this is writing to the framebuffer (with MMUhack on) - MAYBE if the connection to the GTE is fast enough you could quickly stream things to it to work as a coprocessor. But only if it does not need to spit memory back out to you because uncached reads are all kinds of slow. I wish there was some other way to optimize towards this but there is not really, unless you can somehow push an algorithm towards more eager writing instead of reading.
Cache line retiring is slow. If you can manage to organize your data so read-only/write-seldom variables are bunched together in the same cache lines you might mitigate this some, and this would be some improvement for any platform. I do not know how much of a difference it would really make though, and it is damn hard to engineer code like this (in ASM, in anything higher, forget it)
So overall, you can reach full 533MB/sec bus bandwidth writing things (if you do an optimized fractal generator it would probably be super fast) - you can even shove other instructions in 3 of those 4 cycle slots. But reading things you would get nowhere near that. The best I could get is 78 cycles spanning a cache line read, no amount of unrolling would improve that, and 72-73 cycles for a single word read from uncached (so cache is definitely the way to go, even if you thrash it to bits - the spatial locality bonus the line reading gives is huge). That means 9.75 cycles per word, which is over twice as bad as the write bandwidth, or 218.667MB/sec, while thrashing up your cache (and in the process probably causing a bunch of retirement writebacks slowing you down a lot more unless you are reading a ton). Pretty unfortunate.
I would love to have explanations for where the cycles are actually going. If any kind of expert wants to give some theories then by all means.
