Gpmark

[CKeichel]

Optimus released a graphical benchmark for the Wiz and Caanoo, that helps to compare both machines and the different firmware versions.

Firmware 1.5.0 at 533 MHz my Caanoo gives me this results:

Blitting Test: 298.2
Plasma: 216.6
Rotozoomer: 146.0
Rotozoomer Near: 215.9
Rotozoomer Far: 56.7
Radial Blur: 18.9
3D Bunny: 4.7

Maybe anyone with a Wiz can run the benchmark and post the results.

 

[Optimus]

Hi there!

Those are my results:


Caanoo at 533Mhz, Firmware 1.06:

Blitting Test: 331.5
Plasma: 252.3
Rotozoomer: 162.4
Rotozoomer Near: 251.5
Rotozoomer Far: 59.3
Radial Blur: 22.7
3D Bunny: 5.1


Wiz at 533Mhz, Firmware 1.2.6:

Blitting Test: 277.8
Plasma: 203.4
Rotozoomer: 142.2
Rotozoomer Near: 196.1
Rotozoomer Far: 62.0
Radial Blur: 17.9
3D Bunny: 4.6


As I was saying in one thread, firmware 1.06 on Caanoo seems to get quite faster speeds for reasons I don't know. With 1.5.0 I got results similar to yours. Also, in the Wiz version where they told me about the rotating screen performance, I found and used an lib.sdl.so.2 from a devkit that is supposed to be a faster version (plasma was 172.5 before using that for example) which improves things quite but still not reaching the results from Caanoo. I would be curious to investigate further since most people say that the Caanoo memory is a bit slower than Wiz but it doesn't show here.

 

[CKeichel]

Rotozoomer Far is the only benchmark, were the Wiz is faster, than the Caanoo, so I would guess it is safe to say the assumption, that the different Cas Latency makes the Caanoo 10% slower then the Wiz is false.
It seems more that the Caanoo is generally a little bit faster then the Wiz.
Interesting results about the Firmware 1.0.6, I wonder what they did to get such a performance drop in 1.5.0?

 

[paul3100]

My results using a caanoo with firmware 1.50

Blitting Test: 298.2
Plasma: 216.8
Rotozoomer: 146.0
Rotozoomer Near: 215.9
Rotozoomer Far: 56.7
Radial Blur: 18.3
3D Bunny: 4.7

paul3100

 

[AJack]

My results using a caanoo with firmware 1.50

Blitting Test: 298.2
Plasma: 216.8
Rotozoomer: 146.0
Rotozoomer Near: 215.9
Rotozoomer Far: 56.7
Radial Blur: 18.3
3D Bunny: 4.7

paul3100

My Caanoo (firmware 1.50) results were almost identical. At least we're consistent!

Blitting Test: 298.2
Plasma: 216.6
Rotozoomer: 146.0
Rotozoomer Near: 215.9
Rotozoomer Far: 56.7
Radial Blur: 19.3
3D Bunny: 4.7

 

[Popcorn Dave]

So basically the new firmware makes NO noticeable improvements and actually slows down the console? Niiiiice work, GPH. :rolleyes:

 

[u9i]

I guess i should keep mine at 1.0.6 until i actually buy a wifi donle

 

[Fda]

My Wiz results. FW 1.2.6, no overclock (533mhz):

Blitting test: 332.2
Plasma: 242.9
Rotozoomer: 169,6
Rotozoomer near: 233.9
Rotozoomer far: 73.9
Radial blur: 20.7
3d bunny 5.4

no idea why they're so different from Optimus's........maybe the SD class (mine is class 6)?

 

[Exophase]

Rotozoomer Far is the only benchmark, were the Wiz is faster, than the Caanoo, so I would guess it is safe to say the assumption, that the different Cas Latency makes the Caanoo 10% slower then the Wiz is false.
It seems more that the Caanoo is generally a little bit faster then the Wiz.
Interesting results about the Firmware 1.0.6, I wonder what they did to get such a performance drop in 1.5.0?

You're jumping to conclusions. If this test is using SDL the screen may be soft-rotated on Wiz. Hence why replacing it improved things. There is zero justification whatsoever for Wiz to be slower.

And that's why the effect goes down as framerate goes down, because you're changing the proportion of time spent doing work vs updating the screen.

Note that the fast emulators for both platforms are not using SDL.

 

[Optimus]

Yep, I don't know for sure why the speed differences, and the screen rotation on Wiz could be a reason. I guess I will try running the same tests without displaying something in the screen (maybe comment SDL_Flip(screen) would be enough?). Or try some pure CPU and memory read/write tests. I will definitely put some pure tests like this in a next version too. The Wiz results from Fda are also interesting.

 

[CKeichel]

Rotozoomer Far is the only benchmark, were the Wiz is faster, than the Caanoo, so I would guess it is safe to say the assumption, that the different Cas Latency makes the Caanoo 10% slower then the Wiz is false.
It seems more that the Caanoo is generally a little bit faster then the Wiz.
Interesting results about the Firmware 1.0.6, I wonder what they did to get such a performance drop in 1.5.0?

You're jumping to conclusions. If this test is using SDL the screen may be soft-rotated on Wiz. Hence why replacing it improved things. There is zero justification whatsoever for Wiz to be slower.

And that's why the effect goes down as framerate goes down, because you're changing the proportion of time spent doing work vs updating the screen.

Note that the fast emulators for both platforms are not using SDL.

No, I don't jump to conclusions, you said, the Caanoo is 10% slower, because of the different Cas Latency, I said, that I doubt that, because other factors may be significant for how fast a program runs. It seems I was right and you were wrong. In the end, it doesn't matter, why something runs faster or slower. Your exact words were:

I have heard numbers from people testing showing that a program running in Caanoo can be over 10% slower, with all of the other clocks set the same.

and then a little bit later:

I didn't hear 10% CKeichel, I heard benchmark results from which I derived 10%. I'm not disclosing who benchmarked what because I don't know if he's comfortable with me doing so, but it was for several emulators, so pretty representative of the kind of thing people will run on the platform

Until today I wait to hear, which emulators and which games for the emulators were benched, but as these benchmarks are "not public" we will never know were these mysterious 10% results come from.

Note that the fast emulators for both platforms are not using SDL.

Note that you are introducing a new term here, that was never in discussion, it's "fast emulators". Before you always talkes about "several emulators, so pretty representative of the kind of thing people will run on the platform".

 

[Exophase]

No, I don't jump to conclusions, you said, the Caanoo is 10% slower, because of the different Cas Latency, I said, that I doubt that, because other factors may be significant for how fast a program runs. It seems I was right and you were wrong.

Okay, maybe I will write a program that runs a suite of benchmarks but first detects if you're running a Caanoo then runs twice as many iterations if you are. Would that prove me right and you wrong? No, because it's an unfair comparison. It does matter what you're running, if the playing field isn't level.

You're so obsessed to prove that I was lying that you're willing to believe anything without critically questioning it. Nowhere on your other factors did you take into account reliance on libraries that rotate the screen. The one and only factor you gave is that Caanoo has more RAM, so tell me, do you think that's what's happening here? Caanoo is faster because it has more RAM?

Yeah, I talk about emulators, because that's still what most people run on these things. For the "more RAM makes it faster" example the only thing you could refer to was one emulator, so...

Don't worry, when I release the next version of Temper for Wiz and Lordus ports it to Caanoo I'm sure he'll be happy to benchmark both versions for you, and we'll know that there's no critical reliance on something rotating the screen in the way. Oh, and I said Caanoo CAN be 10% slower, not that it ALWAYS IS 10% slower, but I guess you're offended by this simple notion.

 

[Darkknight512]

My Wiz results. FW 1.2.6, no overclock (533mhz):

Blitting test: 332.2
Plasma: 242.9
Rotozoomer: 169,6
Rotozoomer near: 233.9
Rotozoomer far: 73.9
Radial blur: 20.7
3d bunny 5.4

no idea why they're so different from Optimus's........maybe the SD class (mine is class 6)?

Mine are +/- 0.1 from you, Wiz at 533 Mhz running 1.2.6. Class 10 SD card.

I also tried these tests at 800 Mhz, about a 50% higher clock. It gives about 23-32% increase in performance. I'm guessing that is because of the lack of cache?

 

[Exophase]

So now two people have roduced the Wiz 1.2.6 results, three people the Caanoo 1.5.0 results, and one person the Caanoo 1.0.6 results..

By the way, maybe this test should be done for a long enough time to verify against an external time source (or using the hardware timers directly), because there have been reports in the past of the Wiz kernel giving inaccurate timing numbers. This can apply or not to both platforms, for any of the tests, and can in differing amounts.

 

[CKeichel]

No, I don't jump to conclusions, you said, the Caanoo is 10% slower, because of the different Cas Latency, I said, that I doubt that, because other factors may be significant for how fast a program runs. It seems I was right and you were wrong.

Okay, maybe I will write a program that runs a suite of benchmarks but first detects if you're running a Caanoo then runs half as many iterations if you are. Would that prove me right and you wrong? No, because it's an unfair comparison.

You're so obsessed to prove that I was lying that you're willing to believe anything without critically questioning it. Nowhere on your other factors did you take into account reliance on libraries that rotate the screen. The one and only factor you gave is that Caanoo has more RAM, so tell me, do you think that's what's happening here? Caanoo is faster because it has more RAM?

Again, you said:

I didn't hear 10% CKeichel, I heard benchmark results from which I derived 10%. I'm not disclosing who benchmarked what because I don't know if he's comfortable with me doing so, but it was for several emulators, so pretty representative of the kind of thing people will run on the platform

Until today I wait for a proof for that. And I really don't see the need for longer tests, cause the Caanoo results are pretty much the same for everybody and I find it very unlikely, that the results are inaccurate as you suggest. At least, this benchmark is public and it's results are public for everyone. Maybe the benchmarks, you were referring to earlier had some flaws?

 

[tyke]

My Caanoo results. FW 1.5.0, no overclock (533mhz):

Blitting test: 331.5
Plasma: 252.3
Rotozoomer: 162.5
Rotozoomer near: 251.3
Rotozoomer far: 59.4
Radial blur: 22.6
3d bunny 5.1

 

[CKeichel]

My Caanoo results. FW 1.5.0, no overclock (533mhz):

Blitting test: 331.5
Plasma: 252.3
Rotozoomer: 162.5
Rotozoomer near: 251.3
Rotozoomer far: 59.4
Radial blur: 22.6
3d bunny 5.1

Interesting, you are getting the exact results of the 1.0.6 firmware in 1.5.0

 

[Exophase]

Until today I wait for a proof for that.

And what do you want from me? We've explained that it can't be happening, you've implied several times now that it's just a lie...

And I really don't see the need for longer tests, cause the Caanoo results are pretty much the same for everybody and I find it very unlikely, that the results are inaccurate as you suggest. At least, this benchmark is public and it's results are public for everyone. Maybe the benchmarks, you were referring to earlier had some flaws?

I said they can be inaccurate - multiple people getting the same thing does NOT suggest that the time base is correct, just that if it's off it's the off the same for everyone. This can easily happen if bad scale factors are used for clocks.

If the tests Lordus did had some flaws it wouldn't be due to screen rotation because I know his emulators weren't doing it, something we already KNOW happens with SDL on Wiz. But who cares, if you're going to deny Optimus's can have problems then why claim Lordus's could?

And yes, I do think emulators are representative of what people run on the platforms, or at least more representative than these sort of demos.

 

[Darkknight512]

Exo - If what you say is true then none of this data could be trusted. By my understanding you are suggesting it is possible the device's timer is not always accurate and it might make 1 second actually say 0.8 seconds in real time which would skew our results by 20%? (if the device sees 0.8 actual seconds in 1 second then the reported FPS would be 20% lower)

 

[Exophase]

Exo - If what you say is true then none of this data could be trusted. By my understanding you are suggesting it is possible the device's timer is not always accurate and it might make 1 second actually say 0.8 seconds in real time which would skew our results by 20%? (if the device sees 0.8 actual seconds in 1 second then the reported FPS would be 20% lower)

It's possible, not saying it's likely or anything - I just know that a few people have gotten bad timing using the standard kernel time functions. Pickle had a hard time synchronizing framerate with this and there was someone who was getting time deltas of less than zero using it. That doesn't say much about the nature of the problem, or whether or not it has been fixed, but just that it's worth confirming to be accurate.

For what it's worth, I've confirmed with Lordus that he was using hardware timers on both, and was fixing the clock sources to both and set the RAM timings to fastest on both - so the only differences between the two were that Caanoo had a worse CAS latency and Wiz had a worse tRS latency (by one clock, from 8 to 9). Like he reported in another thread, he got 10% worse performance in Picodrive and not just the emulator he won't disclose. So someone else could reproduce this test. This shows that with very close to everything else kept exactly the same the worse CL value can reduce performance by as much as 10%.

It's also possible that GPH is using otherwise worse timing than possible somewhere for the Wiz settings. But this can be corrected. Someone should do a full dump of all the RAM timings and clock settings as part of the test. But I still contend that SDL can be a variable factor.