Release Pcsx-Rearmed

[notaz]

This is caused by Ari64's dynarec recompiling stuff (there is lots of invalidation going on), game is reloading some code modules between fights, I'm not sure if I can fix that. The problem should be less apparent after first few fights though.

The dynarec was designed to handle this. If the old code module is still in the cache, it will recognize this and not recompile it. Can you profile this and see where the time is being spent?

How do you profile recompilation itself? Oprofile gives the following:

Counted CPU_CYCLES events (Number of CPU cycles) with a unit mask of 0x00 (No unit mask) count 100000
samples  cum. samples  %        cum. %     app name                 symbol name
11964    11964         31.3769  31.3769    libc-2.9.so              memset
4450     16414         11.6706  43.0475    pcsx                     new_recompile_block
3296     19710          8.6441  51.6916    pcsx                     GetTextureTransColG32
2702     22412          7.0863  58.7779    pcsx                     anon (tgid:2622 range:0x1000000-0x2000000)

0x1000000-0x2000000 is the translation cache; memset could be coming from GPU plugin, but I don't really know.

For me it looks like normal operation, it happens only first time battle is loaded, but the whole thing goes for 5-10sec and causes varying slowdowns and sound tempo changes which people notice and keep complaining about. Maybe something like liveness analysis depth could be reduced just for that game, I don't know.

This is the same situation that caused slowness because of __clear_cache() calls, it would get stuck for several seconds, now it's a lot better but people still complain.

You have the latest kernel? From a git repository or something? I'd like to replicate your software setup to confirm if this is the problem or not.

The kernel is at http://openpandora.org/firmware/uImage , can be copied to /boot/ but it will likely break things like sgx and wifi and needs installing corresponding module .ipks to fix.

OK, yes it's the hooligans one I'm using(couldn't find any others at the time but it was handy that it had trainers included). Bit of a bummer because now I can't cheat

...any chance of action replay support?

There is some cheat support in the emu already, I just haven't made it accessible in the GUI yet. And I think I fixed the bug so that trainer should work in the next version.

 

[TitanUranus]

Counted CPU_CYCLES events (Number of CPU cycles) with a unit mask of 0x00 (No unit mask) count 100000
samples  cum. samples  %        cum. %     app name                 symbol name
11964    11964         31.3769  31.3769    libc-2.9.so              memset
4450     16414         11.6706  43.0475    pcsx                     new_recompile_block
3296     19710          8.6441  51.6916    pcsx                     GetTextureTransColG32
2702     22412          7.0863  58.7779    pcsx                     anon (tgid:2622 range:0x1000000-0x2000000)

I know emulation is exciting, but surely you don't need to have so many "cum samples" on your pandora?

 

[naples39]

I know emulation is exciting, but surely you don't need to have so many "cum samples" on your pandora?

Fitting post given your avatar pic.

 

[Lobo]

OK, yes it's the hooligans one I'm using(couldn't find any others at the time but it was handy that it had trainers included). Bit of a bummer because now I can't cheat

...any chance of action replay support?

There is some cheat support in the emu already, I just haven't made it accessible in the GUI yet. And I think I fixed the bug so that trainer should work in the next version.

I love you!

 

[Critical_Impact]

So I tried out Final Fantasy 8 and Resident Evil 3.
I could get FF8 to start, but the first battle I had, it froze, the music kept playing though.

As for resident evil, I tried using multiple BIOS's and I couldn't get it to work at all.

I also gave Digimon Rumble Arena a go, that worked fine with an external BIOS file.

 

[ari64]

How do you profile recompilation itself?

In most cases, new_recompile_block() is a small percentage of the total CPU time, so I haven't looked at where the time is being spent. A lot of things get inlined into this function, so you might have to remove "static" from some functions if you want them as separate symbols.

Oprofile gives the following:

Counted CPU_CYCLES events (Number of CPU cycles) with a unit mask of 0x00 (No unit mask) count 100000
samples  cum. samples  %        cum. %     app name                 symbol name
11964    11964         31.3769  31.3769    libc-2.9.so              memset
4450     16414         11.6706  43.0475    pcsx                     new_recompile_block
3296     19710          8.6441  51.6916    pcsx                     GetTextureTransColG32
2702     22412          7.0863  58.7779    pcsx                     anon (tgid:2622 range:0x1000000-0x2000000)

0x1000000-0x2000000 is the translation cache; memset could be coming from GPU plugin, but I don't really know.

For me it looks like normal operation, it happens only first time battle is loaded, but the whole thing goes for 5-10sec and causes varying slowdowns and sound tempo changes which people notice and keep complaining about. Maybe something like liveness analysis depth could be reduced just for that game, I don't know.

This is the same situation that caused slowness because of __clear_cache() calls, it would get stuck for several seconds, now it's a lot better but people still complain.

If it's only happening the first time, that seems like normal operation. Sure you could run it in the interpreter or compile with fewer optimizations but then it would be slow every time.

The liveness analysis is limited to three recursive calls, because I did find a few cases where that would take a long time. If that was the problem here, you'd see time being spent in unneeded_registers() instead of new_recompile_block() since this is one of the few functions that does not get inlined.

What's with all the calls to memset? If this is coming from alloc_reg(), it'd be better to do this with inline asm, since it's only writing 44 or 46 bytes. Also, lsn() can be called more than once per instruction, this could be optimized if it's a problem.

 

[notaz]

I have found one issue:

restore candidate: 800d3360 (211) d=1
invalidate_addr: 800d3544
INVALIDATE: 800d3000 (211)
restore candidate: 800d2e58 (210) d=0
restore candidate: 800d2e7c (210) d=0
restore candidate: 800d33e8 (211) d=1
invalidate_addr: 800d3544
INVALIDATE: 800d3000 (211)
restore candidate: 800d2d7c (210) d=0
restore candidate: 800d2e68 (210) d=0
restore candidate: 800d389c (211) d=1
invalidate_addr: 800d3544
INVALIDATE: 800d3000 (211)
restore candidate: 800d33c4 (211) d=1
invalidate_addr: 800d3544
INVALIDATE: 800d3000 (211)
restore candidate: 800d3274 (211) d=1
invalidate_addr: 800d3544
INVALIDATE: 800d3000 (211)
restore candidate: 800d2afc (210) d=0
restore candidate: 800d3854 (211) d=1
invalidate_addr: 800d3544
INVALIDATE: 800d3000 (211)
restore candidate: 800d38d4 (211) d=1
invalidate_addr: 800d3544
INVALIDATE: 800d3000 (211)
restore candidate: 800d3310 (211) d=1
invalidate_addr: 800d3544
INVALIDATE: 800d3000 (211)
restore candidate: 800d336c (211) d=1
invalidate_addr: 800d3544
INVALIDATE: 800d3000 (211)
- tick -
restore candidate: 800d3740 (211) d=1
invalidate_addr: 800d3544
INVALIDATE: 800d3000 (211)
restore candidate: 800d3390 (211) d=1
invalidate_addr: 800d3544
INVALIDATE: 800d3000 (211)
restore candidate: 800d2668 (210) d=0

This happens hundreds of times during that scene. Any suggestions how to avoid this?
I'll try to trace that memset later today.

Edit: thinking about it, it's probably not an issue as when everything is compiled it causes no slowdowns.
Edit2: That memset anomaly is my own fault:
http://notaz.gp2x.de/cgi-bin/gitweb.cgi?p=pcsx_rearmed.git;a=blob;f=libpcsxcore/new_dynarec/new_dynarec.c;h=641c673dcda5c1b98e31b8da869033e77ea9a19f;hb=HEAD#l8572

Silly me, did not realize I'm memseting 32K bytes for every compiled instruction!

 

[ari64]

This happens hundreds of times during that scene. Any suggestions how to avoid this?

This isn't necessarily a problem. The dynarec cache is invalidated by a write to this page (800d3000) however get_addr() later finds that the code is unmodified, so it marks the page as possibly valid. Another write then marks the page invalid again.

The real work occurs in clean_blocks, which is called from cc_interrupt every 2^26 cycles. This is approximately every two seconds in mupen64plus, but probably closer to four seconds using psx timings. It's possible that you get unlucky here and this gets called right after the block is marked.

I wouldn't worry too much about the calls to invalidate_block unless it's taking an excessive amount of cpu time. If it really becomes a problem, you could have get_addr only check for a match every tenth time, or whatever. Just don't do it too infrequently or you will accumulate a long list of dirty blocks, and spend a lot of time in verify_code.

It's a balance between write-protecting the page, and having to check for modifications. Finding the optimal strategy is a process of trial and error.

Edit2: That memset anomaly is my own fault:
http://notaz.gp2x.de/cgi-bin/gitweb.cgi?p=pcsx_rearmed.git;a=blob;f=libpcsxcore/new_dynarec/new_dynarec.c;h=641c673dcda5c1b98e31b8da869033e77ea9a19f;hb=HEAD#l8572

Silly me, did not realize I'm memseting 32K bytes for every compiled instruction!

You're lucky that gcc handled this badly, otherwise you probably wouldn't have seen it. GCC 4.4 on x86 inlines the memset, and unrolls the loop.

You should be able to remove pass 7 entirely. It figures out which registers are likely to be 32-bit. Since all your registers are 32-bit, you don't need it.

 

[larry]

A minor--minor!--visual glitch with PCSX-Rearmed:

In Spyro The Dragon (the first Spyro game), there are these updrafts that carry Spyro aloft and fly him to a new location. AFAICT they're rendered with billboards of little sparkles. In PCXS-Rearmed they aren't visible at all.

Sometimes this doesn't matter; on Dream Weaver's World, the updrafts are often anchored over cyan dishes in the landscape, so you know right where they are. But on Beast Maker's World, you have to hunt around for them. And on the cruel Tree Tops, the updrafts are out in space, which means if you miss them you plummet to your doom. You have to guess by ear (they make a noise) and use lots of save/restore.

In Spyro 2 updrafts are visible as as little lines going up, which I bet are polygons. I haven't played any Spyro 3 yet so I wouldn't know about that.

If you're interested in investigating this, how can I help? Screenshots? Save files? Is there something I should try?

 

[notaz]

You should be able to remove pass 7 entirely. It figures out which registers are likely to be 32-bit. Since all your registers are 32-bit, you don't need it.

Oh, that memset was in pass 3 provisional_32bit thing, which I removed (along with the call to provisional_32bit()) and the original issue became hardly noticeable now. I'll try to kill pass 7 too then.

If you're interested in investigating this, how can I help? Screenshots? Save files? Is there something I should try?

Yeah a savestate would help. Also try builtin GPU plugin (if not using it already) with frameskip turned off.

 

[larry]

If you're interested in investigating this, how can I help? Screenshots? Save files? Is there something I should try?

Yeah a savestate would help. Also try builtin GPU plugin (if not using it already) with frameskip turned off.

Here's a savestate for Spyro The Dragon (EU):
http://www.midwinter.com/~larry/lch.spyro.eu.savestate.zip

In the savestate, Spyro is near the end of the "Beast Makers Home" level, pointed towards a corner. If he walks forward, he'll walk into an updraft which will lift him up and glide him to the nearby raised platform. The updraft is invisible on PCSX-Rearmed but shouldn't be. You can see how it's supposed to look in this video:
http://www.dailymotion.com/video/x3tfo4_sypro-lp-beast-makers-home_videogames

Skip forward to 6m 10s or so (about 90% in). Those light-blue-and-white + and - signs floating up in a spiral in the corner are what's missing.

Also, I tried all three GPU plugins, with frameskip on and off, and none of them drew it. Not a good sign!

 

[gibberish]

instead of posting all your results here, drop me an email and get them on the compatibility list.
it takes the same amount of time and means that there is one place to go to find whether a game works or not, instead of trawling through threads.

https://spreadsheets.google.com/ccc?key=0AkBB6e4g1lGtdFVwODhsQzdlWVdrMTRKdDFXa1laZ2c&hl=en_GB#gid=0

 

[TitanUranus]

instead of posting all your results here, drop me an email and get them on the compatibility list.
it takes the same amount of time and means that there is one place to go to find whether a game works or not, instead of trawling through threads.

https://spreadsheets.google.com/ccc?key=0AkBB6e4g1lGtdFVwODhsQzdlWVdrMTRKdDFXa1laZ2c&hl=en_GB#gid=0

Isn't that spreadsheet for a different emulator (psx4pandora not Pcsx-Rearmed)?

 

[gibberish]

it doesnt really matter, you can put the emulator version in it.
i made it when i thought psx4pandora would be the main emulator, but the aim is the same - a list of the psx games that work on pandora and with what emulator version. 1 spreadsheet will suffice perfectly well for that don't you think?

 

[quartercast]

You want this one:
https://spreadsheets.google.com/ccc?key=0ArSWWAWRjErldHZVZlFxY0tBVnRRNXM5U3ZqWFNuN0E&hl=en#gid=0

But unfortunately it looks like it's been f*ed up by user editing

 

[gibberish]

ah fair enough, i wasn't aware of another one already made. i guess i'll remove the old psx4pandora one then.

 

[Chaser]

Why remove the old one? Its for a different emu and at one point Zodttd produced an update with the announcement he was going to post further ones with a new gui interface and more options etc. When ED said a better emu was coming but before notaz posted PCSX-reArmed.

That hasnt happened yet and I guess he may have dropped it in the wake of notaz work. There's still some games that work well in his for now aswell, that may still have issues here.

At the least the old data could be added to this one, as you suggested, denoting which emulator versions in use.

 

[Chaser]

You want this one:
https://spreadsheets.google.com/ccc?key=0ArSWWAWRjErldHZVZlFxY0tBVnRRNXM5U3ZqWFNuN0E&hl=en#gid=0

But unfortunately it looks like it's been f*ed up by user editing

Wow its totally screwed now. I know I introduced some errors when I did the bulk of it, as the copy pasted text from word, rather than direct from this thread caused different fonts and font sizes and HackModFord quickly stopped my already realised poor emu revision notes but now theres nothing there, no comments nada.
I've not used it since r5 and still haven't tested r6 yet. Oh well

edit: Incase its any use to repair or rebuild it fresh, here's an OpenOffice spreadsheet without formulas with 173 of the games and a txt doc with attributed contributions from everyone in this thread upto r5 release, as it turns out I'm still on r4. http://www.megaupload.com/?d=BIVKOILZ

 

[Coldbird]

Just wanted to state that Legend of Dragoon runs fine with OPP5 + 1000MHZ, complex cutscenes (ingame graphic in Hoax) slow down a bit, but run just nice when you drive the Pandora to its 1100MHZ limits.

 

[Schnatterplatsch]

Just played some Vagrant Story. Plays fine, better than I did expect it to run. Some times, especially in cut scenes, the framerate drops a bit (even to 1 FPS :lol: ). I think I will continue playing this anyway.

How are the speed improvements going, notaz?
A lot of games I tried were running at a good framerate, often it's just so little to really enjoyable fullspeed...

Anyway, thanks once more. I think I found something to play now B)