ASM, copy carry flag to APSR

[Thorium]

It's not hard to think of situations where self-modifying code would be a huge win performance-wise, especially on a cache coherent platform like x86. But you do probably see less and less of it these days because it's pretty taboo.

Intel advices against using it. Maybe i just dont know the advanced concepts for it. But classicly you would use it inside a loop to change the actual loop, or i am wrong with that?


If thats it than you screw around with the loop optimizer. Current gen Intel CPU's recognise loops and try to optimize them by translating the instructions into mirco ops and do some optimizations with them, like reordering them for maximum parallel execution. In Intels Optimization Reference Manual they actualy advise to not use unrolling on loops because the automatic optimizer will do it better. I did some tests with my Core i7 system and in some cases unrolling realy does not improve performance, but i had many cases it still did improve. I didnt hat a case in that reordering of instructions to break dependencys had any impact on performance. The Core i7 does a realy great job there.


Back to the problem: I am pretty sure the optimizer cant handle self modifieing code. I didnt do any tests for that but i guess it will detect it and just disable the optimization. Which could result in a big performance lose.

Like I said, it's not really a problem for dynamic recompilation. But if your ideas work for static recompilation, I would like to know more about them.

Well, looks like i should go with dynamic recompilation. I dont have a optimization in mind thats only possible on SR. My main concern about DR was the time optimizations need and the overhead they add to the performance. But it seems thats not as big of a issue as i thought, after i read what all you experienced people wrote.


And i thought it would save me some work. As the decoding of the x86 instructions looks complicated and actualy very tedious to implement because you have so many opcodes for one and the same instruction just for different registers and addressings.


On the other hand ARM encoding looks very well structured and fairly easy.


With SR i wouldnt need to care about if the decoder is fast it would be just a ASM text parser, very easy to implement.


I am still on the protyping so i play more around than do serious work. Looks like i switch to DR for the final implementation.

For instance Fallout. Here it's used in playing ingame videos (video decoding to be precise).


Eliminating self-modifying code is possible, I did it in Albion, but in Fallout it's much more complicated. Eliminating it might not be feasible.

Sounds like your 2 games that are running are fallout 1 and 2? That would be so awesome. I must say: I enjoy all you recompiled games a lot. Seems we have a similar taste for games.


Will your 2 new games come for the Wiz too? I dont think i will get my Pandora this year and dont have money for a upgrade.

I didn't give it a lot of thought initially in my project and later it turned out to be more complicated than I thought and adding support for it in the current project would be a lot of work.

Thanks for the advise. Can you tell me what exactly is the problem with multithreading. Isnt the Linux on the Pandora multithreading capable? I thought it's a full fledged Linux.

 

[M-HT]

It's not hard to think of situations where self-modifying code would be a huge win performance-wise, especially on a cache coherent platform like x86. But you do probably see less and less of it these days because it's pretty taboo.

Intel advices against using it. Maybe i just dont know the advanced concepts for it. But classicly you would use it inside a loop to change the actual loop, or i am wrong with that?


If thats it than you screw around with the loop optimizer. Current gen Intel CPU's recognise loops and try to optimize them by translating the instructions into mirco ops and do some optimizations with them, like reordering them for maximum parallel execution. In Intels Optimization Reference Manual they actualy advise to not use unrolling on loops because the automatic optimizer will do it better. I did some tests with my Core i7 system and in some cases unrolling realy does not improve performance, but i had many cases it still did improve. I didnt hat a case in that reordering of instructions to break dependencys had any impact on performance. The Core i7 does a realy great job there.

First of all, the games we are talking here about, i.e. games that are potentially playable on Pandora (speedwise) after recompilation were released around 15 years ago. For these games the target CPUs were somewhere in the range 486 - Pentium MMX and their inner working was really different from current CPUs, so the concerns about self-modifying code were different than they are now.


Back to using self-modifying code. You can either use it outside a loop to modify the loop based on some parameters or you can use it inside a loop. But even when it's used inside a loop, the loop is usually much larger than the pipeline of a CPU, so I'm not sure that the concern about micro ops, optimization and reordering really applies in this situation.

Well, looks like i should go with dynamic recompilation. I dont have a optimization in mind thats only possible on SR. My main concern about DR was the time optimizations need and the overhead they add to the performance. But it seems thats not as big of a issue as i thought, after i read what all you experienced people wrote.


And i thought it would save me some work. As the decoding of the x86 instructions looks complicated and actualy very tedious to implement because you have so many opcodes for one and the same instruction just for different registers and addressings.


On the other hand ARM encoding looks very well structured and fairly easy.


With SR i wouldnt need to care about if the decoder is fast it would be just a ASM text parser, very easy to implement.

This looks like you want(ed) to use a text output of some x86 disassembler and then process it further. If so, then which disassembler do you use and how can you be sure that the output is correct? Because disassemblers use heuristics to distinguish code and data, especially on executables without relocation data.

Sounds like your 2 games that are running are fallout 1 and 2? That would be so awesome. I must say: I enjoy all you recompiled games a lot. Seems we have a similar taste for games.

That's correct.

Will your 2 new games come for the Wiz too? I dont think i will get my Pandora this year and dont have money for a upgrade.

No. First reason is, that the Wiz is not fast enough to run the games. The second reason is, that I'm targeting at least ARMv7 processors and the Wiz has ARMv5 processor.


This also good question for you. Which ARM architecture do you want to target? You might also want to look at Thumb-2 instruction set.

I didn't give it a lot of thought initially in my project and later it turned out to be more complicated than I thought and adding support for it in the current project would be a lot of work.

Thanks for the advise. Can you tell me what exactly is the problem with multithreading. Isnt the Linux on the Pandora multithreading capable? I thought it's a full fledged Linux.

This is an area where static recompilation has an advantage over dynamic recompilation.


The problem is how exactly do you plan on doing it. Do you want to map one 'guest' thread to one 'host' thread, or do you want to map all 'guest' threads to one 'host' thread or something in between? If you want to use more 'host' threads, how do you want to share and synchronize data needed for dynamic recompilation between threads (translated code cache, ...) - do you want to have one shared copy of the data or does each thread have a copy?

 

[Thorium]

This looks like you want(ed) to use a text output of some x86 disassembler and then process it further. If so, then which disassembler do you use and how can you be sure that the output is correct? Because disassemblers use heuristics to distinguish code and data, especially on executables without relocation data.

Yes, i wanted to use OllyDbg. And if needed get some data directly form the exe. I am familiar with the PE format.

Will your 2 new games come for the Wiz too? I dont think i will get my Pandora this year and dont have money for a upgrade.

No. First reason is, that the Wiz is not fast enough to run the games. The second reason is, that I'm targeting at least ARMv7 processors and the Wiz has ARMv5 processor.


This also good question for you. Which ARM architecture do you want to target? You might also want to look at Thumb-2 instruction set.

I want to use ARMv7, i saw some instructions that are very usefull on ARMv7 and Exophase mentioned also that there are usefull ARMv7 instructions.


I didnt looked into Thumb-2, the manual said it performs not as good as ARM, so i didnt looked at it any further.

This is an area where static recompilation has an advantage over dynamic recompilation.


The problem is how exactly do you plan on doing it. Do you want to map one 'guest' thread to one 'host' thread, or do you want to map all 'guest' threads to one 'host' thread or something in between? If you want to use more 'host' threads, how do you want to share and synchronize data needed for dynamic recompilation between threads (translated code cache, ...) - do you want to have one shared copy of the data or does each thread have a copy?

Ah, now i get it.


Didnt saw a big problem because i dont thougt much about the dynamic recompiler.

 

[M-HT]

This looks like you want(ed) to use a text output of some x86 disassembler and then process it further. If so, then which disassembler do you use and how can you be sure that the output is correct? Because disassemblers use heuristics to distinguish code and data, especially on executables without relocation data.

Yes, i wanted to use OllyDbg. And if needed get some data directly form the exe. I am familiar with the PE format.

I never used this one.


Originally I too wanted to use an output from disassembler for my static recompilations, but because of disassembling errors I wrote my own disassembler using the udis86 library.

I want to use ARMv7, i saw some instructions that are very usefull on ARMv7 and Exophase mentioned also that there are usefull ARMv7 instructions.


I didnt looked into Thumb-2, the manual said it performs not as good as ARM, so i didnt looked at it any further.

Weird. Because from what I read it was created to perform better than arm instruction set because of better code density. In my project I created dynamic recompiler backends for both arm and thumb-2 instruction sets and the thumb-2 backend does perform better than the arm backend (not much, but still better, and only when I used "-mthumb-interwork" compile parameter and the proper instructions to use it).


Of course thumb-2 instructions are not the same as arm instructions and some of them don't have the nondestructive properties of their arm counterparts, so depending on your recompiler, you might not be able to use the thumb-2 instruction set at all.

 

[Thorium]

This looks like you want(ed) to use a text output of some x86 disassembler and then process it further. If so, then which disassembler do you use and how can you be sure that the output is correct? Because disassemblers use heuristics to distinguish code and data, especially on executables without relocation data.

Yes, i wanted to use OllyDbg. And if needed get some data directly form the exe. I am familiar with the PE format.

I never used this one.

It's a realy good user mode debugger for windows. I use it all the time. Did some inofficial game patches with it.


Using it as a disassembler should be optimal if i want to have text input. Because ist creates the process for the exe and loads it befor it disassembles it. Also you can just run the game with it and it will detect if there is code in a data section, etc. And you can do some manual analyses on the code.


I prefer OlyDbg over a kernel mode debugger like softice, because it's much easier to use.


http://www.ollydbg.de/

I want to use ARMv7, i saw some instructions that are very usefull on ARMv7 and Exophase mentioned also that there are usefull ARMv7 instructions.


I didnt looked into Thumb-2, the manual said it performs not as good as ARM, so i didnt looked at it any further.

Weird. Because from what I read it was created to perform better than arm instruction set because of better code density. In my project I created dynamic recompiler backends for both arm and thumb-2 instruction sets and the thumb-2 backend does perform better than the arm backend (not much, but still better, and only when I used "-mthumb-interwork" compile parameter and the proper instructions to use it).


Of course thumb-2 instructions are not the same as arm instructions and some of them don't have the nondestructive properties of their arm counterparts, so depending on your recompiler, you might not be able to use the thumb-2 instruction set at all.

ARM® Architecture Reference Manual ARM®v7-A and ARM®v7-R edition


The Thumb instruction set was developed as a 16-bit instruction set with a subset of the functionality of the


ARM instruction set. It provides significantly improved code density, at a cost of some reduction in


performance. A processor executing Thumb instructions can change to executing ARM instructions for


performance critical segments, in particular for handling interrupts.


In ARMv6T2, Thumb-2 technology is introduced. This technology makes it possible to extend the original


Thumb instruction set with many 32-bit instructions. The range of 32-bit Thumb instructions included in


ARMv6T2 permits Thumb code to achieve performance similar to ARM code, with code density better than


that of earlier Thumb code.

"ARMv6T2 permits Thumb code to achieve performance similar to ARM code"


I interpreted that as: It can be fast as ARM, but in some cases it's slower.