8086 (ibm Pc) Static Recompiler For Gp2x

[___]

Here is some suggestions that should be compatible:
Crystal Caves: Awesome platformer.
Framed: Pretty weird platformer with a nice story and some adventure elements.
DukeNukem 1 and 2 (may not be possible, VGA graphics?)

Apogee platformers as they were the shiznit.

 

[YakumoFuji]

You brought up some good points YakumoFuji.
Just to clarify, it won't matter what they're coded in, since we're dealing with the compiled code.
What is tough, is this dynamic overlay stuff. Hilary understands it may more than I do. Could someone point me at a good resource or help describe dynamic overlays to me? In the meantime maybe Wiki will suffice.
The compression of games makes it extremely difficult to do something general, but since we're going by a game-by-game template process, it might be possible to decompress them first.

its what does apps did before DLL's came along. they loaded, some/all/parts of external executables into memory, relocated/linked and called them and unloaded them.

As for using DOSBox and giving it a static recompiler. Since the static recompiler processes the 8086 executable and outputs C code which is compiled, you pretty much lose the need for DOSBox. Having the 8086 executable run as a native ARM binary pretty much eliminates the need for "emulation". Though I'm not quite sure how DOSBox handles things. If I'm wrong on this, please let me know.

you static recompile chunks into a buffer, execute chunks, etc. when the recompiler hits things like hardware register mods and screen addresses, interrupts, doscalls, dos low mem manipulation, you let dosbox handle that then you return to static recompiled code.

What I'd like to know is, how much harder is it to make a 286/386/486 static recompiler over 8086? By the sound of things from YakumoFuji, we might have a harder time with 8086 games over the later processor arch's? I'm still new to the interworking of Intel processors, and going back to pre-486, things get fuzzy.

its a lot harder, as you suddenly have more memory addressing modes (paged, segmented, flat, voodoo), access control (read/write/execute), protection levels (with x86 0,1,2,3), floating point included (som 386 had it, some didnt same with 486slc , etc etc etc)... lets throw in mmx, 3dnow, sse1, sse2 etc, a20 for protected mode.. mapping hi+low irq's (0-8, 70-78)... call gates, task gates, amd fastcall, intel fastcall...

lets not forget sound chips, gfx chips etc, bios calls, undocumented dos memory poking. single step interrupts, etc

Time to deal with self modifying code...

when dealing with self modding code, dont forget prefetch code rules on x86 changed from pre pentium games to pentium=+ games...

I would merge the recompiler into dosbox and have it exec say chunks of up to 16 instructions or whatnot? and cache the recompiled chunks..

the fact you have something working is impressive already.....

 

[zodttd]

YakumoFuji, thanks! Extremely informative!

I personally like the idea of merging this with DOSBox. I don't know how it would work with HilaryCheng's static recompiler though. See there's no translation cache, as nothings being translated to even be cached. I think you may be confused with this being a dynamic recompiler, where a code block is translated, pushed to a buffer/cache, and executed. A dynamic recompiler for DOSBox is an awesome idea. Though what we have now is a static recompiler, where the entire executable is translated at compile time, not runtime.

From what I've been told, handling self modifying code with static recompilation is "impossible". But I'm looking into if it can be handled in this case.

Worst case scenario would be that static recompilation isn't possible/feasible. But then again if that is the case, maybe a dynamic recompiler for DOSBox is possible? We have options.

 

[BradN]

As far as a specific strategy for doing dynamic recompilation... I think typically you start recompiling at what ever instruction you're on, and finish when you reach an unconditional jump, then cache that chunk as what to execute when something jumps to the instruction you were first at. If some other code (or part of the code itself) jumps to somewhere in the middle of that chunk, it might be easiest just to treat it as a fully separate instruction string rather than trying to make every instruction a valid entry point, or guessing where entry points might be.

The nice thing, is that on the gp2x, there's a lot of memory to play with that might allow more suitable caching approaches, as well as keeping a full image of the target system memory. If you want to get fancy, you could use the MMU to help detect memory writes that invalidate cached code. Not sure if this is possible running within linux without extra hacks, but if you can set a memory page as read only and catch faults and determine the circumstances, you're set.

 

[zodttd]

Just to make sure everyone knows, HilaryCheng wrote a *static* recompiler. A static recompiler is quite different from a dynamic recompiler. It's normally faster too.

I have some expierence with dynamic recompilers. I wrote a large portion of the ARM dynamic recompiler for psx4all. And I'm sorting out some issues with the MIPS dynamic recompiler. I'm confident with HilaryCheng's expertise, a dynamic recompiler is possible.

But right now we already have working code for a static recompiler. I just want to make sure that the self modifying code in RomanceOTK1 doesn't make things "impossible", as some have said to me.

The reason I was talking about dynamic recompilation is the idea brought up for doing a DOSBox dynamic recompiler if this project isn't possible to get fully working.

In the meantime, it's VERY cool seeing ROTK1 (Romance Of The Three Kingdoms 1) on the GP2X. It's running further into in-game than before now. Great work Hilary!

 

[BradN]

I have a feeling that self modifying code (except loading overlays) will only be used in a few corner cases that you should be able to patch over. With overlays, it might be a little trickier, but should still be manageable if you can dump the contents and figure out how they link together.

 

[hilarycheng]

I have a feeling that self modifying code (except loading overlays) will only be used in a few corner cases that you should be able to patch over. With overlays, it might be a little trickier, but should still be manageable if you can dump the contents and figure out how they link together.

Yes, it is right, but finding out these self-modifying code is time consuming. , overlay too.

I doubt the gold box games will convert. They are pascal and use dynamic overlays and such, its not a simple .exe...

sounds a bit like a nightmare. lots of games are compressed + packed like Rose Hackstop, protect! 6, etc..

i would rather see some of tis static recomp embedded into dosbox and make it more general...

then you have stuff hitting the hardware registers..

now if you could show me "copper" by Surprise!Productions, then i'd be REALLY impressed

well, yes, lots of games are using protected scheme to encrypt their code. I agreed!

But for the goldbox game, heehee, it should be convertable. I have used IDAPro to disassemble. well, it is using Borland Pascal Overlay which is a standard in Borland. It means someone has been already find out the overlay structure. IMO, I think it is possible.

Hardware Registers is not worst. Since X86 is using lots of IN/OUT Instruction. It is not memory-mapped except 80286 or 80386 or upper. so, it is very easily to deal with it

You brought up some good points YakumoFuji.
Just to clarify, it won't matter what they're coded in, since we're dealing with the compiled code.
What is tough, is this dynamic overlay stuff. Hilary understands it may more than I do. Could someone point me at a good resource or help describe dynamic overlays to me? In the meantime maybe Wiki will suffice.
The compression of games makes it extremely difficult to do something general, but since we're going by a game-by-game template process, it might be possible to decompress them first.

As for using DOSBox and giving it a static recompiler. Since the static recompiler processes the 8086 executable and outputs C code which is compiled, you pretty much lose the need for DOSBox. Having the 8086 executable run as a native ARM binary pretty much eliminates the need for "emulation". Though I'm not quite sure how DOSBox handles things. If I'm wrong on this, please let me know.

What I'd like to know is, how much harder is it to make a 286/386/486 static recompiler over 8086? By the sound of things from YakumoFuji, we might have a harder time with 8086 games over the later processor arch's? I'm still new to the interworking of Intel processors, and going back to pre-486, things get fuzzy.

Time to deal with self modifying code...

For the compression, except the game company buy a code fragment from PKware or elsewhere and intergrate into their code. Otherwise, it get some util to decompress it. I have tried RTK2 and Pool of Radiance. It can be decompressable. but POR need to find our the right entry point.

For DOSBox, they should have dynrec. But no one know the state. Also, Default DOSBOX are using normal interpeter mode. You should configure yourself.

Well, I think the first target this prototype is 8086/8088 CPU. Since 286, 386, 486 got protect mode. the Memory Managment Unit is extremely horrible. We need to handle lots of field.

 

[Yono]

Hrmmm... 8086 suggestions... ?

The Ancient Art of War

[edit] Actually, just tried to play it - it may have too many keyboard-centric controls to be suitable, depending on exactly how you create your 'templates'. [/edit]
[edit2] Plays great on DOSbox (PC) - perhaps I should try it on the GP2X equivalent... [/edit2]

http://wiki.gp2x.org/wiki/DosBox:Ancient_Art_of_War if you want the config files.

Here is some suggestions that should be compatible:
...
DukeNukem 1 and 2 (may not be possible, VGA graphics?)

It runs for me. Config: http://wiki.gp2x.org/wiki/DosBoxuke_Nukem

Welcome to the community Hilary!

 

[hilarycheng]

One more things, for this project, my aim is not just recompile those game and stop. Today, dynrec or interperter emulation is non-power friendly which used up lots of battery. You cannot play for long time. If you wanna to play longer, you need to buy/wait for a new machine. But it is non-possible.

A Static Recompiler (SREC) can generate a brunch of C Code. Once it is working fine. You can replace part of code to *FULLY NATIVE* code. e.g. Waiting for Key. When I programming for 8086, I need to run a loop to wait for a key. or call a interrupt to wait. In RTK1, it will check for key, flashing cursor (most of the time), and nth to do. We can replace these routine to hardware specific and that is doing much faster. Although not every game possible to do it, you can have a chance to finding out from SREC

 

[zodttd]

Hilary: I have some suggestions on improving speed even further. They're pretty basic stuff to start with. I don't think performance will be an issue for the initial release. It's already running pretty nicely in it's current state.

X86 is still weird to me. But I'm getting the hang of it, so more optimization idea's will come to me after these.

I too believe that 8086 is just the start. It's a good first step into possibly extending to more robust X86 processors. And the fact that static recompilation is even faster than a dynarec, it's the best solution for the GP2X.

 

[hilarycheng]

zodttd: thx for your great work. I have tried your GP2X version, wow!!! Speed is acceptable for 240MHz.

For other guy who is familiar with 8086 and IBM PC Programming (Low Level One) and Interested in this project, just let me know. I can send you some "uncommented", "non-organized" code for you to have fun. HeeHee.

 

[zodttd]

Yeah, I'm sure there's some others out there that could help with this project. I must say the code is very well written, so it's really not hard to jump right in.

Hilary: Performance has improved since that build for the GP2X version.

 

[YakumoFuji]

i have plenty of low level x86 cough. hm. just no gp2x (still saving up for one)...

i'm quite familiar with packing/unpacking old dos execs.. i used to crack + train them and other things we wont write about.

pool of rad isnt compressed (my original isnt). its standard turbo pascal large memory model exe with overlay (game.ovr). its entry point is right at the front of the code (next to the hidden codeword 'STING' for loading without splashscreen).

for things that are really simple, I think you will be ok, but for more complex things i think you will have big problems.

I hiope you continue to make it look easy and have lots of success.

if i have time i might help out (in between writing my own gp2x game, working on scummvm and lots of other things like a wife + house etc etc etc)

 

[zodttd]

What if you got a GP2X sent to you for working on this project? Bribe him!

 

[hilarycheng]

Fuji, I don't think you need to work it on the GP2X. Just tried to work on a win32 version first. Since I will work the 8086 Instruction and Decompiling Procedure. The Simple Job is just keep tracking the POR Execution Flow to make sure that it works correctly. Porting Job will leave for others to do it like zodttd helping on some gp2x issue.

i have plenty of low level x86 cough. hm. just no gp2x (still saving up for one)...

i'm quite familiar with packing/unpacking old dos execs.. i used to crack + train them and other things we wont write about.

pool of rad isnt compressed (my original isnt). its standard turbo pascal large memory model exe with overlay (game.ovr). its entry point is right at the front of the code (next to the hidden codeword 'STING' for loading without splashscreen).

for things that are really simple, I think you will be ok, but for more complex things i think you will have big problems.

I hiope you continue to make it look easy and have lots of success.

if i have time i might help out (in between writing my own gp2x game, working on scummvm and lots of other things like a wife + house etc etc etc)

 

[hilarycheng]

I have finally resovled a strange bug (careless coding). I have sent to zodttd for the update. It can run until in-game menu and can do some basic operation like harvest wood. but it cannot have war with other countries yet.

 

[zodttd]

Very cool!
Time to update!

 

[dsd28]

Ooo, this sounds awesome. I'd love to see the AD&D gold box games and M&M2 running! (I know they are available on other platforms, no need to state so...)

 

[hilarycheng]

Ooo, this sounds awesome. I'd love to see the AD&D gold box games and M&M2 running! (I know they are available on other platforms, no need to state so...)

me2. I love these game

 

[sand_man]

Ooo, this sounds awesome. I'd love to see the AD&D gold box games and M&M2 running! (I know they are available on other platforms, no need to state so...)

Yeah but PC versions are better