GP2X Gba Emu And Static Translation

[TKF15H]

lol, no problem. Thanks for the offer anyway.

 

[DaveC]

which way did you choose, the ARM -> C -> ARM way ?

Yup. Which means you'll need GCC to compile games. Or you could get someone else to do it for you, provided you both have the exact same ROM.

Maybe I am dense but why translate ARM instructions to "c" and then run that on an ARM (ARM -> C -> ARM) ? Why emulate a CPU when it IS the CPU? I just don't get it, isn't that alot of wasted cycles? Why not run the ARM code natively wouldn't that be much faster you know ARM -> ARM?

The way you are doing it is like a Chinese person trying to communicate with another Chinese person, only you are translating to English from the first guy, then back to Chinese for the other when the two could just speak to each other in Chinese and eliminate alot of work. What am I missing here?

 

[dockthepod]

Yeah, I was wondering that as well. Is the GBA ARM (ARM7?) not compatible with the GP2X one (ARM9?)?

Also, how I read it... you translate to C as you actually play the game initially? Or did I read that wrong? Because... that makes no sense to me. I don't see how every code path could get exercised. You must be loading up the CODE portion of the rom and decompiling it.

 

[BradN]

Well, the problem is that you could run the code directly, except...

- Need to be able to catch memory accesses intended for hardware
- Need to be able to position RAM whereever necessary (linux may reserve ranges)
- Makes accurate cycle counting impossible (which may or may not be a big deal)

So to fix the first two problems, you need to use the MMU, so you've got to either modify linux, or go to an HH style program (which I don't believe is practical yet).

Then, once you've got the base environment to run the code set up, you still need to emulate the video/sound hardware. It would probably be a pain to modify VBA to run under HH mode, and I have no idea how it'd be linked with a modified linux to give it the control it needs.

So, I don't think the strategy he's using is that bad really - it preserves compatability so a custom kernel isn't needed, should require minimal other modifications to the VBA source, and hopefully will run fast enough to be worthwhile.

The only thing I'd do differently is skipping the C step and doing an ARM -> ARM dynamic recompilation, or possibly gank some code from qemu, as they already have most of the necessary stuff made. But, qemu actually uses gcc to generate the code fragments for the different opcodes anyway, and qemu can achieve reasonable speeds (1/10 to 1/5 speed), plus keep in mind the GBA cartridge interface is somewhat crippled, and there's probably less if any cache (I think the RAM even runs with 2 waitstates by default, cartridges are worse) - so this means there's a little less processing ability to emulate.

The one problem I see with static recompilation, is it's gonna be difficult to make it work right for code that's loaded into RAM and then executed. I'm thinking a lot of game specific hacks might be needed that way.

Edit: Just looked up the GBA specs, and I guess it does have 32KB cache, so it might be slightly more powerful than I thought, but still well within emulatability.

 

[dwelch]

Yeah, I was wondering that as well. Is the GBA ARM (ARM7?) not compatible with the GP2X one (ARM9?)?

Also, how I read it... you translate to C as you actually play the game initially? Or did I read that wrong? Because... that makes no sense to me. I don't see how every code path could get exercised. You must be loading up the CODE portion of the rom and decompiling it.

Do a search for static recompilers. It is/was a yahoo groups thing. Graham Toal did a very nice writeup that summed up what a number of us were doing at the time on the subject.

The ARM in ARM mode is beautiful, one 32 bit word per instruction, no more, no less, easy to disassemble, easy to convert the disassembly to C.

Take any variable length ISA (instruction set architecture) and try to do that. You will find that you cannot, there is always going to be data embedded in the code, that data can fool you into thinking you have found another branch instruction which leads you to an address that you think has an opcode. You can only get so far with a simple disassembly approach (actually you get nowhere if you simply start at one end and disassemble).

Here is one thing I did on the 6502. Start with the reset vector, this gives you an address. Add that address to your list of addresses that contain opcodes. Make a pass on the rom dissassembling from every address in your list and stopping each instruction chain when it hits an unconditional branch. As you go through each instruction sequence make a note of all conditional branch destinations as valid opcodes. Repeat this over and over again until it cannot find any new branch destinations. What I found, at least with Asteroids, is that I would have an address in the list that pointed to a three byte instruction, but some other address in the list pointed to the the second or third byte in that instruction, which is impossible. It made sense once I figured out one/some of the sources of the bad addresses. I dont remember exactly what it was now, but if for example the compiler and or assembly programmer chose to preced a conditional branch with an instruction that would guarantee the branch, thus making the instruction pair an unconditional branch, then follow that with some data that when disassembled gets you out of sequence with the instructions that follow it, it goes down hill from there. If you are lucky you may end up with every single byte in the rom as an opcode even if the processor has no single byte instructions.

Eventually you figure out that disassembly (of a variable length ISA) cannot reach every execution path and have to search for another solution...Hmm, like executing in an emulator and making a note of every execution path.

You dont have to play test in the emulator to build an opcode list, but it will save you a considerable amount of time. I think you have to build your translation or test translations such that instruction sequences that dont end in a branch fall into a handler that halts execution and displays information about where the halt occured. This is actually pretty simple, once you have your opcode list, then disasseble the rom from the beginning to end. Any bytes that are not part of an instruction generate code that calls a handler passing in the rom address of that byte. Then you can manually examine the disassembly near and at that byte (had it been considered an instruction) and determine what to do.

On a variable instruction length architecture I argue you have do this process for every individual rom.

Anyway with the ARM, so long as you know there is no thumb in there, it is super easy.

Now here is a problem, we are talking about a platform (GBA) dominated by thumb mode. Thumb mode cannot stand on its own it will periodically have to switch to arm mode to do some things then back to thumb, nothing you can do about that. So even on this translation, you are dealing with a variable length ISA. So back to plan, A, play test the emulator some, capture some addresses, disassemble and translate some, play the translation some, back and forth.

Another thing here is that you have to go through and hand type C code for hundreds of opcodes (depending on the ISA), this is boring and error prone. Which means your translation is buggy. There are variations on this theme, but basically you have to run the emulator and translation along the same instructions and compare something, register contents and/or memory reads and writes, and/or condition registers, etc. When you get a mismatch you have to sort it out, fix the translation, start again. I chose to do it by running the emulator and logging all of this information to a file, Then running the translator and comparing the emulation output to the translated output. The solution I liked but didnt try myself was to run the emulation and translation at the same time in essense and compare cycle for cycle, this allows you to run infinitely. My solution started to fall apart as the files got into the gigabytes (Well I assume the magic 2gb to be exact).

I know at least during a 4kbyte gba coding competition many were trying to take advantage of the built in compression in the gba. Their program would start by decompressing the rest of the program into ram then branching to the decompressed program in ram. Try and disassemble that. Even playtesting that in an emulator will knock your socks off at first, wait the rom is here and this is executing there, what is going on!

Also it was not uncommon, at least in the developer groups. to talk about moving some code into the fast ram for performance, the heavily used or code that needed to execute fast. A pure disassembly of this code when it is sitting in the rom as data, before it is copied to ram to be executed, will disassemble wrong.

Any of the programs that are multiplayer require that the client version at least runs strictly out of on board ram. This entire program as it were would live somewhere in the rom at the wrong address, this will disasseble wrong, only executing in an emulator will sort this out (actually you might never sort this one out as you might have to simulate the master to slave transfer to figure out what gets transferred and where).

This project, at least from a translation standpoint, just keeps sounding harder and harder.

 

[dwelch]

Edit: Just looked up the GBA specs, and I guess it does have 32KB cache, so it might be slightly more powerful than I thought, but still well within emulatability.

There were a number of hardwareisms to try to compensate for the drastic crippling of the arm on that platform. The roms actually had a pipeline fetch buffer thing, not sure if it was rom or mio. Basically if you fetched instructions in address order the first one took more than one clock but then you could read sequential addresses one clock at a time, something like that. I think I figured out that running from rom was actually the fastest way to go. http://www.dwelch.com/gba/dhry.htm

The more I think about how incredibly difficult a GBA translator would be, I think taking an emulator and modifying the emulator to generate the C code while executing it is an ideal solution, you know you are only going to get valid instructions, in the right mode (arm/thumb) at their proper execution address, and not necessarily as data before being copied to ram to run. The only alternative is to have the emulator sort out the instruction sequences, leaving them as raw arm/thumb instructions but preserving their as executed addresses. From this you create a smart image of your rom that is ready for translation.

If I am not careful I may end up going back and translating Asteroids for a fourth time, with all of this translator talk...

 

[kajuberdut]

Ah, dwelch, dont say that. I look forward to this second only to PSX4All. I really would love to be able to play golden sun, kingdom hearts, FFtactics advanced, tactics ogre, megaman zero.... so many good GBA games! please keep up the great work!

 

[hannanshah]

You sound very confident about getting GBA emulation working well. Much better than the usual pessimism around these forums! If you ever need any motivation then I guess you could think about all the donations that will come piling in once (if!) you can give us a nice release. Good luck mate.

 

[TKF15H]

The way you are doing it is like a Chinese person trying to communicate with another Chinese person, only you are translating to English from the first guy, then back to Chinese for the other when the two could just speak to each other in Chinese and eliminate alot of work. What am I missing here?

Except one is a man, the other a woman. Despite the fact that they speak the same language, they'll never understand each other without the help of a counseler, so they went out looking for the best they could afford, and it happend to be an englishman.

Letting the GP2X attempt to execute the GBA's code is possible, but requires more work than I can afford. So much of the GBA code will have to be changed (all SWIs, instructions that access RAM, and branches) that you might as well use a dynarec or a static recompiler.

Therefore I could have written a dynarec and not have to bother with needing GCC in the first place. But due to limited output-code optimization I thought that the speed boost might not be enough. Rather than go through all the work just to find out if it would be a good idea or not, I went for option 3, a static recompiler with a GCC back-end. This optimizes the code far more than I can, and is only done once. Dynarecs typically flush their code cache when the emulation stops. Here, you can't afford to flush the cache.

Also, how I read it... you translate to C as you actually play the game initially? Or did I read that wrong? Because... that makes no sense to me. I don't see how every code path could get exercised. You must be loading up the CODE portion of the rom and decompiling it.

There is no code portion. As said previously, the code may be compressed, so the only way to get to it is on run-time (unless you manually decompress ).

So to fix the first two problems, you need to use the MMU, so you've got to either modify linux, or go to an HH style program (which I don't believe is practical yet).

Or Squidge's MMU hack could be used to control that, and there is space for that (rather complicated?) optimization in the future.

 

[dockthepod]

Very interesting guys. Thanks for the long explanations. I'm a C++ guy, so this machine code stuff is pretty far removed from my daily experience. I probably should learn a little ASM at some point here.

 

[DavidBeoulve]

I remain happy that TKFISH has taken up the challenge.

 

[TKF15H]

Daily Update:
Stopping for now on opcode "LDR Rd, [Rn, Rm, LSR #]", line 4537.
Roughly 60% done. My right wrist is hurting from all this typing, I'm gonna go watch a movie!

 

[xinfernoofdantex]

This seems to be coming along pretty fast, good luck with everything, I can't wait for the results

 

[A_SN]

Daily Update:
Stopping for now on opcode "LDR Rd, [Rn, Rm, LSR #]", line 4537.
Roughly 60% done. My right wrist is hurting from all this typing, I'm gonna go watch a movie!

Coolness! I'm sure I express many people's feeling when I say that the hard work you're doing is very appreciated anyways what comes after you're done with 100% of the opcodes?

 

[TKF15H]

Write a bit of code that manages what has been executed before and what not, so that the emulator won't output the same opcode more than once. I'll then probably have quite a bit of debugging to do. Then make a jump table writer, which is used for finding the translated point to jump to based on the GBA's instruction pointer. At that point I'll have something that's executable, though its speed will actually be slower than the current emulator. I'll have to make it call previously compiled code, then probably do more debugging. Depending on the speed I get, a release would be made available at this stage. Otherwise, back to the old drawing board for some optimizations.

 

[xinfernoofdantex]

has anyone tried implementing the MMU hack into one of the current GBA emu's?

 

[A_SN]

has anyone tried implementing the MMU hack into one of the current GBA emu's?

No, Squidge himself told me that implementing the MMU hack into VBA would only help by 1 or 2 fps, so it's definitly not worth it.

 

[Shikaku]

has anyone tried implementing the MMU hack into one of the current GBA emu's?

No, Squidge himself told me that implementing the MMU hack into VBA would only help by 1 or 2 fps, so it's definitly not worth it.

I'd say just do it for that, any speed increase is good in an emulator.

However don't make it a priority on getting it to work if it's hard to set up...

 

[mahavailo7]

way to go man! Now, I'm no programer (not good at dedicating a lot of time to one thing, particularly not a monotinous string of code), but this is looking like it's coming along.

I read through most of the thread (couldn't understand a lot of the code talk... buncha robot people you guys are) and saw something about tkfish only having to do like 75% of his coding to get 100% gba emulation. I read this, then think of gba on the PSP, and get very excited. Now I go onto read what has recently been posted and read about the steps tkfish must take after completing his code. So I figure, if he ends up making a decent gba emulator, then works on it after he's done, as to polish it; things are looking good for gba on the gp2x.

way to go man

 

[TKF15H]

The MMU hack is good for emulators that use hardware blitting. IBA is currently using software blitting as I'm concentrating on another bottleneck: The CPU. After that I'll change to hardware, with the MMU hack.

something about tkfish only having to do like 75% of his coding to get 100% gba emulation.

Oi, keep in mind a lot of this is in the "future hopefull tense" as that 75% should be enough but you never know.
For example, some games may rely on background transparency / rotating a lot, which is pretty hard to do quickly on the GP2X, so they will be pretty hard to run fullspeed. Quite a few other games don't need that, so they should be much easier on the emulator.