Share Common Snippets?

[JayFoxRox]

I was wondering wether it would be good or not to share common code like:

- Shader code recompiler
- CPU code recompiler
- DSP code recompiler

or other similar things which are important for alot of projects, reaching from emulators over games to more complex applications.
Is it really necessary that every developer writes their own recompiler? I don't think that will help any of us.
The contra is that I m very picky about my code myself and even if there is an existing solution I write it on my own . Another problem is that my code is usually formatted and styled in a very "loose" way.
Possibly we could even write all the code in some pseudo style which is converted into real code with variable names and syntax (to support more than 1 programming language) defined by the developer.
However, maybe we are able to write some code together to make our lifes a bit easier?

PS: Ofcourse, we could just "steal" each others code from the others open-source application, but I don't want to go open-source until my code is cleaned up and I assume that others think the same way. Also having an existing open-source implementation doesn't mean that everybody is going to use it. If there was a special place (and not an existing projects source directory) with the important snippets more developers would use it probably.

 

[lulzfish]

I'd be interested in checking this out just to know how it works. Recompiling seems very cryptic to me at the moment.

But as for actual code re-use, I dunno. I was looking for a type of Qt widget last night, and there seemed to be an open-source project that offered it, but it looked pretty outdated, so I think I'll just re-do it myself to keep it current.

 

[sindbad]

Sounds like a good idea. How does LLVM sound as a base? It already handles CPUs and some shaders (look at gallium3d).

 

[JayFoxRox]

I was thinking about a very lightweight implementation of functions for recompilers like:
CODE
armRecomiler->nop();
armRecomiler->mov(ARM_REGISTER_PC, ARM_REGISTER_LR);
armRecompiler->run();

which would write:
CODE
nop
mov pc,lr

to the memory and then execute it. I m not sure what LLVM does, but it sounds a bit too much.
I have already started to look at the OpenGL ES 2.0 binary shaders and old ARB Assembler code to see if I can write a dynarec for that. That would be pretty handy as it would allow it to rewrite the fixed function pipeline in a much easier way. I had to fight one problem already though: the PVR SDK emulator doesn't allow binary shaders so I have to re-assemble them to GLSL.

 

[hlidegp2x]

I was thinking about a very lightweight implementation of functions for recompilers like:
CODE
armRecomiler->nop();
armRecomiler->mov(ARM_REGISTER_PC, ARM_REGISTER_LR);
armRecompiler->run();

which would write:
CODE
nop
mov pc,lr

to the memory and then execute it. I m not sure what LLVM does, but it sounds a bit too much.
I have already started to look at the OpenGL ES 2.0 binary shaders and old ARB Assembler code to see if I can write a dynarec for that. That would be pretty handy as it would allow it to rewrite the fixed function pipeline in a much easier way. I had to fight one problem already though: the PVR SDK emulator doesn't allow binary shaders so I have to re-assemble them to GLSL.

I have some future projects for an arm (re)compiler with the ability to optimize or reorder some instructions when emitting arm dynarec code. Not sure if it must be independent of a decompiler (well mostly for me a MIPS to ARM).

 

[Exophase]

You want a common source-driven ARM assembler, not a recompiler. At least one has actually already been written - I'm sure you can do something more friendly, but using C++ for such a basic thing is going to piss off people who want to use it but are using plain C.

Writing this is actually much easier than the difficult parts of a recompiler (about as hard as writing a disassembler, which most emulators have), although ARM has a decent amount of stuff to worry about. On the other hand, doing something that can schedule instructions would be much more work. If only working on the output set (and not from an intermediate language) then it'd be more limited in what it can do.

I have seen a lot of people talk about wanting to do portable recompilers and they almost always include some crippling limitations in performance. The best way is probably to go to an intermediate language that's very comprehensive in the feature set it supports. Then you have to have frontends going from the source languages to the IR and backends going from the IR to the destination languages, probably sharing some common code between all of this. The higher level frontend block management code can also be common, but you would want different strategies for different platforms.

 

[sindbad]

I have seen a lot of people talk about wanting to do portable recompilers and they almost always include some crippling limitations in performance. The best way is probably to go to an intermediate language that's very comprehensive in the feature set it supports. Then you have to have frontends going from the source languages to the IR and backends going from the IR to the destination languages, probably sharing some common code between all of this. The higher level frontend block management code can also be common, but you would want different strategies for different platforms.
You're describing LLVM

 

[Exophase]

You're describing LLVM

I know. But for all of the hype surrounding LLVM, the results aren't very good (at least when used as a C compiler backend, it's noticably behind the typical GCC output for ARM which wasn't that great to begin with). Not encouraging, and I wouldn't want to know the overhead involved in using it otherwise. Dynarecs need to have very tight compilers since they're recompiling on the fly.

Also, I doubt LLVM has the facilities for code management that are necessary for recompiled code, and a good recompiler needs to have very direct integration for some things that are arch specific, not just CPU specific, such as address space. There are also some tricky things such as self-modifying code that may require very involved strategies to counter.

 

[sindbad]

You're describing LLVM
I know. But for all of the hype surrounding LLVM, the results aren't very good (at least when used as a C compiler backend, it's noticably behind the typical GCC output for ARM which wasn't that great to begin with). Not encouraging, and I wouldn't want to know the overhead involved in using it otherwise. Dynarecs need to have very tight compilers since they're recompiling on the fly.

Also, I doubt LLVM has the facilities for code management that are necessary for recompiled code, and a good recompiler needs to have very direct integration for some things that are arch specific, not just CPU specific, such as address space. There are also some tricky things such as self-modifying code that may require very involved strategies to counter.Yes, I know about the ARM backend issues. And I don't know that much about dynarecs, my reasoning could just be wrong.

However, an optimising low level virtual machine seems to me like just the thing for building emulators on. Improving LLVM may or may not be easier/better than making some helper library for scratch.

 

[JayFoxRox]

You have a valid point about the term recompiler, Exophase. I was thinking about which term to use and ended up with "recompiler" because the code assembles some code itself at runtime, and re-assembler just seemed wrong. But you got a point there
A recompiler (- meaning a code which recompiles code from one arch for another) is not necessary as thats completly up to the developer imo. A generic re-assembler (- meaning code which assembles code to memory at runtime) could assist in writing recompilers for example and would be almost the same for most projects (with features like optimization and instruction re-ordering which were already mentioned by hlide).
As I said before: We should look for a solution which allows multiple programming languages and coding styles and not limit the code to one specific language.
Maybe I will upload my findings on the OpenGL ES 2.0 shader instruction set (I m figuring out instructions by hex-editing the binarys right now - if someone knows a REAL instruction set inform me about that please . I thought it must be in the OGL specification, but I found nothing about that in there) in the next days so someone can start to write an re-assembler for that.

 

[Exophase]

Maybe I will upload my findings on the OpenGL ES 2.0 shader instruction set (I m figuring out instructions by hex-editing the binarys right now - if someone knows a REAL instruction set inform me about that please . I thought it must be in the OGL specification, but I found nothing about that in there) in the next days so someone can start to write an re-assembler for that.

Don't the OpenGL ES 2 drivers do just that for you?

 

[JayFoxRox]

OpenGL ES 2.0 allows you to load a shader binary or source. If you use the source you have to load the included shader compiler and it takes a short time until the source is compiled. Another problem is that you have to include the full shader glsl source (or you have to reconstruct it somehow). The PowerVR SDK provides a tool to compile the shaders to binary shaders (closed source - atleast the source didn't come with my SDK). My idea is to re-compile all required shaders at startup so they can be loaded in binary form because my current shader is really long already as I have to include all emulated functions like blending, lights, vertex weights etc. even if I don't want to use them. In theory I could also generate GLSL code for all combinations but I m not a friend of having ascii-codes which are compiled at runtime. I m also sure that it would be alot slower (although speed shouldn't matter if all shaders are compiled at startup).

 

[darkblu]

re SGX shader reverse engineering, you want to talk to maciek urbanski about that.

re LLVM, i gained tons of respect for it recently, when i saw how it manages to 'plug-in the wholes' of the gma950 under glsl on osx. basically, it makes glsl possible for the gma9xx architecture (one of the most confused pieces of GPU in the recent history, while at the same time being insanely popular).

 

[hlidegp2x]

re SGX shader reverse engineering, you want to talk to maciek urbanski about that.

re LLVM, i gained tons of respect for it recently, when i saw how it manages to 'plug-in the wholes' of the gma950 under glsl on osx. basically, it makes glsl possible for the gma9xx architecture (one of the most confused pieces of GPU in the recent history, while at the same time being insanely popular).

still LLVM is not that easy to handle and ARM backend seems to have a lot of issues.