X86 Linux & Arm Linux Apps

[Poem58]

I just don't fully understand yet the process that was outlined by WizardStan when it comes to "having the right packages installed" to compile.

I know it doesn't apply in all cases (or answer all your questions, unfortunately), but quite often the readme file that will most likely accompany any source code you download will tell you what else you need to have installed before you can compile it (or perhaps the place you get it from might have a note to this effect published there). For example, it might say that it needs "libsuchandsuch", so you should make sure you have that before trying to compile the code. You'll likely see these things referred to as "dependencies", since they're what the program you're trying to compile depends upon to work.

Clear, concise, thank you!
Once you've done that does it incorporate said file into the output so as the next person doesn't need to search out said file?

If so, that sounds quite doable and sound easy enough that many people might do this then. So what was being said by Aninhumer was that if there is no mention of libsuchandsuch for some reason, that there would be an error saying libsuchandsuch was missing right? Does the compiling process always know then what files are needed. Am I understanding this correctly?
And do you just google the libsuchandsuch as a file or as something you install? Just asking for knowing if the libsuchandsuch is something that has to be Pandora specific.
I'm sure it will make more sense when we actually have Pandora's to do it on, but I think I'm understanding a bit more now. Haven't learned about libraries etc yet. I have a lot of googling to do! LOL

 

[Aninhumer]

The vast majority of stuff I've had to install from source have been
1) download source (as a tar.gz file)
2) uncompress tar.gz file to a temporary directory
3) start a terminal, and "cd" to that temporary directory
4) type "make configure"
5) type "make install" (may need root privileges for this)
6) start the program by typing its command, and/or adding a shortcut to the menu or desktop
An absolute beginner would need a step-by-step guide (how exactly to start a terminal, or uncompress a tar.gz file, for example) but if you've got at least basic understanding, it should be pretty simple.
This assumes that you have all the right packages already installed. Making sure that you have everything you need first is what will cause the greatest challenge.

... I appreciate that you responded however, asking someone else to do it was kinda against the point. Some of us would like to learn how to do it and my questions were asked to determine how likely it is someone like myself would be able to do this. ... I'd rather do it myself and share it out if possible, than spend weeks asking someone to do it for me.

Yeah, I was more answering the original question than your post.

I don't need an entire manual or anything, I'm sure someone will post a guide, but I was curious if this really was going to be something I could do, I just don't fully understand yet the process that was outlined by WizardStan when it comes to "having the right packages installed" to compile. How does one figure that out when something doesn't compile, and once you compile it is it fully ready to go or does the downloader of that package have to "have the right packages installed"?
Keep in mind I haven never compiled anything, barely used linux (Ubuntu live CD for an hour) So saying it "fails at the configuration stage" (???) and "guess by the error" (How?) doesn't really help much. If someone could just give me a simple example of what the error might say? Does it just say, "compile failed, missing ABCD.XXX" or something?
Or is there some guide somewhere I could read that explains all this? Anything I can learn would be helpful.

When you run "make configure", the script will output a series of lines explaining what packages and tools it is searching for, and whether it found them.
So if it says something like: "Searching for SDL... No" at the end, you'd know you needed the SDL libraries.
(I can't remember the exact wording offhand, but it's something like "Searching for")

Unfortunately the name it lists might not be something you know about, and the package name might not even be similar, and that's where the fun starts.

Oh another important thing, I don't know whether the Pandora will ship with a build environment (compilers and stuff) ready, so you might have to install those as well.
On Ubuntu you can run: "sudo apt-get install build-essential" to set up for compiling, there will probably be something similar for Angstrom.

 

[Exophase]

I just wanted to add (without going into details), that a lot of speed in my static recompiler comes from the fact that it's targeted at a single executable.
If I wrote a dynamic recompiler targeting a single executable, then it would be slower than the static recompiler, but still much faster than a dynamic recompiler targeting a lot of executables. The advantage would be less manual input than the static recompiler.

I know you said you wouldn't go into details, but could you qualify this?

From what I'm aware of what you're doing I don't see where a substantial amount of speed is gained by being either static or from targeting specific executables. I'm sure various assumptions are being made about the simplicity of the code and the address space, but this could apply to a whole set of applications. If you're stubbing in native code for certain functions (ie, HLE) then there'd be that, so are you doing anything like that?

That aside, the main thing I'm interested in is what you think static is gaining you performance-wise. Because from what you've described, the only multi-instruction analysis you're performing is dead-flag elimination, which probably isn't spanning multiple blocks and you wouldn't gain that much. Register allocation can be and is completely static and therefore wouldn't benefit from block spanning analysis at all. Indirect branches would have to go through some kind of translation process either way, the dynamic case would just have links to a recompiler stub for translation misses, while the static one would have to have no misses. The translation sounds simple enough that its generation could pretty easily outpace loading from flash.

By the way, that reminds me.. WizardStan earlier said that an advantage of static recompilation is that you don't need to have the original executable loaded in memory. Actually, that's not true, because executables typically tend to intersperse code throughout the data, for instance string literals stored inbetween functions. This is especially true on platforms with PC-relative addressing. So the whole original program has to be loaded, not to mention invariably a lot of recompiled code that won't get used. mmapping the file could mitigate the cost from loading, but if the constants are well dispersed then you'll end up pulling in a lot of the pages.

 

[Poem58]

When you run "make configure", the script will output a series of lines explaining what packages and tools it is searching for, and whether it found them.
So if it says something like: "Searching for SDL... No" at the end, you'd know you needed the SDL libraries.
(I can't remember the exact wording offhand, but it's something like "Searching for")

Unfortunately the name it lists might not be something you know about, and the package name might not even be similar, and that's where the fun starts.

Oh another important thing, I don't know whether the Pandora will ship with a build environment (compilers and stuff) ready, so you might have to install those as well.
On Ubuntu you can run: "sudo apt-get install build-essential" to set up for compiling, there will probably be something similar for Angstrom.

Thanks, that clears it up well. So it will be easy, but possibly difficult. LOL
I'm sure once they all ship someone will make guides on how to set this all up (or I will be asking) For someone like myself, I would try to compile stuff I find and if I ran into the ones I didn't know what to install I'd google, then ask for help here. I just don't want to find all kinds of stuff and making lists for other people to cater to me. It's good to know, unlike coding, this is something I should have a relative amount of success. If I knew how, I would set it up on my PC (which can be done right?) and start learning. I want to learn coding as well (even if just enough to draw circles and lines on screen, just something) I've tried reading websites about python, but I guess it's the way they describe it that doesn't make me know what to do (learned basic with a list of names of functions, what they did and sample syntax, believe it or not I understood that more than today's python for newbies sites!)

One last thing not answered, once completed compiling, the resulting package is ready to go right? Or do others who use the package need to install what you had to to get it to compile?
Just curious as I've seen people say things like "you'd need to apt-get, or you need to have libsuchandsuch", not sure if thats mainly when something is being written/compiled or just altogether.

So thanks to you and Prometheus for answering without continuing to use unfamiliar terminology. Lots of little things to get used to, which is half the fun of the Pandora! I guess for the most part, an old fogey like me remembers Pre MSX Extended Basic and early DOS as the "command line" typing and since "sudo apt-get install build-essential" doesn't exacly freely spring to mind, one can get an anticipation of all this being confusing and difficult. Microsux (I do actually like Windows, then again, I should I have spent so much time working on repairing 95-Vista machines, it's become a part of my everyday life and has spoiled me to the point and click, and download and install mentality. Which is probably why I want to learn to compile and code etc. again. Get back to what made computers fun, actually DOING something with them, nt just surf, edit, print, etc.

 

[Exophase]

One last thing not answered, once completed compiling, the resulting package is ready to go right? Or do others who use the package need to install what you had to to get it to compile?
Just curious as I've seen people say things like "you'd need to apt-get, or you need to have libsuchandsuch", not sure if thats mainly when something is being written/compiled or just altogether.

That depends on whether or not what was compiled is statically linked or dynamically linked (or some combination of both, which is unusual).

Statically linked means that all the libraries, the things you had to get in order to compile, are included in the program you compiled.

Dynamically linked means they're not included, and library files have to be present on the computer. On Windows this would mean you need to have a DLL, which is often bundled with the program. On Linux this would be an SO (shared object file). They're not usually bundled with the program, since that defeats the purpose of compiling dynamically, which is to avoid having to drag in the library.

With repositories like apt-get usually libraries have a runtime version, which is what other people need in order to run stuff using it (in other words, those SO files), and a dev version, which is what you need to compile things that use it (so the library and header files).

 

[M-HT]

I just wanted to add (without going into details), that a lot of speed in my static recompiler comes from the fact that it's targeted at a single executable.
If I wrote a dynamic recompiler targeting a single executable, then it would be slower than the static recompiler, but still much faster than a dynamic recompiler targeting a lot of executables. The advantage would be less manual input than the static recompiler.

I know you said you wouldn't go into details, but could you qualify this?

From what I'm aware of what you're doing I don't see where a substantial amount of speed is gained by being either static or from targeting specific executables. I'm sure various assumptions are being made about the simplicity of the code and the address space, but this could apply to a whole set of applications. If you're stubbing in native code for certain functions (ie, HLE) then there'd be that, so are you doing anything like that?

The things (some of them) that I'm doing (when targeting specific executable) are:
- HLE (stubbing in native code for certain functions) - I'm doing this for clib functions - not for performance reasons, but because I don't want to implement it myself. I'm also doing it for sound/music functions - for performance reasons and because I don't want (and I don't know how) to emulate sound HW.

- I'm replacing accesses to memory mapped IO (ie. video memory, bios time, ...) with my implementation. That also means that all other memory accesses are direct, without memory maps or other checking (this is a performance killer) - memory address in translated code is an address in host memory.

- I'm also sometimes "patching" the original code. For example rewriting self-modifying code with non-self-modifying code or rewriting a parameter in graphics library to use vesa mode instead of xmode.

This all can be done either in static or in dynamic recompiler.

That aside, the main thing I'm interested in is what you think static is gaining you performance-wise. Because from what you've described, the only multi-instruction analysis you're performing is dead-flag elimination, which probably isn't spanning multiple blocks and you wouldn't gain that much. Register allocation can be and is completely static and therefore wouldn't benefit from block spanning analysis at all. Indirect branches would have to go through some kind of translation process either way, the dynamic case would just have links to a recompiler stub for translation misses, while the static one would have to have no misses. The translation sounds simple enough that its generation could pretty easily outpace loading from flash.

Actually, I am doing dead-flag elimination spanning multiple blocks. Also indirect calls don't go through any kind of translation process, because the original addresses were properly translated into host addresses.
I didn't say that the speed difference is great, only that statically recompiled code is faster.

By the way, that reminds me.. WizardStan earlier said that an advantage of static recompilation is that you don't need to have the original executable loaded in memory. Actually, that's not true, because executables typically tend to intersperse code throughout the data, for instance string literals stored inbetween functions. This is especially true on platforms with PC-relative addressing. So the whole original program has to be loaded, not to mention invariably a lot of recompiled code that won't get used. mmapping the file could mitigate the cost from loading, but if the constants are well dispersed then you'll end up pulling in a lot of the pages.

It's true that with static recompilation the original executable doesn't have to be loaded in memory.
If the original executable looks like this: code - data - code - data - ..., then the translated executable looks like this: translated code - data - translated code - data - ... (at least with my recompiler).
It's also true that static recompilation translates code that will never be used (and translated code is larger than original code).
But I think that the size of original code is larger than the size of translated code that will never be used. So, static recompilation will use less memory, but only if the dynamic recompilation stores all translated blocks into memory and doesn't discard old translated blocks.

 

[ldesnogu]

That aside, the main thing I'm interested in is what you think static is gaining you performance-wise. Because from what you've described, the only multi-instruction analysis you're performing is dead-flag elimination, which probably isn't spanning multiple blocks and you wouldn't gain that much. Register allocation can be and is completely static and therefore wouldn't benefit from block spanning analysis at all. Indirect branches would have to go through some kind of translation process either way, the dynamic case would just have links to a recompiler stub for translation misses, while the static one would have to have no misses. The translation sounds simple enough that its generation could pretty easily outpace loading from flash.

By the way, that reminds me.. WizardStan earlier said that an advantage of static recompilation is that you don't need to have the original executable loaded in memory. Actually, that's not true, because executables typically tend to intersperse code throughout the data, for instance string literals stored inbetween functions. This is especially true on platforms with PC-relative addressing. So the whole original program has to be loaded, not to mention invariably a lot of recompiled code that won't get used. mmapping the file could mitigate the cost from loading, but if the constants are well dispersed then you'll end up pulling in a lot of the pages.

Sorry for taking part into the conversation in the middle: are you commenting about M-HT specific code or about human-assisted static translation in general? Because for the only static translation I did, I made sure indirect jumps were translated into what they should (switch and return), and I grouped data. And that was for a game written in assembly language; the effort took several weeks of work (and that included most of what M-HT describes: replacing I/O functions and so on), but for a C program I guess less manual assistance would be needed.

 

[Poem58]

One last thing not answered, once completed compiling, the resulting package is ready to go right? Or do others who use the package need to install what you had to to get it to compile?
Just curious as I've seen people say things like "you'd need to apt-get, or you need to have libsuchandsuch", not sure if thats mainly when something is being written/compiled or just altogether.

That depends on whether or not what was compiled is statically linked or dynamically linked (or some combination of both, which is unusual).

Statically linked means that all the libraries, the things you had to get in order to compile, are included in the program you compiled.

Dynamically linked means they're not included, and library files have to be present on the computer. On Windows this would mean you need to have a DLL, which is often bundled with the program. On Linux this would be an SO (shared object file). They're not usually bundled with the program, since that defeats the purpose of compiling dynamically, which is to avoid having to drag in the library.

With repositories like apt-get usually libraries have a runtime version, which is what other people need in order to run stuff using it (in other words, those SO files), and a dev version, which is what you need to compile things that use it (so the library and header files).

eeeech, so what once sounded rather easy yet again rears an ugly head. So if I were to say just want to find stuff on the web and compile it for the Pandora. Is it up to me to do it statically or dynamically, Or based on type of program or type of libraries? Obviously everyday ordinary Joe's can compile or people wouldn't say "Just ask and someone might do it for you" so is there a good place to read how to do this in terms that aren't supposing you practically wrote linux and it's terminology? Or is something I should just not worry about as it will become clear once we get our Panda's and I try to do it? I don't have anything specific I want to compile, heck I may never find anything. But it sounds like something interesting to learn and since I'm not up on the terminology it makes the stuff I have run across to read seem like mumbo jumbo. I do appreciate you guys for answering though, helps me learn in a better way, and I love to learn, just can't always make sense of the websites that are supposed to guide you (much like that Python one which made some sense but didn't help me understand anything about the language)

 

[Exophase]

Actually, I am doing dead-flag elimination spanning multiple blocks.

This is relatively easy/fast to do on a dynarec too, you just have to recursively translate the blocks you're linking to before you translate the current block and store their initial flag status somewhere. These aren't time consuming things.

Also indirect calls don't go through any kind of translation process, because the original addresses were properly translated into host addresses.
I didn't say that the speed difference is great, only that statically recompiled code is faster.

This may work for calls/returns, but it wouldn't work any worse in a dynarec. It won't succeed any more often than in a dynarec without manual intervention, which falls outside of the scope of the comparison. Really if user feedback is required we're talking about an entirely different thing, more of a very thorough porting tool.

In fact, a dynarec could likely be more successful with this conversion by patching the code at runtime using runtime information.

It's true that with static recompilation the original executable doesn't have to be loaded in memory.
If the original executable looks like this: code - data - code - data - ..., then the translated executable looks like this: translated code - data - translated code - data - ... (at least with my recompiler).
It's also true that static recompilation translates code that will never be used (and translated code is larger than original code).
But I think that the size of original code is larger than the size of translated code that will never be used. So, static recompilation will use less memory, but only if the dynamic recompilation stores all translated blocks into memory and doesn't discard old translated blocks.

Like the above, what you're saying sounds very error prone with user intervention necessary or at least low compatibility. You'd have to statically determine the address in order to up and move it somewhere else. This gets worse for platforms that aren't x86 and don't have purely immediate memory offsets.

Sorry for taking part into the conversation in the middle: are you commenting about M-HT specific code or about human-assisted static translation in general? Because for the only static translation I did, I made sure indirect jumps were translated into what they should (switch and return), and I grouped data. And that was for a game written in assembly language; the effort took several weeks of work (and that included most of what M-HT describes: replacing I/O functions and so on), but for a C program I guess less manual assistance would be needed.

My comments were never about human-assisted static translation. You can't compare it to an automatic dynarec. It's apples and oranges.

 

[Tom`]

eeeech, so what once sounded rather easy yet again rears an ugly head. So if I were to say just want to find stuff on the web and compile it for the Pandora. Is it up to me to do it statically or dynamically, Or based on type of program or type of libraries? Obviously everyday ordinary Joe's can compile or people wouldn't say "Just ask and someone might do it for you" so is there a good place to read how to do this in terms that aren't supposing you practically wrote linux and it's terminology? Or is something I should just not worry about as it will become clear once we get our Panda's and I try to do it? I don't have anything specific I want to compile, heck I may never find anything. But it sounds like something interesting to learn and since I'm not up on the terminology it makes the stuff I have run across to read seem like mumbo jumbo. I do appreciate you guys for answering though, helps me learn in a better way, and I love to learn, just can't always make sense of the websites that are supposed to guide you (much like that Python one which made some sense but didn't help me understand anything about the language)

I have no idea how to statically link the libraries, this isn't what a typical Linux user would do. All you usually do is:
1. Download the source (usually a .tar.gz or .tar.bz2 file)
2. Fire up your favorite terminal program.
3. Type

tar xzvf <filename>.tar.gz

(OR, if it's a tar.bz2 file)

 tar xjvf <filename.tar.gz>

Then cd to the newly-created directory.
4. Type

./configure

.
5. If the configuration process fails, it will tell you "Library libsomething.so version <some number> was not found, cannot continue" or something to that effect, so you open up your package manager and try to find the libsomething-dev package. If it's not in the repo, find source code for this library and repeat steps 1-5 for that. Do this as many times as necessary for all the required dependencies (this can take forever, or one of the dependencies may not compile right - in this case, I would usually just give up and/or ask someone knowledgeable to do it, since I'm no programmer or kernel expert and don't know how to fix any non-obvious errors.)
6. If the configuration process succeeds, it will usually tell you what options the software was configured with. Now type "make".
7. The binary should be somewhere in the current directory (or some subdirectory). If you want to install it, you can type "make install" as root now, or, since you'll probably be cross-compiling for the Pandora, you can run whatever PND packaging tools may be available (I'm not too clear on how this works). Otherwise, you can just copy the binary straight to your SD card and execute it from a terminal or shell script on the Pandora itself, but this method is probably discouraged for most users.

It's really not as difficult as it sounds, but I've never tried to package anything for distribution before, so I don't know what that involves.

 

[Shaun199three]

This is kinda irrelevant to this topic but I think it might be interesting. While were talking about Linux I decided to check if there was some sought of readyboost alternative for linux devices and I found something called Swapboost. I know alot of people argue that Readyboost does nothing or very little but it should do alot to device with only 256mb ram. Does anyone know if this should work or not on the Pandora?

 

[ldesnogu]

My comments were never about human-assisted static translation. You can't compare it to an automatic dynarec. It's apples and oranges.

OK then I don't understand the point and will leave that thread alone For me (and you IIRC) non human-assisted static translation is just an intractable problem.

 

[M-HT]

Also indirect calls don't go through any kind of translation process, because the original addresses were properly translated into host addresses.
I didn't say that the speed difference is great, only that statically recompiled code is faster.

This may work for calls/returns, but it wouldn't work any worse in a dynarec. It won't succeed any more often than in a dynarec without manual intervention, which falls outside of the scope of the comparison. Really if user feedback is required we're talking about an entirely different thing, more of a very thorough porting tool.

In fact, a dynarec could likely be more successful with this conversion by patching the code at runtime using runtime information.

In case it wasn't clear, I was talking about my static recompiler and comparing it to dynamic recompiler targeting a single executable. That means that manual intervention is part of it.
And since at least the point about not requiring any kind of translation process in indirect calls still holds, I think that statically recompiled code is faster than dynamically recompiled code.

It's true that with static recompilation the original executable doesn't have to be loaded in memory.
If the original executable looks like this: code - data - code - data - ..., then the translated executable looks like this: translated code - data - translated code - data - ... (at least with my recompiler).
It's also true that static recompilation translates code that will never be used (and translated code is larger than original code).
But I think that the size of original code is larger than the size of translated code that will never be used. So, static recompilation will use less memory, but only if the dynamic recompilation stores all translated blocks into memory and doesn't discard old translated blocks.

Like the above, what you're saying sounds very error prone with user intervention necessary or at least low compatibility. You'd have to statically determine the address in order to up and move it somewhere else. This gets worse for platforms that aren't x86 and don't have purely immediate memory offsets.

Actually, it has high percent of success and requires only little user intervention (in my recompiler). And since I'm talking about my recompiler, I'm talking about X86 and DOS executables.

 

[Aninhumer]

eeeech, so what once sounded rather easy yet again rears an ugly head. So if I were to say just want to find stuff on the web and compile it for the Pandora. Is it up to me to do it statically or dynamically, Or based on type of program or type of libraries? Obviously everyday ordinary Joe's can compile or people wouldn't say "Just ask and someone might do it for you" so is there a good place to read how to do this in terms that aren't supposing you practically wrote linux and it's terminology? Or is something I should just not worry about as it will become clear once we get our Panda's and I try to do it? I don't have anything specific I want to compile, heck I may never find anything. But it sounds like something interesting to learn and since I'm not up on the terminology it makes the stuff I have run across to read seem like mumbo jumbo. I do appreciate you guys for answering though, helps me learn in a better way, and I love to learn, just can't always make sense of the websites that are supposed to guide you (much like that Python one which made some sense but didn't help me understand anything about the language)

Actually, I'm not sure how the PND system works, but I think there is some way of telling it what packages are needed to run a game.
So you'd keep track of what packages you installed, and do something with them when making the PND, and the Pandora will install the packages needed.
I don't think many people know much more than that right now.

I'm not sure how the Pandora will get packages if it can't connect though...

 

[dflemstr]

Actually, I'm not sure how the PND system works, but I think there is some way of telling it what packages are needed to run a game.

Nope, there isn't.

If you make an app and package it in a PND, you have to include all of the package dependencies in that same PND. No dependency resolution available.

 

[Exophase]

In case it wasn't clear, I was talking about my static recompiler and comparing it to dynamic recompiler targeting a single executable. That means that manual intervention is part of it.
And since at least the point about not requiring any kind of translation process in indirect calls still holds, I think that statically recompiled code is faster than dynamically recompiled code.

For the case of manually assisted translation, which is really only partial translation. In that case you wouldn't really have an argument for dynamic recompilation since you'd be better off doing whatever was necessary to make it not required. But like I said, this was never about that, you can't compare it. When you have the power to fix up the result yourself and patch it arbitrarily on a program by program basis then really no rules apply and the translators are tools, not emulators.

Actually, it has high percent of success and requires only little user intervention (in my recompiler). And since I'm talking about my recompiler, I'm talking about X86 and DOS executables.

It doesn't matter if it's high or not, if it's not 100% then it doesn't apply to the automatic translation case. Now of course this is assuming that it's possible at all, meaning that it can determine all branch targets on its own and the code is static, for some chosen target. If that's the case you still have to translate the targets at runtime or it can fail, meaning you have to load the original executable.

In your post you said that even for an automatic translator static translation has performance benefits. I'm trying to determine what these are.

 

[juniorm33]

Actually, I'm not sure how the PND system works, but I think there is some way of telling it what packages are needed to run a game.

Nope, there isn't.

If you make an app and package it in a PND, you have to include all of the package dependencies in that same PND. No dependency resolution available.

Does that include already included packages that are on the pandora by default, or do you mean just the additional ones?

 

[dflemstr]

Actually, I'm not sure how the PND system works, but I think there is some way of telling it what packages are needed to run a game.

Nope, there isn't.

If you make an app and package it in a PND, you have to include all of the package dependencies in that same PND. No dependency resolution available.

Does that include already included packages that are on the pandora by default, or do you mean just the additional ones?

The Pandora uses IPKG for core packages, but developers have the option to distribute games and similar simple apps in PND packages. So, if you want to install a new browser or a new window manager something, you use IPKG to do so, but if you want to play PandoraPanic, you download a PND file and drop it in a folder instead.

 

[M-HT]

In your post you said that even for an automatic translator static translation has performance benefits. I'm trying to determine what these are.

I apologize if it sounded that way, but I didn't talk about automatic translators - and that applies to everything I said in this thread.
My original point was that my recompiler is fast because it targets single executable (instead of multiple executables). Not because it's a static recompiler instead of dynamic one.

 

[ldesnogu]

I'm glad I had understood what M-HT wrote the way it was supposed to be