X86 Linux & Arm Linux Apps

[Near_Kstor]

Now that you guys mentioned, something came in my mind...

Suppose I am running Ubuntu ARM on the Pandora... When I go to the Synaptics to search and download some apps, will I be able do download stuff for my Pandora (already ARM-Compiled)? Or the Synaptics can't really see which kind of processor I have and just suppose I am running on a x86 device?

 

[closetwam]

If you are running Ubuntu ARM, the /etc/apt/sources.list file will have entries for ARM repositories specifically. Synaptic will download the correct apps for your platform.

 

[kloplop321]

@Near-kun
I suppose so, the repositories are most likely going to be pointed towards apps compatible with the system you're running. I don't think that the makers of Ubuntu would make the mistake of assigning the incorrect repository locations.

Synaptics doesn't see the Processor, but I would bet that Ubuntu ARM edition or what ever would have the repositories for ARM binaries

 

[Exophase]

This doesn't really apply to exe files, but other ROMs and com files don't necessarily have a definitive break between code and data. Static recompilation you can direct, and ensure that what you recompile is code, not data. Dynamic recompilation has other tricks that are less effective.

I think you have that totally backwards, a dynamic recompiler is going to know what's code since the program will be trying to execute it. At static time anything really could be code.

 

[Mr.Confuzed]

I think you have that totally backwards, a dynamic recompiler is going to know what's code since the program will be trying to execute it. At static time anything really could be code.

I agree that this particular problem will be much more difficult with a static recompiler. However, some of your other posts seem to suggest that there is something wrong with static recompilation. I'm not sure where that's coming from, but I am curious: Have you tried writing one? Is there some sort of problem with the method? I don't see one.

 

[WizardStan]

I think you have that totally backwards, a dynamic recompiler is going to know what's code since the program will be trying to execute it. At static time anything really could be code.

Well, a dynamic recompiler only knows what is code because it's about to try and run it; if it wants to work ahead of itself, it's hard because it hasn't got there yet.
A static recompiler can afford to try everything, even spend the time to work backwards to see whether a given block is ever referenced.

 

[VRAndy]

It's not really that bad. In fact, it's pretty normal for Linux apps.
Compiling stuff is super-easy on Linux, and the vast majority of apps are distributed as source.

:blink: Eh? Wai-wait... how easy is it? Like for a beginner who's never once compiled an app? If I could learn how to do it, I think I'd gladly do it quite often.

You download it. You unzip it. You type "make install". Done.

There are probably GUI ways of doing this, but I've never tried.

Also, stuff created specifically for Pandora will probably have ready-made packages that are even EASIER (I hope) to install.

Really, it's nothing to worry about.

 

[dflemstr]

I think you have that totally backwards, a dynamic recompiler is going to know what's code since the program will be trying to execute it. At static time anything really could be code.

Well, a dynamic recompiler only knows what is code because it's about to try and run it; if it wants to work ahead of itself, it's hard because it hasn't got there yet.
A static recompiler can afford to try everything, even spend the time to work backwards to see whether a given block is ever referenced.

Ever heard about the Halting Problem, and other Turing laws? The only way to find out what's data and what's not is to execute the code and see how the code treats its context, and if you have to do that in a static recompiler, it kinda defeats the purpose now doesn't it.

(I learned this after Exophase pointed it out so I thought it was unintuitive too at first)

 

[WizardStan]

Ever heard about the Halting Problem, and other Turing laws? The only way to find out what's data and what's not is to execute the code and see how the code treats its context, and if you have to do that in a static recompiler, it kinda defeats the purpose now doesn't it.

(I learned this after Exophase pointed it out so I thought it was unintuitive too at first)

All that proves is that static recompilation doesn't work in all cases. It doesn't make a case as to whether the cases it does work are large or small in number, or whether those cases perform better or worse than the dynamic alternative.

 

[dflemstr]

Ever heard about the Halting Problem, and other Turing laws? The only way to find out what's data and what's not is to execute the code and see how the code treats its context, and if you have to do that in a static recompiler, it kinda defeats the purpose now doesn't it.

(I learned this after Exophase pointed it out so I thought it was unintuitive too at first)

All that proves is that static recompilation doesn't work in all cases. It doesn't make a case as to whether the cases it does work are large or small in number, or whether those cases perform better or worse than the dynamic alternative.

Absolutely. However, I haven't heard of a program that operates without data, so unless you have access to every resource your app is going to load at runtime (DOS) or your executable separates data from code (like the JVM, CLI, and LLVM; they are artificially verbose for exactly that reason), you can't really do much with static recompilation.

 

[Tor]

It becomes much simpler if the source CPU in question uses a Harvard architecture.. i.e. separate data and code. E.g. MC68k.

 

[Poem58]

This is cool reading, while it doesn't sound too difficult, can a Pandora app be created to simplify/speed up these steps (or would it not really be any easier that way?)

I think even a newbie like myself would love to be able to compile software and with a whole bunch of us doing it, we could have a rather large amount of available software from people in a relatively short amount of time. In any case, I'm glad to know that it's a relatively simple process to find nearly anything with a source file and make it workable on the Pandora. Even if someone like myself has to learn about installing other things to get it to compile.

Question, if you have to install things to make the software convert from say x86 to ARM, let me rephrase, as a newbie, I find a cool looking program (be it game or app) I try the steps for compiling and it doesn't work, how do I find out what is missing and needs to be installed, and secondly, once I convert it, can I share it to the rest of the group and it will just work on their Pandora or do you have to have them know what you installed to make it compile. There is so much I don't know yet about the whole linux process and I want to learn it, I hope these questions don't seem to dumb, the idea of even a linux ignorant person like myself can help add to the available software list seems exciting and important for understanding and learning the OS and it's differences to Microsuck Winblows. ( I pick on Microsoft because once again a person I know's machine died (drive died) and now his windows install (on hard drive) is gone with no way to reinstall so I used an Ubuntu live disk and must say it was rather impressive to see how easily it could do things over windows)

in other words, Me Likey!

 

[Exophase]

I agree that this particular problem will be much more difficult with a static recompiler. However, some of your other posts seem to suggest that there is something wrong with static recompilation. I'm not sure where that's coming from, but I am curious: Have you tried writing one? Is there some sort of problem with the method? I don't see one.

I've written dynamic recompilers. People bring up static recompilers a lot, but if they're so viable then why do so few real static recompilers exist, much less ones that are completely automatic and run a significant amount of real software? Sometimes one will come along that calls itself a static recompiler but it's actually just a deeply recursive dynamic one. It still generates code at runtime - as much as it can, but it'll stop when it hits a dead end and has to be capable of picking up again later.

Every program has indirect branches, and potential indirect branch targets could be basically anything. There are heuristics to try to guess these, and it helps when you're recompiling an executable that has some sort of structure to it, but most emulators are emulating hardware, not operating systems, and only have a binary image of some sort to work with. Ultimately heuristics will certainly fail in some circumstances, especially considering that even with structured executables you'll find code doing things like putting jump tables in the code segment, or for instance on something like ARM, incrementing the PC based on a computed value. Static recompilers I've seen literature on go to extremes such as making everything in the data segment a branch target and recompiling that. This creates huge bloat and is still not foolproof because branch targets don't have to be stored verbatim. The only way to really get all branch targets, if you're certain that they all exist in your binary, is to recompile a block starting at every possible starting location in the binary. This gets pretty out of hand with an architecture that has variable width instructions like x86. You'll also end up with a huge indirect branch table that you have to hash through at runtime, unlike the code which could get swapped out.

Of course, this still only works if your code is resident at load time. Anything dynamically loading, generating, or modifying code will fail. So if your program, say, uses overlays or loads DLLs in a way that's not determined by the executable structure at compile time, then it'll fail. Like for instance, plugins in a media player or emulator.

Meanwhile, people assume that static recompilers will have benefits because you can spend more time performing optimizations, but they should understand that recompilers aren't the same as high level language compilers and a lot of optimizations don't apply in the same fashion.

Well, a dynamic recompiler only knows what is code because it's about to try and run it; if it wants to work ahead of itself, it's hard because it hasn't got there yet.
A static recompiler can afford to try everything, even spend the time to work backwards to see whether a given block is ever referenced.

A static recompiler can't try anything, since it has no idea if it's right or not. A dynamic recompiler can work ahead as much as is actually advantageous for recompilation, and it can recompile again later based on new information if that's somehow desirable. Besides, this has little to do with what you claimed, that a static recompilation has benefits in determining what's code vs what's data and in dynamic recompilation you can't be sure. The complete opposite is true.

It becomes much simpler if the source CPU in question uses a Harvard architecture.. i.e. separate data and code. E.g. MC68k.

You're thinking of Harvard address spaces, not Harvard internal buses (ie, icache/dcache, they both go to the same bus at some level up). Few architectures that are that interesting to recompile actually employ split address spaces. 68k certainly doesn't. Even architectures that do employ this in some capacity usually have a way to load data from the code space and code from the data space.

 

[Aninhumer]

Poem58, there are two different conversations going on here unfortunately. <_<
A lot of devs have hijacked this topic which was asking a simple question about regular compiling of source code (which is easy) to talk about recompiling binaries (which isn't). Guys go make a topic in the dev board or something, and stop confusing people!

If you find an app you want to run on the pandora, but it doesn't work, usually it will fail at the configure stage with an error, and you might be able to guess what package is needed from the output.

If you think the app would be useful/desirable/cool for other people, mention it on the forums and someone can probably package up a PND file and put it on the archive.


Overall, although it's likely some apps will compile fine, it's probably better for a less experienced user to wait for a properly packaged version, as what might be hours of confusion for a newbie could be a few minutes for someone else, who doesn't mind doing it for the community.

 

[WizardStan]

A static recompiler can't try anything, since it has no idea if it's right or not. A dynamic recompiler can work ahead as much as is actually advantageous for recompilation, and it can recompile again later based on new information if that's somehow desirable. Besides, this has little to do with what you claimed, that a static recompilation has benefits in determining what's code vs what's data and in dynamic recompilation you can't be sure. The complete opposite is true.

I knew it was a trap. Walked into it anyway. Now I need to defend my thoughts against a master. No way out but forward.
Anywho, here's how I saw it. A static recompiler can follow the code until it hits a branch, and then assume one side is true until it reaches some sort of termination (recursively following other branches until it reaches a return or exit statement, or hits a block of code it has already recompiled). Then it just winds back and tries the other half of the branch, until it has no branches left. At the same time it's following the code, it's seeing requests for memory locations that get flagged as data. By the end, you've profiled theoretically the entire executable. Anything else is dummy data. A clever recompiler might then be able to see certain paradigms (loops, switch/case, etc...) ripe for optimization that a dynamic compiler wouldn't have the time to look for.
Bearing in mind that I've never written a recompiler, just straight up compilers; that's probably biasing my thinking to want to get a full snapshot of every branch of the code, leading me to conclusions that this would lead to better optimization than what the dynamic compiler could come up with.

 

[Polossatik]

Just wished I could compile static beer into dynamic beer...

 

[Exophase]

Poem58, there are two different conversations going on here unfortunately. <_<
A lot of devs have hijacked this topic which was asking a simple question about regular compiling of source code (which is easy) to talk about recompiling binaries (which isn't). Guys go make a topic in the dev board or something, and stop confusing people!

Really? What was asked:

"The question is whether or not it will be possible to run regular linux apps on pandora."

From the context it looks more like he was asking about running binaries than recompiling source. So I think we're actually more on topic ;p

I knew it was a trap. Walked into it anyway. Now I need to defend my thoughts against a master. No way out but forward.
Anywho, here's how I saw it. A static recompiler can follow the code until it hits a branch, and then assume one side is true until it reaches some sort of termination (recursively following other branches until it reaches a return or exit statement, or hits a block of code it has already recompiled). Then it just winds back and tries the other half of the branch, until it has no branches left. At the same time it's following the code, it's seeing requests for memory locations that get flagged as data. By the end, you've profiled theoretically the entire executable. Anything else is dummy data. A clever recompiler might then be able to see certain paradigms (loops, switch/case, etc...) ripe for optimization that a dynamic compiler wouldn't have the time to look for.
Bearing in mind that I've never written a recompiler, just straight up compilers; that's probably biasing my thinking to want to get a full snapshot of every branch of the code, leading me to conclusions that this would lead to better optimization than what the dynamic compiler could come up with.

It's a common misconception that a program's code space can be exhausted statically by following the execution tree. This only works while following direct branches. Indirect branches mean that the branch target is taken from a register and thus could be anything since a register could be anything. Indirect branches are used for returning from functions and calling function pointers. They're also often used for switches and they're used for vtables in C++. You can predict that the location after a function call will be a branch target for the functions, if they play nice and don't return somewhere else. But for those other things you don't really know what their targets could be. Some targets will be branched to both directly and indirectly. Others won't. You don't know. There are heuristics but they aren't absolute.

 

[SNESFAN]

A lot of devs have hijacked this topic which was asking a simple question
snip snip

the dev's have itchy programming fingers.... can't wait to get their cheeto encrusted hands on their precious

they need something to do while they wait.... it's the about the same as us normal users just complaining while we wait... i think the question the OP was asking got answered in post 2

 

[dflemstr]

It's a common misconception that a program's code space can be exhausted statically by following the execution tree. This only works while following direct branches. Indirect branches mean that the branch target is taken from a register and thus could be anything since a register could be anything. Indirect branches are used for returning from functions and calling function pointers. They're also often used for switches and they're used for vtables in C++. You can predict that the location after a function call will be a branch target for the functions, if they play nice and don't return somewhere else. But for those other things you don't really know what their targets could be. Some targets will be branched to both directly and indirectly. Others won't. You don't know. There are heuristics but they aren't absolute.

In other words:

#include <stdlib.h>
#include <stdio.h>

void a(void) {
  printf("a");
}

void b(void) {
  printf("b");
}

int main(int argc, char *argv[])
{
  void (*funcs[2])(void);
  int i;

  funcs[0] = &a;
  funcs[1] = &b;

  for(i = 0; i < 500; i++)
    funcs[rand() & 1]();

  return 0;
}

Provided that rand() has been seeded properly, can you tell what this program will output and which functions will be called under which conditions? Can a good recompiler even detect the branch and all of its alternatives without executing the code?

 

[Aninhumer]

Really? What was asked:

"The question is whether or not it will be possible to run regular linux apps on pandora."

From the context it looks more like he was asking about running binaries than recompiling source. So I think we're actually more on topic ;p

Hmm, rereading it's possible the question was "can I run x86 on ARM", but I think the answer in this context is "no, but compiling for ARM is easy so don't worry about it", rather than a multipage discussion about binary recompilers.
I just got the impression it was an honest newb question, rather than a dev question.