GP2X The Quest For The Open2x Firmware ..


torpor

hack hack hack, the little machines fight back
Joined
Oct 21, 2005
Messages
2,475
Location
vienna, austria
Website
w1xer.at
As discussed with the gang (DJWillis mostly) on #gp2xdev, I would like to get the OPEN2X project rolling along to where its possible for us to build our own, tweaked, custom firmware. Yes, thats right: all the way from building our own 2.4.25 kernel, modules, libs, and apps, rolling it right down on into a tight set of gp2x*.img files which can be used to safely and reliably boot the GP2X, and run all your favourite apps, still compatible with the factory firmware.

So let me just report what I have found so far:

1. The OPEN2X Project. In previous weeks, there was quite some difficulty for OPEN2X hackers to get sync'ed with the Sourceforge SVN repository; this problem now seems to have gone away, and its possible to get yourself a full copy of the OPEN2X tree, such that it is in its current state, build it, and use it.

2. The Toolchain. OPEN2X comes with scripts to build the toolchain, and then the libs.

The toolchain consists of:

binutils-2.16.1
glibc-2.3.5
linux-headers-2.6.12.0
gcc-4.0.2

This is all fine and dandy: the script runs, you get a 4.0.2 gcc, you can then use it to build libs. The one problem:

**The current gcc-4.0.2 compiler cannot be used to build the open2x linux-2.4.25 kernel**

However, if you don't wish to do kernel work, this combo works just fine.

3. The Kernel. OPEN2X has, of course, a copy of the GPH-provided kernel (linux-open2x in the svn tree), and I can confirm that this kernel can be built easily enough if you have arm-linux-gcc 2.95.3 onboard, and you can boot it, and the GP2X will run along just fine with it .. in fact I have been kernel hacking on the GP2X for the last few months, and it works just fine, except for one rather cantankerous gotcha, more on that later ..

4. The Libs. Included with the OPEN2X repository is a grand archive of libs - SDL (of course, which I think includes the latest and greatest hacks/patches from paeryn to exploit the GP2X acceleration features..), as well as all the standards: libtiff/mikmod/mpeg/SDL_*/etc. The only thing I want to say about this is that I've added cairo/libsvg/libsvg-cairo and its ancilliary libs (fontconfig/glitz) to the libs/ tree, so that we can very easily use SVG 'out of the box' in the OPEN2X firmware - more on this later (hint: for a new menu).

5. There is a huge problem with all this: You cannot use the same compiler to build the kernel as you use to build the libs. The toolchain dilemna is such that you have to have multiple toolchains onboard to make a completed OPEN2X firmware, and this is wraught with issues: 2.95.3 is 'ancient' (but still works fine), so its not really fun to build all the LIBS with it, whereas 4.0.2 will not compile the linux 2.4.x series of kernels, etc.

I have recently run into an even thornier issue with this problem in my efforts to port MidiShare to the GP2X. Right now, MidiShare works just fine with my custom kernel, built with 2.95.3, and MidiShare apps can be written and run just fine on the GP2X: BUT because MidiShare consists of a kernel module+libs, it is not possible to mix and match compilers on projects using MidiShare: this means you cannot use gcc-4.0.2 from the OPEN2X toolchain to compile against MidiShare, thus: none of the 4.0.2 libs are available to a MidiShare hacker. The reason: the dreaded soft float versus hard float ogre.


So .. where does this leave us? It means that, in order to be 'complete', the OPEN2X project needs another compiler in the toolchain: one which can be used to build both the kernel, and all the libs, in one fell swoop.

My hacking so far has led me to conclude that the following toolchains, pre-built and downloaded from mysterious places all over the 'net, do not work to build the OPEN2X kernel: GCC3.0/GCC3.2.3/GCC3.3.1. I have not built these toolchains myself and so I do not believe that I've necessarily gotten these toolchains right for a kernel build .. for me the most optimal solution would be to simply get GCC3.4.4 (the highest version of GCC which can, apparently, 'safely' build the linux 2.4.x series of kernels) added to the toolchain script .. if someone has already built it for the GP2X. I 'feel' like someone has, I've heard about it, but I can't work out who it was..

Now, if you're still following along with me, the point is this: whats the point? I know that there are other toolchain hackers in this community who have also worked on similar issues: oopo/theoddbot/djwillis. What success have you guys had with building a toolchain that can boot a linux 2.4.x kernel on your GP2X?

Are you guys willing to throw your toolchain buildscripts into the OPEN2X mix so that we get either: a) one compiler to build all (kernel/libs/apps) even if its not the 'latest 4.x' whizbang, or B) two compilers easily integrated into the OPEN2X mix which produce compatible (i.e. *SAME* float specs) objects that can be mixed and matched.. so we can use 3.4.4 to build the kernel+modules, 4.0.2 to build libs+apps, and they work happily together ..

Ideally, one of you guys has already solved this issue and we can add a patch to the OPEN2X toolchain script .. but if you've got more details about the work you've done so far on this issue, please lets hear about it .. here, or on #gp2xdev (where there is much interest in getting the OPEN2X firmware buildable to the final point: gp2xkernel.img/gp2xfile.img, etc.)
 

Blah

Wanna Be Programmer
Joined
Dec 18, 2003
Messages
3,253
Age
33
Location
Oregon, USA
Website
Visit site
I'd kill myself compiling more than 3 toolchains in one day. Seriously.

Anyhoo, thats what I call dedication :)

Also you may see some Cairo work from me someday, I've had a few ideas bouncing around in my head about that one.
 

torpor

hack hack hack, the little machines fight back
Joined
Oct 21, 2005
Messages
2,475
Location
vienna, austria
Website
w1xer.at
Cairo works great on the GP2X .. paired with libsvg-cairo of course, it means that a whole world of great content can be produced very easily for game coders to use ..

But first we have to solve this toolchain issue. One compiler to build it all, I think, is whats needed ..
 

Blah

Wanna Be Programmer
Joined
Dec 18, 2003
Messages
3,253
Age
33
Location
Oregon, USA
Website
Visit site
I wish I could help with the toolchain issue, but right now I'm in no position :( My GP2X hasn't arrived yet, and I also need to buy a new hard drive so I'll have some space to set up a nice Linux based dev environment, right now all I can do is make the best of the stupid WinXP and the great-yet-inferior MSYS.
 

BradN

Member
Joined
Jun 22, 2003
Messages
577
Maybe there's some other way around the hard/soft float issue... what exactly is the problem? 2.95 always builds with hard floats? Object file flags (soft/hard) don't work correctly between them? Object files are just plain binary incompatible between the versions when using soft floats?

If it's the 2nd one, it might be possible to make a utility to hack up the object file a little so things like it better.
 

Blah

Wanna Be Programmer
Joined
Dec 18, 2003
Messages
3,253
Age
33
Location
Oregon, USA
Website
Visit site
It won't allow you to link stuff it thinks is hard-float stuff with soft-float stuff. So it bitches about it a millon times and then exits.
 

BradN

Member
Joined
Jun 22, 2003
Messages
577
Yeah, I was aware of that, but just wondering why soft float from 2.95 doesn't work with soft float from 4.x or whatever. If it's an issue where it only thinks it's hard float, but it's really soft float (and actually would be compatable otherwise), it seems that it'd be worth tracking down exactly how it determines it and finding a workaround that way. But if the soft float "methods" are different and incompatible, then I guess one's kinda screwed.
 

Blah

Wanna Be Programmer
Joined
Dec 18, 2003
Messages
3,253
Age
33
Location
Oregon, USA
Website
Visit site
It happened to me when I was compiling GP32 stuff with the last GCC 3.x version of DevkitARM. I forget exactly which GCC it was.
 

dwelch

Member
Joined
Jul 7, 2006
Messages
119
I have built gcc more times than I can count, from 2.95.3 to the present. I only have used newlib though and only build for C (not c++). I dont know any of the glibc or c++ issues, but assume that when building for linux you probably just put a stock glibc in there yes? Do the kernel mode drivers make the connection between generic C calls and the hardware? Soft/hard float is always a thorny subject, dont remember 2.95 being restricted. The .elf file format, if I remember right, marks hard or soft in the header, easily hacked with a utility if need be. I have an aweful lot of fpu experience with arm and gcc and quite frankly dont know what the fuss is all about, soft float should be the default, at one time I was well versed at this soft/hard float issue, unfortunately it has escaped me. If you follow the Keil page to find the eval version of their ARM tool, the arm demo download page also has a gcc 3.3.1 that is free do what you want with it. Dont know much about its build. I am linux and windows savvy, would prefer to do this on windows (dont start, I am a linux fan just prefer windows for this stuff right now) and recently worked thorugh building 4.0.2 on mingw (preferred over cygwin)(there is a msys/mingw related bug that prevents gcc building, it actually has to do with the soft float library). The soft float library used by pretty much every compiler I have tried is quite old, written by Sun. So gcc 2.x, 3.x, 4.x should all have the same soft float library calls, I would be surprised if they didnt. I would do everything you can to move to 3.x from 2.95.x. Why doesnt the kernel build on 4.x.x? Is that a gp2x or arm related issue or is that a Linus Torvalds issue and we should just wait until his crew figures it out? I assume the open2x gcc build script does not work on 3.x? I think devkitarm is in the 4.x range yes (gcc --version)? I for one would like to be ABLE to build the kernel and/or modified bootloader from source without having to jump through gcc hoops. The one binary I found for the modified bootloader was what made the difference between continuing with the gp2x and chucking it in a box with my gbas and other short lived gadgets...I am a HH person using the HH term I have seen in this gp32/gp2x forum, so you probably wont like me for that. Granted if I new more about how linux is implemented on this platform and could work around it as desired, I might find not have the desire to replace it.

Anyway if you can sort out my questions from comments and have any answers that would be great. There appear to be a lot of good gcc builds for arm out there, but just like Linux distros, each one has some persons or groups agenda tied to it. Anything built for the GBA is going to be centered on thumb mode and we need to center on arm mode (preferrably not have thumb in there). And there is the FPU preference (who is using one of those old FPUs with an arm anyway, the cirrus fpu was quite buggy, and arm was replacing it with a whole new instruction set the last time I looked at it (a few years ago). In theory if we build everything with the same compiler we wont have the hard/soft issue any more.
 

slurry

Still Fresh
Joined
Aug 17, 2006
Messages
3
Hi all,
I'm *very* new to GP2X having just got mine yesterday, please let me down gently if I'm talking crack.

Why is there (seemingly) no effort to port kernel 2.6 to GP2X??
This perplexes me somewhat. It would seem to solve part (if not all) of the toolchain dilema.

- the *complete* GP2X kernel 2.4 source is provided right?
- there arent any additional binary modules?
- the interface/games dont depend on the kernel do they?

Has anyone produced a diff with vanilla 2.4? Can we port the changes to 2.6? What is stopping this from occuring? It seems like a good thing to do for the future. Official firmwares would shurely make the switch too if the community did all the work already.

I am familiar with the openwrt build system and it is great, you just check it out and run make. The toolchain is downloaded and built followed by the kernel, then all the apps. Finally, a complete firmware (kernel + squashfs or jffs filesystem image) is spat out. Your also then left with a perfect development environment for further hacking. I'd absolutely love to see something simillar for GP2X and I gather this is the goal of Open2X? wouldn't kernel 2.6 be a great start?

Again, sorry if I've missed something obvious!

Dave
 

torpor

hack hack hack, the little machines fight back
Joined
Oct 21, 2005
Messages
2,475
Location
vienna, austria
Website
w1xer.at
i think the point really is just to 'complete' the open2x work thats been done so far as a distro building platform, whatever kernel is used, so we can start easily producing 'complete' firmware.img files .. i'm familiar (no pun intended) with openwrt as well, and of course there is a grand plethora of good examples for how to set up a decent linux-based toolkit+distro+apps in a fashion similar to that which you describe: checkout, shell-script, toolchain, distro, final firmware image .. this exact process is a goal of the Open2X makefiles so far, and it its pretty close except for the kernel build lacking its toolchain.

as for the 2.4->2.6 port, this comes up a lot .. sure, it would be 'nice' to port 2.6, and there are good reasons to do such a thing, but keep in mind for now that:

- 2.4 is still an excellent kernel. it is not 'old'. all gp2x users are using it just fine.
- we want as much RAM as we can get on the gp2x .. 2.6 > 2.4 for RAM requirements.
- 2.4 can still be set up to smoke a 2.6 kernel (at least on ARM)
- open2x is, in fact, 'almost' at that point where it will go through the stages you describe, from checkout to firmware.img, at least with what you can get out of SVN right now .. ( -* )

so .. given all this .. all thats needed is the discovery/configuration/inclusion of a compiler that can build the linux-open2x/ (2.4.25+gp2x_patches), and we can get it pretty damn tight. this seems like it would be easier than to attack a port of the 2.4.25 gp2x-patch to a 2.6 kernel right now (not that we might not want to do that once Open2X is 'process-complete' for 2.4...)

in other words, if someones' got a build-script for a 2.95.3-gp2x/arm-linux gcc that can be used to also build the open2x libs/distro, and slides right in with the existing 4.0 compiler thats already there (i.e. builds the libs/distro/app/firmware.img too), that'd give us a 'complete task queue' for open2x .. and with that, we can either continue to refine a 2.4.x kernel/build setup (not a bad idea), or use it as a platform to complete a 2.4->2.6 port.

that said, i'd encourage anyone coming to GP2X with a linux bent to check out the open2x SVN tree as it stands right now, and give it all a whirl.. its pretty close to being 'usable', and particularly now that there has been a lot of other frontier hacking on the 'special' stuff (squidges mmu/libthe2ndCPU.o's/hacks in app land/etc.) we maybe want to get things to the point where we can, in fact, boot an open2xkernel.img/open2xfile.img through u-boot .. ;)


*- maybe other Open2X hackers are ahead of the curve on this - i.e. SVN is being ignored in favour of rapid progress ..
 

DJWillis

Random GP32/GP2X/Wiz & Pandora Moocher
Joined
Jul 20, 2003
Messages
1,470
Age
44
Location
Bath, Somerset, UK
Website
www.distant-earth.com
torpor,

I have just been looking through my tool-chain scripts from a long while ago.
Just noticed that I seem to have a set of 2.95.3 build scripts I tweaked (tied to the correct gLibC etc.) that I think I used at some point.

I will use them to build up a chain over this weekend and see if it will build current SVN, do you want to mail me over your latest patch so I can try like with like?

As for hard/soft float well if you want to be evil you can just remark the libs as ARM is always going to be soft FP anyway ;) it's just the marking that is borked. Again, I have a remarking tool somewhere I wrote for the GP32, should work if I can find the code but I find this idea EVIL (tm).

As for the 2.4 > 2.6 debate. My gut feeling is to clean and stablise a like for like 2.4 1st and maybe move to the latest 2.4 ARM release.

Getting diffs for 2.4 against stock is easy. You will need to diff of 2.4.25 ARM (not Kernel.Org). The results are not to scary I guess but as it stands 2.6 would break things like MMU hack and the like and most lightly right now would mess up backwards compatability (don't forget about all the mapping of the upper 32MB that goes on and all the 2.4 exploits that are used to get at some of the hardware ;)). With that in mind I think getting an accepted 2.4 based Open2x image is a nicer goal right now. A lot of people have looked at 2.6 but backed off for now for some of the reasons above.

slurry, if you want to help on 2.6 feel free to join up :). Its a nobel goal, just a bit up hill from where we are ;).
 

torpor

hack hack hack, the little machines fight back
Joined
Oct 21, 2005
Messages
2,475
Location
vienna, austria
Website
w1xer.at
DJWillis: can we do it the other way around - you send me the 2.95.3 buildscripts, i try to fold it into the Open2X makefile, and we just try to push that compiler into Open2X?

i guess i should get out of bed, boot up my gp2x dev box, try to find you on IRC today before its too late ..
 

slurry

Still Fresh
Joined
Aug 17, 2006
Messages
3
Righto, thanks for clearing up the issues!

Regardless of the kernel, a totally free build system which builds a totally free firmware would be super-sweet!
In the short term, sticking with 2.4 is a very sensible idea, especially given the hacks out there. I'm going to grab open2x and have a play.

I do think that 2.6 is the future for this device though. I doubt that ram usage would really be an issue. We're quite fortunate to have opensource drivers for everything, pushing it all upstream would be awesome, and great for maintainability. ARM development all happens upstream (kernel.org) in 2.6 these days I think.

Dave
 

torpor

hack hack hack, the little machines fight back
Joined
Oct 21, 2005
Messages
2,475
Location
vienna, austria
Website
w1xer.at
slurry posted on Aug 19 2006 at 12:22 PM said:
ARM development all happens upstream (kernel.org) in 2.6 these days I think.

its true that there's a lot of activity with ARM at 2.6, but its also true that there's a grand history of ARM at 2.4, and thus no reason not to be able to balance between both.. for now, we should get 2.4 pinned all the way to open2xkernel.img/file.img, and get some sort of parity/stability vis a vis the factory (GPH) firmware .. this will make it more appealing/easier for the front-end of the spectrum (people who use their GP2X for gaming/app fun) to tag along as well ..

i'm going to have another swing at the compiler situation tonight/tomorrow, maybe i'll get some sort of interesting progress ..


BradN posted on Aug 17 2006 at 11:06 AM said:
Yeah, I was aware of that, but just wondering why soft float from 2.95 doesn't work with soft float from 4.x or whatever.

for me, its like this 'should' work anyway: open2x kernel built with gcc2.95.3, open2x kernel mods built with 2.95.3, libs and distro built with gcc4.x working with some libs also built with 2.95.3.

If it's an issue where it only thinks it's hard float, but it's really soft float (and actually would be compatable otherwise), it seems that it'd be worth tracking down exactly how it determines it and finding a workaround that way. But if the soft float "methods" are different and incompatible, then I guess one's kinda screwed.

there is a lot of confusion in float land, even among the gcc compilers ..
 
Last edited by a moderator:
Top