Debian In An Extend

[SomeGuy99]

Right, I think I get the problem...?

The way I understood it, if you mounted a RW aufs file ontop of existing aufs mounts, then installed onto that entire stack, the the new files only get put onto the RW one. So if that file was completely brand new, and empty, then you have effectively isolated the files that are installed when you do apt-get or whatever?

I thought that you could then extract those files into a folder, then ISO the thing up. Whack a PXML and icon on the end and... Bob's your uncle?

The problem you describe is that programs are chucking files in locations of the filesystem which are difficult to capture? Like usr/share?

 

[Stuckie]

You can't add an AUFS on top of an existing AUFS - you can only "daisy chain" them at mount time ( but I think you probably got that earlier... )
But yes, you're correct.. if you daisy chained from mount time, your files would only be in the RW AUFS and your RO AUFS systems would not see them.

And as said, a standard program will generally splatter it's files everywhere!
I'm trying to be cautious with AUFSing everything as it doesn't always come off cleanly, so you need to be careful in what you bind. The /usr hierarchy in particular is a minefield, and that's generally where all application data goes! AUFSing in there is very tricky if you want to get it back off.

The reason it's tricky to get things in /usr unmounted is that because everything goes in there, there's usually something attempting to read or write in there as you're trying to unmount, and the system will correctly give you a bollocking for it and stop you from doing so.

 

[SomeGuy99]

You can't add an AUFS on top of an existing AUFS - you can only "daisy chain" them at mount time ( but I think you probably got that earlier... )
But yes, you're correct.. if you daisy chained from mount time, your files would only be in the RW AUFS and your RO AUFS systems would not see them.

And as said, a standard program will generally splatter it's files everywhere!
I'm trying to be cautious with AUFSing everything as it doesn't always come off cleanly, so you need to be careful in what you bind. The /usr hierarchy in particular is a minefield, and that's generally where all application data goes! AUFSing in there is very tricky if you want to get it back off.

The reason it's tricky to get things in /usr unmounted is that because everything goes in there, there's usually something attempting to read or write in there as you're trying to unmount, and the system will correctly give you a bollocking for it and stop you from doing so.

Isn't it easier to do with our chrooted jail than a live system? I mean you have a RW overlay on that debian chroot, then you quit the whole thing and see what you captured? Convert or copy to PND?

You have to understand how new I am to this stuff. Sometimes I have to read your posts a few times to make any sense. Exciting though!

Can't you just kill the read/write attempts by quitting the chroot jail?

 

[Stuckie]

Oh, I see what you're doing now! You sneaky bugger!

Essentially you're wanting to do this:
Bind Debian.
Bind a Root Overlay over the Debian chroot, marking the Overlay as RW and Debian as RO.
Install stuff.
Exit the chroot and clean up - which should be safe for the most part, though I've noticed the /Angstrom and /media binds seem to stay active.. might have to look at them again.
Remount the Overlay in Angstrom and see what you picked up ( not necessarily meaning mount it over / .. just to a random folder to see what's in it )
Try and clean it a bit ( bunch libraries and binaries together, see if it's spread any crap all over /usr/share that'll need dealt with - this is the tricky part )
PND the remains.
Test.
???
PROFIT!

*cough*

That would work.. that's also somewhat insane.
You would need to be careful with /etc however.. as the current script sets that up as an AUFS bind from Angstrom/etc to Debian/etc .. but for what you're wanting to do, you'd possibly be better off writing some scripts and doing bits of it manually so you've better control over what's going where.

 

[SomeGuy99]

I didn't realise there were limits to the insanity in this thread.

It's a shame that you'd have to do this manually... some kind of Apt to PND making machine would rock.

Can you just not bind /etc on the RW overlay, or do all the daisy chained ones have to share the same bindings?

Still, there is option 2: Ignoring the PND factory dream, how about these tiny little loopbacks with single programs on them be put into one of the bigger ones? Even a brute method force of copying files over and rebuilding? Then they could be shared! This is what happens with big stuff in Puppy (wine/gnome/chromium). People use a utility to combine them together into big mega-squashfs mounts.

Or how about this for madness:

A text box and a button on the frontend. In the textbox you type the name of a program in the Debian repos. The program then does the entire thing (mounting, chroot, unmounting etc), including apt-install with no input. Some kind of error trapping is saved to a log file, and then a popup says 'yeah that installed' or 'sorry, no such program' or whatever. Afterwards you have a nice little loopback file with the title of the program, and the program on it?

 

[Stuckie]

You'd just need to bind /etc manually with something like this:
sudo mount -t aufs -o dirs=/etc=ro:/tmp/debextend/etc=ro:/tmp/overlay/etc=rw -t aufs /tmp/debextend/etc
Doesn't that look like a fun command?

As for the slamming together of extends into a bigger extend, that's essentially what my patching process was going to involve.
Copy ExtendA's contents into ExtendB.
If need be, Copy ExtendA and ExtendB contents into MegaExtendC ( cause you've ran out of space in B )

Trying to keep all this generic enough that it won't matter what's in the Extend - could be a chroot-enabled filesystem, a root overlay, a home overlay, or just some random file you want to stash stuff!

Oh, and I forgot to add support in for this, but Extend files can also be used as a swap file.

Your idea of stuffing a single program into a loop back file would work.. then smash them together later into a group of them to keep under the 7 loop limit.
You couldn't directly take that and turn it into a PND though, you'd need to do some fiddling depending on where it's installed it's data, but I think you're wanting to keep this to within the chroot extend stuff?

 

[SomeGuy99]

I think we should take the PND generating machine idea off the table for now. Save that for later?

It seems if a frontend is made, a nice screen for merging extends into mega-extends is in order.

Say for example you have the folder /appdata/fluffykittenabuse/addons and in here you have a bunch of files with various things installed. Let's say one has Gnome on, another has a massive collection of command line programs, another has KDE etc. Some of these you will have made yourself, and some you will have downloaded from this forum or whatever, those ones other people have made.

So our mounting stack looks like:

Base OS - Debian/Ubuntu base in squashFS
User overlay - RW portion. Not just home folders, but anything else, including installed apps.
Home overlay - RW If you want to split home and user installed software. Puppy doesn't bother with this.
Software overlay - Load an individual squashfs addon file. Users will want to merge them into mega-extensions.
Installer overlay - An alternative to User/Home/Software and used to quickly install a single application and dump into the addon folder. It gets converted into squashfs.

So yeah, a screen to smash together extensions into bigger ones would be brilliant. We could take the addon/software extensions and merge them into big ones. We could take that and smash it into the user extension, and then only use two.

In short: We need a screen for kitten fusing.

 

[Stuckie]

Last post before I crawl to bed for a few hours:

I see exactly what you're up to now, and this is quite possible.
The tricky bit is ensuring the AUFS binds all get concatenated in the one go so they're daisy-chained properly.
I also think that the GUI should be cross platform to an extent... have you tried copying the contents of Extend files on a Pandora? it's agonisingly slow! Smashing together Extends on a Pandora would be a task only for the patient.

I have two Ubuntu Extends ready just now.. I haven't tested them, but they're uploading to the Extend dump at the moment. There's a 1gb and 2gb variant, and they'll be up in about half an hour or so, if you want a fiddle.
I nicked the rootfs from here: https://wiki.ubuntu.com/ARM/RootfsFromScratch as I wanted something quick to test ( that also seems to do crazy stuff with QEMU ... worth a look! ) but I haven't had time to look for it, so I offer them as untested. Should work. Don't see why they wouldn't. No idea what's in them though!

 

[SomeGuy99]

Yeah, I imagined that kitten fusing would be incredibly slow. We'll need to be using the best equipment available for this!

I'm off to bed too. Thanks for the stimulating exchange sir. I hope my wild suggestions didn't make any heads spin - I think this could be dead usable for people if it happens. At the very least, I'll write some decent documentation on how to use it and you can include this as in-program help?

I'll test the Ubuntu tommorow and let you know how it goes! I'm assuming we have no X, but I can launch all the including programs and stuff?

This would be the easiest, coolest and most deliciously insane way to run Ubuntu on the Pandora... as an app! I'm just hoping Paddy won't be expecting child payments or anything.

 

[skeezix]

Sick sick (heroic awesome) man

Aside and irrelevent: You know, you could just slap Debian onto SD, and use the pandora kernel, and just boot it No one says you have to chroot run it, but so totally awesome to do so

I don't think I've seen anyone put this much abuse onto a union filesystem before; you sir, are a giant among.. those who live in island empires polotting to overthrow the world

jeff

I can give you one good reason why I prefer this: Copying a linux filesystem to an SD card takes hours, because of all the tiny files. It takes even longer on the Pandora itself... it was still running overnight!

Am I an evil henchman then?

You have no idea how many thousands of times I set up a linux firmware on SD (multiple times _per night_ due to SD corrupting problems in early kernels), and I can tell you _Exactly_ how long it takes to extract an OS onto ext2 and ext3 filesystems on SD. And you know what -- it was about 10 minutes (ext2 was win though, since it did the final sync about 5x faster than ext3).

*woot*

Now, FAT on the other hand, would be bad

jeff

 

[skeezix]

This thread is epic, boys And I dont' say that too often

jeff

 

[Stuckie]

You have no idea how many thousands of times I set up a linux firmware on SD (multiple times _per night_ due to SD corrupting problems in early kernels), and I can tell you _Exactly_ how long it takes to extract an OS onto ext2 and ext3 filesystems on SD. And you know what -- it was about 10 minutes (ext2 was win though, since it did the final sync about 5x faster than ext3).

*woot*

Now, FAT on the other hand, would be bad

jeff

Think he means on the Pandora itself... and I can confirm that, it seems to take an absolute age to copy files between SD cards or on the same one.. granted my SD cards are ancient 2Giggers so they're probably not the speediest of things to begin with, but it's still quite mad!


Anyway, I've had a look at the Ubuntu images now.
1Gb: http://www.stuckiegamez.co.uk/apps/pandora/SimpleDev/ubuntu-karmic-arm.1GbExtend.zip
2Gb: http://www.stuckiegamez.co.uk/apps/pandora/SimpleDev/ubuntu-karmic-arm.2GbExtend.zip
It's Karmic Koala by default, and I noticed that mangling it's /etc causes it not to work.. so ExtendUtils has been updated again to not blanket mount /etc and to hardlink mtab and copy in /etc/resolv.conf.
Here's some fun I noticed you can do though!
When you get to your terminal, type: startlxde
Now stand back

Getting back out of that can be fun, however... but you should be able to do CTRL+C in the terminal and it should dump you back to XFCE - though perhaps with a buggered up theme.

Exciting Stuff!

 

[SomeGuy99]

Woah. I just did that... and it was kind of sick and wrong! My XFCE panel was running behind the LXDE one, so I switched it to the top. :lol:

I noticed this image has Gnome and Qemu. Very exciting indeed B)

 

[Stuckie]

Oh it's actually underneath the LXDE one?

Now that's interesting... open your Debian apps with the LXDE panel, and your Angstrom apps with the XFCE panel!
And yes, that image seems to have a lot of crap scattered about it.. so I think I will have to run through the guide and do a clean one.

 

[SomeGuy99]

Oh it's actually underneath the LXDE one?

Now that's interesting... open your Debian apps with the LXDE panel, and your Angstrom apps with the XFCE panel!
And yes, that image seems to have a lot of crap scattered about it.. so I think I will have to run through the guide and do a clean one.

Ah, the Qemu binary is to load ARM stuff on a x86 machine. Interesting!

 

[paddy]

Tested lots of games pushing polygons but none work as they are opengl,also tested Amsn,gnupaint and audacity and they work perfectly,audacity is nice and i can confirm the mic works and sounds very nice and loud.

I was using the 2gb extend and found that when installing stuff it takes up some amount of space,must be all the packages also when trying to remove those packages it asks to install more data just to remove stuff,no good when you have run out of space halfway through installing totem player that took you 2 hours.

anyway advice would be to view the installed file size carefully and keep count of what you already have,now my question would be . . is there anyway i can get to that audacity install and move it over to nand or make a pnd ? seeing as its already compiled and nstalled would thsi not be a bit more easy than having to port over to the pandora.

Even if i could move the folder over to nand and run it like the way the n64 emulator and one of the amiga emus are currently it would still be a start then someone or myself could pnd it at some point.

 

[Stuckie]

Unfortunately, that's the way of the system... 2Gb is quite small for a desktop distribution for the most part.. 4Gb is a good sweet spot ( I know this as my system fits comfortably in 4Gb and I've a fair amount of crap as well! )

As for moving stuff into PND files.. this was discussed in here in great length.. it's not easy.
SomeGuy has some ingenious ideas of splatting an overlay over the filesystem before you install stuff and literally catching it in the overlay, so that when you unmount, only what was installed is in there and can be more easily split up and dealt with.

If a program depends on libraries, they need packaged up, and if a program spreads stuff all over /usr, you've a bit of a problem as you need to do some creative solutions to get that stuff working cleanly when inside a PND. If you followed what I did with GIMP in Developer's Corner ( the packaging Angstrom Apps as PNDs thread ) you'll see just what kind of fun you're in for!

It's not quite as easy as just copying files over into a PND and running it.. I wish it were.. though having said that, there are times when you can get away with it

 

[SomeGuy99]

Any luck getting a chroot X going?

I had some further ideas for those temporary overlays becoming software addons. Basically, why don't we append a text file to the end like with PND? What's in the text file? Well, a description of the contents! This would make managing the files a lot easier.

Say we do the auto-mount-aptget-exit-unmount thing... why not use the original parameter you supplied to the program, which was then given to apt? For example 'VLC' or maybe even capture the package info and put that in there too.

If you were to smash together one or more into a super-extend, you could combine the text for each, and effectively get a nice summary/catalogue of what's inside. We can have this displayed when browsing in our gui perhaps.

 

[Stuckie]

You wouldn't be able to use those extend files as PNDs as you wont be able to unmount them cleanly.. you'd only be able to mount them in a chroot so that when the chroot dies, the mount points can be cleaned up.

As for chrooted X, haven't had a chance yet, sorry.. not long back from work and rather hungry!

 

[SomeGuy99]

You wouldn't be able to use those extend files as PNDs as you wont be able to unmount them cleanly.. you'd only be able to mount them in a chroot so that when the chroot dies, the mount points can be cleaned up.

As for chrooted X, haven't had a chance yet, sorry.. not long back from work and rather hungry!

Yeah, i'm forgetting the PND generating stuff for now. This is more along the lines of puppy SFS files. The difference here is that Puppy doesn't have a package management system and can't benefit from the Debian repos (there's a spiritual successor called Woof that can though, I'm also reading that Puppy 5 is Debian/Ubuntu based).

You work pretty late then, crikey!