Apt Repositories For Pandora

[AireTamStorm]

Alright, it looks like we're all gravitating to everyone being happy now.

QUOTE
Of course there are impossible to recover problems that may possibly occur- but you're assuring everyone that this happens often. It doesn't. If it did, nobody would use it.

Don't get me wrong - I also use and like APT. But that's on a fixed harddisk, not a removable media. That's where I see the problems.I did use ipkg on My iPAQ 3600, 5500, and 4700hx. All very happily worked on removable media (CF, SD and SD). I installed apache on my SD card, and hosted my webserver on it for a few days when my server died, on an old 802.11b Wifi card.

However, that was the card mounted on /mnt/sd. The Union that I keep talking about makes the SD card transparent to the software on the system. When you remove the card and replace it with another, everything installed on it, including the modified database, go with it. If you take an out-of-box Pandora and union mount the SD card to /, then install 5,000 packages, delete 2000 of them, turn the device off and remove the SD card, when you turn the device back on, you get an out-of-box Pandora. One SD card should be used for a union, the other for data, music, ROMs, etc. Corruption of the data is prevented by using a Journaled file system. Windows can't read it? You can do all kinds of things to make it readable, like installing a filesystem driver, sharing the filesystem via Samba, or using FTP over USB, as was done with the GP2X. With Samba, you even get the benefit of being able to see the files on the Pandora without it being physically connected to that one computer (WiFi). Note that nobody /has/ to do this, but it is one very viable option.

QUOTE
Much appreciated. As with the two other projects I'm working on at the moment, once I have something to show for, we'll talk. Right now I've gotta learn how to create packages and maintain a repo. Just to confirm: The PM software on the Pandora /is/ APT, right?

Yes. As it looks like, the main OS the Panda will based is a specially enhanced xubuntu.
That is very good to know, as it is as cosurgi said- the Ubuntu ARMEL repo can be used to satisfy many packages.

 

[Squidge]

My problem with aptget is that i am a newbee to linux and i have had to reinstall Ubuntu 6 times in a week since i have been installing things that i guess i shouldnt have even tho the apps i had been installing are
Virtual machines are ideal for this kind of playing with Linux (and Windows). You play without worrying about breaking things for educational reasons. When something breaks, you roll back the virtual machine to "just installed" state and carry on. No reinstallation required. Of course, you need the harddisk space to actually do this, but those are so cheap these days. I use a VM to try new apps in the Linux and Windows VM's before applying them to my real systems.

ohh and as far as the package installer goes ,cant we have both ,or cant the important files just be kept in nand
and then only download the main program files without the bloat ? there must be a simple way to do this

Yes, you can have both (already discussed in this thread), and yes, the important and commonly used files will be kept in nand so apps don't get bloated (also discussed in this thread).

When you remove the card and replace it with another, everything installed on it, including the modified database, go with it. If you take an out-of-box Pandora and union mount the SD card to /, then install 5,000 packages, delete 2000 of them, turn the device off and remove the SD card, when you turn the device back on, you get an out-of-box Pandora.

Even though you say one card should be apps and another data, I suppose you could use two cards as along they are not in the system at the same time? So you could remove one card and install 5,000 packages, and then remove that card and insert another, and install a different 2,500 packages. Then depending on which sd card was inserted would result in a different array of installed packages? To change the cards live what would you do? umount? I've never played with file system unions.

 

[AireTamStorm]

When you remove the card and replace it with another, everything installed on it, including the modified database, go with it. If you take an out-of-box Pandora and union mount the SD card to /, then install 5,000 packages, delete 2000 of them, turn the device off and remove the SD card, when you turn the device back on, you get an out-of-box Pandora.

Even though you say one card should be apps and another data, I suppose you could use two cards as along they are not in the system at the same time? So you could remove one card and install 5,000 packages, and then remove that card and insert another, and install a different 2,500 packages. Then depending on which sd card was inserted would result in a different array of installed packages?Yes. So if you install a pygame app and pygame / python on one card, then remove it and install Quake 3 on another card, python does not get carried over. All deps that needed to be installed go with the card and therefore the card contains only what it needs. You can mount an "Apps" card like any other "Data" card. All changes to the root filesystem get recorded to the card rather than the NAND.

At this point, I'm suggesting it so that you do /not/ use the Package manager on the NAND, but rather, update the NAND through images and keep the applications on the SD card, which wouldn't use as many writes. It also allows you to install packages using the package management without having define specific directories, potentially getting the db out of sync. When you remove the SD card, the modified db goes with it.

 

[cosurgi]

So you could remove one card and install 5,000 packages, and then remove that card and insert another, and install a different 2,500 packages. Then depending on which sd card was inserted would result in a different array of installed packages? To change the cards live what would you do? umount? I've never played with file system unions.

Me neither, and I admit that it really sounds very inetersting, and like it is exactly what we need with Pandora. Install some games on one card, onstall some other games on another cards, swap the cards and voila! Just like a cartridge in commodore And works by simple swapping of them. Awesome!

That is very good to know, as it is as cosurgi said- the Ubuntu ARMEL repo can be used to satisfy many packages.

@AireTamStorm when you start hosting stuff in your repo, please make sure that you provide deb-src acces also. Because you'll be hosting ubuntu binaries (I guess), so I'll use deb-src to automatically create debian binaries for me.

 

[AireTamStorm]

I've put together a UnionFS example:

CODE

luna@nayukios:~$ sudo apt-get install unionfs-tools
[sudo] password for luna:
luna@nayukios:~$ mkdir unionfs_example
luna@nayukios:~$ cd unionfs_example/
luna@nayukios:~/unionfs_example$ mkdir layer
luna@nayukios:~/unionfs_example$ mkdir union
luna@nayukios:~/unionfs_example$ echo "Text files are cool." >> example.txt
luna@nayukios:~/unionfs_example$ cp example.txt union/
luna@nayukios:~/unionfs_example$ cp example.txt layer/
luna@nayukios:~/unionfs_example$ echo "So are Unions." >> layer/example.txt
luna@nayukios:~/unionfs_example$ cat union/example.txt
Text files are cool.
luna@nayukios:~/unionfs_example$ sudo mount -t unionfs -o dirs=./layer/=rw:./union/=ro none ./union
luna@nayukios:~/unionfs_example$ cat union/example.txt
Text files are cool.
So are Unions.
luna@nayukios:~/unionfs_example$ echo "And this is why." >> union/example.txt
luna@nayukios:~/unionfs_example$ sudo umount union/
luna@nayukios:~/unionfs_example$ cat union/example.txt
Text files are cool.
luna@nayukios:~/unionfs_example$ cat layer/example.txt
Text files are cool.
So are Unions.
And this is why.
luna@nayukios:~/unionfs_example$



Coolest thing is, the Union is (as I'm sure you noticed) folder based rather than device based, so it works regardless of Filesystem.

EDIT: Changed mount command.

 

[cosam]

The Union that I keep talking about makes the SD card transparent to the software on the system. When you remove the card and replace it with another, everything installed on it, including the modified database, go with it. If you take an out-of-box Pandora and union mount the SD card to /, then install 5,000 packages, delete 2000 of them, turn the device off and remove the SD card, when you turn the device back on, you get an out-of-box Pandora.

If this works as well as it sounds it will, it really would be the business.

@AireTamStorm when you start hosting stuff in your repo, please make sure that you provide deb-src acces also. Because you'll be hosting ubuntu binaries (I guess), so I'll use deb-src to automatically create debian binaries for me.

Yep, deb-src would be great for the stuff where the sources are available. I should imagine it wouldn't be too hard to at least partially automate conversion from PXML/zip archives to .deb packages for any software that doesn't come with source.

Any thoughts on integrating PXML into the .deb packages, so that the standard menu system can be used? The only issue I can think of there would be that PXML uses relative paths and, as I'd assume the .debs would be installed under /usr/bin, /usr/lib, etc. you probably don't want to keep them with the executables. And for non-PXML-aware armel packages, how about a .desktop -> PXML conversion tool?

 

[Squidge]

I've put together a UnionFS example:

So basically, when an sd card is present and mounted, it overlays the repo db normally stored on nand with the db on the sd card, along with all the apps. Once you remove the card, the db reverts to the nand one. Considering the fact that the db is only ever accessed when you install, update, or remove a package, I think this will work.

Is it not possible to have both the nand and sd databases in use at the same time? I'm assuming not, as they will be seperate files, but maybe thats a apt config thingy? If you had two databases on nand, and one of them was called the same nand on the sd card, which would take precedence? (I'm assuming one of them will be unavailable due to file collision avoidance).

 

[Etinin]

Can apt even handle two databases?

 

[bigtruck]

Can apt even handle two databases?

It wouldn't have to. That's how union mounts work. When the SD card isn't present, apt would use the database on the Pandora. When it is present, it would use the one on the SD card, which is where all the packages would be installed (assuming a / union mount). Take the SD card out and there's no trace of the other database or any changes to the system at all.

I certainly do like this idea. It would simplify the use of existing debian armel packages.

 

[AireTamStorm]

I've put together a UnionFS example:
So basically, when an sd card is present and mounted, it overlays the repo db normally stored on nand with the db on the sd card, along with all the apps. Once you remove the card, the db reverts to the nand one. Considering the fact that the db is only ever accessed when you install, update, or remove a package, I think this will work. Yep. Thus I'm advocating this approach. We could even have a directory on the SD card ( /mnt/sd/pandora ?) rather than using the root of the SD card for the changes. All of this can be automated via script.

Whoops, my mount command should have been:
CODE

luna@nayukios:~/unionfs_example$ sudo mount -t unionfs -o dirs=./layer/=rw:./union/=ro none ./union

Otherwise if the file does not exist on the layer path, the orignal file will be written to. This way, even without the original file on the layer, it will be created there. No writes happen to the underlying union/ .

 

[Creature XL]

I've put together a UnionFS example:

So basically, when an sd card is present and mounted, it overlays the repo db normally stored on nand with the db on the sd card,

Wouldn't APT then install libc, libstdc++, libSDL*, libGL*, and all the other standard libs which are on the NAND on the SD itself? Because the apps which get installed on the SD will most probbably need some of these and APT can't know they are already there because the SD overshadows the NAND-ATP-DB?

 

[Squidge]

Wouldn't APT then install libc, libstdc++, libSDL*, libGL*, and all the other standard libs which are on the NAND on the SD itself? Because the apps which get installed on the SD will most probbably need some of these and APT can't know they are already there because the SD overshadows the NAND-ATP-DB?

Hence my "is it possible to have 2 databases" question, but all I got was "It wouldn't have to", maybe I should have rephrased it better. Yours is perfect. If this is case, it would seem that these redundant libraries would take up more room than the few libs the apps would require seperately themselves.

AireTamStorm: I assume the [ b ] and [ /b ] are bbcode that failed to work?

 

[bigtruck]

Wouldn't APT then install libc, libstdc++, libSDL*, libGL*, and all the other standard libs which are on the NAND on the SD itself? Because the apps which get installed on the SD will most probbably need some of these and APT can't know they are already there because the SD overshadows the NAND-ATP-DB?

Hence my "is it possible to have 2 databases" question, but all I got was "It wouldn't have to", maybe I should have rephrased it better. Yours is perfect. If this is case, it would seem that these redundant libraries would take up more room than the few libs the apps would require seperately themselves.

*Slaps self* Doh! Yeah, I guess I had a bit of a brain fart there. Now I see what you're getting at, and I'm curious as to the answer. I suppose one possibility could be some sort of script to combine the two databases when mounting, but that might be messy. At worst you'd have an extra copy of the NAND stuff on the SD card, but then I suppose you might as well just boot directly from the SD card.

 

[cosam]

I think the idea is that when the union is first created, there's no database on SD, so the one on NAND is visible. The moment the database on the union is written to, the updated version would be written to SD, as it has presidence. So a second database wouldn't be necessary - the original would return as soon as the SD card is unmounted.

 

[bigtruck]

But then what do you do when you update the packages on the NAND after you already have a bunch of stuff installed on the SD card? Would you have to fix the database on the SD card by hand? Or I suppose you could never update the NAND from the day you take it out of the box, but that's certainly a drawback (albeit perhaps a minor one, depending on the way you want to use your Pandora).

 

[Creature XL]

thanks SteveM for clearing this up. Sounds good AFAI understood the unionFS.

But then what do you do when you update the packages on the NAND after you already have a bunch of stuff installed on the SD card? Would you have to fix the database on the SD card by hand? Or I suppose you could never update the NAND from the day you take it out of the box, but that's certainly a drawback (albeit perhaps a minor one, depending on the way you want to use your Pandora).

Yepp thats a problem. But what about this solution:

You insert the SD (with db) and do the firmware update. I now assume that the old files get overwritten ON THE NAND and the db on the SD is updated. Drawback: you have to do the firmware update which each SD card. This should work as long as 1.) my assumtion is correct and 2.) the updates of the firmware dont introduce new files

 

[AireTamStorm]

Hence my "is it possible to have 2 databases" question, but all I got was "It wouldn't have to", maybe I should have rephrased it better. Yours is perfect. If this is case, it would seem that these redundant libraries would take up more room than the few libs the apps would require seperately themselves.
When APT is called, it loads the database and creates a lock on the database. It does its thing, commits the database to disk, deletes the lock file, and terminates. I'm not sure I get what you mean by two databases- if what you mean is, will the original database take effect once the (in our case) unioned database is removed, then yes. When APT is called again, it reads the database from disk, which would be whatever copy is "on top".

AireTamStorm: I assume the [ b ] and [ /b ] are bbcode that failed to work?
Yes, I removed it. Code tags don't allow highlights I guess?

Edit: I'll make a quick example showing APT do its thing on UnionFS, give me a few minutes.

 

[cosam]

Aah, I got the wrong end of the stick with the dual database thing - I see where you're going with that now. Updating the NAND then remounting SD over it would indeed be a problem. You'd either have to, well, not do it, or merge the updated database with it's SD counterpart. The latter should be doable, possibly even automatically, but I'd have to look deeper into the database format to tell you exactly how.

 

[BookTracker]

thanks SteveM for clearing this up. Sounds good AFAI understood the unionFS.

But then what do you do when you update the packages on the NAND after you already have a bunch of stuff installed on the SD card? Would you have to fix the database on the SD card by hand? Or I suppose you could never update the NAND from the day you take it out of the box, but that's certainly a drawback (albeit perhaps a minor one, depending on the way you want to use your Pandora).

Yepp thats a problem. But what about this solution:

You insert the SD (with db) and do the firmware update. I now assume that the old files get overwritten ON THE NAND and the db on the SD is updated. Drawback: you have to do the firmware update which each SD card. This should work as long as 1.) my assumtion is correct and 2.) the updates of the firmware dont introduce new files


guys it probably works like this:

nand - libfile.bla
sd-nothing
when u set the union(insert your SD),
sd gets libfile.bla
lets say libfile.bla2 comes out,
it should overwrite the libfile.bla, which really means that it copies libfile.bla2 to the sd card, and leaves libfile.bla on the nand.
so like everyone said, with the sd you got your system, without it... out-of-the-box pandora.

so in other words, i suppose if u got 2 files with the same name, the one on the sd would be favored.
(so lets say if you insert another sd card, and want latest verions, you would have to update again. but is that a problem? )

this is how i think it would work(how ti usually does, i guess...), correct me if wrong.

 

[bigtruck]

guys it probably works like this:

nand - libfile.bla
sd-nothing
when u set the union(insert your SD),
sd gets libfile.bla
lets say libfile.bla2 comes out,
it should overwrite the libfile.bla, which really means that it copies libfile.bla2 to the sd card, and leaves libfile.bla on the nand.
so like everyone said, with the sd you got your system, without it... out-of-the-box pandora.

so in other words, i suppose if u got 2 files with the same name, the one on the sd would be favored.
(so lets say if you insert another sd card, and want latest verions, you would have to update again. but is that a problem? )

this is how i think it would work(how ti usually does, i guess...), correct me if wrong.

Yes, that would work fine, but that means you'd be wasting space on the SD card. Every time a package in the firmware is updated, the new package would be installed on the SD card. That's what I meant by never updating the NAND. 256MB at most (more or less depending on compression) isn't really that huge of a deal considering how cheap 16GB SD cards are now, but nonetheless it's still a waste of space.