Discussion About Additions To The Pnd Format

[bzar]

Seriously, the Applications I add to the Pandora will be placed in NAND!? Everything I read before said you drop the files on your SD card and they are just installed. Seems you could fillup the NAND mighty quick. Not liking the sound of this.

Edit: or do you mean Non-PND software will have to be placed in the NAND? Which I guess would be an actual install where if I understand correctly the PND is an installation free way right?

Yes, non-pnd software in the NAND. PND's on SD cards.

EDIT: To clarify, nothing stops you from putting non-pnd software on sd cards. However, the ipkg package manager used for core system updates and such installs stuff to NAND. This package manager can also be used to install other programs/libraries/whatnot. If you use it, you'll be installing to NAND.

 

[Poem58]

Seriously, the Applications I add to the Pandora will be placed in NAND!? Everything I read before said you drop the files on your SD card and they are just installed. Seems you could fillup the NAND mighty quick. Not liking the sound of this.

Edit: or do you mean Non-PND software will have to be placed in the NAND? Which I guess would be an actual install where if I understand correctly the PND is an installation free way right?

Yes, non-pnd software in the NAND. PND's on SD cards.

Ah, thanks. And more begins to make sense. Thanks. So Hopefully most everything will be in PND format as I would hate to have to make decisions of what software I can have based solely on what's left of the NAND while I have multiple gigs of free space on an SD card! OK, I guess this is about all I can need to know right now. I'm sure once I read the middle pages of this thread I will be so lost there won't even be a point of asking more.
Thanks guys.

 

[viridior]

There are 3 generic ways you can add software/perform upgrades.

1. ipkg installs on NAND.
2. unionfs mount SD mapped over NAND. (You can then set the NAND OS as read-only and anything you install would be on the SD, doesn't use any more of NAND then what was with default, but then you also have to keep the SD in it's slot in order to keep your updates)
3. Install OS on SD and run completely from there, along with upgrades. (Also requires you reserving one of the SD slots for the OS).

For many instances in this discussion ipkg and pnd are interchangeable in the above methods.

 

[rv6502]

2a: I love how you guys who don't want the NAND to be touched suggest we use ipkg. Doesn't that install on the NAND?

no, I'm suggesting we use ipkg (or other already existing packaging systems) for stuff that would install on the system, and keep PND for stuff that does not install and remains on the removable device.

 

[dflemstr]

There are 3 generic ways you can add software/perform upgrades.

1. ipkg installs on NAND.
2. unionfs mount SD mapped over NAND. (You can then set the NAND OS as read-only and anything you install would be on the SD, doesn't use any more of NAND then what was with default, but then you also have to keep the SD in it's slot in order to keep your updates)
3. Install OS on SD and run completely from there, along with upgrades. (Also requires you reserving one of the SD slots for the OS).

For many instances in this discussion ipkg and pnd are interchangeable in the above methods.

You are absolutely correct, and it makes me wonder if it's actually worth it to modify the PND system in convoluted ways to make it possible for PNDs to provide files to the file system,

OR if it wouldn't be better to apply all of the knowledge we've gathered in this discussion to create an SD-overlay system (@polyroy, that would mean max 2 overlay mounts, which would be acceptable at boot) instead of a PND overlay system, so that IPKG becomes usable (not to ordinary folks, but maybe to more tech-savvy folks).

Let's think about it for a second...
There are a couple of scenarios for such an SD overlay system:

1. The SD is formatted with FAT32
   • Solution: Create an image file in the SD root that contains the overlay. So, first, the image file gets mounted, and after that, its contents get overlain.
   • Problem: How can we make the overlay expand/contract to fit it's contents?
   • Problem: FAT32 can only take file sizes of max 4 GB; workaround?
2. The SD card is formatted with ext*
   • Solution: Just create a directory in the SD root that contains the overlay.
3. There are two SD cards inserted, and both contain an overlay.
   • Problem: What to do? Which overlay is preferred and should we mount both?
4. There are services installed by IPKG that have to activate at boot
   • Problem: How do we do all of this dynamic checking and mounting before init (it's gotta be done in initramfs + kernel modules...)?
5. There might be folks who want to hotswap overlay SDs.
   • Problem: Can this be implemented, and how?
6. There will be a separate /home dir per overlay
   • Problem: People might not want that if they have more than one overlay (very unlikely). Workaround?

 

[polyroy]

You are absolutely correct, and it makes me wonder if it's actually worth it to modify the PND system in convoluted ways to make it possible for PNDs to provide files to the file system,

I think it should still be possible for static apps to have some kind of PND overlay system, this is the only way in which we can cleany "install/remove" without actually touching (read: permanently changing) stuff. If we just tell the user that the PND requires an overlay with the OS (whatever that may mean for the user), they will know that this integrated functionality is in no way permanent for the system. And for every other app that can run without integration, no such overlay needs to be present at all. It will just be loopback mounted and run from there upon application start.

We could even go as far as to merge the data inside the overlay permanently to the underlying filesystem. Some kind of 'permanent install' for those that want this. Removal is then also quite simple. Then we provide the best of both worlds. Two buttons: "Just run this app from this location" and "Install this app for permanent use", or something.

OR if it wouldn't be better to apply all of the knowledge we've gathered in this discussion to create an SD-overlay system (@polyroy, that would mean max 2 overlay mounts, which would be acceptable at boot) instead of a PND overlay system, so that IPKG becomes usable (not to ordinary folks, but maybe to more tech-savvy folks).

The OS is best delivered in a compressed firmware image, to save space on the NAND. An overlay on which the contents can be edited is then required anyway. Any extra overlays from SD cards can be activated by the user, either represented as entire partitions, subfolders or image files. Mounting multiple overlays should be no problem. In fact, I can imagine a user interface showing the branch stack (list of overlays), which allows the user to move them around (raise visibility/importance, with the topmost having priority). This would show the way the system is setup in a very obvious way, and not be too confusing for most users I guess.

1: If people want permissionless stuff in fat32 directories, let them It is not supposed to be a high-security device. And those that do value security should simply avoid FAT32 for overlays.

You should also consider multiple partitions on SD cards - one FAT32 for compatibility (phones, camera's, etc), the other meant for an overlay.

3: I'd vote for "first overlay found on slot 1 takes priority" or something. Let this be obvious, and user-configurable, such that users can also tell the system that file x on USB holds their preferred overlay.

4: I don't get what the problem is. rc-scripts can be set to run the first thing during boot after initrd set up aufs on root, which can then be extended with remounts when the details become known. If you worry about selecting the medium, the bootloader should handle that for you.

5: Still need to test. Running applications shouldn't be hindered (they just terminate themselves, or crash). Timeouts might be the bigger issue.

6: Selectable via the overlay menu I guess. Some sort of "warn me when a PND/overlay is activated which contains files in the following folders" list.

 

[dflemstr]

4: I don't get what the problem is. rc-scripts can be set to run the first thing during boot after initrd set up aufs on root, which can then be extended with remounts when the details become known. If you worry about selecting the medium, the bootloader should handle that for you.

6: Selectable via the overlay menu I guess. Some sort of "warn me when a PND/overlay is activated which contains files in the following folders" list.

4: BUT if the overlay contains entries for /etc/init.d or /etc/rc*.d, those services won't be started (which is what's supposed to happen if you install some IPKG packages that need that functionality).

6: I don't think that anyone will have more than one overlay anyways, so maybe this isn't an issue... But of course, this can be handled in an UI of some sort etc; just wondering how it would actually be implemented technically. AKA, how can we tell aufs to "not let that overlay provide files in that directory" or "redirect all writes to overlay X if they are targetted at a subnode of folder Y"; we really shouldn't have to go the ramdisk route again.

 

[God Ginrai]

Seriously, the Applications I add to the Pandora will be placed in NAND!? Everything I read before said you drop the files on your SD card and they are just installed. Seems you could fillup the NAND mighty quick. Not liking the sound of this.

Edit: or do you mean Non-PND software will have to be placed in the NAND? Which I guess would be an actual install where if I understand correctly the PND is an installation free way right?

Non-PND on the NAND. You won't find PNDs on the Angstrom repository. you'll go to the Angstrom repository if there is something you want that there isn't a PND of. The PND is just the easier way, not the free way.

EDIT: And dflemster, I think I like the current PND-overlay idea better than an SD overlay. It just seems like it would work better.

-God Ginrai

 

[polyroy]

4: BUT if the overlay contains entries for /etc/init.d or /etc/rc*.d, those services won't be started (which is what's supposed to happen if you install some IPKG packages that need that functionality).

The init.d scripts will run by prefix order. If you remount (and thus make available) the overlay in, say, S10 of the boot runlevel, which makes available services above S10, they will still be started right after the remount script finishes. S-scripts in other runlevels aren't even influenced by this at all, and most services only run when multi-user (2 and above) is encountered. The only issues are with init.d scripts less than S10 in the boot runlevel, but we'll just have to make sure that the overlay remounts are the first thing to happen, and the rest will follow.

6: I don't think that anyone will have more than one overlay anyways, so maybe this isn't an issue... But of course, this can be handled in an UI of some sort etc; just wondering how it would actually be implemented technically. AKA, how can we tell aufs to "not let that overlay provide files in that directory" or "redirect all writes to overlay X if they are targetted at a subnode of folder Y"; we really shouldn't have to go the ramdisk route again.

The only way is to either fully accept or deny those overlays, and if we think they do something they shouldn't, let the user know and have them decide. If the user just wants to access these things, they are already mounted in other locations. The user can still access (and possibly manipulate) those overlays to either inspect or fix the issue.

EDIT: And dflemster, I think I like the current PND-overlay idea better than an SD overlay. It just seems like it would work better.

You're going to end up with an overlay for the readonly (cramfs'd) firmware on NAND anyway. I can imagine the firmware in a regular filesystem to exceed that 512MB with ease. It doesn't matter where the overlay is going to be stored if we can just check that removal issues are nonexistant.

 

[Vorporeal]

Hmm... I'm getting kinda confused. Weren't we thinking of the overlays being a lot of symlinks that could be overlaid onto the root filesystem to allow for PNDs to be "integrated" with the root filesystem? If so, we wouldn't need to worry about resizing the overlay file for FAT32 - we'd just make one that's large enough to hold as many symlinks as we'd realistically need (10Mb?). Also, what's the deal with this single-overlay? I thought we'd have an overlay for each SD card, which would make hot-swapping them more plausible.

 

[dflemstr]

Hmm... I'm getting kinda confused. Weren't we thinking of the overlays being a lot of symlinks that could be overlaid onto the root filesystem to allow for PNDs to be "integrated" with the root filesystem? If so, we wouldn't need to worry about resizing the overlay file for FAT32 - we'd just make one that's large enough to hold as many symlinks as we'd realistically need (10Mb?). Also, what's the deal with this single-overlay? I thought we'd have an overlay for each SD card, which would make hot-swapping them more plausible.

Yeah, you're right, but now we've gone over to discussing root overlays, so that you can save files all you want all over the place → IPKG will be usable, but the files aren't actually stored on NAND but instead in an overlay on your SD card, and that overlay would need to be huge since it has to contain all those files that you "modify" in the root fs.

Oh, and we never actually talked about using one overlay per SD card or anything; if you pull out a mounted SD card and have PNDs mounted on it, things will break (and they will break with the current system, too; never pull out something without unmounting it first) so you'll have to unmount PNDs whatever you do, and if you unmount a PND its overlay is removed so then it's not a problem.

 

[polyroy]

if you pull out a mounted SD card and have PNDs mounted on it, things will break (and they will break with the current system, too; never pull out something without unmounting it first) so you'll have to unmount PNDs whatever you do, and if you unmount a PND its overlay is removed so then it's not a problem.

Why does it break? PND's are read-only, so they cannot be corrupted. The containing filesystem (if read/write) better be using something with a journal. Writable persistent overlays themselves can be corrupted, but that's the user's fault for not unmounting, and even then journalling will help recovery. And if a filesystem suddenly disappears, it would be a serious kernel bug if things crash, no matter how deeply stacked things are (unless the root FS is on an SD card, but AFAIK we could make root a ramfs with aufs stacking the real root on top, removing the crash). The only issue we need to deal with should be setting some sane timeouts.

Pulling a SD-card should, at most, make the system stall until the timeouts pass and some recovery is done (e.g. those filesystems are forcefully removed in the kernel).

Mind you, I haven't tested any of this, so now may be the time.

 

[dflemstr]

Yeah, my mind was still processing the SD overlay idea so for some reason, I thought of that when I talked about pulling out the SD card. You're right: unless you have applications that use files on the PND (directly or indirectly) and are in the iddle of a read, nothing serious should happen.

Mind you, I haven't tested any of this, so now may be the time.

I can't use QEmu because I've compiled my kernel in a convoluted way and have the virtualbox driver loaded (so my virtualization thingie is taken up by it) and have several VMs running (and won't create a nother one for several reasons). So, unless VBox suddendly gets ARM support, I can't test anything in the actual environment. It's of course possible to just chroot something together on my normal machine so that what I'll do once I've found the aufs modules in AUR (because I'll be using the git trunk version of aufs because I guess that that's what on the Pandora; not that it matters, probably).

The main issue will be that for every situation mentioned so far, you will need to move the root mount point, and it has to happen after init has started, so that'll be difficult to simulate without the actual Ångström environment since I won't modify my initramfs or reboot my PC a million times.

 

[Pickle]

I can't use QEmu because I've compiled my kernel in a convoluted way and have the virtualbox driver loaded (so my virtualization thingie is taken up by it) and have several VMs running (and won't create a nother one for several reasons). So, unless VBox suddendly gets ARM support, I can't test anything in the actual environment

Doesnt really have to be ARM with Angstrom, this test could be run on any arch just to prove it can function.

 

[dflemstr]

I can't use QEmu because I've compiled my kernel in a convoluted way and have the virtualbox driver loaded (so my virtualization thingie is taken up by it) and have several VMs running (and won't create a nother one for several reasons). So, unless VBox suddendly gets ARM support, I can't test anything in the actual environment

Doesnt really have to be ARM with Angstrom, this test could be run on any arch just to prove it can function.

Yes, but my point rather was that I don't have a VM available at all. Didn't really come out right but oh well....

 

[God Ginrai]

EDIT: And dflemster, I think I like the current PND-overlay idea better than an SD overlay. It just seems like it would work better.

You're going to end up with an overlay for the readonly (cramfs'd) firmware on NAND anyway. I can imagine the firmware in a regular filesystem to exceed that 512MB with ease. It doesn't matter where the overlay is going to be stored if we can just check that removal issues are nonexistant.

yes, but with the PND overlay, you don't need a file sitting on your SD card specifically because it's not formatted with an ext filesystem.

-God Ginrai

 

[polyroy]

EDIT: And dflemster, I think I like the current PND-overlay idea better than an SD overlay. It just seems like it would work better.

You're going to end up with an overlay for the readonly (cramfs'd) firmware on NAND anyway. I can imagine the firmware in a regular filesystem to exceed that 512MB with ease. It doesn't matter where the overlay is going to be stored if we can just check that removal issues are nonexistant.

yes, but with the PND overlay, you don't need a file sitting on your SD card specifically because it's not formatted with an ext filesystem.

I agree with your point though. PND overlays allow for way more flexible and user-friendly configuration. The only problem could be performance, so if we just provide a way to permanently fuse a PND with a filesystem (read: install), there is at least one certain way to deal with performance issues since no mounts would need to happen at all for those PND's. Then we benefit both from an on-demand nonpermanant solution (PND the way it has been proposed up until now), and at the same time allow for everything that IPKG-packages could do. We could even go as far as to allow the user to "convert" a PND to a pure IPKG-package, making the system use the underlying IPKG system for those permanently installed PND's (which allows for easy removal, etc.). Since we're using overlays, we can have the user choose which overlay the installed data would go to, in essence having the best of both worlds (PND's and the SD-overlay idea).

Edit: naturally you'd need to have some kind of "primary" overlay which keeps the IPKG metadata in tact (all of it). But the data itself can be spread among multiple overlays.

Edit2: Now that I think about it, most IPKG packages could probably be converted into PND's as well, and thus be overlaid instead of installed, should the user want this. This should be fun

 

[Bagera]

I don't know if I'm way off here but wouldn't one sollution be to force the user to declare a home SD? If there is none they will be asked upon inserting a SD. This way everything can just be stored there and if you want to swap "main" SD you have to copy that data first?

 

[God Ginrai]

EDIT: And dflemster, I think I like the current PND-overlay idea better than an SD overlay. It just seems like it would work better.

You're going to end up with an overlay for the readonly (cramfs'd) firmware on NAND anyway. I can imagine the firmware in a regular filesystem to exceed that 512MB with ease. It doesn't matter where the overlay is going to be stored if we can just check that removal issues are nonexistant.

yes, but with the PND overlay, you don't need a file sitting on your SD card specifically because it's not formatted with an ext filesystem.

I agree with your point though. PND overlays allow for way more flexible and user-friendly configuration. The only problem could be performance, so if we just provide a way to permanently fuse a PND with a filesystem (read: install), there is at least one certain way to deal with performance issues since no mounts would need to happen at all for those PND's. Then we benefit both from an on-demand nonpermanant solution (PND the way it has been proposed up until now), and at the same time allow for everything that IPKG-packages could do. We could even go as far as to allow the user to "convert" a PND to a pure IPKG-package, making the system use the underlying IPKG system for those permanently installed PND's (which allows for easy removal, etc.). Since we're using overlays, we can have the user choose which overlay the installed data would go to, in essence having the best of both worlds (PND's and the SD-overlay idea).

Edit: naturally you'd need to have some kind of "primary" overlay which keeps the IPKG metadata in tact (all of it). But the data itself can be spread among multiple overlays.

Edit2: Now that I think about it, most IPKG packages could probably be converted into PND's as well, and thus be overlaid instead of installed, should the user want this. This should be fun

I don't know. One of the nice things about the overlay was that you could take the PND off your card and you have no changes. The speed issue would only exist when you first mount the SD, right? (and maybe during unmount)

-God Ginrai

 

[Sphinxter]

Was /etc/mtab a file or a symbolic link to /proc/mounts?

NoA file, it seems to be generated. Not sure what does that though, this is a standard Debian install. Good find, but I can't see how much of the time this is eating. Anyway, wouldn't such generation be in the 0.2 seconds of user time? It still leaves about ~100ms of sys time per mount.

Perhaps someone can test this in more environments?

Any more advice on speeding this up?

Edit: I should really be ashamed that I answered false to a question which can only be true.

Maybe move /etc/mtab ~ and make a symbolic link to /proc/mounts like an embedded device and subtract the difference? Opening and closing a file is one of the slower things you can do.