Discussion About Additions To The Pnd Format

[WizardStan]

And WizardStan, it's never a good idea to make specific solutions. That just leaves you with a good system that is now overflowing with hacks to make it work for specific things. It's better to come up with a general solution. What's wrong with the Ramdisk UnionFS idea?

It's never a good idea to make absolute statements. Specific solutions can and do work all the time, if the number of use cases are small enough.
I just find the idea of an application being able to put itself into any random place it so chooses a little disconcerting, and I'm surprised that those who complained my idea of a user controlled autostart process ignored security are not complaining about applications that can effectively inject libraries and things entirely automatically by design.

 

[God Ginrai]

I can see PND being used by:
1. Users who have made homebrew games or emulators and want to deploy them.
2. Users that have packaged nice applications (for instance MPlayer, or GEdit, or GNOME, or Apache) in a neat way and want to share them amongst others because the installation process is so easy with PNDs.
3. Users who have created artwork, levels, games etc that are too big to fit on the internal NAND.
4. Companies who want to deploy commercial applications.

NAND should not be touched by SD card / PND insertion.
NAND does not need to be touched by unix-style HOME configuration settings in .dot directories

from uses you mentioned
PND use 1:
game executable with included library bloat: 5 to 20 MB,
100% sure it works because the library are included, no risk of crashing the game with a new library that introduce a bug
"it just works" and that's what you want in a game console when you just want to play a game.

PND use 2a:
install applications thru rpm, ipkg, etc.
PND should be for removable apps, there are already two dozen permanent app installation distribution formats, no need for a new one.
really, for the love of all that is good pleeeeeeeease don't create a new installation package format for linux.

PND use 2b:
include a 10MB MPlayer PND package with each SD cards containing movies
you know that mplayer version plays your movies fine and will always play it fine, pop in the card, it works.
10MB mplayer vs 700MB movie = 1.4% waste
10MB mplayer vs 4.7GB DVD dump = 0.2% waste
10MB mplayer vs 32GB SDHC card = 0.03125% waste

PND use 3:
a. install the expansion pack on the SD card with the original game on the SD card.
b. have the game (open source, special pandora version) look for expansion packs on the SD cards

PND use 4:
as a professional, last thing I want is some updated, optimized, suddenly more memory hungry library to crash my game for whatever reason.
so I want my games to ALWAYS and ONLY run with the included libraries.
otherwise I get emails and complains from users: "the game crashes on my machine" which I can't reproduce, can't fix, get a bad reputation for, and I get unhappy users: everybody loses.
the user will not be able to guess that its because game Y updated the ALSA libs to a new DSP-accelerated version with an accidental +1 buffer overrun / alignment round off error or whatever reason.
http://www.mobygames.com/developer/sheet/view/developerId,68260/
http://www.google.ca/search?hl=en&q=stephane+hockenhull

as for saving in HOME, have HOME point to the SD card when running an game off a PND, you want to keep the saves with your game or Apps on the SD card.
this way, whatever apps or games are running will find its associated files on the SD card.

no UnionFS, no NAND writes, pure plug and play.
its been done, tested and proven to death in the professional game industry in the last 20+ years.

and anyone who'd want to install apps or update the system, can easily do so without using PND.

last thing anyone wants is worry that a PND will suddenly trash its system or other apps/games.
and why not run the games and PND apps (like firefox, gimp, etc) as a special extremely limited user, all the game/apps needs to access is its own SD card or at most both SD cards.

KISS and cheers

1 and 4: Strict versioning keeps the libraries seperate. Your application would never use the updated library without some user meddling.
2a: I love how you guys who don't want the NAND to be touched suggest we use ipkg. Doesn't that install on the NAND?
2b: Don't really get your point here, Mplayer is a program that I saw would benefit from this symlink system.
3: That only works for specific things.

And WizardStan, it's never a good idea to make specific solutions. That just leaves you with a good system that is now overflowing with hacks to make it work for specific things. It's better to come up with a general solution. What's wrong with the Ramdisk UnionFS idea?

It's never a good idea to make absolute statements. Specific solutions can and do work all the time, if the number of use cases are small enough.

The problem with that is that it's only a couple of cases at first. Then you need more, so you go "oh, it's only a couple more". By the end of it, it's bloated.

I just find the idea of an application being able to put itself into any random place it so chooses a little disconcerting, and I'm surprised that those who complained my idea of a user controlled autostart process ignored security are not complaining about
applications that can effectively inject libraries and things entirely automatically by design.

They are symlinks and they can't overwrite existing files. We have stressed that god knows how many times.

-God Ginrai

 

[WizardStan]

The problem with that is that it's only a couple of cases at first. Then you need more, so you go "oh, it's only a couple more". By the end of it, it's bloated.

Is it though? What if we never do need more than four or six subtypes? You could be wasting a large amount of time coming up with a generic solution when this specific solution could be implemented fairly easily.
There's also an argument for ease of use for the developer. "I'm creating an icon pack. Set subtype to icon."
I also reject the idea that it would necessarily bloat to infinity. There's only a finite number of use cases, the question is whether there's few enough to be controllable, or are we talking dozens?

They are symlinks and they can't overwrite existing files. We have stressed that god knows how many times.

Creating a symlink for a library is effectively the same as just copying it in anyway, so long as the SD card is in place. Even if it doesn't overwrite anything, it has still "installed" code into a place you normally need root access to write to, entirely without user intervention. What if I don't want that to happen?

 

[polyroy]

as for saving in HOME, have HOME point to the SD card when running an game off a PND, you want to keep the saves with your game or Apps on the SD card.
this way, whatever apps or games are running will find its associated files on the SD card.

Good point. I'd advocate an export HOME=$PNDROOT for single-app like PND's. And a $REALHOME for, naturally, the real location of the user's home. This will cause issues with common configuration files, but nothing that can't be handled by the apps if they are specifically packaged for Pandora anyway.

I also reject the idea that it would necessarily bloat to infinity. There's only a finite number of use cases, the question is whether there's few enough to be controllable, or are we talking dozens?

Well, there is the so-called standard UNIX directory hierarchy, I say we stick to that and define our own for every potential service type. No need for complex metadata stuff when everything is just a file. If we can just decide that this directory structure is never changed (or that at least backward compatibility is maintained), we're cool right? No need for complex stuff, the directory location indicates its use. Done.

Creating a symlink for a library is effectively the same as just copying it in anyway, so long as the SD card is in place. Even if it doesn't overwrite anything, it has still "installed" code into a place you normally need root access to write to, entirely without user intervention. What if I don't want that to happen?

I agree. This is an argument about sandboxing apps. Since some apps require interoperability, they need to provide some things to the system (to really integrate), e.g. the symlinking. Instead of forcing permanent OS changes here, we package them as PND's too. So we have to just inform the user that the app will be doing some temporary OS modifications (allow/deny) when those PND's specifically put files outside their "home zone", whatever that may be. For most apps, this is not needed and and no such warnings need to be issued. These should be the random apps you can trust somewhat. I don't think there will be many security issues when it is checked if the critical unioned areas (e.g. /bin or /var/lib) are left unpopulated.

 

[God Ginrai]

The problem with that is that it's only a couple of cases at first. Then you need more, so you go "oh, it's only a couple more". By the end of it, it's bloated.

Is it though? What if we never do need more than four or six subtypes? You could be wasting a large amount of time coming up with a generic solution when this specific solution could be implemented fairly easily.
There's also an argument for ease of use for the developer. "I'm creating an icon pack. Set subtype to icon."
I also reject the idea that it would necessarily bloat to infinity. There's only a finite number of use cases, the question is whether there's few enough to be controllable, or are we talking dozens?

That is the kind of thinking that leads to unclean source code with lots of bugs. (like Windows)

They are symlinks and they can't overwrite existing files. We have stressed that god knows how many times.

Creating a symlink for a library is effectively the same as just copying it in anyway, so long as the SD card is in place. Even if it doesn't overwrite anything, it has still "installed" code into a place you normally need root access to write to, entirely without user intervention. What if I don't want that to happen?

My point is that they can't "inject" libraries to subvert the use of an application because they can't overwrite system files. Also, once again, if we use the RAMdisk UnionFS, it never would be writing these files to the NAND.

-God Ginrai

 

[bzar]

And WizardStan, it's never a good idea to make specific solutions. That just leaves you with a good system that is now overflowing with hacks to make it work for specific things. It's better to come up with a general solution. What's wrong with the Ramdisk UnionFS idea?

It's never a good idea to make absolute statements. Specific solutions can and do work all the time, if the number of use cases are small enough.
I just find the idea of an application being able to put itself into any random place it so chooses a little disconcerting, and I'm surprised that those who complained my idea of a user controlled autostart process ignored security are not complaining about applications that can effectively inject libraries and things entirely automatically by design.

What if the UnionFS would be overlaid to only a part of the fs tree? Something like "/overlay" (I recall someone mentioning something like this earlier)? Then the PND's would have no way to affect critical systems. The "/overlay" could have directories like "lib" and "media/themes" for different usages. Am I just saying the same stuff here that's already been said? I don't see any potential problems, security or otherwise. If it's possible (I don't know) some directories could be symlinked to places where the system keeps the default stuff, like /usr/lib and so forth, so that pnd's could only "write" to those locations. Ok, I'll try an example:

NAND:
/
--/usr
--/etc
...
--/overlay
----/overlay/lib -> /usr/lib
----/overlay/media/themes -> /usr/share/themes
----/overlay/games/quake/maps

A lib PND on SD1:
/overlay/lib/lib.so

A theme PND on SD2:
/overlay/media/themes/theme.zip (or something)

A game map PND on SD1:
/overlay/games/quake/maps/map.bsp

All overlaid:
/
--/usr
----/usr/lib/lib.so
----/usr/share/themes/theme.zip
--/etc
...
--/overlay
----/overlay/lib -> /usr/lib
----/overlay/media/themes -> /usr/share/themes
----/overlay/games/quake/maps/map.bsp

Is something like this possible?


About the home directory. If you use the nand as home, people will use it as one. Most programs will automatically suggest saving everything there, so people will. If you put home to an sd card, you lose settings as most programs save them there. The problem is that home directory is not supposed to be like this. The whole idea is not designed to be moved around media. If we could have separate settings and default save directories, that would be better (conf on NAND, save dir primarily on SD1, backed up by SD2, backed up by NAND), but still wouldn't solve the problem. Best way I see would be to keep the home on NAND, but make clearly labeled "SD1" and "SD2" symlinks to the cards' respective mount points to make it absolutely clear that you are not saving on SD's when in home dir.

 

[WizardStan]

That is the kind of thinking that leads to unclean source code with lots of bugs. (like Windows)

No, it really isn't. Again, if the use cases can be enumerated, why come up with a fully generic solution?

My point is that they can't "inject" libraries to subvert the use of an application because they can't overwrite system files. Also, once again, if we use the RAMdisk UnionFS, it never would be writing these files to the NAND.

From the existing file overlay proposal, something like this is possible:
/etc/rc3.d/ordinary_script -> /media/pnd12345/overlay/install-adware
and then, so long as the SD card is inserted, every time you restart, install-adware gets run, without any user interaction whatsoever, and no way to disable that behavior.
So you implement it to not allow symlinks into /etc/rc directories. Which is fine for now, until someone discovers another flaw and you block that one. And then another and another and then you're into the same bloat you were accusing my solution of.
Even if it does remain at just the single directory set, and you block symlinks into /etc/rc, you now no longer have the ability to start services like Apache automatically.

 

[God Ginrai]

That is the kind of thinking that leads to unclean source code with lots of bugs. (like Windows)

No, it really isn't. Again, if the use cases can be enumerated, why come up with a fully generic solution?

My point is that they can't "inject" libraries to subvert the use of an application because they can't overwrite system files. Also, once again, if we use the RAMdisk UnionFS, it never would be writing these files to the NAND.

From the existing file overlay proposal, something like this is possible:
/etc/rc3.d/ordinary_script -> /media/pnd12345/overlay/install-adware
and then, so long as the SD card is inserted, every time you restart, install-adware gets run, without any user interaction whatsoever, and no way to disable that behavior.
So you implement it to not allow symlinks into /etc/rc directories. Which is fine for now, until someone discovers another flaw and you block that one. And then another and another and then you're into the same bloat you were accusing my solution of.
Even if it does remain at just the single directory set, and you block symlinks into /etc/rc, you now no longer have the ability to start services like Apache automatically.

Wait, so the system just runs anything in the /etc/rc directories? (I'm still not familiar with this directory yet) Second, you could request root password for when it tries to add to /etc/rc directories. That way the user knows where it is trying to add stuff, and can decide whether they believe that it should be doing that.

-God Ginrai

 

[dflemstr]

OK, first of all, I have to say the following, and no one will be respected by me who hasn't first read through these points before making another response:

1. The plan as it currently is, will not ever require any writes to the NAND, ever. It will even reduce the writes to and reads from NAND, because this system will replace some parts of the IPKG system (Not all of it of course, because the IPKG system is good for very many things. This system only fills in the holes where the IPKG system doesn't reach).
2. Someone who makes a PND may choose not to use an overlay at all. If this idea gets implemented, the overlay will be purely optional. You will still be able to statically link your application to libraries inside of the PND and make a self-contained PND.
3. No files will ever be replaced or even touched by the PND system that exist on the NAND. The PND only has the option to provide files that don't already exist on the system.
4. Discussions about where the /home directory should be located have nothing to do with this discussion. Neither the PND system nor I care about where irrelevant folders are mounted (at the moment; I might discuss the issue another time). Please refrain from discussiong that particular point.

Now, as a response to the previous post by B-ZaR: The way UnionFS works makes it possible to put the resulting, "merged" system (the overlays + the NAND) anywhere you want.
Look at this line of code, taken from my post on page 3:

sudo mount -t unionfs -o dirs=/:/dev/shm/overlays none /

Change it to:

sudo mount -t unionfs -o dirs=/:/dev/shm/overlays none /overlay

and *poof*, the whole merged file system moves to the /overlay folder. You can chroot that folder, do anything you want with it, your root file system will be untouched, and no applications that you currently have on NAND will suffer from the additional files coming from PNDs. It's even a win-win situation: The PNDs that are chrooted to the overlay won't even notice that they are in a constrained environment, and can write to the /home directory etc. all they want without issue (the writes will be stored in the actual /home folder; let's not discuss where that folder is stored but it'll still be possible to write to it). So, you'd have exactly the same environment that you had before, running in the / folder, and the overlay environment run in the /overlay folder where the PNDs will reign. I don't say that this would be the preferred modus operandi, but it's possible, and there won't be issues with it.

EDIT: @ the two posts that spawned above my post while I was typing: Yes, on a system like Debian, something in /etc/rc*.d/ would be regarded as a service and be part of the autostart process. However, since PNDs aren't mounted before the init process does its job, the adding of malware won't be possible because services have to exist before init starts for them to be noticed, and the pnd daemon will start after init for sure (since it itself is a daemon started by init). Also, as has been said many times before: all other packaging systems known to man are also allowed to write to /etc/rc*.d/ so why shouldn't we be? You have to trust your sources, after all.

If you think this isn't good practice, then let's just say that the pnddaemon will have the ability to have filters like "/etc/rc*" added to it, where it isn't allowed to add symlinks in the overlay. BAM, issue fixed.

 

[WizardStan]

Wait, so the system just runs anything in the /etc/rc directories? (I'm still not familiar with this directory yet)

Yes. scripts in /etc/rc1/* get called at runlevel 1, /etc/rc2/* get called at runlevel 2, etc...

Second, you could request root password for when it tries to add to /etc/rc directories. That way the user knows where it is trying to add stuff, and can decide whether they believe that it should be doing that.

The PND format is supposed to be for users that don't (or don't want to) know about the fiddly bits. They won't care that AwesomeGame is putting stuff into /etc/rc directories, they just want to get at the awesome game they downloaded. It really should be handled automatically somehow, or at least made so simple that the user can make a good guess as to what they want.

 

[bzar]

Now, as a response to the previous post by B-ZaR: The way UnionFS works makes it possible to put the resulting, "merged" system (the overlays + the NAND) anywhere you want.
Look at this line of code, taken from my post on page 3:

sudo mount -t unionfs -o dirs=/:/dev/shm/overlays none /

Change it to:

sudo mount -t unionfs -o dirs=/:/dev/shm/overlays none /overlay

and *poof*, the whole merged file system moves to the /overlay folder. You can chroot that folder, do anything you want with it, your root file system will be untouched, and no applications that you currently have on NAND will suffer from the additional files coming from PNDs. It's even a win-win situation: The PNDs that are chrooted to the overlay won't even notice that they are in a constrained environment, and can write to the /home directory etc. all they want without issue (the writes will be stored in the actual /home folder; let's not discuss where that folder is stored but it'll still be possible to write to it). So, you'd have exactly the same environment that you had before, running in the / folder, and the overlay environment run in the /overlay folder where the PNDs will reign. I don't say that this would be the preferred modus operandi, but it's possible, and there won't be issues with it.

Well, I didn't mean that the system would be chrooted to the overlay directory. My idea was to use the overlay directory as a "hub" to overlay selected parts of the file system pointed out with symbolic links. This way PND's could only overlay data to previously designated directories. The actual "will this work" question was whether unionfs follows symlinks in such way, that if I have symlink /overlay/lib pointing to /usr/lib and I mount a union filesystem to /overlay, which has /lib/lib.so, will lib.so be visible in /usr/lib? This would allow PND applications to share files (for example mounting /overlay/games/quake/maps) and add core functionality (stuff mounted via symbolic links to system directories) without giving PNDs the possibility to mount wherever they want.

I'm just throwing ideas here, give it to me straight if I'm way off . I can see from the discussion before that this is no trivial matter.

 

[dflemstr]

Wait, so the system just runs anything in the /etc/rc directories? (I'm still not familiar with this directory yet)

Yes. scripts in /etc/rc1/* get called at runlevel 1, /etc/rc2/* get called at runlevel 2, etc...

Second, you could request root password for when it tries to add to /etc/rc directories. That way the user knows where it is trying to add stuff, and can decide whether they believe that it should be doing that.

The PND format is supposed to be for users that don't (or don't want to) know about the fiddly bits. They won't care that AwesomeGame is putting stuff into /etc/rc directories, they just want to get at the awesome game they downloaded. It really should be handled automatically somehow, or at least made so simple that the user can make a good guess as to what they want.

The system as it is planned to be at the moment cannot compromise the system, with or even without the need for root passwords and user authentication. As said above, the whole merged environment can be put anywhere, it's very easy to constrain the pnddaemon so that it's only allowed to overlay certain directories (you could even enforce this on a kernel level), and no PNDs are able to provide daemons that will be launched at boot.

 

[dflemstr]

Now, as a response to the previous post by B-ZaR: The way UnionFS works makes it possible to put the resulting, "merged" system (the overlays + the NAND) anywhere you want.
Look at this line of code, taken from my post on page 3:

sudo mount -t unionfs -o dirs=/:/dev/shm/overlays none /

Change it to:

sudo mount -t unionfs -o dirs=/:/dev/shm/overlays none /overlay

and *poof*, the whole merged file system moves to the /overlay folder. You can chroot that folder, do anything you want with it, your root file system will be untouched, and no applications that you currently have on NAND will suffer from the additional files coming from PNDs. It's even a win-win situation: The PNDs that are chrooted to the overlay won't even notice that they are in a constrained environment, and can write to the /home directory etc. all they want without issue (the writes will be stored in the actual /home folder; let's not discuss where that folder is stored but it'll still be possible to write to it). So, you'd have exactly the same environment that you had before, running in the / folder, and the overlay environment run in the /overlay folder where the PNDs will reign. I don't say that this would be the preferred modus operandi, but it's possible, and there won't be issues with it.

Well, I didn't mean that the system would be chrooted to the overlay directory. My idea was to use the overlay directory as a "hub" to overlay selected parts of the file system pointed out with symbolic links. This way PND's could only overlay data to previously designated directories. The actual "will this work" question was whether unionfs follows symlinks in such way, that if I have symlink /overlay/lib pointing to /usr/lib and I mount a union filesystem to /overlay, which has /lib/lib.so, will lib.so be visible in /usr/lib? This would allow PND applications to share files (for example mounting /overlay/games/quake/maps) and add core functionality (stuff mounted via symbolic links to system directories) without giving PNDs the possibility to mount wherever they want.

I'm just throwing ideas here, give it to me straight if I'm way off . I can see from the discussion before that this is no trivial matter.

I don't fully udnerstand what you mean, but I'll try to give you a satisfactory answer anyway.

So, the problem here seems to be that people are afraid that PNDs will add files where they don't want them to be added. Now, I suggested a solution where you will have your normal system (the one booting from NAND and the one you'd be running normally if this suggestion wouldn't be implemented) absolutely untouched. No PND will be able to change anything there, and no applications will see any changes, because there are none to see.

Instead, we would have the /overlay directory. Now, think about it like this: the /overlay directory will contain an exact clone of your entire system that exists under /, but with the sole exception that in THAT folder, the overlay files from the PNDs will be visible. Your / system will have no changes in it at all, but a "changed version" of your system with additional files from PNDs will exist under /overlay.

Then, you'd be able to safely chroot your PNDs to that directory, and they would act normally because in their /overlay environment, all the files they need exist, and the applications started from PNDs will be able to write to the user's /home/user directory by writing to /overlay/home/user (like they would be able to anyways) without issue, becuase the changes they make will be reflected in the normal, /-based system.

It would even be possible to *hide* directories in /overlay, so that PNDs not only can't write to them, but also cannot *read* them, even if they would have permissions to do so. The environment would be completely isolated.

Have I made myself clear? I thinkt hat we're both after the same thing here.

 

[WizardStan]

The system as it is planned to be at the moment cannot compromise the system, with or even without the need for root passwords and user authentication. As said above, the whole merged environment can be put anywhere, it's very easy to constrain the pnddaemon so that it's only allowed to overlay certain directories (you could even enforce this on a kernel level), and no PNDs are able to provide daemons that will be launched at boot.

There were three use cases you provided beyond the original "click to run" that you wanted for the PND format.
1) Start daemons like Apache automatically
2) Install icon themes
3) Extra libraries and codecs
By your design, (1) is completely out. It's a use case that can never be achieved.
I was actually expecting a different response to the /etc/rc problem, and was preparing an argument that would render (2) useless as well, but simply rejecting overlays to /etc/rc solves that problem, even if it does eliminate one of the other use cases.
Except that right now it's /etc/rc. What happens if someone else thinks up a way of abusing a different directory? Block that one as well?
For some reason, inclusive feels better than exclusive: "you can overlay into these directories, adding more overtime as needed" sounds safer than "you cannot write into these directories, adding more as flaws are found".

 

[bzar]

Now, as a response to the previous post by B-ZaR: The way UnionFS works makes it possible to put the resulting, "merged" system (the overlays + the NAND) anywhere you want.
Look at this line of code, taken from my post on page 3:

sudo mount -t unionfs -o dirs=/:/dev/shm/overlays none /

Change it to:

sudo mount -t unionfs -o dirs=/:/dev/shm/overlays none /overlay

and *poof*, the whole merged file system moves to the /overlay folder. You can chroot that folder, do anything you want with it, your root file system will be untouched, and no applications that you currently have on NAND will suffer from the additional files coming from PNDs. It's even a win-win situation: The PNDs that are chrooted to the overlay won't even notice that they are in a constrained environment, and can write to the /home directory etc. all they want without issue (the writes will be stored in the actual /home folder; let's not discuss where that folder is stored but it'll still be possible to write to it). So, you'd have exactly the same environment that you had before, running in the / folder, and the overlay environment run in the /overlay folder where the PNDs will reign. I don't say that this would be the preferred modus operandi, but it's possible, and there won't be issues with it.

Well, I didn't mean that the system would be chrooted to the overlay directory. My idea was to use the overlay directory as a "hub" to overlay selected parts of the file system pointed out with symbolic links. This way PND's could only overlay data to previously designated directories. The actual "will this work" question was whether unionfs follows symlinks in such way, that if I have symlink /overlay/lib pointing to /usr/lib and I mount a union filesystem to /overlay, which has /lib/lib.so, will lib.so be visible in /usr/lib? This would allow PND applications to share files (for example mounting /overlay/games/quake/maps) and add core functionality (stuff mounted via symbolic links to system directories) without giving PNDs the possibility to mount wherever they want.

I'm just throwing ideas here, give it to me straight if I'm way off . I can see from the discussion before that this is no trivial matter.

I don't fully udnerstand what you mean, but I'll try to give you a satisfactory answer anyway.

So, the problem here seems to be that people are afraid that PNDs will add files where they don't want them to be added. Now, I suggested a solution where you will have your normal system (the one booting from NAND and the one you'd be running normally if this suggestion wouldn't be implemented) absolutely untouched. No PND will be able to change anything there, and no applications will see any changes, because there are none to see.

Instead, we would have the /overlay directory. Now, think about it like this: the /overlay directory will contain an exact clone of your entire system that exists under /, but with the sole exception that in THAT folder, the overlay files from the PNDs will be visible. Your / system will have no changes in it at all, but a "changed version" of your system with additional files from PNDs will exist under /overlay.

Then, you'd be able to safely chroot your PNDs to that directory, and they would act normally because in their /overlay environment, all the files they need exist, and the applications started from PNDs will be able to write to the user's /home/user directory by writing to /overlay/home/user (like they would be able to anyways) without issue, becuase the changes they make will be reflected in the normal, /-based system.

Have I made myself clear? I thinkt hat we're both after the same thing here.

I understand. Now the thing people seem to be concerned about is PND's adding files to security-critical locations such as rc-directories. The other concern was that giving the PND makers the whole filesystem as a playground of potential places to put their stuff via overlaying would cause chaos, because everyone would just put anything anywhere (addressing this was one of the bigger advantages I saw in WizardStan's pnd profiles). My suggestion addresses these both by giving only small parts of the core file system (files on nand) available for overlay by mounting the overlay to a directory which has symlinks to these specific locations. This way the overlay could only add files to directories explicitly symlinked to from the overlay directory pre-mounting. It would also make it easy to create clearly and shortly named directories for application data such as the /games/quake/maps.

Maybe
a ) I'm just too tired to explain this clearly (I should be sleeping already )
b ) I've understood the whole problem wrong
c ) my solution is just so silly that smart people can't think wrong enough to get it
d ) my solution is not possible with the tools given, so I can't explain it so that it makes sense

Anyways, I understand your idea, which is the best generic one I've heard so far. I think WizardStan's profile idea has merit as well. It would certainly make managing easier by forcing directory protocol from system end.

 

[Pickle]

I think im for the UnionFS idea. Ive been wondering how it could work and i can see this would be great for data files.
So now we could have the branch fileystem on the sd become union with the trunk filesystem on the nand. For example it seems I could create the folder /usr/local/share/quake on the sd card with the data pak's and then create the union and the system would see them as if they were on the nand in /usr/local/share/quake.
This would also be good for home, now all writes to home can go to the sd card rather than the nand.

Although is the unionfs smart enough where to put the files? I read you can make parts of the union read only. For example if something runs on the nand and needs to put some file in /home I want them on nand:/home and not sd:/home, but it something is on the sd I want them the opposite way.
Also can unionfs handle fat32?

 

[bzar]

This would also be good for home, now all writes to home can go to the sd card rather than the nand.

Wouldn't this make things hard for the configurations stored in the home directory? Pop the SD, lose the configurations? If both SD's are in place, which would hold home?

 

[peca]

NAND should not be touched by SD card / PND insertion.
NAND does not need to be touched by unix-style HOME configuration settings in .dot directories

from uses you mentioned
...shortened...
no UnionFS, no NAND writes, pure plug and play.
its been done, tested and proven to death in the professional game industry in the last 20+ years.
...shortened...
and anyone who'd want to install apps or update the system, can easily do so without using PND.
last thing anyone wants is worry that a PND will suddenly trash its system or other apps/games.
and why not run the games and PND apps (like firefox, gimp, etc) as a special extremely limited user, all the game/apps needs to access is its own SD card or at most both SD cards.
KISS and cheers

Exactly!

 

[God Ginrai]

I think im for the UnionFS idea. Ive been wondering how it could work and i can see this would be great for data files.
So now we could have the branch fileystem on the sd become union with the trunk filesystem on the nand. For example it seems I could create the folder /usr/local/share/quake on the sd card with the data pak's and then create the union and the system would see them as if they were on the nand in /usr/local/share/quake.
This would also be good for home, now all writes to home can go to the sd card rather than the nand.

Although is the unionfs smart enough where to put the files? I read you can make parts of the union read only. For example if something runs on the nand and needs to put some file in /home I want them on nand:/home and not sd:/home, but it something is on the sd I want them the opposite way.
Also can unionfs handle fat32?

The unionFS idea was actually supposed to work in RAM, if I'm not mistaken. (dflemster, correct me if I'm wrong)

EDIT:

NAND should not be touched by SD card / PND insertion.
NAND does not need to be touched by unix-style HOME configuration settings in .dot directories

from uses you mentioned
...shortened...
no UnionFS, no NAND writes, pure plug and play.
its been done, tested and proven to death in the professional game industry in the last 20+ years.
...shortened...
and anyone who'd want to install apps or update the system, can easily do so without using PND.
last thing anyone wants is worry that a PND will suddenly trash its system or other apps/games.
and why not run the games and PND apps (like firefox, gimp, etc) as a special extremely limited user, all the game/apps needs to access is its own SD card or at most both SD cards.
KISS and cheers

Exactly!

You really don't understand this. Go back and read this post:

OK, first of all, I have to say the following, and no one will be respected by me who hasn't first read through these points before making another response:

1. The plan as it currently is, will not ever require any writes to the NAND, ever. It will even reduce the writes to and reads from NAND, because this system will replace some parts of the IPKG system (Not all of it of course, because the IPKG system is good for very many things. This system only fills in the holes where the IPKG system doesn't reach).
2. Someone who makes a PND may choose not to use an overlay at all. If this idea gets implemented, the overlay will be purely optional. You will still be able to statically link your application to libraries inside of the PND and make a self-contained PND.
3. No files will ever be replaced or even touched by the PND system that exist on the NAND. The PND only has the option to provide files that don't already exist on the system.
4. Discussions about where the /home directory should be located have nothing to do with this discussion. Neither the PND system nor I care about where irrelevant folders are mounted (at the moment; I might discuss the issue another time). Please refrain from discussiong that particular point.

Now, as a response to the previous post by B-ZaR: The way UnionFS works makes it possible to put the resulting, "merged" system (the overlays + the NAND) anywhere you want.
Look at this line of code, taken from my post on page 3:

sudo mount -t unionfs -o dirs=/:/dev/shm/overlays none /

Change it to:

sudo mount -t unionfs -o dirs=/:/dev/shm/overlays none /overlay

and *poof*, the whole merged file system moves to the /overlay folder. You can chroot that folder, do anything you want with it, your root file system will be untouched, and no applications that you currently have on NAND will suffer from the additional files coming from PNDs. It's even a win-win situation: The PNDs that are chrooted to the overlay won't even notice that they are in a constrained environment, and can write to the /home directory etc. all they want without issue (the writes will be stored in the actual /home folder; let's not discuss where that folder is stored but it'll still be possible to write to it). So, you'd have exactly the same environment that you had before, running in the / folder, and the overlay environment run in the /overlay folder where the PNDs will reign. I don't say that this would be the preferred modus operandi, but it's possible, and there won't be issues with it.

EDIT: @ the two posts that spawned above my post while I was typing: Yes, on a system like Debian, something in /etc/rc*.d/ would be regarded as a service and be part of the autostart process. However, since PNDs aren't mounted before the init process does its job, the adding of malware won't be possible because services have to exist before init starts for them to be noticed, and the pnd daemon will start after init for sure (since it itself is a daemon started by init). Also, as has been said many times before: all other packaging systems known to man are also allowed to write to /etc/rc*.d/ so why shouldn't we be? You have to trust your sources, after all.

If you think this isn't good practice, then let's just say that the pnddaemon will have the ability to have filters like "/etc/rc*" added to it, where it isn't allowed to add symlinks in the overlay. BAM, issue fixed.

-God Ginrai

 

[peca]

This would also be good for home, now all writes to home can go to the sd card rather than the nand.

Wouldn't this make things hard for the configurations stored in the home directory? Pop the SD, lose the configurations? If both SD's are in place, which would hold home?

In this case you can (or rather must) have config on every card. If home is on both inserted cards, then first inserted is mounted or slot 1 is taken as primary.