About Sharing

[sebt3]

First, I'm not realy sure this belong to the dev section, as I'm not realy planning to code something for this (My C coding skillz are weak). On the other side, this only interest packagers and coders, so it's best here.

As we all know, the nand space is limited. I'm fine with the library selection on it (well I could see a fit for a package or 2 more, but I digress).
But there are many thing that can be shared between PNDs. For what I've worked on :
- timidity
- kde-libs (in fact kdelibs, kdesupport, kdebase and dependencies, that's a 80M compressed PND)
- And I'm sure we could find more of them

I would like to see some kind of a limited dependency system within PND. I'm not thinking about a fine grained dependency system (like the one used in package management, this would exactly kill the idea behind PNDs : simplicity)

My idea was to have a _few_ PND being libraries (stored in /pandora/libs ?) that other PNDs could declare to depend on (in the PXML).
The library would be mounted at the same time as the application by pnd_run.sh.

When I introduced my idea to Jeff, he sugested that these PND could be mounted on card insert and unmounted when you remove the media.

As I stated in the first line, I don't have the skillz to code this in libpnd But I plan on doing the work required in pnd_run.sh (I'm realy confortable with shell scripting, a newer and a bit cleaned version might even pop-up within HF4)

What do you think ? Someone for the job ?

 

[skeezix]

Thats the two main approaches we'd discussed before releasing the Pandora to public; I never got around to it since I figuredf it best to get public involvement afgter we'd gotten real world usage experience with the existing system -- I'm a huge fan of getting something out, working, simpler, and then complicating it later; it speeds adoption when things work

Options I'd thought of..

1) On media (ie: boot, insert, eject), scan for /pandora/autorun or /pandora/lib or something, and mount all the stuff found there (sort of like how old MAc OS did things, I suppose.) The advantage is -- you want a lib, or a dictionairy data file, or wallpaper packs, or whatever shared resource made available without speed hit .. drop your pnd there, and good to go; disadvantage -- ties up the media. ie: I don't know current state, but with union filesystem mounts, things can be unstable -- if you're long term mounting a pnd, and then an hour later just yank out the SD card without explicitly unmounting these long lived pnds, will the device hang, or crash? Certainly, any apps depending on those libs would explode. (ie: Consider -- if you'd put long lived lib pnds on SD1, and run an app from SD2 that depended on them...); we'd also need to add some PXML hints, so the pnd could advertise it wants a given lib or not, so th user coudl avoid 'silent fails' on app runs with missing files, etc. Also, slow boot/insert time.

2) On run -- as with (1), we change the PXML so it can advertise some basic dependancies, and we change the run system so it auto-mounts the needed guys (and posts a zenity fail box if some are missing), at run time; advantages -- no risks for the scenario I mentioned above, since its all mounted and unmounted at runtime of the app pnd; disadvantage is -- slower invocation of apps; the union mount is the slow part of the pnd run (not really slow, but a second or so per mount; if you depended on 10 lib mounts, it could take awhile to fire the app up .. for every new run.) A bit of complexity in pnd_run.sh so it knows what to mount/unmount around apps (if you have 5 pnd files running at the same time, with a big web of dependancies, you have to be careful.. but no biggy. Session counters for each pnd-lib or something would be enough.)

3) Both; really, its just a question of -- boot a lib at first available time (Take the tme hit once, some risk of stability), or mount as needed (time hit later, when yo want to run an app, but perhaps more stable.) Code wise, not big difference. So we could make two directories: /pandora/autostart and /pandora/demand-start sort of thing, so the user could decide what camp they want to be in.

4) We push developers to package necessary libs into their pnd files, and avoid this whole mess; _this should be the default_, for small/trivial libs (and be nice if we could override those libs if newer versions show in the firmware down the road?); ie: ITs nice when an app just works, with no dependancies; it sounds bad to have 5 apps have 5 copies of some small lib in them, but in practice, it doesn't take up a lot more space or make clutter (see OSX for instance.) But for any _large_ or clumsy or important libs, thats what we're talking about here in this posting -- KDE for instance; a big pile of libs and tools, that woudl be really ugly to include into 10 pnd files.

Not rteally complicated to implement on the surface, but lots of fiddly bits that could make users lives hard; hence, we left it for later.

Perhaps theres another clever idea here (other than making people run their firmware off SD andjust install all the libs into the firmware

jeff

Anyway, I'm crazy busy in and out today, but hopefully this is a sensible post

 

[sebt3]

if you depended on 10 lib mounts, it could take awhile to fire the app up ..

Then you would have basically broken the PND design : I f a package depend on 10 libs, then the user will be forced to install them, and manage the dependency manually. Which crazy and on the other end of the PND system.

(if you have 5 pnd files running at the same time, with a big web of dependancies, you have to be careful.. but no biggy. Session counters for each pnd-lib or something would be enough.)

I was going to make pnd_run.sh open a file for write (writing nothing) within the union mount. This way no library would be umounted before each apps using it would have finished

/pandora/autostart

A "man" PND could use this

We push developers to package necessary libs into their pnd files, and avoid this whole mess; _this should be the default_, for small/trivial libs (and be nice if we could override those libs if newer versions show in the firmware down the road?); ie: ITs nice when an app just works, with no dependancies; it sounds bad to have 5 apps have 5 copies of some small lib in them, but in practice, it doesn't take up a lot more space or make clutter (see OSX for instance.) But for any _large_ or clumsy or important libs, thats what we're talking about here in this posting -- KDE for instance; a big pile of libs and tools, that woudl be really ugly to include into 10 pnd files.

I agree with this.
Audacious and deadbeef depend on libflac, and both package it. There is no need of a flac.pnd as the file would be a few Kb. Each these apps should contain it and that good this way.
We are only talking of a few large (over 10M compressed) and shared (more than 2 apps) dependency (timidity, kde-libs).
EFL might worth a package too, but I cant see many more apps then enna and enlightement using it on the P, so this sound pretty lame too.
My current shoot at kde-libs is a 70M squashfs-file ; this is slimed down (stripped, unusefull data removed) and it still miss some dependency (kde-base at least)

Anyway, I'm crazy busy in and out today, but hopefully this is a sensible post

It is, as always, thanks

 

[Tobriand]

Some of the user headache might be avoided if these special pnds could be obtained automatically from Pandora Apps if they were not already present. I'm thinking of essentially a download url being included in the shell script that, if run_pnd fails, Pandora offers the user the opportunity to download the required libs to /pandora/libs automatically.

It'd be annoying if I had no net access, but a lot less annoying than manually obtaining a series of cryptically named files that I wouldn't necessarily even know the need for prior to attempting to run the lib-less app.

 

[Sphinxter]

Think I'm missing something here*, what's wrong with a simple dependency list? If pnd checks the mount list and doesn't find a libflac version ??-?? or better pnd mounted then look on the cards and mount it else if not found bitch.

*besides a pandora.

 

[sebt3]

what's wrong with a simple dependency list?

Simple ? nothing wrong. It is exactly the topic

Complex ? Did you ever maintain a debian box offline ? Before this it was a HELL. I don't want a pandora user going trough all this. Beside, pnd aim to be easy on the user.
Every package should be self-contain... to the point where it become stupid to stay with this rule...
As I reached this point with my kdeGame package, I opened this thread

 

[God Ginrai]

Thats the two main approaches we'd discussed before releasing the Pandora to public; I never got around to it since I figuredf it best to get public involvement afgter we'd gotten real world usage experience with the existing system -- I'm a huge fan of getting something out, working, simpler, and then complicating it later; it speeds adoption when things work

Options I'd thought of..

1) On media (ie: boot, insert, eject), scan for /pandora/autorun or /pandora/lib or something, and mount all the stuff found there (sort of like how old MAc OS did things, I suppose.) The advantage is -- you want a lib, or a dictionairy data file, or wallpaper packs, or whatever shared resource made available without speed hit .. drop your pnd there, and good to go; disadvantage -- ties up the media. ie: I don't know current state, but with union filesystem mounts, things can be unstable -- if you're long term mounting a pnd, and then an hour later just yank out the SD card without explicitly unmounting these long lived pnds, will the device hang, or crash? Certainly, any apps depending on those libs would explode. (ie: Consider -- if you'd put long lived lib pnds on SD1, and run an app from SD2 that depended on them...); we'd also need to add some PXML hints, so the pnd could advertise it wants a given lib or not, so th user coudl avoid 'silent fails' on app runs with missing files, etc. Also, slow boot/insert time.

2) On run -- as with (1), we change the PXML so it can advertise some basic dependancies, and we change the run system so it auto-mounts the needed guys (and posts a zenity fail box if some are missing), at run time; advantages -- no risks for the scenario I mentioned above, since its all mounted and unmounted at runtime of the app pnd; disadvantage is -- slower invocation of apps; the union mount is the slow part of the pnd run (not really slow, but a second or so per mount; if you depended on 10 lib mounts, it could take awhile to fire the app up .. for every new run.) A bit of complexity in pnd_run.sh so it knows what to mount/unmount around apps (if you have 5 pnd files running at the same time, with a big web of dependancies, you have to be careful.. but no biggy. Session counters for each pnd-lib or something would be enough.)

3) Both; really, its just a question of -- boot a lib at first available time (Take the tme hit once, some risk of stability), or mount as needed (time hit later, when yo want to run an app, but perhaps more stable.) Code wise, not big difference. So we could make two directories: /pandora/autostart and /pandora/demand-start sort of thing, so the user could decide what camp they want to be in.

4) We push developers to package necessary libs into their pnd files, and avoid this whole mess; _this should be the default_, for small/trivial libs (and be nice if we could override those libs if newer versions show in the firmware down the road?); ie: ITs nice when an app just works, with no dependancies; it sounds bad to have 5 apps have 5 copies of some small lib in them, but in practice, it doesn't take up a lot more space or make clutter (see OSX for instance.) But for any _large_ or clumsy or important libs, thats what we're talking about here in this posting -- KDE for instance; a big pile of libs and tools, that woudl be really ugly to include into 10 pnd files.

Not rteally complicated to implement on the surface, but lots of fiddly bits that could make users lives hard; hence, we left it for later.

Perhaps theres another clever idea here (other than making people run their firmware off SD andjust install all the libs into the firmware

jeff

Anyway, I'm crazy busy in and out today, but hopefully this is a sensible post

I'm fond of a mixture of 3 and 4.

EDIT: Note, I very much like the idea of demandstart and autostart

/pandora/autostart

A "man" PND could use this

+1.

Some of the user headache might be avoided if these special pnds could be obtained automatically from Pandora Apps if they were not already present. I'm thinking of essentially a download url being included in the shell script that, if run_pnd fails, Pandora offers the user the opportunity to download the required libs to /pandora/libs automatically.

It'd be annoying if I had no net access, but a lot less annoying than manually obtaining a series of cryptically named files that I wouldn't necessarily even know the need for prior to attempting to run the lib-less app.

Well, wouldn't you add something to the PXML spec where you can list dependencies? That way, anything that interprets the PNDs can look for and make you aware of the dependencies. I could see this working really well in something like The Box.

-God Ginrai

 

[Tempel]

I know this is a bit premature, but if PNDs are ever to become useful on systems other than the Pandora, dependency listing is a must. As I envisioned it, you could download a PND for your desktop, then when you try to run it, libpnd would interface with the package manager (through PackageKit?) to ensure that all dependencies are satisfied. Obviously this is not a solution on the Pandora, since installing through the package manager means installing to the small space on the NAND.

But this suggests that we could decouple two different problems here and solve them separately. First, a PND needs to be able to specify its dependencies; this would likely just need a change to the PXML spec. Second, we need a way to carry libraries on an SD card, independently of the rootfs but easily accessible by any program; this might mean an extension of what PNDs can do, or it might mean a different system entirely for libraries.

As for solving that second problem, there was once some talk about making PNDs able to overlay files on the rootfs, but then we run in to the question of whether to mount such PNDs at insertion or on demand, as skeezix has already brought up.

 

[Stuckie]

I would be very careful in having "dependency PNDs" to be honest.
The current setup is that you can only have 7 loop devices active, and each PND takes up one of these, so you can quickly stop yourself from loading up more PNDs if you're not careful.
You could also have interesting issues if you AUFS/Union mount multiple things to the same folder - these HAVE to be done at the same time, otherwise mount throws hissy fits when trying to mount an AUFS/Union on top of another. I have the same issues with my Extend Utils.. and the latest script was a down right *bad word* to get right, because of this!

If you are going to start overlaying files over the rootfs, I'd be careful of where you do it, as you could inadvertently tie up the SD card.
Going anywhere near /usr will more than likely tie you up, for instance.. as nearly everything touches /usr/bin and /usr/lib at regular intervals making them utter swines to try and unbind; since they'll be in use and you'll be denied.

It may be wise to take a leaf out of Apple's book with their .app packages, in that the PND will be self-satisfied in as far as non-standard dependencies go. If a library is not on a vanilla Pandora ( or HotFix ) install, then it MUST be included in the PND. It may mean that PNDs could end up fat bloaters, but you're at least guaranteed not to have to go on a dependency hunt, or ensuring you've got enough loop slots open for whatever you're trying to run.

But that's just my opinion from hacking about in the basement.. don't mind me

 

[God Ginrai]

I would be very careful in having "dependency PNDs" to be honest.
The current setup is that you can only have 7 loop devices active, and each PND takes up one of these, so you can quickly stop yourself from loading up more PNDs if you're not careful.
You could also have interesting issues if you AUFS/Union mount multiple things to the same folder - these HAVE to be done at the same time, otherwise mount throws hissy fits when trying to mount an AUFS/Union on top of another. I have the same issues with my Extend Utils.. and the latest script was a down right *bad word* to get right, because of this!

If you are going to start overlaying files over the rootfs, I'd be careful of where you do it, as you could inadvertently tie up the SD card.
Going anywhere near /usr will more than likely tie you up, for instance.. as nearly everything touches /usr/bin and /usr/lib at regular intervals making them utter swines to try and unbind; since they'll be in use and you'll be denied.

It may be wise to take a leaf out of Apple's book with their .app packages, in that the PND will be self-satisfied in as far as non-standard dependencies go. If a library is not on a vanilla Pandora ( or HotFix ) install, then it MUST be included in the PND. It may mean that PNDs could end up fat bloaters, but you're at least guaranteed not to have to go on a dependency hunt, or ensuring you've got enough loop slots open for whatever you're trying to run.

But that's just my opinion from hacking about in the basement.. don't mind me

But wouldn't your problem not arise if you take skeezix's suggestion and mount the needed dependency only when needed, and only for large non-obscure dependencies that would take up too much space on NAND?

-God Ginrai

 

[Stuckie]

Unfortunately not, because if you mount to a directory which is actively in use, you cant unmount from it - stuff is being written/read, but where to or from? The underlying NAND, or the overlay(s) you've mounted on top?

It's an arse... and near enough every directory in /usr is affected, as generally that's where dependencies get stored. Union and AUFS mounts are meant to function as one directory, so as soon as something can differentiate between the overlay(s) and the NAND, it's pretty much a broken system.

And if you can't unmount, that's both a loop device taken up, and the SD card locked.

You possibly could get away with it, if you do somewhat hacky things like I originally did with the GIMP PND prototype - symlinks and AUFS mounts to folders within a child directory of /usr rather than a parent ( so /usr/share/gimp instead of /usr/share. )
This requires writing at least folders/symlinks to the NAND for the linking to work, but should allow you to pull it back off relatively sanely, assuming there's nothing else accessing them or again, you can't unmount.

 

[sebt3]

Unfortunately not, because if you mount to a directory which is actively in use, you cant unmount from it - stuff is being written/read, but where to or from? The underlying NAND, or the overlay(s) you've mounted on top?

It's an arse... and near enough every directory in /usr is affected, as generally that's where dependencies get stored. Union and AUFS mounts are meant to function as one directory, so as soon as something can differentiate between the overlay(s) and the NAND, it's pretty much a broken system.

And if you can't unmount, that's both a loop device taken up, and the SD card locked.

You possibly could get away with it, if you do somewhat hacky things like I originally did with the GIMP PND prototype - symlinks and AUFS mounts to folders within a child directory of /usr rather than a parent ( so /usr/share/gimp instead of /usr/share. )
This requires writing at least folders/symlinks to the NAND for the linking to work, but should allow you to pull it back off relatively sanely, assuming there's nothing else accessing them or again, you can't unmount.

Who speak about /usr in the first place ?
Does kde-libs requiere to live in there ? no. Does timidity ? neither

I know about the loop limits, but I wasn't planning to have tons of pnds library, only those that are _large_ and _shared_.

Usage scenario :
I want to listen to music : I fire amarok.
pnd_run.sh discover that amarok requiere kde-libs and it's not mounted. So it mount kde-libs to /mnt/pnd/kde-libs and create it's overlay (/mnt/utmp/kde-libs). It now mount amarok pnd as usual, set the environments variables kde-libs ask to set (LD_LIBRARY_PATH, QT_DIR etc...), open a file in the kde-libs overlay. and finaly start amarok.
Now I want to play grandfather (my favorite card game ), so I fire kpat. pnd_run.sh found that kde-libs is already opened, load the environement variables open it's file in the kde-libs overlay and start kpat.
I stop amarok, pnd_libs.sh destroy its file in the overlay, unmount amarok pnd and try to unmount kde-libs but fail : the file created by the kpat pnd_run instance is still opened.
Now I stop kpat, pnd_libs.sh destroy its file in the overlay, unmount kpat pnd and unmount kde-libs as it's the last one to use it.
Sure I could only open 6 pnd at the same time instead of 7, big deal....

In this scenario I find this realy helpfull.
But sure, otherwise, I could create a kde.pnd that will be around 500M but then many will find this to be realy heavy to just have amarok or smb4k.
Or I could create about 100 pnds each 90M each as they will all carry the whole KDE-libs stack and then some won't like to spend 1G of SDcard space because they want the full kde desktop experience (I sure will). Oh and don't forget we are memory constraint. If kde-libs are bundled into many files, so as the files are different for the kernel, they won't be shared. So where is the point of having "*.so" ?

 

[skeezix]

Another option I suppose .. sort of crappy --

For 'libs' (or some class of libs), we could _unpack them_ for the user, for the lifetime of the mount, onto SD. (Or perhaps just unpack them, and then during next mount, just verify the versions are correct and leave them there.) ie: In this case, using pnd as a rudimentary nasty but easy distribution that self unpacks (like a hotfix, but targeting SD cards.)

So you download 'kde libs' and 'kde games'; kde games autoruns kde-libs each mount, and kde-libs just 1) unpacks to SD, or 2) verifies previous unpack is correct versions still. pnd_run.sh could then include the new path in LD_LIBRARY_PATH and good to go. /pandora/.unpacked-magic-libs or something.

*shrug*

No aufs mount pain, but sort of goofy and encrouching on the territory of real packagers

jeff

 

[DaveC]

I like the approach of keeping smallish libs in the PND that way no differing version compatibility problems ie. update the main lib break older orphaned apps.

For common big things like timidity that could just be placed in the appdata directory in a directory called "timidity". Now any app can be pointed there and make use of it. So for example Doom could just look for the files in a timidity directory in appdata on the same SD as the app. This was kind of done on the GP2X with timidity. It was established by DOOM and the rest just used the same directory for timidity from then on. The only problem I see is that some kind of standard would have to be followed by devs. From what I see though, no offense, but getting many devs to follow one standard can be like herding cats. The first to use the dir name would have to publish it and the following users would choose the same name.

 

[sebt3]

So you download 'kde libs' and 'kde games'; kde games autoruns kde-libs each mount, and kde-libs just 1) unpacks to SD, or 2) verifies previous unpack is correct versions still. pnd_run.sh could then include the new path in LD_LIBRARY_PATH and good to go. /pandora/.unpacked-magic-libs or something.

kde-libs does contain symlinks. this won't work (or cost about 3 times the storage) on a vfat sdcard

 

[Stuckie]

Who speak about /usr in the first place ?
Does kde-libs requiere to live in there ? no. Does timidity ? neither

I know about the loop limits, but I wasn't planning to have tons of pnds library, only those that are _large_ and _shared_.

Usage scenario :
I want to listen to music : I fire amarok.
pnd_run.sh discover that amarok requiere kde-libs and it's not mounted. So it mount kde-libs to /mnt/pnd/kde-libs and create it's overlay (/mnt/utmp/kde-libs). It now mount amarok pnd as usual, set the environments variables kde-libs ask to set (LD_LIBRARY_PATH, QT_DIR etc...), open a file in the kde-libs overlay. and finaly start amarok.

Ah, I misunderstood what you were up to.. I thought you were trying to get libraries and dependencies installed into the _default_ locations ( which generally is /usr ) so that anything which doesn't respect the environment settings can find them without having to be specially compiled to look for your /mnt/utmp/kde-libs path.

In that case, yea it'd be easy to pull them off and most of my fears are moot
I still don't trust all applications to behave themselves, however.. but I'm sure they're few enough to deal with manually.

Anyway, my mistake.. probably shouldn't read interesting forum posts while stupidly tired!

--edit--
Also, if no one has time to look at this, I probably will.. it's rather close to the madness I get up to with Extend Utils anyway.

 

[Rockthesmurf]

How big are these shared libs? Because unless they are taking up a decent amount of space, I'd prefer to keep things how they are, as in a pnd always contains everything it needs to run, except some real core stuff which is available on the NAND. I like the idea of keeping everything as simple as possible unless there really is a problem that can't be overcome.

Steve

 

[sebt3]

How big are these shared libs? Because unless they are taking up a decent amount of space, I'd prefer to keep things how they are, as in a pnd always contains everything it needs to run, except some real core stuff which is available on the NAND. I like the idea of keeping everything as simple as possible unless there really is a problem that can't be overcome.

Steve

We are only talking of a few large (over 10M compressed) and shared (more than 2 apps) dependency (timidity, kde-libs).
My current shoot at kde-libs is a 70M squashfs-file ; this is slimed down (stripped, unusefull data removed) and it still miss some dependency (kde-base at least)

 

[Rockthesmurf]

Ah I see, what are peoples thoughts as to what is an acceptable amount of space to be wasted on an SD card? I normally find my SD cards end up with mainly data files for applications (ROMs/music/movies/game data files/etc.) rather than actual executables/libraries. At a guess I might say on a 16 GB SD card, I might end up with 2 GB of executables/libraries and 14 GB of data files. If out of that 2 GB of executables/libraries there was 1 GB of duplicated libraries, that would be okay for me. I'm not entirely sure what my upper limit would be, 1.5 GB of duplicates out of 2.0 GB seems like an awful lot, but if that is a trade for keeping things simple I'm not sure I'd be that bothered, so my limit might be 75%. With the current system, is it likely we are actually going to go past that limit?

I hope it is appreciated that I am just talking on my own behalf, some people might have completely different ideas as to what is acceptable! I am just trying to end up with a rough idea of how bad it'd need to be before we start making things more complicated than they are now...

Finally, if there are a couple of examples of really bad duplications (maybe KDE stuff is one), I'd be happy enough having them as special cases, where you have a kde-libs.pnd file that *must* be on the SD card, along side any applications that access it. But for it to be a special case I'd want there to be a bunch of applications that would share the libraries, and I would want the size to be reasonable.

Steve

 

[God Ginrai]

Ah I see, what are peoples thoughts as to what is an acceptable amount of space to be wasted on an SD card? I normally find my SD cards end up with mainly data files for applications (ROMs/music/movies/game data files/etc.) rather than actual executables/libraries. At a guess I might say on a 16 GB SD card, I might end up with 2 GB of executables/libraries and 14 GB of data files. If out of that 2 GB of executables/libraries there was 1 GB of duplicated libraries, that would be okay for me. I'm not entirely sure what my upper limit would be, 1.5 GB of duplicates out of 2.0 GB seems like an awful lot, but if that is a trade for keeping things simple I'm not sure I'd be that bothered, so my limit might be 75%. With the current system, is it likely we are actually going to go past that limit?

I hope it is appreciated that I am just talking on my own behalf, some people might have completely different ideas as to what is acceptable! I am just trying to end up with a rough idea of how bad it'd need to be before we start making things more complicated than they are now...

Finally, if there are a couple of examples of really bad duplications (maybe KDE stuff is one), I'd be happy enough having them as special cases, where you have a kde-libs.pnd file that *must* be on the SD card, along side any applications that access it. But for it to be a special case I'd want there to be a bunch of applications that would share the libraries, and I would want the size to be reasonable.

Steve

I see and understand where you are coming from, but think about it this way. Larger filesize of applications due to these libraries creates longer download and copy times, which can be a pain for people trying to do it from their Pandora, or on bad internet connections. Also, not everyone can afford 16 GB cards.

-God Ginrai