Re-targeting the list.
---------- Forwarded message ---------- From: Jeff Mitchell skeezix@skeleton.org To: "[ISO-8859-1] S�bastien Huss" sebastien.huss@gmail.com
On Fri, 25 Feb 2011, [ISO-8859-1] S�bastien Huss wrote:
# The nice thing of having the filesystem at first and the rest appended at # the end is that the filesystem is immediatly available to be mounted : no # need to do ugly hack to have mount to see the filesystem
Actually the loopback mechanism (these days, not years ago) can usually take an offset, so you canhave a binheader and not too much work .. as long as calculating where the filesystem is is easy :)
# > - payload filesystem change; # Might be nice, but does zipfs support Unix like priviledge-system ?
Support enough I think; I doubt pnd_run.sh itself cares about the appended parts, just the main payload; but libpnd cares about the appended stuff, and its easy to read a basic zip, without mounting zipfs.
I've often thought it would be cute to do iso+zip, with the zip part having one PXML.xml or the .desktops, and icon(s), and anything the dev wants to publish easily (audio clip or screenshots.)
# > - appended format; # That sound like the simplest, yet best solution so far. No change to # pnd_run.sh, only adding the previews and the icons there. sdound resonable # for me and will fixes thoses little limitation we have currently
It'd be easy to add to libpnd.
In fact, I added iso/squash/whatever+zip in a couple of times a year ago, its very easy.
Folks in IRC tended to dislike the idea though, so its not in place, but it could be.
atc -- your time to speak up!
# > - theres 'crazy like a fox' ideas too; # > # the thing I like here is that you could then have a modified bitbake to # build pnd more easily than with current situation. But all these overlay is # the best way to shoot into your feet imho. (once you start 2 apps and # more...)
You could just use debian packages at this point too, or other crazy tricks; but having 300 overlays is just not going to work imho :)
Still, floor is open for other ideas; I admit, I'm ovrworked and fried in the end, so really outrageous creative solutions aren't coming to mind for this .. so its very possible some truly awesome idea is out there.
But that side, the conventional ideas I think I listed there.
So I'm a bit of a fan of iso(whatever)+zip, but I've also not actually run down the pros/cons yet to see if its a worthy addition.
filesystem + zip-appended
Pro: - doesn't fight with current system - trivial to add with no user confusion - improve performance of some operations, with changes to libpnd to understand it; ie: fast fetch of screenshots
Cons: - still goofy filesystem plus something, which I don't personally mind
Before we consider acting on that, best to consider all the options, and pick the best overall one. Just adding this one would end up with two systems, and then if we find a better, we'd have 3 ... better to let it sit, and find a better solution, and go to that.
I also like libpnd nice and simple, so we can hack on it and grow/change it easily, with little baggage; keeping it simple makes for speedier adoption by developers, reliability makes it adoptable by users. Complexity is bad for both scenarios.
So libpnd hasn't changed pnd-files since inception a year or two back, but thats okay. We want adoption.
But nows a good time to figure out if there is changes to be.
jeff
-- If everyone would put barbecue sauce on their food, there would be no war.
And since all the chatter is in IRC ..
Try to keep it on list ;)
- .desktop files could replace PXML for sure; someone needs to itemize the value of PXML and compare to .desktop and see if .desktop can do it all, and if so, we could switch. We moved fast last time (we thought pandoras were shipping like 6 weeks later :), so I think we latched onto PXML quickly and just stuck with it. We also liked the idea of XML validators, etc, but we all know XML makes baby jesus cry.
- zip appending could be cute; would be uncompressed probably, since we're not worried about the compression (most stuff in the appdned bits are small, or uncompressible anyway, like audio clips or pics.) We're concerned about speed of locating, and the index, that a zip would provide .. faster access to _find and pull_ stuff. Nothing to do with runtime performance.. pnds already perform pretty well. App discovery and pulling PXLM and icons, thats slow.
- appending tar is no better than what we have now since it has no tail-index :)
Now, keep it on list :)
jeff
On Fri, 25 Feb 2011, Jeff Mitchell wrote:
# # Re-targeting the list. # # ---------- Forwarded message ---------- # From: Jeff Mitchell skeezix@skeleton.org # To: "[ISO-8859-1] S�bastien Huss" sebastien.huss@gmail.com # # On Fri, 25 Feb 2011, [ISO-8859-1] S�bastien Huss wrote: # # # The nice thing of having the filesystem at first and the rest appended at # # the end is that the filesystem is immediatly available to be mounted : no # # need to do ugly hack to have mount to see the filesystem # # Actually the loopback mechanism (these days, not years ago) can # usually take an offset, so you canhave a binheader and not too much work # .. as long as calculating where the filesystem is is easy :) # # # > - payload filesystem change; # # Might be nice, but does zipfs support Unix like priviledge-system ? # # Support enough I think; I doubt pnd_run.sh itself cares about the # appended parts, just the main payload; but libpnd cares about the appended # stuff, and its easy to read a basic zip, without mounting zipfs. # # I've often thought it would be cute to do iso+zip, with the zip # part having one PXML.xml or the .desktops, and icon(s), and anything the # dev wants to publish easily (audio clip or screenshots.) # # # > - appended format; # # That sound like the simplest, yet best solution so far. No change to # # pnd_run.sh, only adding the previews and the icons there. sdound resonable # # for me and will fixes thoses little limitation we have currently # # It'd be easy to add to libpnd. # # In fact, I added iso/squash/whatever+zip in a couple of times a # year ago, its very easy. # # Folks in IRC tended to dislike the idea though, so its not in # place, but it could be. # # atc -- your time to speak up! # # # > - theres 'crazy like a fox' ideas too; # # > # # the thing I like here is that you could then have a modified bitbake to # # build pnd more easily than with current situation. But all these overlay is # # the best way to shoot into your feet imho. (once you start 2 apps and # # more...) # # You could just use debian packages at this point too, or other # crazy tricks; but having 300 overlays is just not going to work imho :) # # Still, floor is open for other ideas; I admit, I'm ovrworked and # fried in the end, so really outrageous creative solutions aren't coming to # mind for this .. so its very possible some truly awesome idea is out # there. # # But that side, the conventional ideas I think I listed there. # # So I'm a bit of a fan of iso(whatever)+zip, but I've also not # actually run down the pros/cons yet to see if its a worthy addition. # # filesystem + zip-appended # # Pro: # - doesn't fight with current system # - trivial to add with no user confusion # - improve performance of some operations, with changes to libpnd # to understand it; ie: fast fetch of screenshots # # Cons: # - still goofy filesystem plus something, which I don't personally # mind # # Before we consider acting on that, best to consider all the # options, and pick the best overall one. Just adding this one would end up # with two systems, and then if we find a better, we'd have 3 ... better to # let it sit, and find a better solution, and go to that. # # I also like libpnd nice and simple, so we can hack on it and # grow/change it easily, with little baggage; keeping it simple makes for # speedier adoption by developers, reliability makes it adoptable by users. # Complexity is bad for both scenarios. # # So libpnd hasn't changed pnd-files since inception a year or two # back, but thats okay. We want adoption. # # But nows a good time to figure out if there is changes to be. # # jeff # # # -- # If everyone would put barbecue sauce on their food, there would be no war. # # !DSPAM:4d680190334175616312847! #
-- If everyone would put barbecue sauce on their food, there would be no war.
For various reasons I'd prefer a raw format to the appended pxml, pxml+png, zip, cranberryjuice, etc
There are various benefits and various hurdles, I'll try to outline the ones I see below.
First off though, this does not primarily care about what data is present in the pxml as that is discussed in separate threads, also, I'd argue we do want to keep the pxml as the definition format and still bundle it with the filesystem, benefit being that it's easy to edit with human-readable tools and application dirs still work unchanged.
Why change at all?
Well first off the current system is _slow_, not sure if the primary bottleneck is the back-seeking for the pxml or the xml parsing and dom scanning but by cutting both out and caching (on disk mind you) I was able to cut the mmenu load time in half with 172 pnds (and only 125 of those were be cached as some exist on the nand). Not to mention the fact this was despite all the static startup costs etc nor did cache any of the png loading (which did shave another 0.5 seconds off the startup).
An example format would be as follows:
First it's always prudent to begin with a file type identifier, I like four bytes as it aligns nicely with 32 bit boundaries and doesn't waste too much data, possibly PND followed by a null byte or 0xff.
Second follows the basic header in four 32 bit words (unsigned ints), I prefer 32 bits as they align with the pandora bitsize, and are usually large enough to give plenty of room for any enumeration as well as sizes/offsets.
pnd_version (first word should be self explanatory but likely to be prudent to keep it first just in case we go with something crazy like a streamed variant in the future making the following fields irrelevant)
header_size (in bytes, I'd include everything from the start of the PND)
file_size (in bytes, thus filesystem-size would be file_size - header_size)
checksum (could be md5 which would be word-sized but likely better to go with sha1 which should provide a good trade-off between cryptographic strength and speed, weighs in at 5 bytes (160 bits), not sure what would be the easiest to verify but could be the whole file with the checksum zero padded, possibly we could hash the header and filesystem separately)
filesystem_type (possibly not needed, and possibly this isn't the right place for it but I'd put it here for good measure)
The header payload could simply be an index of offsets into the for the pxml and png and we could call it a day, I have some thoughts regarding that as well but I'd say we can evaluate that separately once we've pondered what we have here.
What are the benefits of all this then?
1. We can discard bad or incomplete pnds early, in a single fixed size read of 25 (or 24 for md5) bytes we can first discard it if the file_size doesn't match the stat-ed result.
2. We have a simple integrity check using both size and checksum.
3. If we have a strong checksum signing the whole pnd is equivalent of signing only this header.
4. A repository can compare pnds directly against the hash provided here (if it fails the pnd is invalid anyway).
5. It's both simple and fast, we can get the entire header in two read()s and cast it directly to a C struct.
Oh, and we could append this header and maintain some additional compability but it would make testing for complete pnds more difficult as we'd have to back-seek for the PXML tag if we don't get the full file and find the appended PND 0xff. Also, if I understand correctly part of the reason for appending is that mounting loopback devices didn't support offsets when it was first brought up, this is no longer an issue as all loopback mounts support both and offset and a max_length option these days.
Any thoughts so far? Am I thinking too hard, making things overly complicated or missing the obvious? ;)
-- Anders
This summary of the suggested format should have been part of the original mail:
file-id (PND0xff) - 4 bytes version - 4 bytes header size - 4 bytes file size - 4 bytes checksum (md5/sha1) - 16/20 bytes filesystem type - 4 bytes header payload - x bytes filesystem - y bytes
On Tue, Mar 1, 2011 at 7:13 PM, Anders Petersson demitar@gmail.com wrote:
checksum (could be md5 which would be word-sized but likely better to go with sha1 which should provide a good trade-off between cryptographic strength and speed, weighs in at 5 bytes (160 bits), not sure what would be the easiest to verify but could be the whole file with the checksum zero padded, possibly we could hash the header and filesystem separately)
And this should of course have been 4 and 5 _words_ respectively (16 or 20 bytes), this affects the header sizes mentioned below as well.
-- Anders
On Tue, 1 Mar 2011, Anders Petersson wrote:
# There are various benefits and various hurdles, I'll try to outline the ones # I see below.
Thats what need more.. the pro/cons comparison, and relative to existing.
For historical perspective -- we did try a binary "TOC" very early on, but I don't recall why we discarded it. Perhaps it was problematic with the loopback mount, or didn't perform well enough, or any number of things .. the earlier kernels causes us lots of headaches thaqt changed our decisions :)
The TOC we did try took all of about 20 mins to add into libpnd; fragments of it might still be in there, or old gits; we had just put in a header 4bytes to define the PND-ness, and then a number of entries in the TOC, and for each entry an offset and size; thus we could read 4b and get the PNDness, then read the TOC-size and jump straight to the icon or PXML with a seek.
We did also test speed for SD -- knowing that all SD reads of certain sizes get blocked up to SD-chunk-size; so reading 1b is same speed as reading 30k.
One con that comes to mind: a binary header of any complexity means its harder for the devs to bundle up; nolonger just a make-this, make-that, cat 'em together.
# argue we do want to keep the pxml as the definition format and still # bundle it with the filesystem, benefit being that it's easy to edit with # human-readable tools and application dirs still work unchanged.
My well formedness rule require 'appended' tyep data (icon, PXML.xml, sound clips, whatever) to always be inside the main iso/whatever, so that it is not lost during unpack/repack, and there during a mount, etc. IMHO it is an error to omit the PXML.xml from the iso but we don't enforce it.
# Why change at all? # # Well first off the current system is _slow_, not sure if the primary # bottleneck is the back-seeking for the pxml or the xml parsing and dom # scanning but by cutting both out and caching (on disk mind you) I was able # to cut the mmenu load time in half with 172 pnds (and only 125 of those were # be cached as some exist on the nand). Not to mention the fact this was # despite all the static startup costs etc nor did cache any of the png # loading (which did shave another 0.5 seconds off the startup).
As noted in IRC; caching works for sure; I put on a cache at one poiint in mmenu, but pulled it out as in my implementation it cached the app scan result list, and thus the user had to hit 'refresh cache' (you'll notice rescan is still in the mmenu select menu.)
It did speed things up for sur; your approach of just caching the PXML and image surface can work without operator trigger, but a few caveats .. (ie: what if the guy changes themes, etc.) But all stuff we can handle easily.
So once HF5 goes out, submit the diff and I'll take a look, and we cna merge her in; we can slap a 'enable cache' (or possibly 'disable' if it seems reliable enough all the time), so user can manage it.
I've not tested in a year or more, but if can reduce mmenu start up in half, thats a big win ;)
Another option I toyed with was more drastic -- make mmwrapper maintain the app cache; so when mmenu exits and restarts, it pulls or shares the data from mmwrapper, and is _instant_ statr time. That was a bit dirty fealing but worked really well in testing, but I've not had time to work it over.
So an 'app pool' in mmwrapper might be the way to go right out, but it turns mmwrapper from a tiny little job into a bigger thing.
# First it's always prudent to begin with a file type identifier, I like four : # The header payload could simply be an index of offsets into the for the pxml # and png and we could call it a day, I have some thoughts regarding that as # well but I'd say we can evaluate that separately once we've pondered what we # have here.
When we fiddled with this before, we allowed for >1 icon and other bits too; we had always intended to have up to 1 icon per sub-app, and optional screenshot/sound-clips for the menus to try (you'll see this in PXML spec, but only ever worked for dir-apps of course, since the pnd-format we went with is overly restrictive.)
So theres definately an average of 2 attachments (PXML and icon), but quite a few apps would have 3-4 attachments (multiple sub-app icons.)
# Oh, and we could append this header and maintain some additional compability # but it would make testing for complete pnds more difficult as we'd have to # back-seek for the PXML tag if we don't get the full file and find the # appended PND 0xff. Also, if I understand correctly part of the reason for # appending is that mounting loopback devices didn't support offsets when it # was first brought up, this is no longer an issue as all loopback mounts # support both and offset and a max_length option these days.
A fun alternative is..
zip (uncompressed) everything. zip contains iso, PXML.xml, icon(s). Since its uncompressed, asking the zip tailer for the offset to the iso is trivial, and can use loopback to mount it (since everythig is uncompressed.) (the iso itself could be squash or crammed or whatever still, of course.)
So no custom binary header at all, just a zip of the whole lot.
(I've often brought up iso+zip, but I think zip-all would be the way to go)
# Any thoughts so far? Am I thinking too hard, making things overly # complicated or missing the obvious? ;)
We're coders, our job is to think way too hard about how to do simple things :)
jeff
-- If everyone would put barbecue sauce on their food, there would be no war.
On Tue, 1 Mar 2011, Jeff Mitchell wrote:
# A fun alternative is.. # # zip (uncompressed) everything. # zip contains iso, PXML.xml, icon(s). # Since its uncompressed, asking the zip tailer for the offset to # the iso is trivial, and can use loopback to mount it (since everythig is # uncompressed.) # (the iso itself could be squash or crammed or whatever still, of # course.) # # So no custom binary header at all, just a zip of the whole lot. # # (I've often brought up iso+zip, but I think zip-all would be the # way to go)
atc brought up something we also trie dlong ago but discarded at the time..
Pure zip pnd; not even an iso inside. Just a zip filesystem.
When we (very briefly!) tried it before, I think we had both zipfs performance issues, and FUSE was broken badly anyway. I seem to recall zipfs was really slow or something.
I really should go back to those ancient IRC chats and see why we decideed or un-decided some of these things!
Anyway, a pure-zip pnd would make life easier for all devs, so if its feasible now, it might just be the way to go. (All them jar's and Quake paks in the wild would suggest it works okay. Everything else about java might be crazy, but jar's are okay ;)
So .. anyone got time to make performance tests across the different formats?
jeff
-- If everyone would put barbecue sauce on their food, there would be no war.
(Stupid gmail, do I have to look in labs for a reply to list? >.<)
On Tue, Mar 1, 2011 at 8:44 PM, Jeff Mitchell skeezix@skeleton.org wrote:
atc brought up something we also trie dlong ago but discarded at
the time..
Pure zip pnd; not even an iso inside. Just a zip filesystem.
I had some ideas which while still applicable now would apply on different parts not the pnd itself.
To put it simply, this has so many attractive properties I'd be happy to work around the quirks we have to deal with. Unzip, edit and rezip would be awesome.
Now there are a number of issues to look into.
1. My impression is that there's actually a plethora of zip versions out there, and which ones can we support? All the following points come back to this one, if it's in a zip version we cannot reliably mount it's not an option.
2. Filenames, how many characters, etc
3. Can we get a checksum in here or would checking integrity have to come from the outside? (I'm sure zip has some kind of checksum but not sure there's any of cryptographic strength we can realiably use given 1 above, again strong hashes would only be needed for signing.)
4. How much will potential compression (which I imagine we don't want to do if we can avoid it for runtime performance) mess with binary deltas of pnds?
A side effect which is good just as it is with an iso is that we cannot set permissions, that also means no permissions can mess up (at most we'd like the x bit not to be set for everything).
One thing we should consider is to either just before zipping or by rezipping on the repo add a raw version of the pxml data in the same format as the cache described below.
However while I'd not be as worried about multiple versions of the same pnds as I perhaps should, it would complicate rezipping to ensure we don't have a stale cache in the zip, so perhaps we should just take the one-time performance hit and get on with our lives.
So .. anyone got time to make performance tests across the
different formats?
This would be useful, but I'd expect all uncompressed formats to be equal, I wonder how compressed fare against uncompressed though.
We did also test speed for SD -- knowing that all SD reads of
certain sizes get blocked up to SD-chunk-size; so reading 1b is same speed
as reading 30k.
I was considering this but not taking the SD-card reads into account but just regular block sizes, but this just makes one other optimization all the more useful:
Hopefully the directory data including the directory index and the stat data will be cached when read by the kernel from the SD-card but this is outside of our control and we cannot usefully avoid walking the /media/*/pandora/... path. _However_ we don't need to read _any_ data from the pnd file itself if we store all such data in a separate cache file for all the pnds on that media along with any useful indexes.
Such an index could usefully be stored in pandora/cache/<cache version>/pnd.cache or similar and likely be read in a single SD-read and most likely mmap()ed by the library.
Not sure if it'd be worth the trouble to update in-place or not.
-- Anders