(Stupid gmail, do I have to look in labs for a reply to list? >.<)

On Tue, Mar 1, 2011 at 8:44 PM, Jeff Mitchell <skeezix@skeleton.org> wrote:
       atc brought up something we also trie dlong ago but discarded at
the time..

       Pure zip pnd; not even an iso inside. Just a zip filesystem.

I had some ideas which while still applicable now would apply on different parts not the pnd itself.

To put it simply, this has so many attractive properties I'd be happy to work around the quirks we have to deal with. Unzip, edit and rezip would be awesome.

Now there are a number of issues to look into.

1. My impression is that there's actually a plethora of zip versions out there, and which ones can we support? All the following points come back to this one, if it's in a zip version we cannot reliably mount it's not an option.

2. Filenames, how many characters, etc

3. Can we get a checksum in here or would checking integrity have to come from the outside? (I'm sure zip has some kind of checksum but not sure there's any of cryptographic strength we can realiably use given 1 above, again strong hashes would only be needed for signing.)

4. How much will potential compression (which I imagine we don't want to do if we can avoid it for runtime performance) mess with binary deltas of pnds?

A side effect which is good just as it is with an iso is that we cannot set permissions, that also means no permissions can mess up (at most we'd like the x bit not to be set for everything).


One thing we should consider is to either just before zipping or by rezipping on the repo add a raw version of the pxml data in the same format as the cache described below.

However while I'd not be as worried about multiple versions of the same pnds as I perhaps should, it would complicate rezipping to ensure we don't have a stale cache in the zip, so perhaps we should just take the one-time performance hit and get on with our lives.


       So .. anyone got time to make performance tests across the
different formats?

This would be useful, but I'd expect all uncompressed formats to be equal, I wonder how compressed fare against uncompressed though.
 

        We did also test speed for SD -- knowing that all SD reads of 
certain sizes get blocked up to SD-chunk-size; so reading 1b is same speed
as reading 30k.

I was considering this but not taking the SD-card reads into account but just regular block sizes, but this just makes one other optimization all the more useful:

Hopefully the directory data including the directory index and the stat data will be cached when read by the kernel from the SD-card but this is outside of our control and we cannot usefully avoid walking the /media/*/pandora/... path. _However_ we don't need to read _any_ data from the pnd file itself if we store all such data in a separate cache file for all the pnds on that media along with any useful indexes.

Such an index could usefully be stored in pandora/cache/<cache version>/pnd.cache or similar and likely be read in a single SD-read and most likely mmap()ed by the library.

Not sure if it'd be worth the trouble to update in-place or not.

--
Anders