Forwarding back to list
--
If everyone would put barbecue sauce on their food, there would be no war.
---------- Forwarded message ----------
Date: Tue, 1 Mar 2011 16:21:48 -0500 (EST)
From: Jeff Mitchell <skeezix(a)skeleton.org>
To: Anders Petersson <demitar(a)gmail.com>
Subject: Re: [Firmware-dev] on pnd-file format changes
On Tue, 1 Mar 2011, Anders Petersson wrote:
# 1. My impression is that there's actually a plethora of zip versions out
# there, and which ones can we support? All the following points come back to
# this one, if it's in a zip version we cannot reliably mount it's not an
# option.
Zip is pretty well locked down, but more.. I think as soon as you
say 'uncompressed zip' you needn't worry too much.
But if we're talking full zip with no iso, then yeah, we'd want to
constrain the type of compression used perhaps, but not sure how many zip
tools actually let you control that, so might be packaging trouble. But I
honestly doubt theres many incompaible zip versions .. some bleeding edge
zip tools have some oddities for sure, but I'd say 99% of implementations
are fine.
One thing I do like about zips is they were designed to go
anywhere in the stream; the tail-index they use is handy, and anything
after the index is 'comment'.
(Did you know that zip tools use backwards seek to find the zip
header? I bet you didn't know that, but they do! libpnd is not the only
goofy one doing it; just so few people use comments, you never notice ;) I
think some zip implementatins tried to append negative-offset-to-header
after comment, but thats not to spec IIRC. (But then, its not like I'm
reading zip spec in the last 20 years either :P)
# 2. Filenames, how many characters, etc
yeah, we'd have to well define that.
# 3. Can we get a checksum in here or would checking integrity have to
# come from the outside? (I'm sure zip has some kind of checksum but not
# sure there's any of cryptographic strength we can realiably use given 1
# above, again strong hashes would only be needed for signing.)
zip includes a checksum; probably crc32 or something simple.
Could mandate putting md5's in there on the part of the dev if we
needed, but we dont' really need anything hard core in there.
# 4. How much will potential compression (which I imagine we don't want to do
# if we can avoid it for runtime performance) mess with binary deltas of pnds?
We do squashfs now, so why worry?
# One thing we should consider is to either just before zipping or by
# rezipping on the repo add a raw version of the pxml data in the same format
# as the cache described below.
eh?
# However while I'd not be as worried about multiple versions of the same pnds
# as I perhaps should, it would complicate rezipping to ensure we don't have a
# stale cache in the zip, so perhaps we should just take the one-time
# performance hit and get on with our lives.
PXML cache?
# So .. anyone got time to make performance tests across the
# > different formats?
#
# This would be useful, but I'd expect all uncompressed formats to be equal, I
# wonder how compressed fare against uncompressed though.
#
# We did also test speed for SD -- knowing that all SD reads of
# >
# certain sizes get blocked up to SD-chunk-size; so reading 1b is same speed
# > as reading 30k.
#
# I was considering this but not taking the SD-card reads into account but
# just regular block sizes, but this just makes one other optimization all the
# more useful:
#
# Hopefully the directory data including the directory index and the stat data
# will be cached when read by the kernel from the SD-card but this is outside
# of our control and we cannot usefully avoid walking the /media/*/pandora/...
# path. _However_ we don't need to read _any_ data from the pnd file itself if
# we store all such data in a separate cache file for all the pnds on that
# media along with any useful indexes.
Ah, declare up above and fill in later? What is this, C code? ;)
As long as we don't get cache-out-of-sync problems, or the speed
of comparing cache versions to real versions just slows you down, I don't
mind caching; but if we're using zip, we could probably read live pretty
fast. .. but you've really got cache on the mind eh? :)
# Such an index could usefully be stored in pandora/cache/<cache
# version>/pnd.cache or similar and likely be read in a single SD-read and
# most likely mmap()ed by the library.
#
# Not sure if it'd be worth the trouble to update in-place or not.
We never want to update the pnd-file (zip or otherwise inside.)
Its nice for user to just pull the pnd off and give it to someone, and its
fresh and original; all changes go to appdata.
that said, you can put a cache into the guys appdata if you want
(with a suitably crazy filename so as to not likely collide with a real
file in the app.)
jeff
--
If everyone would put barbecue sauce on their food, there would be no war.