For various reasons I'd prefer a raw format to the appended pxml, pxml+png, zip, cranberryjuice, etc

There are various benefits and various hurdles, I'll try to outline the ones I see below.

First off though, this does not primarily care about what data is present in the pxml as that is discussed in separate threads, also, I'd argue we do want to keep the pxml as the definition format and still bundle it with the filesystem, benefit being that it's easy to edit with human-readable tools and application dirs still work unchanged.


Why change at all?

Well first off the current system is _slow_, not sure if the primary bottleneck is the back-seeking for the pxml or the xml parsing and dom scanning but by cutting both out and caching (on disk mind you) I was able to cut the mmenu load time in half with 172 pnds (and only 125 of those were be cached as some exist on the nand). Not to mention the fact this was despite all the static startup costs etc nor did cache any of the png loading (which did shave another 0.5 seconds off the startup).


An example format would be as follows:

First it's always prudent to begin with a file type identifier, I like four bytes as it aligns nicely with 32 bit boundaries and doesn't waste too much data, possibly PND followed by a null byte or 0xff.

Second follows the basic header in four 32 bit words (unsigned ints), I prefer 32 bits as they align with the pandora bitsize, and are usually large enough to give plenty of room for any enumeration as well as sizes/offsets.

pnd_version (first word should be self explanatory but likely to be prudent to keep it first just in case we go with something crazy like a streamed variant in the future making the following fields irrelevant)

header_size (in bytes, I'd include everything from the start of the PND)

file_size (in bytes, thus filesystem-size would be file_size - header_size)

checksum (could be md5 which would be word-sized but likely better to go with sha1 which should provide a good trade-off between cryptographic strength and speed, weighs in at 5 bytes (160 bits), not sure what would be the easiest to verify but could be the whole file with the checksum zero padded, possibly we could hash the header and filesystem separately)

filesystem_type (possibly not needed, and possibly this isn't the right place for it but I'd put it here for good measure)


The header payload could simply be an index of offsets into the for the pxml and png and we could call it a day, I have some thoughts regarding that as well but I'd say we can evaluate that separately once we've pondered what we have here.


What are the benefits of all this then?

1. We can discard bad or incomplete pnds early, in a single fixed size read of 25 (or 24 for md5) bytes we can first discard it if the file_size doesn't match the stat-ed result.

2. We have a simple integrity check using both size and checksum.

3. If we have a strong checksum signing the whole pnd is equivalent of signing only this header.

4. A repository can compare pnds directly against the hash provided here (if it fails the pnd is invalid anyway).

5. It's both simple and fast, we can get the entire header in two read()s and cast it directly to a C struct.


Oh, and we could append this header and maintain some additional compability but it would make testing for complete pnds more difficult as we'd have to back-seek for the PXML tag if we don't get the full file and find the appended PND 0xff. Also, if I understand correctly part of the reason for appending is that mounting loopback devices didn't support offsets when it was first brought up, this is no longer an issue as all loopback mounts support both and offset and a max_length option these days.


Any thoughts so far? Am I thinking too hard, making things overly complicated or missing the obvious? ;)

--
Anders