How To Store Thousands Of 1Kb Files Efficiently?

[zapman]

Hi!

TangoGPS works great on the Pandora, the map data gets downloaded from openstreetmap.org and can be saved on a SD card for offline use. Now the problem is, that the map tiles are lots of very small .png files (<1KB) and the SD segment size seems to be ~4KB. That means that 120mb of map tiles take about 800mb of space on the card :/

Can someone pls tell me what would be the best way to store them efficiently? Maybe putting them all in some fs-file that can be mounted? Or some kind of archive file that would be accessible by tangoGPS? Reformatting the SD card with a smaller block size?

Two things would be nice to have as well:
# beeing able to reduce/ increase the storage file later on
# automating the process, that you don't have to punch in cosole commands each time you want to use the map package.

Thx for your help!

 

[Caine]

Maybe squashfs is an option.

The data could optionally be bundled into the PND file as well (though rebuilding the PND takes some time and dynamic changes would go into the overlay and take up more space again).

Automation can easily be handled by wrapping a shell script around TangoGPS.

 

[Vimacs]

Squashfs is a read only filesystem, so that wont work.

What you could do thou is creating a compressed jffs2(or other r/w compressed fs) image for the data. Currently this would require some work on your side and root privileges on the Pandora. However this is something i would like to implement as a pxml option sooner or later to allow having the appdata as per app compressed fs images.

Alternatively you could fuse mount a .zip for the time being, fuse-zip should be shipped in the default fs - this will also need root thou.

Third option would be to use libz internally and just writing a single compressed file, this is the only proper option right now as it doesn't require root privileges, but its some work on your end.

Edit: A ugly option for the time being would be to keep appending the data to a Squashfs image, that's quite slow, dirty, and inefficient thou.

 

[Caine]

Squashfs is a read only filesystem, so that wont work.

Depends a bit on the usage scenario.

For automatic downloads it wouldn't work very well. You're right there. You could store the auto downloads in a union overlay and repack them weekly or monthly to reduce space usage, but it's not really an ideal solution.

For a not too frequent manual download it's quite feasible to just repack the map data image.

However this is something i would like to implement as a pxml option sooner or later to allow having the appdata as per app compressed fs images.

That sounds like an awesome idea Vimacs. I would definitely like to see that.

 

[zapman]

alright, thx for your thoughts.

I took the easiest way and reformatte my sd-card to ext2 with a block size of 1024 byte. Now the files take only about 10% additial storage space on the SD card, that's acceptable imo.