GP32 Many Small Files Take More Space?

[JRyouK]

don't know anyone has already realised that if you put a lot of small files on your GP32, it will easily use up the SMC easily than having a few big files with the same total size.

i think that's due to the SMC is formatted in FAT12. am i right ?

if so, i think games/demos should be packed into fewer big files or even a single file via some mechanisms. probably like the *.jar file in java? or is if possible to format the SMC in FAT16/FAT32?

hope this help!


JRyouK

 

[Tobriand]

Its partly due to the whole Fat12 thing, but small files always take up more space - more wasted bits of clusters. Its the reason KoF91 uses zipped up characters, since unzipped, a 2 mb character takes 8 mb often...

 

[JRyouK]

so we should advocate programmer/developer to pack small data files into a single file!!

how about having a I/O lib to handle that? Open/modify/close a file in the pack in the same way as in the smc. update/add/delete file into the pack. one game/demo has one pack. how is that?

JRyouK

 

[Akuma no Houkon]

so we should advocate programmer/developer to pack small data files into a single file!!

how about having a I/O lib to handle that? Open/modify/close a file in the pack in the same way as in the smc. update/add/delete file into the pack. one game/demo has one pack. how is that?

JRyouK

Not really a workable solution, some games, like PerfectFit, allow the user to create their own level packs, so the files could be any size and any number. Some apps like emulators are entirely dependant on roms, save states, settings files, and most other games and apps have similar dependancies. For many games and apps, having one large media file is not a workable solution.

 

[rcx21000]

When there are lots of files, it takes up lots of space becuase of clusters. Clusters are little sections of storage that can be taken up by only one file. For example, if the GP32 used 8KB clusters (I don't know what it actually is), a 1KB file would take up 8KB (the whole cluster, only one file per cluster), a 9KB file would use 16KB (2 clusters). If you have 200 1KB files, it will take up 1600KB :wacko:

 

[JRyouK]

(rcx21000 @ May 12 2004, 08:38 PM)
When there are lots of files, it takes up lots of space becuase of clusters. Clusters are little sections of storage that can be taken up by only one file. For example, if the GP32 used 8KB clusters (I don't know what it actually is), a 1KB file would take up 8KB (the whole cluster, only one file per cluster), a 9KB file would use 16KB (2 clusters). If you have 200 1KB files, it will take up 1600KB

that's why i think of the idea to have all files packed into a single file, with library support to load the files as if they are not packed together. something like a virtual file system within a file. as usual we can open a file with specified file name. For those apps/emu require loading/writing files, the library should handle that as well. for roms, we may have a utility packing all roms into a file before uploading to smc. although i don't know how to implement such a library, i think it's not hard for those emu developers who are already able to load files from a rom structure.
Adding such I/O functions to a early-stage library (e.g. Mr.Mirko's) will benefit the whole community, i think

Am i expecting too much? just wanna give my opinion.

JRyouK

 

[Squidge]

It's quite easy to do as long as you expect to change the files in the pak from the gp32 side (ie. most of the time). Modified files can live outside the pak.

For an example, just look at BOR

Unfortunately, it does mean that for every change, you need to delete the pak file and replace it. No many games require changes often though.

 

[RockOut]

This may sound dumb, but why not just have 1kb clusters since the largest smart media cards only have 128 mbs of space? You would have less wasted space and you can fit more files on one card!

 

[ColinR]

But then you have more clusters on the card, and so more places for a piece of file to theoretically be in, meaning that it would take longer for the piece of file that the GP is looking for to be found. Not much of an issue with modern hardware and solid state technology but a pain in the arse and speed killer back in the day when filesystems were originally devised.

8 times as small clusters means 8 times as many places a file can be found meaning 8 times as much data needs to travel to the card to "seek" the sector you require. Cluster size does scale with disk size, but I think 8K (I think - may be lower) is the smallest it goes to with a hard disk (floppies go down to 512 bytes - probably lower with 5.25 disks).

 

[VirtuaLeech]

It's quite easy to do as long as you expect to change the files in the pak from the gp32 side (ie. most of the time). Modified files can live outside the pak.

For an example, just look at BOR

Unfortunately, it does mean that for every change, you need to delete the pak file and replace it. No many games require changes often though.

An easier solution is to put all files into a zip file and have the application read from the zip file. Only takes up one file in the filesystem, is easy to update with new files, allows reading and writing and has a nice compression as a bonus.

 

[Squidge]

Good idea! However, I think for some programs that use a lot of ram, the zip file should not be compressed (otherwise you need to load the compressed data into memory, and then decompress into memory = lots of memory required).

 

[VirtuaLeech]

That's true. You must also consider the time it takes to uncompress to see if it fits your application. But if your application loads most things into memory at the start of the level/round/etc. zip is a nice solution especially since there is already a pretty good zip library ported to the GP32.

 

[slubman]

That's true. You must also consider the time it takes to uncompress to see if it fits your application. But if your application loads most things into memory at the start of the level/round/etc. zip is a nice solution especially since there is already a pretty good zip library ported to the GP32.

What is this library VirtuaLeech ?
Wheres can i found it ?

 

[JRyouK]

That's true. You must also consider the time it takes to uncompress to see if it fits your application. But if your application loads most things into memory at the start of the level/round/etc. zip is a nice solution especially since there is already a pretty good zip library ported to the GP32.

so why not just pack them together without any compression? no speed overhead!


JRyouK

 

[cyberic]

Mr mirko's SDK include ZDA container support, that is a kind of zip file containing lots of files.