Writing Too Much To The Sd (using Mmap)

[Hitnrun]

I've read that SD cards have a limited write life-time.

For Zelda Classic, I need a 70mb memory chunk for the bigger quests, I allocated a 70mb file on the SD, and used mmap to simulate memory to the application. In the beginning, it writes like hell, totally randomly everywhere to the file, but after this initial load, writes are rare. But this 70mb writing is fierce, and takes about 3 minutes.

Seems to be working, but is this too heavy on the card? Could I be sued by user for making their SD cards lifetime smaller?

 

[Squidge]

You could always use a caching system, so you keep a number of chunks in memory and flush out when you have a certain amount in memory.

 

[ledow]

I think the general rule is "avoid writing to the SD as much as possible without interfering with things".

SD do have a limited lifetime that you can, if you are running worst-cases 24-hours a day hit very, very quickly and the card will stop accepting write requests (or even just stop working completely). But modern ones can do all sorts of things to prevent it, and the filesystem used on the GP2X is designed to help.

But yes, mmaping a large file that gets written to all over the place for 3 whole minutes sounds a little sub-optimal (what speed is the SD card? Just how much data is getting written in those 3 minutes?). Your card will have a reduced lifespan compared to one that doesn't do this. I wonder why the writes are so random? Caching the data sounds like a good idea, as does on-the-fly compression or similar to reduce the amount of data actually being written to the card. 70Mb is a lot of data that the GP2X can't hold in it's RAM all at once, so it makes me wonder why you have to generate all that data before you start?

What sort of data is it? Sprite-caches after some sort of graphical maniuplation (I know that some games load one image, rotate it many times and keep the results to save calculating them each time)?

 

[Blah]

It sounds like the SD is being used as virtual memory. Is that correct?

 

[Hitnrun]

I've read that SD cards have a limited write life-time.

For Zelda Classic, I need a 70mb memory chunk for the bigger quests, I allocated a 70mb file on the SD, and used mmap to simulate memory to the application. In the beginning, it writes like hell, totally randomly everywhere to the file, but after this initial load, writes are rare. But this 70mb writing is fierce, and takes about 3 minutes.

Seems to be working, but is this too heavy on the card? Could I be sued by user for making their SD cards lifetime smaller?

You could always use a caching system, so you keep a number of chunks in memory and flush out when you have a certain amount in memory.

The big problem is that I'm porting it, I didn't want to modifiy the source too much, and the way it's designed, to work like this I would need to.

I think the general rule is "avoid writing to the SD as much as possible without interfering with things".

SD do have a limited lifetime that you can, if you are running worst-cases 24-hours a day hit very, very quickly and the card will stop accepting write requests (or even just stop working completely). But modern ones can do all sorts of things to prevent it, and the filesystem used on the GP2X is designed to help.

But yes, mmaping a large file that gets written to all over the place for 3 whole minutes sounds a little sub-optimal (what speed is the SD card? Just how much data is getting written in those 3 minutes?). Your card will have a reduced lifespan compared to one that doesn't do this. I wonder why the writes are so random? Caching the data sounds like a good idea, as does on-the-fly compression or similar to reduce the amount of data actually being written to the card. 70Mb is a lot of data that the GP2X can't hold in it's RAM all at once, so it makes me wonder why you have to generate all that data before you start?

What sort of data is it? Sprite-caches after some sort of graphical maniuplation (I know that some games load one image, rotate it many times and keep the results to save calculating them each time)?

The data are the zelda maps for that level, all maps are loaded on ram at startup. It is an array of very big structs (6272 bytes), multiplied by the a fixed number of screens (136), multiplied by the number of maps of this quest (83), which gives 70798336. This array, on load, isn't written totally sequentially, it depends on how the maps were written on the quest file, and some screens are copied from one position to others and modified, I'm not really sure how it works because it is not my code, but for each of these map positions a memset if performed, 3 members are pointers and are malloc'ed (this will be a problem too), and the rest are read from the disk.

Hmm taking a better look at the code, it is seems it normally is written sequentially, the out of orders writes looks like to be less often than I thought. Still, this specific map takes about 3 minutes to load competly.

After this initial load looks like it is not modified anymore, only read, but then again this time it is really random access, it is accessed as array indices.

Its the way it is designed, so I need to keep it like this to not need to modify it too much...

It sounds like the SD is being used as virtual memory. Is that correct?

You can say that, is like a swap file for data that does not fit on memory.

 

[Squidge]

The big problem is that I'm porting it, I didn't want to modifiy the source too much, and the way it's designed, to work like this I would need to.

Sometimes you need to completely rework a section of code however whilst porting the app, specially if its going to trash an sd card after running it 10 times due to the sheer number of writes.

 

[ledow]

Well, it seems that you may have to rewrite the portions of code that deal with such things. Keeping 70Mb of data around in memory on a 32Mb machine is a big waste of space, and if you're mmapping that memory from SD card, then it's not good.

More likely, you'll have to rewrite any functions that access that data to use wrapper functions which access a smaller, contained area of memory which can "page in" other parts of the map as they are needed, so that only the screens "near" the player are in memory at any one time.

Perhaps another idea would be to introduce a "copy-on-write" principal into the code, so that any writes to that area are actually stored somewhere in memory until there's enough of them to write to disk (and thus, you reduce multiple writes to the same areas from hitting the disk several times).

Either way, 70Mb of what is intended to be in-RAM data sitting in "swap" on an SD card isn't a nice way to handle things, especially on a low-RAM machine. It was probably implemented as a speed/memory tradeoff in the first place - by having it in memory, you increase performance at the cost of RAM. I'd be looking at reversing that tradeoff unless the performance drop of doing so was seriously heavy (in which case you have to ask yourself why you're trying to get such a program running on a GP2X).

 

[Hitnrun]

There's other problem, the quest file is compressed by an allegro function, and the file access is forward-only-very-slow-seek, that's probably the reason it loads everything in memory.

The good thing is that this memory is used only for reading after loading from this (as far as I know), does randomly reading a lot the SD is a problem too? I think I could optimize more the number of writes made to the SD.

The bad thing is, the code isn't very readable, it is plain C without clearly defined objects, so this very same array can be loaded in 3 different places, and memory loaded in one place can be freed in other place. It really wasn't meant for low-memory devices (at least the big quests, the default quests are ~300kb file/55mb memory, which fits very nicelly on the gp2x).

I am mostly asking this, to include a disclaimer with the release (THIS CAN KILL YOUR SD, USE AT YOUR OWN RISK), I really want to make this run on the gp2x even if it wasn't mean to!

 

[Hitnrun]

Ok, I tested with this way and it worked, I was able to play the 70mb quest using this swap technique. It took about 5 minutes to load everything and begin playing, I just needed to remove the memset for the 70mb array, it was crashing the application. But the array is initialized fully anyway, so it shouldn't be needed.

Now I know it is possible, just need to work out what the consequences of doing this.

 

[Kramy]

Ack! That sounds like a really bad way of designing it. This seems like a fine example of where a simpler, higher level language would actually have more efficiency.

 

[Hitnrun]

Well as they say on their website, ZC started on dos, even before the C99 spec was approved, so there's lot of legacy in the code, which if changed could pose lots of compatibilities problems.

There's lots of ifs depending on specific versions, for the older quests continue to work.

 

[gmiller1018]

Not sure if you have not already thought of this but you could wrap all access to the array (maybe only of on the GP2X) and then implement a write-back cache system (as suggested) to minimize the read/write to the SD. If most of the access is sequential then a cache system should be pretty efficient depending on your cache buffer size. I you have some problems with that I could send you a URL for some cache code I wrote for the class I teach on Machine Architecture. The code is not generally applicable but it could be mangled into something that might help. If you need it feel free to use it I only use it to show examples of direct, fully associative, and set associative cache with write through and write back options.

BTW: The wrapping would be easier (in my opinion) of the code used a pointer and the rest of the code used that:

For example:
CODE


struct foo stuff[1000]
struct foo *now;


// old
stuff.junk = 12;
// or
if (stuff.flag == 1) {
}


// new ....
now = stuff + i;

now->junk = 12;
if(new->flag == 1) {
}



this might be too simplistic but I thought I would throw it in ....

 

[Hitnrun]

Not sure if you have not already thought of this but you could wrap all access to the array (maybe only of on the GP2X) and then implement a write-back cache system (as suggested) to minimize the read/write to the SD. If most of the access is sequential then a cache system should be pretty efficient depending on your cache buffer size. I you have some problems with that I could send you a URL for some cache code I wrote for the class I teach on Machine Architecture. The code is not generally applicable but it could be mangled into something that might help. If you need it feel free to use it I only use it to show examples of direct, fully associative, and set associative cache with write through and write back options.

BTW: The wrapping would be easier (in my opinion) of the code used a pointer and the rest of the code used that:

For example:
CODE

struct foo stuff[1000]
struct foo *now;


// old
stuff.junk = 12;
// or
if (stuff.flag == 1) {
}


// new ....
now = stuff + i;

now->junk = 12;
if(new->flag == 1) {
}



this might be too simplistic but I thought I would throw it in ....

Hmm I don't know if I understood, how would this diminish the memory requirements?
As I've studied the code looks like this array is not written after the initial load, maybe only reading (and seeking) does not pose to much problem to the SD?

There's another problem that I can't read the source file if not sequentially from the begging, because it is compressed in an custom format that does not support seek.

 

[gmiller1018]

It does not diminish the need for storage , bit if the references are changes to this form them your "cache" code can return a pointer to the correct cache buffer and you will be able to put the code in simpler. I knew I was going for a too simplistic approach, more details would have been better.