Rewrite Limit

[Undefined Error]

I've heard several people, both on this forum and in countless others, refer to the limited number of erase/write operations that Flash ROM (like the NAND memory in the GP2X or the SD cards) can handle. I am familiar with the problem and why it happens. (Those of you who are not, check out this lovely Wikipedia entry on Flash Memory, paying particular attention to the third paragraph of the Overview section. The HowStuffWorks link at the bottom is also quite informative.)

What I am not clear on is: Is this really a problem that affects people in their regular use of personal electronics? Or is it merely a known spectre of doom that computer afficionados are particularly aware of and fond of referencing?

I rather thought the whole big deal about NAND flash ROM, aside from the speed enhancements, was the raising of the theoretical write limit to and beyond 1 million writes, which seems like an awful lot for anything that's not constantly rewriting to the disk, like a virtual memory system. I understand why common, storage-write-intensive techniques like virtual memory are therefore not used on systems that rely on NAND memory or flash ROM card. But do the normal daily operations of most personal electronics exceed this limit in an amount of time we should be worried about?

 

[Digital Awakening]

The problem with limited writes is when you want to use the memory for swap files as could be needed for some emulations. Basicly using the flash memory to cover up for the lack of RAM (some emus require more then 64MB!) would drasticly shorten the lifespan of any flash based memory. Normal useage is no problem, exept if you constatly delete and install stuff over many years. So for things that you do delete often use an SD card and not the internal NAND if you want to be sure that your unit lasts forever.

 

[Squidge]

Well, on most flash cards there is a system called "Wear levelling". I'm not sure if this is described on the sites you mention, so basically what happens is that the file system is not given physical access to the card, but rather using another layer that performs a logical<->physical relationship. This means that the physical blocks may be stored anywhere on the disk, and can be relocated at any time. The file system is unaware of this as it only handles the logical side.

For example, if the file system decided to update block 64 for example, the lower layer may decide that block 64 is stored on physical block 20. If the same block is updated again, it may decide to store it on block 92, and mark block 20 as free.

This means that on a card that has a erase/write cycle of 100,000, that figure can easily be multiplied by the space available on the disk, so you can get 100,000,000 writes for example.

So, the more free space you have on your card, the longer it'll last

 

[Winterkid]

*looks at all his full SMC's with a shocked look on his face*
Uh....

 

[Tobriand]

Does that mean, rather ironically, that an SMC (or SD or whatever) that is kept near capacity - say with 5 mb left, and fairly frequently has a 1mb block of data written to it (overwriting some other block of data) would actually last longer - or at least, go for longer before bytes started to drop off - if its entire contents were flushed and then re-written every month or so?

 

[Squidge]

Well, if you do that, there's a good chance that the 1mb chunk will have access to more free blocks to play with than if you didn't, so in theory at least, it should last longer, assuming the logical->physical layer doesn't choose the same blocks when you flushed and rewrote the card (but only a braindead card reader would do that...)

 

[Axe_Murderer]

Well, on most flash cards there is a system called "Wear levelling". I'm not sure if this is described on the sites you mention, so basically what happens is that the file system is not given physical access to the card, but rather using another layer that performs a logical<->physical relationship. This means that the physical blocks may be stored anywhere on the disk, and can be relocated at any time. The file system is unaware of this as it only handles the logical side.

For example, if the file system decided to update block 64 for example, the lower layer may decide that block 64 is stored on physical block 20. If the same block is updated again, it may decide to store it on block 92, and mark block 20 as free.

This means that on a card that has a erase/write cycle of 100,000, that figure can easily be multiplied by the space available on the disk, so you can get 100,000,000 writes for example.

So, the more free space you have on your card, the longer it'll last

​

So, where does this 'lower layer' store information to keep track of what it's doing? If it also uses NAND memory, then how does it avoid creating a similar problem for itself? Or is there another bit of non-volatile memory that it uses which isn't prone to the same number of limited writes?

 

[grahf]

In my opinion, who cares? By the time a card actually stops functioning from use, a replacement will be dirt cheap. Its not worth stressing over. Just use it.

 

[junker]

you can't replace the onboard nand memory

 

[Robster]

Tobriand: you're actually right in that each time you flushed the card, the 5MB worth of free space would end up on different physical sectors and you might extend the life of your card. The only thing to be concerned about is corruption of your data while reading it off and then re-writing it, but as long as you verified that it was correctly written, you'd be fine.

Ultimately with Flash memory, the reality is that blocks will go bad, and so you need to have a strategy in place for dealing with that.

Different types of memory have varying levels of intelligence about wear levelling and bad block remapping.

Raw NAND chips don't do any of it, you need to handle it all yourself. This may be a bit of an issue if people are planning to store files on the NAND chip in the GP2X; I'm sure they won't be updated all that often so the wear levelling shouldn't be an issue but bad blocks do need to be dealt with.

SMC is basically just raw NAND in a plastic wrapper, but the SMC specification details a logical-physical mapping scheme designed to deal with wear levelling and bad block remapping. It's a big job for a software developer to implement all that stuff.

SD and MMC cards have intelligence built in to do wear levelling and bad block remapping. That makes implementing drivers for these cards much easier. The problem of blocks going bad still remains though, so in Tobriand's scenario of a card with 5MB free having a 1MB block rewritten often, you may still find that the blocks go bad relatively quickly.

Axe_Murderer: In the Flash chip spec, it says that the first few blocks of Flash must be guaranteed not to have errors over a 10,000 write lifetime (not sure on the exact number). So you should be able to be confident about the initial information about your card contents. But you are right, if the information about the block remapping (which is stored in Flash as well) goes bad, then you will get problems

 

[Squidge]

Axe_Murderer: Some cards use the "guaranteed to be good for at least X cycles" blocks, whilst others use different schemes. For example, the SMC specs state that the blocks are split up into sections, which are again split up into zones (as the SMC was only intended to store 32MB originally). It gets rapidly confusing, but the basic gist is that there's 16-bytes of reserved info at the end of each block, and these must be read upon card insertion to figure out the physical<->logical translation. These also store the checksum of each block so you would know if it went bad.

If the block goes bad, it can simply be relocated and the P2L table modified accordingly (there are so many guaranteed unused blocks per zone just for this). However, if a piece of software falls over and writes crap throughout the card in the reserved areas, don't expect a card reader to even attempt to recognise it.

If you want more info, hunt the web for the various SD information leaked over the years. I think the official information is still NDA'd and requires a license, but there's plenty of info out there now which you can easily piece together. Also, check out the source code to my GpDrive, which does all the above (although the wear levelling isn't brilliant as I couldn't get it to work properly, so it uses a really simple formula; and I don't think I ever implemented bad-block remapping).

 

[theoddbot]

Raw NAND chips don't do any of it, you need to handle it all yourself. This may be a bit of an issue if people are planning to store files on the NAND chip in the GP2X.

​

The JFFS2 filesystem used on internal NAND was designed specifically to deal with these issues.

 

[aQwaBlaz3]

So? Go buy an SD and use the SD...no one making you use the NAND...

Oh man.....I hope my GP2X doesnt brick on me...the stuff you dudes arew saying are.....scaring me.

 

[Robster]

The JFFS2 filesystem used on internal NAND was designed specifically to deal with these issues.

​

That's fine, but the NAND is split up into various sections - for example, bootloader, kernel, filesystem etc. The filesystem part uses JFFS2, but the bootloader and kernel don't so if a bad block appears in those sections, your GP2X will be bricked

 

[whiskthecat]

The way i look at it is is nothing to worry about because once you use your card for so long (probably a very good amount of time) it twon't be a big deal to go out and buy another one. I mean they are pretty cheap and since nothing lasts forever paying a low price to replace an old card doesnt bother me. B)

 

[Prothall]

I choose to read it this way:

Buy a cheap SD card if you don't own one.

Do not become too attached to it.

Well-loved GP2X units will not be found functioning by survivors of the future nuclear apocalypse.

 

[lucoxade]

Ooh man.. :mellow: There should be more slots so that we can RAID our SD cards...

 

[Axe_Murderer]

Robster and Squidge: Very interesting, thanks for taking the time to provide some explanation, guys It certainly makes sense to me that certain critical parts of the flash memory would be designed to have a higher guaranteed level of reliability to help extend the card's overall life when used in conjunction with these various wear levelling techniques.

Incidentally, on the subject of GpDrive, allow me to congratulate you Squidge for creating such a very reliable and useful program, not least because it eliminates the inconvenience of needing and having to use a card reader

 

[Winterkid]

But a card reader with USB 2.0 can be a LOT faster than the GP32, even with GPDrive, I think. The GP32 can only run at Maximum USB 1.1 speeds even in a USB 2.0 socket.

My concern is currently for my SMC cards, the SD are new enough that I don't really expect much problems from them yet

 

[aQwaBlaz3]

Ooh man.. :mellow: There should be more slots so that we can RAID our SD cards...

​

Is that even possible?