Flash memory and filesystem, wear levelling is an answer

[boxybox]

I'm currently looking into the matter of choosing the right SDHC card for my pandora and I have some concern: we know that flash memory is made out of several individual segement which can only deal with a limited number of write cycles and when a particular segment has ran out of cycles the device is permanently corrupted.

I recently read about this in the slax documentation and more specifically about certain filesystems (FAT, EXT2) being designed for a totally different kind of technology and as such make repeated writes in the same location thus decreasing greatly the life of the flash device. I know this is very old news but I'm just mentioning itfor those who don't already know.

Some flash devices address this problem with "wear levelling", and my concern is about identifying such devices in the sea of flash devices.

I know there's already a few threads about speed and SDHC cards but I haven't found any about this particular concern (maybe on the old forums, haven't check there), can anyone with more insight in the domain give us recommendations about SDHC cards ?

 

[phrosty]

i'm pretty sure all sdhc cards use wear leveling

 

[johnx]

I'll expand on that to say that I'm very sure that all memory cards sold directly to consumers have wear leveling of some form. There's a nice white paper on wear leveling on MMC cards that might be interesting: http://www.sandisk.com/Assets/File/OEM/ ... elv1.0.pdf

 

[atomicthumbs]

Any memory card or USB stick that doesn't work by exposing the pins of the flash chip to a controller in the reader (i.e. SmartMedia) has integrated wearleveling. You don't need to worry.

 

[LynxJSA]

All flash cards have this issue. It's the nature of all non-volatile memory, however it is much more pronounced with Flash Cards. The cards are so dirt cheap, though, that it's worth it to just back up your stuff and replace the cards on a regular basis.

 

[j.pickens]

I have a Nokia Internet Tablet, model 770, which uses flash memory, both in the permanently installed device, and in SD card (rsMMC).
It has been in use for years, with heaps of use, including using the SD memory to extend the OS cache, to run programs, and to repeatedly read and write data for myriad programs.
Never had so much as a hiccup, as far as flash memory is concerned, nor have I EVER heard anyone complaining about flash memory failure on the various Nokia forums.
Since, in the nearly four years of use, advances in the write cycle and wear leveling have been dramatic, newer memory should be even more robust. I wouldn't worry about this issue at all.

 

[Shikaku]

All flash cards have this issue. It's the nature of all non-volatile memory, however it is much more pronounced with Flash Cards. The cards are so dirt cheap, though, that it's worth it to just back up your stuff and replace the cards on a regular basis.

I have had 3 failures of flash media, and this advice is worth its weight in gold (then again, you should always have backups...). 2 were permanent flash card failures, and 1 was a random partition wipe.

 

[j.pickens]

All flash cards have this issue. It's the nature of all non-volatile memory, however it is much more pronounced with Flash Cards. The cards are so dirt cheap, though, that it's worth it to just back up your stuff and replace the cards on a regular basis.

I have had 3 failures of flash media, and this advice is worth its weight in gold (then again, you should always have backups...). 2 were permanent flash card failures, and 1 was a random partition wipe.

Agreed that backing up is necessary for any storage medium.
However, I would bet that the failures you experienced had nothing to do with wearing out the memory past its read/write cycle life.
More to do with basic failures due to defects. My point was that wear on these cards is not worth worrying about. For many, many years, at least.

 

[reyawn]

I have had cards that work well still after years of use as well as cards that failed abruptly after a month or so; there certainly seems to be a degree of luck to it (although a good brand can help!). Also, I've had flash drives get lost by falling out of a device during a hard fall (my Treo 650); it was my GPS software!

Thankfully I had made a backup of it and managed to get it working again with the help of online resources; but if I didn't make a backup then I would be in trouble. I would recommend making backups of any and all storage media! If you really want to do it quick and dirty you can do a DD dump in linux to a file that you can use to restore to an equally sized flash disk, or just maintain a running a backup.

I'd imagine that the Pandora's wifi capability will come in really handy as it would be possible to sync / backup the device with remote locations.

 

[LynxJSA]

Pickens, the internal memory on your device is hardwired and managed with a built-in controller to distribute wear leveling. It is much less susceptible to failure than a removable memory card.

Consider the application and consider the user.

First there is the general nature of any mass produced inexpensive device (SD cards). Now add on to it that it has exposed connectors. Tack on that it is very likely that most Pandora users will be using their SD cards not just for their Pandora but for several other devices as well. Since the Pandora is a device that will most likely be used for tinkering around, there will be significantly higher reading and writing to the cards than for a digital camera, cell phone, or even a standard PDA.

In light of that, I'd be hesitant to say that removable memory failure is something that one shouldn't worry about at all.

One thing hobbyists get used to in tinkering around with electronics is that, simply by the nature of what they are subjecting things to, they often drastically reduce the life of the components in the process.

 

[j.pickens]

Pickens, the internal memory on your device is hardwired and managed with a built-in controller to distribute wear leveling. It is much less susceptible to failure than a removable memory card.

Consider the application and consider the user.

First there is the general nature of any mass produced inexpensive device (SD cards). Now add on to it that it has exposed connectors. Tack on that it is very likely that most Pandora users will be using their SD cards not just for their Pandora but for several other devices as well. Since the Pandora is a device that will most likely be used for tinkering around, there will be significantly higher reading and writing to the cards than for a digital camera, cell phone, or even a standard PDA.

In light of that, I'd be hesitant to say that removable memory failure is something that one shouldn't worry about at all.

One thing hobbyists get used to in tinkering around with electronics is that, simply by the nature of what they are subjecting things to, they often drastically reduce the life of the components in the process.

I was not saying that removable memory failure was not worth worrying about, I was saying that failure of SD card memory Due to read/write life cycle being exceeded was not worth worrying about. This was the subject of the original post, to which I responded.

 

[darkblu]

a brand name flash card will last you the lifetime of the device* if used without excessive writes.

* omap3530 is rated at ~5.5 years lifespan (run-time) when used without restrictions in the clocks, i.e. when used with some overclock.

 

[boxybox]

Can you define "excessive writes", "cause as the power user I am, I do intend to make intensive use of the flash memory which includes a lot of writing.

It took about a year for my 1GB, life warranty, usb flash memory to wear off, it was older technology and this has certainly improved since but still I'm not eager to believe what the manufacturers say or what is repeated based on paper spécifications. Usage very often proove to be different and mileage may vary.

 

[Tor]

Even if your way of using the flash cards turns out to damage the card before its manufacturer's spec'ed lifetime, there's not much we can do on the Pandora hardware or software level. Ref. your question from your first post:

Some flash devices address this problem with "wear levelling", and my concern is about identifying such devices in the sea of flash devices.

The answer to that is 'they all do'. That is, all SD cards you can buy have this built-in. But there's probably a quality difference between brands. I've only seen anecdotal evidence as to which brands are good or bad however.

 

[reyawn]

well, as long as you are careful with allocation of temporary directories and swap files (swap? well, what if you need more than 128mb RAM? )...I'd imagine that it could minimize impact.

I've heard that applications like Firefox have a heavy hand when it comes to reading/writing so it would be a good idea to be careful. I personally intend on keeping /home and perhaps other directories in the first SD card slot (easy backup and also to minimize wear on the irreplaceable internal flash).

 

[boxybox]

It's good news to read that all SD cards include wear levelling, and it's quite a good idea to use a SD card for swap and /home to minimize use of internal memory.
About firefox, as I intend to *not* use it, I hope that there will be another web browser available on the pandora.

 

[amnesiasoft]

I believe someone on the xkcd board went and did a series of calculations related to wearing out a flash drive - this was for Windows ReadyBoost though - and concluded with 24 or 8 hours of use a day, I don't remember which, that it would take a couple years to wear out the flash drive.

 

[boxybox]

Reviving this topic with an update. I just read an interesting article about SSDs where it is mentioned that hardware wear levelling just sucks at wear levelling and will not prevent data from silently becoming corrupted (which explains a lot of the data corruption issues I experienced with my gp2x).

The article is available at http://valhenson.livejournal.com/25228.html

Here are a few excerpts I found interesting:

[...]
Let's start with hardware wear-leveling. Basically, nearly all practical implementations of it suck. You'd imagine that it would spread out writes over all the blocks in the drive, only rewriting any particular block after every other block has been written. But I've heard from experts several times that hardware wear-leveling can be as dumb as a ring buffer of 12 blocks; each time you write a block, it pulls something out of the queue and sticks the old block in. If you only write one block over and over, this means that writes will be spread out over a staggering 12 blocks! My direct experience working with corrupted flash with built-in wear-leveling is that corruption was centered around frequently written blocks (with interesting patterns resulting from the interleaving of blocks from different erase blocks). As a file systems person, I know what it takes to do high-quality wear-leveling: it's called a log-structured file system and they are non-trivial pieces of software. Your average consumer SSD is not going to have sufficient hardware to implement even a half-assed log-structured file system, so clearly it's going to be a lot stupider than that.
[...]
When it comes to flash, manufacturers are handicapped when predicting long-term failure rates for a number of reasons. First, it's hard to extrapolate failure rates under stress tests to long-term failure rates. In particular, the failure mode of a flash cell is that the charge leaks out of the cell - slowly, over time. Stress testing by writing to the cell a lot and then reading the data back is not going to test this situation. In general, we simply don't have a lot of experience with testing flash and it will take a few years to build it up. Second, manufacturers of the device using flash are constantly switching suppliers for the actual flash memory itself. When it comes to consumer-grade flash, manufacturers have strong pressure to drive the price down and very little pressure in the direction of quality. Frequently, the manufacturer won't have any idea where the flash chips came from for a particular device based purely on the model number because it used more than one supplier for that model. Third, failure rates will be heavily depending on the pattern of both reads and writes, so a device that checks out fine under one test pattern will failure miserably under another load - not generally a characteristic of disks. Another fascinating aspect of flash-based SSD is that you don't seem to get any report of checksum failures on corruption - at least I haven't seen one in the three confirmed cases of flash corruption I've seen. I don't know if this is because the device isn't reporting it or because the OS driver isn't listening for it, but it's what happens.

The exception to these observations are any flash device that costs a lot of money - commercial-grade flash as opposed to consumer-grade flash. [...]

I guess carefully choosing the filesystem to use with your pandora is the way to go to help prevent data corruption from happening.

 

[Tor]

>I guess carefully choosing the filesystem to use with your pandora is the way to go to help prevent data >corruption from happening.

Not very much, really, or almost really..
The filesystem choice for the NAND matters a lot, because it doesn't come with built-in wear leveling anyway, so we can only use it with an fs that has.
The SD cards all have wear leveling built-in, and it's not much a filesystem on top of that can do with it. It certainly can't apply its own wear leveling algorithm. However (and that's where the 'almost' comes in..), FAT is known to write a lot to its little top dir directory.

On the other hand, all corruptions I've heard of have, when it comes down to it, ended up in the following categories:
- An actual faulty SD card. These are not that common actually.
- A fake (for the lack of a better word) card bought on ebay. Don't buy cards there.
- Corruption due to removing the card while it's not yet fully written (very common)
- Corruption due to device switching off power (or lowering the voltage) before the card has settled (a major headache last year with one of the Nokia internet tablet kernel upgrades - it turned out, and that was tricky to find, that many cards (particularly newer, larger SDHC cards) don't follow the specs exactly and are still not ready when a certain stage is reached. The fix was to wait for another signal, then turn down the voltage to idle level.)

An actual worn-out card is a rare thing to see.. I haven't, yet.

 

[truekaiser]

wait till ext4 and similar linux file systems. it supposedly has optimizations for ssd and flash based mediums.