Filesystem? Please Don't Default To Fat32


Squidge said:
juanvvc said:
it takes a long time to save big files in my SD, up to ten minutes for a 300MB file.
I've had exactly the same problem - taking ages (anything upto 5 minutes) to save a ~300MB file to an SD card. I changed the sd card reader, and now it takes 40-50 seconds. Both are USB2.0 high speed devices. The slow one is SD card only, the fast one is a '7 in 1' device. Maybe the processor is faster, or the way of talking to the SD card is different in the '7 in 1' device.

Sphinxter said:
Wouldn't touch FAT with your ten foot pole;
http://www.news.com/Microsofts-file-system..._3-6025447.html



The FAT file system isn't patented, that was rejected with "prior art". These days, only some features of Microsoft's implementation of the FAT system is patented. So, you can still read a fat file system without getting sued.


good to know.
 
Last edited by a moderator:
Squidge said:
Correct. However, since on the gp2x the power is normally always shut off before the device is unmounted, that means every time you boot the machine with that card in it's going to trawl through the journal looking to see whats faulty due to the 'dirty' bit being set.
Surely Journaling is smarter than that? It should mark off when somethings completely written from the journal, or clear the journal when writing is done. Then it would be able to tell whether it needs to check things or not. It wouldn't use the journal for reading. Would the speed of it even be an issue on a flash drive? Flash drives have an almost instant seek time, so that's one slowdown out of the equation. How much data does the journal really contain?
 
Last edited by a moderator:
The main problem with using something other than FAT32 is the lack of Windows support. That problem can be solved in software, though.
 
The best argument for FAT32 is cross-platform compatability. FAT32 disks work flawlessly on many many OS'.... Windows, OSX, Linux, *BSD. NTFS is supported R/W on Linux, R/O on *BSD (I think, at least on NetBSD), and R/O on OSX(though why snapple can't get this sorted I'll never know)....

And this of course brings us to other devices... mp3 players, cameras, car radios... it's simply one of the most supported FS' of all time.
 
Orkie said:
[
Sorry, I'm confused as to why you'd want to do this?. It's be much simpler to just have /mnt/sd like the GP2X does... Oh, and you can mount two filesystems on the same directory using unionfs.
Well /usr/local is the standard directory for third party/user installed programs (fex. loki installed games go under /usr/local/games), so I thought it would be more logical to mount sd card there instead of messing with /usr/bin. /mnt/sd is fine I suppose, but if it's going to be more like a regular gnu system then having to set path would be problematic. Well, I suppose this depends on the launcher/frontend they're going to be using.

I hadn't heard of unionfs, but good to know theres a solution for that.
 
Last edited by a moderator:
Zarneth said:
Yes RiserFS uses Journaling. I could be wrong but I think NTFS did too.
But I thought the purpose of journaling was to STOP the problems with corruption when the power is shut off? :blink:
Wouldn't MS and Linux still use fat32 if it was superior.
There isn't a single universal superiority order amongst filesystems. There are quite a lot of parameters to be weighed. For instance, a journalling filesystem by design constantly makes small writes to a a specifik file (that is, the journal). This is more or less inconsequential on a hard disk, but not on a flash device which has a limited amount of write cycles. Some file systems are better at many small files, some are better at a few large files. Some devices has loads of storage space and can deal with higher overheads, some are really cramped and needs to squeeze the last drop of storage out of it. And so on.

What makes superiority is rather different from a mainspowered desktop PC with magnetic hard drives and controlled up- and down sequences, to a batterypowered portable device with flash storage which gets switched on and off whenever the local bus stops.

QUOTE
I was under the impression that FAT32 was used on portable drives because of it's universal compatibility.



...which is not an advantage to be taken lightly on a device like this. But, then again, and as been said higher up: There is no problem for a linux kernel to mount SD-cards formated as ext2, Fat32, Reiser...It only needs the kernel module, and off you go.
 
Last edited by a moderator:
There are specialized dedicated NAND filesystems just to solve these problems;
http://en.wikipedia.org/wiki/List_of_file_systems

I would not at all be surprised to see the internal NAND of the Pandora use JFFS2 / YAFFS.

By default the Pandora should read ANY common Filesystem on the SD card.

FAT16, FAT32, NTFS, EXT2/3, JFFS2, ect...

Also who really expects to take out the SD cards that often?

The Pandora has WiFi 802.11a/b, and can function as a USB Host AND Slave device.

That means you'd use SAMBA/NFS/HTTP/FTP/SSH to transfer files to and from other systems, without caring about the native FS.
 
Series-8 said:
There are specialized dedicated NAND filesystems just to solve these problems;
http://en.wikipedia.org/wiki/List_of_file_systems

I would not at all be surprised to see the internal NAND of the Pandora use JFFS2 / YAFFS.

By default the Pandora should read ANY common Filesystem on the SD card.

FAT16, FAT32, NTFS, EXT2/3, JFFS2, ect...

Also who really expects to take out the SD cards that often?

The Pandora has WiFi 802.11a/b, and can function as a USB Host AND Slave device.

That means you'd use SAMBA/NFS/HTTP/FTP/SSH to transfer files to and from other systems, without caring about the native FS.



If I start using it for a pda I'll be probably be removing an encrypted sd card from it every time I set it down.
 
Last edited by a moderator:
Magnulus said:
Zarneth said:
It's a shame it's only 256mb really. 512mb would allow for a much more functional linux install.
I'd say Damn Small Linux is pretty functional, and that's what? 50 megs?


I've put X, ice wm and Mozilla 0.92 in under 24, linux without a gui in less than 4. Your desktop linux, all desktop os for that matter are bloated exercises in wasting space and resources.
 
Last edited by a moderator:
klikklak said:
Orkie said:
[
Sorry, I'm confused as to why you'd want to do this?. It's be much simpler to just have /mnt/sd like the GP2X does... Oh, and you can mount two filesystems on the same directory using unionfs.
Well /usr/local is the standard directory for third party/user installed programs (fex. loki installed games go under /usr/local/games), so I thought it would be more logical to mount sd card there instead of messing with /usr/bin. /mnt/sd is fine I suppose, but if it's going to be more like a regular gnu system then having to set path would be problematic. Well, I suppose this depends on the launcher/frontend they're going to be using.

I hadn't heard of unionfs, but good to know theres a solution for that.

As far as I know (correct me if I'm wrong) removable media is usually mounted on /media/
I think this is since hal and dbus became commonplace.
The micro-mms on my nokia 77 is mounted under /media/mmc1
When I connect an usb-device to my desktop pc (debian) it will always be mounted on /media/<device>
Even a cd/dvd-rom disc will be mounted on /media/cdrom0
 
Last edited by a moderator:
Squidge said:
Zarneth said:
But I thought the purpose of journaling was to STOP the problems with corruption when the power is shut off? :blink:
Correct. However, since on the gp2x the power is normally always shut off before the device is unmounted, that means every time you boot the machine with that card in it's going to trawl through the journal looking to see whats faulty due to the 'dirty' bit being set.

juanvvc said:
Fragmentation really is a problem in flash drivers where files with different sizes are often added and removed, like gamers do. My SD card really needs a defragmentation since fsck does not help any more :(
An SD card is always fragmented, even if you do a full format and then copy a single file to it. Defragging an SD card does nothing but reduce the life cycle. This is because when you write a block to an SD card, the first thing it does is choose a random location to write the block to. This is called wear levelling, and prevents certain parts of the card failing before others (eg. when the FAT is constantly updated).


Something I'm not seeing addressed at all in this thread, that I've been trying to get more information on: to the best of my knowledge, wear leveling on all simple consumer flash devices -- USB flash drives, SD cards, CF cards, etc. depends entirely on the product's controller understanding the underlying filesystem.

If the device doesn't understand the filesystem, it can't know which sectors are in use, and which aren't; thus it can't pull data off a more-heavily-accessed sector and move it to a free sector that's more virginal. If it did that, it might overwrite data that's in use. At least, this is my understanding.

I've spent some time attempting to contact some of the bigger names in UFD/CF/SD manufacturing -- especially Transcend -- to find out if there are any products on the market that speak alternative filesystems. So far, I've gotten no responses at all. Given that, I'm forced to assume that any sealed-case product is much, much safer using FAT32 (or FAT16 for older or smaller products). There's more than one reason that those come preformatted as FAT from the factory, I'm thinking.

Unfortunately, all my research into wear leveling has returned old documents (for 16MB EEROM parts and the like) that confirm that the sealed product's wear leveling depends on understanding FAT, or newer documents for recent products that merely say, "We've got wear leveling!" but decline to explain how it works. The best information I've found is that (some?) CompactFlash controllers have reserved repair blocks and defect management, which would at least partially alleviate the problem.

If you or anyone else have more information on what filesystems can be used with wear leveling, and on what (type? brand?) of products, I'd appreciate it.

TIA
 
Last edited by a moderator:
AloisHammer said:
Something I'm not seeing addressed at all in this thread, that I've been trying to get more information on: to the best of my knowledge, wear leveling on all simple consumer flash devices -- USB flash drives, SD cards, CF cards, etc. depends entirely on the product's controller understanding the underlying filesystem.
USB flash drives & SD cards support wear levelling, but neither care about the underlying file system. You could even write your own file system and wear levelling would still work. They don't care about used and unused sectors. They break the memory up into blocks and tell the OS how many blocks they are (and the size of a block). The OS then typically breaks these blocks up into sectors or whatever it prefers for the file system. When the OS says "Write block X", the storage picks a random block and writes that block, updating its internal map which says where its storing all the blocks (ie, logical block 2 is on physical block 26).

Thats it really.

NAND chips and Compact Flash cards don't have any wear levelling, thats up to the driver; which typically does exactly the same as above using OOB (Out of Band) memory, which is a reserved area at the end of each block for error detection/correction and the physical<->logical translation tables.

The other advantage to these P2L tables is that if a block becomes bad, the OS doesn't even have to know about it unless the storage runs out of "spare" blocks, which are blocks not counted in the number of blocks reported to the OS. Even then, if the medium isn't full, the storage could simply reduce the amount of blocks available, but I've only actually seen this done in one product so far (as the OS could have recorded the number of blocks in the file system itself, and may get upset if the number of blocks change whilst the OS has "mounted" the storage, specially if the device is almost full, so there is only 1 block left - the OS would attempt to write the last remaining block and receive an unexpected error).
 
Last edited by a moderator:
Squidge said:
USB flash drives & SD cards support wear levelling, but neither care about the underlying file system.

NAND chips and Compact Flash cards don't have any wear levelling, thats up to the driver; which typically does exactly the same as above using OOB (Out of Band) memory, which is a reserved area at the end of each block for error detection/correction and the physical<->logical translation tables.
No, the CF controller does infact remap blocks as do ALL modern NAND devices. So does any 'HardDisk'.

(possibly with the exceptions of MRAM or something exotic like that Phase-Change 'RAM')

How the remapping works, depends on the algorithm and design requirements...(and quality of NAND)

In short, it really depends on what kind of NAND is being uses as there are several sub-types (multi-level, single level, and even hybrids), and even the number of chips present on the media (well SD is usually one, but USB-Drives, CF cards could have two ore more. a SSD might have 50+)

Generally unless you are using your SD as a swap file and thrashing the heck out it, it will last many years (~10?).
 
Last edited by a moderator:
Series-8 said:
No, the CF controller does infact remap blocks as do ALL modern NAND devices. So does any 'HardDisk'.
Well, yes, of course the CF controllers remap's blocks, as do modern NAND devices. However, I'm not talking about CF controllers, but CF cards, and NAND chips. Read my post again.

Have you a source for hard disk wear levelling? I don't believe they do, it would complete destroy sequential performance.
 
Last edited by a moderator:
Back
Top