Formatting a SD card : demystifying !

[Linux-SWAT]

I wanted to post about this a long time ago.


So, if you want to properly partition/format a SD, you don't need to use the official SD formatter :


https://www.sdcard.o...ds/formatter_3/


If you noticed that this tool is extremely fast, then you also concluded that he does not check the card. (EDIT : may be wrong about this, see : http://boards.openpa...333#entry103333)


The only "magic" this tool has is that he left 4M of empty space at the beginning of the card.


I saw once a card pre-formatted with 4M free at the beginning of the card AND 4M free at the end, so now, i always do the same.


Doing this with GParted or any Linux partitioner/formatter is a piece of cake.


Another thing, be extremely careful of using a good card reader like transcend ones.


And use a good USB cable. The ones shipped with the iCP rocks because they have a thick grounding, and they are short.


And if you want the best possible setup, avoid the front USB connectors of ATX cases, prefer the ones directly connected to the motherboard.


----EDIT


I forgot to mention that transferring a lot of files once in a row under Linux, may corrupt some cards filesystem, even formatted with the official SD formatter.

 

[Prometheus]

So all this time, here I was thinking it was a real pain that the official formatter leaves out Linux, Mac, BSD, etc., users, and that's all it does? :blink: Seriously?


Now that begs the question: If cards work so well with the official formatter doing this trick, why don't OSes do this by default?

 

[Linux-SWAT]

Maybe because, on a native reader like the OP's one, SD is seens as /dev/mmcblk0


And on a USB reader, the card is shown as /dev/sde.


So an automatic setting seems complicated. Maybe hdparm can read it ?

bash-4.1# hdparm -i /dev/sde

/dev/sde:

HDIO_GET_IDENTITY failed: Invalid argument

Seems not.

 

[Prometheus]

^ Sorry, I should have been a bit clearer: I just meant the formatting tools of operating systems in general, rather than the Pandora's specifically.

 

[Linux-SWAT]

You were clear.

 

[Prometheus]

Ok. I think I'm slightly confused, now, so I'll back out.

 

[Grench]

I wanted to post about this a long time ago.


So, if you want to properly partition/format a SD, you don't need to use the official SD formatter :


https://www.sdcard.o...ds/formatter_3/


If you noticed that this tool is extremely fast, then you also concluded that he does not check the card.


The only "magic" this tool has is that he left 4M of empty space at the beginning of the card.


I saw once a card pre-formatted with 4M free at the beginning of the card AND 4M free at the end, so now, i always do the same.


Doing this with GParted or any Linux partitioner/formatter is a piece of cake.


Another thing, be extremely careful of using a good card reader like transcend ones.


And use a good USB cable. The ones shipped with the iCP rocks because they have a thick grounding, and they are short.


And if you want the best possible setup, avoid the front USB connectors of ATX cases, prefer the ones directly connected to the motherboard.

Very interesting. I've never used the tool mentioned. When I was formatting my boot card (Patriot LX 32GB SDXC) I was able to put the ext2 partition at the very front of the card without issue. I did end up with an unused set of trailing space though.


/dev/mmcblk0p1 ext2 /media/PANDORAROOT 9.77GiB


/dev/mmcblk0p3 fat32 /media/PND 20.29GiB


/dev/mmcblk0p2 linux-swap 23.53MiB (I should make this larger - but it's working and I don't want to mess with it ATM.)


unallocated 7.68MiB


So - I'm wondering - does it just want about 8MiB of 'leftovers' on the card somewhere? I'm wondering if it even matters where it happens in the partition table. I.e. it's solid state - position in the media might not matter at all.


So - why would SD cards need ~8MB of unassigned space...? Maybe something to do with how the cards handle wear leveling and the like? I'm not sure.

 

[neko]

So - why would SD cards need ~8MB of unassigned space...?  Maybe something to do with how the cards handle wear leveling and the like?  I'm not sure.

It's for alignment purposes, so that data is less likely to be split between different blocks which would make writes slower.  The standard partition alignment is 4MB, although that's probably excessive for most cards.

 

[Linux-SWAT]

It's for alignment purposes.

I also read somewhere that the first and last parts can be defective on some cards.

 

[Hoodoo]

I thought you just run gparted and that's it. Didn't know there was anything complicated about this.


Just don't forget to (re)create the partition table.

 

[Prometheus]

I must confess, I just use KMFormat, myself. (It's just a GUI front-end to the standard command-line tools, but I find it a bit quicker. )

 

[Christoph.Krn]

The only "magic" this tool has is that he left 4M of empty space at the beginning of the card.

The official SD Formatter from Panasonic is most definitely not limited to performing the mentioned task - have you conducted an actual analysis? Also see http://forums.adafruit.com/viewtopic.php?t=20861 .


PS: Could you please use underlined text very carefully? It is confusing on websites.

 

[Linux-SWAT]

The official SD Formatter from Panasonic is most definitely not limited to performing the mentioned task - have you conducted an actual analysis? Also see http://forums.adafru...pic.php?t=20861 .

My actual analysis is that this tool left 4M at the beginning of the card, and since i did the same with GParted, all my problems suddently vanished.


Your link is pretty interesting.

PS: Could you please use underlined text very carefully? It is confusing on websites.

I always do ! :lol:

 

[Christoph.Krn]

My actual analysis is that this tool left 4M at the beginning of the card, and since i did the same with GParted, all my problems suddently vanished.

This aligning may have visibly mitigated at least parts of the problems you had encountered, but that doesn't mean that it is best practice for every card; that it is everything the official formatter does in all of the cases; or that it will help in all other cases, too.

 

[Linux-SWAT]

Indeed, but how to proceed with other fs than FAT16/32 ?

 

[Stradian]

I have been having some issues with cards corrupting, recently. Usually I just use the terminal to make a ext 3 file system on the sd card without the free space. I thought the card had gone bad. I'll reformat it and stick in the 4mb free space at the beginning and the end and see if I get the same corruption issues. Then I'll compile something relatively large on the pandora and see if this fixes my issue.


I'll be amazed if this works.

 

[Christoph.Krn]

[...] how to proceed with other fs than FAT16/32 ?

There is usually too much dark voodoo going on in all the proprietary SD card controllers to be able to reliably tell how they really behave.


A good approach would probably be to test as many cards in as many configurations yourself as possible (for maximally reliable results), and to try to get information out of other resources such as the SD compatibility list on pandorawiki, while being aware that evaluating true SD card performance is difficult.


Your suggestion /might/ or might not help, also, but the effect is quite unclear at this point. Did you make any kind of comparisons?


edit-03nov2011-0120GMT+1: If I understood "check the card" from your first post correctly: There is no need for the Formatter to do so (unless it wanted to attempt to make sure that it had successfully overwritten a card). The official SD Formatter does have "full format" functions that do take a comparatively long time to complete. However, the time required for formatting is not really an indicator for what exactly is being done to the card other than how much data is possibly being written/erased. </edit>

 

[Linux-SWAT]

Though i was clear, so i'll be more precise :


Without 4M space = data losses, need to reformat every week.


With 4M space = no problem.


For the hardware hacking part, i'm not that good in electronics, and havn't the needed tools.

I have been having some issues with cards corrupting, recently. Usually I just use the terminal to make a ext 3 file system on the sd card without the free space. I thought the card had gone bad. I'll reformat it and stick in the 4mb free space at the beginning and the end and see if I get the same corruption issues. Then I'll compile something relatively large on the pandora and see if this fixes my issue.


I'll be amazed if this works.

I'm 95% sure you'll be amazed.

 

[Christoph.Krn]

Without 4M space = data losses, need to reformat every week.


With 4M space = no problem.

Which file system(s) was/were this, and what brand and type of card(s) exactly? How exactly were they being formatted?

For the hardware hacking part, i'm not that good in electronics, and havn't the needed tools.

There is no hardware hacking, but rather different ways in which data is being written to the card, which can be used to better determine its behavior.

 

[Linux-SWAT]

FAT16, FAT32, EXT2, EXT3, ReiserFS.


Many well-known brands, kingston, transcend, sandisk... SD, Micro-SD and Mini-SD.


Do you want the color too ?


According to your link, using hardware tools (or software bench) must be wayyyyyy faster than manually making tries.

A good approach would probably be to test as many cards in as many configurations yourself as possible (for maximally reliable results)

I have a far better approach : YOU test as many cards in as many configurations yourself as possible.


For maximally reliable results, of course.