Formatting a SD card : demystifying !

[chaotix]

32GB RAM? That's quite a Pandora you have there... (And I thought you had a lot of USB thumb drives connected )


Yes, the filesize is necessary to avoid false results due to caching. So I don't think you will get realistic results with a 1GB file on a machine with 32GB RAM.


But you can use Bonnie on your Pandora. It will not show you what your card is capable of when connected to your PC as the SD-interface maxes out at about 15 MB/s on the Pandora. You can get a version of Bonnie for the Pandora at http://lxtek.de/pandora/sdcard-filesystem-benchmark/bonnie/ (It's actually just extracted from the Debian armel package).

 

[OmegaPhil]

Its OK - I'm now testing on a 2GB RAM laptop. I did try bonnie++'s -D option (should avoid read/write caches), however the write always fails (EINVAL), probably a bug.

 

[OmegaPhil]

Right, I have now tested the 32GB card with bonnie, with 4, 8 then 0MB empty paritions infront of the used ext4 partition, two runs in all cases:


4MB space:

Version 1.96	 ------Sequential Output------ --Sequential Input- --Random-

Concurrency 1	 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine	 Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP

omega2		 4G 74 96 8038 6 5509 6 752 98 37579 9 743.9 29

Latency			 188ms 2143ms 1322ms 20987us	 104ms	 952ms

Version 1.96	 ------Sequential Create------ --------Random Create--------

omega2			 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

			 files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP

			 256 8458 34 150248 99 650 2 7340 36 185136 97 406 2

Latency			 4101ms 3347us 19658ms 4471ms 1794us 14606ms

1.96,1.96,omega2,1,1340987190,4G,,74,96,8038,6,5509,6,752,98,37579,9,743.9,29,256,,,,,8458,34,150248,99,650,2,7340,36,185136,97,406,2,188ms,2143ms,1322ms,20987us,104ms,952ms,4101ms,3347us,19658ms,4471ms,1794us,14606ms


Version 1.96	 ------Sequential Output------ --Sequential Input- --Random-

Concurrency 1	 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine	 Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP

omega2		 4G 60 86 8157 5 5828 5 813 99 42808 9 750.8 27

Latency			 877ms 2104ms 1382ms 17411us 98221us 1855ms

Version 1.96	 ------Sequential Create------ --------Random Create--------

omega2			 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

			 files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP

			 256 8562 33 156519 100 646 2 7013 26 205901 98 421 1

Latency			 4314ms 3293us 15126ms 4354ms	 143us 17579ms

1.96,1.96,omega2,1,1341008943,4G,,60,86,8157,5,5828,5,813,99,42808,9,750.8,27,256,,,,,8562,33,156519,100,646,2,7013,26,205901,98,421,1,877ms,2104ms,1382ms,17411us,98221us,1855ms,4314ms,3293us,15126ms,4354ms,143us,17579ms

8MB space:

Version 1.96	 ------Sequential Output------ --Sequential Input- --Random-

Concurrency 1	 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine	 Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP

omega2		 4G 28 52 7387 6 5774 4 327 86 37725 9 724.1 30

Latency			 3159ms 3138ms 2518ms	 258ms	 105ms 1140ms

Version 1.96	 ------Sequential Create------ --------Random Create--------

omega2			 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

			 files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP

			 256 8450 41 95936 95 601 2 6141 24 170515 98 456 2

Latency			 2631ms 14866us 11671ms 6092ms	 416us 18239ms

1.96,1.96,omega2,1,1340998657,4G,,28,52,7387,6,5774,4,327,86,37725,9,724.1,30,256,,,,,8450,41,95936,95,601,2,6141,24,170515,98,456,2,3159ms,3138ms,2518ms,258ms,105ms,1140ms,2631ms,14866us,11671ms,6092ms,416us,18239ms


Version 1.96	 ------Sequential Output------ --Sequential Input- --Random-

Concurrency 1	 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine	 Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP

omega2		 4G 53 87 7989 5 6105 4 759 95 42971 9 844.4 29

Latency			 1546ms 2362ms 1783ms	 203ms 64182us 2450ms

Version 1.96	 ------Sequential Create------ --------Random Create--------

omega2			 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

			 files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP

			 256 7721 30 146191 99 757 2 8163 31 192236 99 390 1

Latency			 6196ms 8088us 18530ms 5449ms 1196us 20475ms

1.96,1.96,omega2,1,1341008621,4G,,53,87,7989,5,6105,4,759,95,42971,9,844.4,29,256,,,,,7721,30,146191,99,757,2,8163,31,192236,99,390,1,1546ms,2362ms,1783ms,203ms,64182us,2450ms,6196ms,8088us,18530ms,5449ms,1196us,20475ms

0MB space:

Version 1.96	 ------Sequential Output------ --Sequential Input- --Random-

Concurrency 1	 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine	 Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP

omega2		 4G 57 84 6758 4 5608 4 1065 99 43455 9 781.1 28

Latency			 2884ms 4664ms 3917ms 21536us 57881us 1426ms

Version 1.96	 ------Sequential Create------ --------Random Create--------

omega2			 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

			 files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP

			 256 7834 30 157401 99 527 2 6766 25 197294 99 416 1

Latency			 5754ms 3180us 18520ms 10537ms	 445us 23744ms

1.96,1.96,omega2,1,1341053160,4G,,57,84,6758,4,5608,4,1065,99,43455,9,781.1,28,256,,,,,7834,30,157401,99,527,2,6766,25,197294,99,416,1,2884ms,4664ms,3917ms,21536us,57881us,1426ms,5754ms,3180us,18520ms,10537ms,445us,23744ms


Version 1.96	 ------Sequential Output------ --Sequential Input- --Random-

Concurrency 1	 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine	 Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP

omega2		 4G 52 85 7793 5 5989 4 1065 99 43264 9 830.8 30

Latency			 1543ms 2262ms 1938ms 15895us	 426ms 1307ms

Version 1.96	 ------Sequential Create------ --------Random Create--------

omega2			 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

			 files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP

			 256 5994 23 156241 99 722 2 6493 24 200789 99 385 1

Latency			 9747ms 4427us 18464ms 9340ms 1719us 23217ms

1.96,1.96,omega2,1,1341057692,4G,,52,85,7793,5,5989,4,1065,99,43264,9,830.8,30,256,,,,,5994,23,156241,99,722,2,6493,24,200789,99,385,1,1543ms,2262ms,1938ms,15895us,426ms,1307ms,9747ms,4427us,18464ms,9340ms,1719us,23217ms

From this, I cant easily see any significant differences between the different partitioning schemes - so because I havent found any bad blocks, I will leave it at 0MB.

 

[chaotix]

This is strange. Three explanations come to (my) mind:

1. Your card is somehow "optimized" for this kind of unaligned access.
   
   
2. The computer had too much RAM and the results are invalidated by the cache.
   
   
3. Your card is really, really fast and all partitions where not align correctly (and hence show the same but really fast results for unaligned access).
   

Is this the same card that gave the "strange" flashbench readings even after filling it with data?


Did you check with "fdisk -ul" that the partitions where indeed aligned correctly.


Maybe you could run bonnie again this time on the Pandora? That would reduce the effect of caching and give results that would be more realistic for usage on the Pandora and more compareable to other SD-card measurements like the ones I posted on http://lxtek.de/pandora/flashbench/ (which clearly show the effect of misalignment).


But either way, I'm confused

 

[OmegaPhil]

Remember, 2GB RAM on the laptop - bonnie shouldve managed the correct filesize this time. I am now using this card normally (already dug it up too many times), once the original data is back in the card I will run a bonnie test on the Pandora, but I dont plan to do any further major testing with the card (perhaps the next time?).


There is still plenty for me to understand (why ext4 mounted with sync causes ~500KB/Sec reported write speed in SpaceFM on this main machine??), but I dont really want to spend more time on this.

 

[chaotix]

Remember, 2GB RAM on the laptop - bonnie shouldve managed the correct filesize this time.

The bulk writes/reads should not really be affected by the partition alignment. So the filesize is not that relevant for this. I was more thinking about the file creation and deletion tests. Maybe 2GB was enough to cache most/all of the metadata for the test. On my 512MB Pandora that was obviously not the case.

I am now using this card normally (already dug it up too many times), once the original data is back in the card I will run a bonnie test on the Pandora, but I dont plan to do any further major testing with the card (perhaps the next time?).

I can understand that.

There is still plenty for me to understand (why ext4 mounted with sync causes ~500KB/Sec reported write speed in SpaceFM on this main machine??), but I dont really want to spend more time on this.

I don't know what you mean with SpaceFM, but that the write speed drops dramatically with the "sync" option should not be too much of a surprise. Write caching is completely disabled with that option. So each erase block will probably be written several times before it is "full".

 

[OmegaPhil]

SpaceFM is The Best™ UNIX file manager.


I have downloaded your bonnie binary and am testing the card on the Pandora now.


Ah, thankyou for the information - I can see sync is a Very Bad Thing in this case... I just got disgusted when (under mount organised by Disk Utility) writes went at RAM speed and it took forever to later umount the card. I understand exactly what it is doing, but I expect to do a transfer then have the file manager actually tell me the genuine progression of the operation. I guess its something else for me to look into at some point (the correct balance).

 

[OmegaPhil]

I have now ran two runs of bonnie++ on the Pandora. Firstly, the card's partitions:

Disk /dev/mmcblk1: 31.9 GB, 31914983424 bytes

64 heads, 32 sectors/track, 30436 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

Disk identifier: 0x00018bea


	    Device Boot	  Start		 End	  Blocks   Id  System

/dev/mmcblk1p1			   1	   30436    31166448   83  Linux

Bonnie tests:

Version  1.96	   ------Sequential Output------ --Sequential Input- --Random-

Concurrency   1	 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine	    Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP

omega-openpandor 1G    39  93  7046   8  4858   7   514  97 21226  15 654.4  60

Latency			   984ms   10987ms    1687ms   51422us   13885us    1266ms

Version  1.96	   ------Sequential Create------ --------Random Create--------

omega-openpandora   -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

			  files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %

CP

			    256   100   2 46885  98    64   1    99   2 46854  97    51   1

Latency			  4360ms    9276us    5605ms    5018ms    7994us    6169ms

1.96,1.96,omega-openpandora,1,1341037907,1G,,39,93,7046,8,4858,7,514,97,21226,15,654.4,60,256,,,,,100,2,46885,98,64,1,99,2,46854,97,51,1,984ms,10987ms,168

7ms,51422us,13885us,1266ms,4360ms,9276us,5605ms,5018ms,7994us,6169ms

Version  1.96	   ------Sequential Output------ --Sequential Input- --Random-

Concurrency   1	 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine	    Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP

omega-openpandor 1G    38  92  6955   8  4819   7   557  98 20975  14 792.0  68

Latency			   948ms   12289ms    2378ms   38788us   14038us	 790ms

Version  1.96	   ------Sequential Create------ --------Random Create--------

omega-openpandora   -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

			  files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP

			    256   101   2 55643  98    64   1   101   2 71541  99    50   1

Latency			  4021ms   15350us    6219ms    3957ms	 457us    4850ms

1.96,1.96,omega-openpandora,1,1341087357,1G,,38,92,6955,8,4819,7,557,98,20975,14,792.0,68,256,,,,,101,2,55643,98,64,1,101,2,71541,99,50,1,948ms,12289ms,2378ms,38788us,14038us,790ms,4021ms,15350us,6219ms,3957ms,457us,4850ms

 

[chaotix]

What filesystem and mount options did you use? (mount | grep mmcblk)

 

[OmegaPhil]

ext4 (rw,dirsync,noatime,user_xattr,barrier=1,data=ordered)

 

[chaotix]

Block output is slower than in my tests for ext4, but that's probably just the write speed of your card. I don't think it has anything to do with partition alignment.


"dirsync" explains the slow create and delete operations which are about at the same level as the tests I did.


So it seems that the alignment has no relevance for your card. Strange. But that would explain the inconclusive flashbench results for your card. What kind of card are you using?

 

[Link]

So if you were getting a new 128GB Lexar card, what is your recommended way to set it up. I think I'll want to keep it FAT32, but do I put 16MB at the beginning or end. I'm really confused.

 

[chaotix]

I would use flashbench (http://lxtek.de/pandora/flashbench) to test what erase block size the card has and then partition it accordingly. If the results are inconclusive I would write several GB of data to the card and try it again.


If that is too much work, I guess 16MB is a safe size.

 

[Linux-SWAT]

Yup, 16M.


I think 16M at end is not mandatory, but it's safe.

 

[Link]

Thanks for the info ^^

 

[OmegaPhil]

What kind of card are you using?

SanDisk Ultra SDHC card.


Time for some drama - I have just noticed something. This is what I get on the Debian machine from this card:

Disk /dev/sdm: 31.9 GB, 31914983424 bytes

64 heads, 32 sectors/track, 30436 cylinders, total 62333952 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x00018bea


   Device Boot	  Start		 End	  Blocks   Id  System

/dev/sdm1			  32    62332927    31166448   83  Linux

So now I am confused. It looks like fdisk on the two machines are using different units - 2048B on the Pandora, and 1B on the Debian machine. Does this mean that the first 'track' is basically the MBR? I havent really thought too hard about how partitioning is actually implemented, and this would make sense - and therefore not using an empty first partition = always misaligned.

 

[chaotix]

What kind of card are you using?

SanDisk Ultra SDHC card.

I just bought a SanDisk Ultra Class 10 microSDHC card and am running some benchmarks now.

Time for some drama - I have just noticed something. This is what I get on the Debian machine from this card:

Ah. Did not see that in your first post. You did not give the -u option to fdisk.

Disk /dev/mmcblk1: 31.9 GB, 31914983424 bytes

64 heads, 32 sectors/track, 30436 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

Disk identifier: 0x00018bea


	 Device Boot	 Start		 End	 Blocks Id System

/dev/mmcblk1p1			 1	 30436 31166448 83 Linux

Disk /dev/sdm: 31.9 GB, 31914983424 bytes

64 heads, 32 sectors/track, 30436 cylinders, total 62333952 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x00018bea


Device Boot	 Start		 End	 Blocks Id System

/dev/sdm1			 32 62332927 31166448 83 Linux

That's the same partition layout. The first one reports values in an ancient DOS compatibility mode which uses CHS (cylinder/head/sector) addressing (which has nothing to do with cylinders, heads and sectors on modern hard drives). The second one uses the real (LBA) sector numbers. If you want to see the same partition table on both machines use "fdisk -ul"

So now I am confused. It looks like fdisk on the two machines are using different units - 2048B on the Pandora, and 1B on the Debian machine. Does this mean that the first 'track' is basically the MBR? I havent really thought too hard about how partitioning is actually implemented, and this would make sense - and therefore not using an empty first partition = always misaligned.

The MBR is the first sector of a hard disk. It contains the partition table. And yes, that's also why you partition is always misaligned on SD cards or modern hard drives with 4K sectors if you start you partition directly on the second sector as the second sector starts with byte 513.


So, if you don't want to be confused by some old compatibility mode which you really don't need anymore, use "fdisk -ul" to display partitions and "fdisk -cul" to partition. (Although fdisk on the Pandora does not understand the -c switch as it is too old. Which is why its not a good tool for the job. You need at least fdisk version 2.17.1 for that.)

 

[chaotix]

According to Wikipedia (https://en.wikipedia...i/TRIM#SD.2FMMC) it is at least possible to erase a block on an SD-card. I read somewhere that this Windows formatter takes a very long time. That correlates to the Wikipedia statement that the block has to be actually overwritten to be erased.


But even if the formatter does actually marks the blocks as unused the question remains what the benefits of this are on an SD-card. As I wrote in the filesystem shootout thread (http://boards.openpa...ut/#entry160742) I have my doubt that an ElCheapo SD-card controller can profit from the information which blocks are unused.


Although testing if there is an improvement should be possible:


1. Run the formatter


2. Use fdisk to show partition table.


3. Create a new filesystem on the partition (e.g. ext4 or vfat)


4. Run a benchmark test like bonnie.


5. Fill the complete card with data. Something like dd if=/dev/zero of=/dev/mmcblk<whatevere> bs=1M.


6. Repartition the card according to 2.


7. Create the same filesystem with the same options as in step 3.


8. Run bonnie again.

OK, I did this now. (I did not run step 1 as the card was new.)


And the result shows there are no performance differences between running bonnie before filling the whole card and then running it again after I filled the whole partition.


This implies that running the official formatter does not have any effect on performance even if it somehow manages to convince the card that blocks are unused. (Provided you aligned the partition correctly when you partitioned it by hand.)


This also implies that leaving some space at the end of the card empty/unpartitioned should have no impact on performance.

 

[Limeberry]

SDFormatter does not just align the beginning of the partition to 4mb, it also aligns the first data sector by changing the number of reserved sectors on a fat32 partition. I remember trying gparted and it aligned the partitions starting offset but failed to align the first data sector that starts inside of the fat32 partition.

 

[Linux-SWAT]

Is this something achievable by the Linux tools ?