Formatting a SD card : demystifying !


OmegaPhil: On ext2/3/4 filesystem a certain amount of space is reserved for the user root by default. This is useful to keep essential utilities (like syslog) on a regular unix system running if a user should fill the entire partition. It also keeps fragmentation low. But you can disable this at filesystem creation time with the "-m 0" option to mkfs.ext2/3/4 or later with the tune2fs utility that is unfortunately not installed on the official Pandora firmware. "tune2fs -m 0 /dev/yoursdcardblockdevice" should do the trick.


If you have not disabled the journal for your ext3 filesystem this will also occupy some space. Although this is shown as used with "df".
 
And on another note regarding partition alignment:


It's really no use talking about it if you can test it yourself :) (and see that it does make a huge difference).


I compiled a small utility called "flashbench" for the Pandora. With it you can guess the erase block size of your SD-cards. According to the list at https://wiki.linaro....FlashCardSurvey it is probably between 1 MB and 8MB. But don't count on the details in the list. I have a card that's listed which has a different erase block size than the one on the list. So just measure it yourself. It will only take about one minute.


I wrote a small HowTo here: http://lxtek.de/pandora/flashbench/


Bottomline:


File creation times from a bonnie++ run on an ext4 filesystem with journaling enabled and mounted with relatime.


Wrong alignment (Partition starts at block 1): 69 files created / sec


Corrected alignment (Partition starts at 16384): 4380 files created / sec


To be fair this only affects small random writes in this magnitude. Linear reads and writes are more or less unaffected by the partition alignment.
 
chaotix said:
OmegaPhil: On ext2/3/4 filesystem a certain amount of space is reserved for the user root by default. This is useful to keep essential utilities (like syslog) on a regular unix system running if a user should fill the entire partition. It also keeps fragmentation low. But you can disable this at filesystem creation time with the "-m 0" option to mkfs.ext2/3/4 or later with the tune2fs utility that is unfortunately not installed on the official Pandora firmware. "tune2fs -m 0 /dev/yoursdcardblockdevice" should do the trick.


If you have not disabled the journal for your ext3 filesystem this will also occupy some space. Although this is shown as used with "df".
Click to expand...

Yes, root reserved blocks had already been killed off. Even if I hadn't, much more than 14.4MB would be stolen.
 
Last edited by a moderator:
chaotix said:
It's really no use talking about it if you can test it yourself :) (and see that it does make a huge difference).
Click to expand...
The question isn't whether correct alignment does have an effect, because it does have an effect that cannot be denied.


The question in this thread was whether the only special thing that Panasonic's official formatter does is to use a 4MiB offset, and this could not be verified.


Theses are two different, albeit related, issues.
 
chaotix, thanks for linking flashbench. I have ran it on my Sandisk 16GB Extreme card on this Debian machine with the following results:



Code: 
sudo ./flashbench -a /dev/sdm --blocksize=1024

align 4294967296	pre 1.22ms	on 1.42ms	post 1.21ms	diff 197µs

align 2147483648	pre 1.21ms	on 1.49ms	post 1.04ms	diff 367µs

align 1073741824	pre 1.12ms	on 1.41ms	post 1.07ms	diff 319µs

align 536870912	pre 1.19ms	on 1.44ms	post 1.03ms	diff 330µs

align 268435456	pre 1.22ms	on 1.44ms	post 1.17ms	diff 243µs

align 134217728	pre 1.19ms	on 1.46ms	post 1.02ms	diff 354µs

align 67108864	pre 1.18ms	on 1.44ms	post 1.24ms	diff 233µs

align 33554432	pre 1.19ms	on 1.4ms	post 1.23ms	diff 187µs

align 16777216	pre 1.18ms	on 1.43ms	post 1.21ms	diff 240µs

align 8388608	pre 1.14ms	on 1.46ms	post 1.21ms	diff 280µs

align 4194304	pre 951µs	on 1.12ms	post 920µs	diff 179µs

align 2097152	pre 981µs	on 1.07ms	post 993µs	diff 78.1µs

align 1048576	pre 993µs	on 1.09ms	post 1.01ms	diff 93.8µs

align 524288	pre 996µs	on 1.09ms	post 1.01ms	diff 86.5µs

align 262144	pre 987µs	on 1.09ms	post 1ms	diff 97.6µs

align 131072	pre 977µs	on 1.09ms	post 1ms	diff 102µs

align 65536	pre 1.05ms	on 1.1ms	post 982µs	diff 89.3µs

align 32768	pre 1.03ms	on 1.07ms	post 1.07ms	diff 24.7µs

align 16384	pre 1.01ms	on 1.08ms	post 1.02ms	diff 66.6µs

align 8192	pre 1ms	on 1.08ms	post 1.01ms	diff 68.3µs

align 4096	pre 994µs	on 1.07ms	post 1.02ms	diff 56.4µs

align 2048	pre 1.02ms	on 1.08ms	post 1.02ms	diff 58.7µs


I cant read the diff column too easily but it seems the first jump is at 4194304B, then the second at 8388608B - but 4MB sounds far too large for the page size, and 8MB must be the erase block size - I have repartitioned the card now to have the main filesystem start at 16384 blocks, 8MB.
 
I'd be curious to know the result of the official SD formater about that 8M start on your 16G card.
 
OmegaPhil said:
I cant read the diff column too easily but it seems the first jump is at 4194304B, then the second at 8388608B - but 4MB sounds far too large for the page size, and 8MB must be the erase block size - I have repartitioned the card now to have the main filesystem start at 16384 blocks, 8MB.
Click to expand...

That's strange. I have no idea what the result at 4MB means. But I'm not an expert on SD-cards. I guess, 8MB is the safe choice. Although you could try it with 4MB alignment and run some benchmark test.

Christoph.Krn said:
The question isn't whether correct alignment does have an effect, because it does have an effect that cannot be denied.


The question in this thread was whether the only special thing that Panasonic's official formatter does is to use a 4MiB offset, and this could not be verified.
Click to expand...

OK, sorry for the misunderstanding.


According to Wikipedia (https://en.wikipedia...i/TRIM#SD.2FMMC) it is at least possible to erase a block on an SD-card. I read somewhere that this Windows formatter takes a very long time. That correlates to the Wikipedia statement that the block has to be actually overwritten to be erased.


But even if the formatter does actually marks the blocks as unused the question remains what the benefits of this are on an SD-card. As I wrote in the filesystem shootout thread (http://boards.openpa...ut/#entry160742) I have my doubt that an ElCheapo SD-card controller can profit from the information which blocks are unused.


Although testing if there is an improvement should be possible:


1. Run the formatter


2. Use fdisk to show partition table.


3. Create a new filesystem on the partition (e.g. ext4 or vfat)


4. Run a benchmark test like bonnie.


5. Fill the complete card with data. Something like dd if=/dev/zero of=/dev/mmcblk<whatevere> bs=1M.


6. Repartition the card according to 2.


7. Create the same filesystem with the same options as in step 3.


8. Run bonnie again.


(Step 2 is necessary to ensure that we can reproduce the exact same filesystem in step 7.)

Linux-SWAT said:
I'd be curious to know the result of the official SD formater about that 8M start on your 16G card.
Click to expand...

Yes. Me too. When I first used flashbench on a fresh card the results were non-conclusive. After writing to the card results got stable.


Although even if the results do not change after running the formatter it does not necessarily mean that there is no performance benefit from using it. As the sporadic results on an unused card might be due to an uninitilized controller.
 
Last edited by a moderator:
^ For what it's worth, the last time I tried it, it didn't work under WINE anyway. It would run, but it wouldn't allow you to actually manipulate the cards. That was a year or two ago, though, so it may well have improved since.
 
Last edited by a moderator:
I have now tested my Sandisk 32GB Ultra SDHC card, and it successfully reads and writes all sectors :)


Flashbench however seems to be giving nonsensical results. The following was done after the card was completely written to due to the above test:


First run:



Code: 
sudo ./flashbench -a /dev/sdm --blocksize=1024

align 8589934592    pre 1.1ms    on 1.37ms    post 1.04ms    diff 305µs

align 4294967296    pre 1.14ms    on 1.5ms    post 1.11ms    diff 380µs

align 2147483648    pre 1.14ms    on 1.43ms    post 1.01ms    diff 353µs

align 1073741824    pre 1.09ms    on 1.5ms    post 1.1ms    diff 403µs

align 536870912    pre 1.11ms    on 1.41ms    post 1.02ms    diff 346µs

align 268435456    pre 1.17ms    on 1.46ms    post 1.09ms    diff 330µs

align 134217728    pre 1.15ms    on 1.42ms    post 1.03ms    diff 332µs

align 67108864    pre 1.16ms    on 1.46ms    post 1.09ms    diff 339µs

align 33554432    pre 1.13ms    on 1.45ms    post 1.06ms    diff 352µs

align 16777216    pre 1.1ms    on 1.46ms    post 1.03ms    diff 396µs

align 8388608    pre 1.04ms    on 1.32ms    post 1.03ms    diff 291µs

align 4194304    pre 1.04ms    on 1.31ms    post 1.01ms    diff 282µs

align 2097152    pre 1.12ms    on 1.13ms    post 1.04ms    diff 49.4µs

align 1048576    pre 1.1ms    on 1.16ms    post 1.06ms    diff 75.1µs

align 524288    pre 1.06ms    on 1.25ms    post 1.02ms    diff 213µs

align 262144    pre 1.11ms    on 1.13ms    post 1.06ms    diff 47.1µs

align 131072    pre 1.1ms    on 1.14ms    post 1.04ms    diff 65.4µs

align 65536    pre 1.09ms    on 1.15ms    post 1.07ms    diff 70.3µs

align 32768    pre 1.13ms    on 1.15ms    post 1.04ms    diff 60.9µs

align 16384    pre 1.13ms    on 1.11ms    post 1.06ms    diff 15.2µs

align 8192    pre 1.01ms    on 1.03ms    post 999µs    diff 25.3µs

align 4096    pre 1.02ms    on 1.01ms    post 1.01ms    diff -5667ns

align 2048    pre 991µs    on 1ms    post 974µs    diff 20.6µs



Second run:





Code: 
sudo ./flashbench -a /dev/sdm --blocksize=1024

align 8589934592    pre 1.13ms    on 1.32ms    post 1.02ms    diff 248µs

align 4294967296    pre 1.16ms    on 1.47ms    post 1.09ms    diff 340µs

align 2147483648    pre 1.13ms    on 1.47ms    post 1.11ms    diff 351µs

align 1073741824    pre 1.12ms    on 1.48ms    post 1.09ms    diff 381µs

align 536870912    pre 1.11ms    on 1.44ms    post 1.04ms    diff 359µs

align 268435456    pre 1.13ms    on 1.46ms    post 1.07ms    diff 365µs

align 134217728    pre 1.12ms    on 1.44ms    post 1.09ms    diff 336µs

align 67108864    pre 1.13ms    on 1.46ms    post 1.02ms    diff 383µs

align 33554432    pre 1.12ms    on 1.45ms    post 1.08ms    diff 356µs

align 16777216    pre 1.09ms    on 1.43ms    post 1.03ms    diff 370µs

align 8388608    pre 1.05ms    on 1.35ms    post 1.03ms    diff 307µs

align 4194304    pre 1.06ms    on 1.3ms    post 993µs    diff 276µs

align 2097152    pre 1.1ms    on 1.12ms    post 1.05ms    diff 42.7µs

align 1048576    pre 1.1ms    on 1.13ms    post 1.04ms    diff 60.8µs

align 524288    pre 1.04ms    on 1.26ms    post 985µs    diff 249µs

align 262144    pre 1.1ms    on 1.13ms    post 1.07ms    diff 43.9µs

align 131072    pre 1.11ms    on 1.16ms    post 1.03ms    diff 92.1µs

align 65536    pre 1.09ms    on 1.12ms    post 1.06ms    diff 40.4µs

align 32768    pre 1.14ms    on 1.16ms    post 1.04ms    diff 65.3µs

align 16384    pre 1.12ms    on 1.12ms    post 1.03ms    diff 43µs

align 8192    pre 1.02ms    on 1.01ms    post 1.01ms    diff -5782ns

align 4096    pre 1.01ms    on 1.02ms    post 990µs    diff 19µs

align 2048    pre 956µs    on 1.01ms    post 976µs    diff 40µs


I do like the negative results :)


As long as bonnie works, I plan to test 4MB, 8MB then 0MB partition offsets (presumably 0MB is correct in this case since all bytes in the card are fine).
 
Bonnie seems not to be able to tell how big the card is either... this happened before with my other card:



Code: 
/usr/sbin/bonnie -d /media/pandora_2/ -n 256

Writing a byte at a time...done

Writing intelligently...Can't write block.: No space left on device

Can't write block 3606702.


Annoying as I cant then get a standardised test.
 
OmegaPhil said:
Bonnie seems not to be able to tell how big the card is either... this happened before with my other card:



Code: 
/usr/sbin/bonnie -d /media/pandora_2/ -n 256

Writing a byte at a time...done

Writing intelligently...Can't write block.: No space left on device

Can't write block 3606702.


Annoying as I cant then get a standardised test.
Click to expand...

Can you post the output of "df /media/pandora_2" and "cat /proc/partitions"?


P.S.


You can skip the character write test by adding "-f" to the command line options.
 
Last edited by a moderator:
Code: 
df /media/pandora_2

Filesystem	 1K-blocks    Used Available Use% Mounted on

/dev/sdm2	   31086428 5510180  24018132  19% /media/pandora_2



The used space is another bonnie test going on currently - it has been freshly formatted.





Code: 
cat /proc/partitions

major minor  #blocks  name


   8	   16  146523384 sdb

   8	   17    9764864 sdb1

   8	   18  136757248 sdb2

   8	    0  146523384 sda

   8	    1    9764864 sda1

   8	    2  136757248 sda2

   8	   32  976762584 sdc

   8	   33  976760001 sdc1

   8	   48  976762584 sdd

   8	   49  976760001 sdd1

   8	   64 1953514584 sde

   8	   65 1953512001 sde1

   8	   80 1953514584 sdf

   8	   96  976762584 sdg

   8	   97  976760001 sdg1

   8	  112 1953514584 sdh

   8	  113 1953512001 sdh1

   8	  192   31166976 sdm

   8	  193	   4080 sdm1

   8	  194   31162368 sdm2

  11	    0    4574368 sr0

  11	    1    1048575 sr1

   8	  144 1953514584 sdj

   8	  145 1953512001 sdj1

   8	  128 1953514584 sdi

   8	  129 1953512001 sdi1

   8	  160  245116416 sdk

   8	  161    9764864 sdk1

   8	  162  136757248 sdk2

   9	  128    9763768 md128

   9	  129  136756152 md129

   9	  127  976759865 md127


I use a few disks ;)
 
Mmmh. This is really strange....


Why is there 5GB of data in use on sdm2? If you are just runnig a Bonnie test and the partition is otherwise empty it should not be more than 1GB + some overhead (if you did not increase the testfile size).


Do you get any errors in 'dmesg'? What filesystem are you using? FAT32?
 
Currently the 'Bonnie.1019' file is 22.2GB, so its clearly growing as part of the test?


Filesystem is ext4, no errors.
 
OmegaPhil said:
Currently the 'Bonnie.1019' file is 22.2GB, so its clearly growing as part of the test?
Click to expand...

Ok. That's not right :) Bonnie should not use more than 512MB or 1G (twice the RAM-size) which should be more than enough.


You can force Bonnie to use 1G:



Code: 
$ bonnie -d /media/pandora_2 -s 1G


If it complains that this is too small it somehow thinks you have more RAM (that would explain why it is making such a huge file). You can convince it with "-r 512" that you only have 512MB.
 
Ah yes, it is presumably to avoid caching or something? I have 32GB RAM here ;) That would explain everything then - I will restart the test with 1GB.
 
You must log in or register to reply here.

Similar threads

superatarian
Formatting SD card
Replies
14
Views
7K
superatarian
superatarian
Wrath Of Khan
Anyone have trouble with a sandisk 32 gig class 2 sdhc card?
Replies
8
Views
2K
Tor
Tor
M
Copy from one SD card to another on the Pandora
Replies
1
Views
958
WizardStan
WizardStan
J
Sd Card Database?
Replies
10
Views
1K
Prometheus
Prometheus
sinoth
Using An Sd Card As Swap Space
Replies
4
Views
2K
Stuckie
Stuckie
Back
Top