Re: SSD & mechanical disc in RAID 1

[Wil Reichert <wil.reichert@gmail.com>]

On Sun, Jan 31, 2010 at 12:21 PM, Aryeh Gregor
<Simetrical+list@gmail.com> wrote:
> On Fri, Jan 22, 2010 at 11:37 AM, Goswin von Brederlow
> <goswin-v-b@web.de> wrote:
>> I think his initial idea of raid1 is verry good. With write-mostly all
>> reads should come from the fast SSD. I would also add write-behind.
>> That way writes will not wait for the rotating disk and get the full
>> SSD speed as well.
>
> Recently I got an Intel X25-M 80G, and have been trying this out.  I
> figured some people here might be interested in the results.  I've
> mirrored the SSD to a slow 80G partition on an old spinning-rust 160G
> disk, which is set to write-mostly, with write-behind of 16384 (the
> max).  The bitmap is on a separate unmirrored partition on the SSD.
> My root filesystem is on it, ext4 on top of LVM, using 60G out of
> 73.77G allocated to LVM, with a separate 500M-ish mirrored boot
> partition at the start, and a gap in between for storing unmirrored
> things (like all my RAID bitmaps).
>
> I did six runs of bonnie++ overnight:
>
> 1) 128 files, both disks working.
> 2) 1024 files, both disks working.
> 3) 128 files, conventional disk failed and removed.
> 4) 1024 files, conventional disk failed and removed.
> 5) Same as (2).
> 6) Same as (4).  (Note that in this case, the disk ended up being
> failed but not removed, but this should make no difference.)
>
> At the end of this e-mail, I've attached the full script I ran and its
> full output.  These are the bonnie++ numbers for the six tests, best
> viewed in a fixed-width font on a pretty wide screen (126 chars):
>
>         |      Sequential out      |  Sequential in   | Random  |
> |     Sequential create     |       Random create      |
>         |per char| block  |rewrite |per char|  block  | Seeks   |
> | Create |  Read   | Delete | Create |  Read   |Delete |
> monoid,6G,47571,90,55343,30,41390,24,55803,96,233594,54,4923.0,36,
> 128,25973,94,+++++,+++,15910,41,27179,85,+++++,+++,11922,38
> monoid,6G,48358,90,55378,25,41338,25,56883,95,233689,52,4233.0,29,1024,19819,70,64728,
> 75,  848, 4,14928,60,55484, 79,  463, 3
> monoid,6G,48751,90,77485,27,45263,25,55660,94,238757,49,5107.7,33,
> 128,28027,86,+++++,+++,20161,49,29618,88,+++++,+++,14780,43
> monoid,6G,44503,83,73894,29,42975,26,56463,95,237173,53,5052.9,36,1024,18073,73,66151,
> 81, 5348,28,18102,65,60213, 77, 3441,23
> monoid,6G,48580,91,55567,24,40748,22,56803,95,237006,49,4828.2,29,1024,20332,67,73175,
> 80,  783, 4,13615,45,58066, 74,  489, 3
> monoid,6G,48966,92,77517,29,43999,24,56863,94,198557,38,3553.8,16,1024,19273,68,72414,
> 86, 6326,31,16499,60,56438, 75, 2784,15
>
> (Where two numbers are given, the first is KB out per second and the
> second is CPU usage, as far as I can figure.) Note that there was lots
> of memory free here, about 2.5G (-/+ buffers/cache as reported by
> free).  My conclusions:
>
> 1) The file creation tests with only 128 files should just be ignored,
> they look like they're in-memory.  The 1024-file tests look more
> plausible.
>
> 2) All reads perform comparably whether the extra disk is present or
> not, as expected.  There's some variation, but then, I wasn't being
> very scientific.
>
> 3) Tests 3, 4, and 6 (only SSD) are about 35% faster on the sequential
> out rewrite test.  The per-char sequential out test was CPU-bound, and
> everything performed the same.  Everything also performed the same on
> the rewrite test; I'm not sure why.
>
> 4) Sequential file creation is about the same between tests 4 and 6
> (only SSD) and tests 2 and 5 (both disks).  Random file creation gives
> the disk+SSD about a 20% lead.  But both sequential and random
> deletion have an order-of-magnitude difference between the two.  I
> don't know why this might be -- lots of buffering for some operations
> but not others?
>
> Overall, this seems like a very feasible setup, and I'll certainly be
> sticking with it, even though it will obviously slow down some writes.
>  Hope this data will be useful (or at least interesting) to someone.
>
> The script that I ran from the at job follows, with a couple of
> comments added.  I ran it using ionice -c1 to reduce the effects of
> any concurrent operations.
>
>
>
> #!/bin/bash
> cd /tmp
> echo Test 1; echo
> free -m
> bonnie -u 0 -n 128
> echo; echo Test 2; echo
> free -m
> echo
> bonnie -u 0 -n 1024
> echo; echo Test 3; echo
> free -m
> echo
> mdadm --fail /dev/md1 /dev/sda2
> sleep 5
> mdadm --remove /dev/md1 /dev/sda2
> echo
> bonnie -u 0 -n 128
> echo; echo Test 4; echo
> free -m
> echo
> bonnie -u 0 -n 1024
> echo
> free -m
> echo; echo 'Re-adding sda2 and waiting for sync'; echo
> sudo mdadm --add /dev/md1 --write-mostly /dev/sda2
> sleep 1800
> cat /proc/mdstat
> echo; echo Test 5; echo
> free -m
> echo
> bonnie -u 0 -n 1024
> echo
> mdadm --fail /dev/md1 /dev/sda2
> # The second command was run too fast and failed, I forgot the sleep 5
> from above.
> mdadm --remove /dev/md1 /dev/sda2
> echo; echo Test 6; echo
> free -m
> echo
> bonnie -u 0 -n 1024
> echo
> free -m
> echo
> # Also failed, since the device hadn't been removed.
> sudo mdadm --add /dev/md1 --write-mostly /dev/sda2
>
>
>
>
> Output of script:
>
>
>
>
> Test 1
>
>            total       used       free     shared    buffers     cached
> Mem:          3024       2085        938          0        589        973
> -/+ buffers/cache:        522       2501
> Swap:         1023        526        497
> Using uid:0, gid:0.
> Writing with putc()...done
> Writing intelligently...done
> Rewriting...done
> Reading with getc()...done
> Reading intelligently...done
> start 'em...done...done...done...
> Create files in sequential order...done.
> Stat files in sequential order...done.
> Delete files in sequential order...done.
> Create files in random order...done.
> Stat files in random order...done.
> Delete files in random order...done.
> Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
>                   -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
> monoid           6G 47571  90 55343  30 41390  24 55803  96 233594  54  4923  36
>                   ------Sequential Create------ --------Random Create--------
>                   -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>             files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>               128 25973  94 +++++ +++ 15910  41 27179  85 +++++ +++ 11922  38
> monoid,6G,47571,90,55343,30,41390,24,55803,96,233594,54,4923.0,36,128,25973,94,+++++,+++,15910,41,27179,85,+++++,+++,11922,38
>
> Test 2
>
>            total       used       free     shared    buffers     cached
> Mem:          3024        544       2480          0         51         30
> -/+ buffers/cache:        462       2562
> Swap:         1023        526        497
>
> Using uid:0, gid:0.
> Writing with putc()...done
> Writing intelligently...done
> Rewriting...done
> Reading with getc()...done
> Reading intelligently...done
> start 'em...done...done...done...
> Create files in sequential order...done.
> Stat files in sequential order...done.
> Delete files in sequential order...done.
> Create files in random order...done.
> Stat files in random order...done.
> Delete files in random order...done.
> Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
>                   -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
> monoid           6G 48358  90 55378  25 41338  25 56883  95 233689  52  4233  29
>                   ------Sequential Create------ --------Random Create--------
>                   -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>             files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>              1024 19819  70 64728  75   848   4 14928  60 55484  79   463   3
> monoid,6G,48358,90,55378,25,41338,25,56883,95,233689,52,4233.0,29,1024,19819,70,64728,75,848,4,14928,60,55484,79,463,3
>
> Test 3
>
>            total       used       free     shared    buffers     cached
> Mem:          3024        788       2235          0        332         50
> -/+ buffers/cache:        405       2619
> Swap:         1023        565        458
>
> mdadm: set /dev/sda2 faulty in /dev/md1
> mdadm: hot removed /dev/sda2
>
> Using uid:0, gid:0.
> Writing with putc()...done
> Writing intelligently...done
> Rewriting...done
> Reading with getc()...done
> Reading intelligently...done
> start 'em...done...done...done...
> Create files in sequential order...done.
> Stat files in sequential order...done.
> Delete files in sequential order...done.
> Create files in random order...done.
> Stat files in random order...done.
> Delete files in random order...done.
> Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
>                   -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
> monoid           6G 48751  90 77485  27 45263  25 55660  94 238757  49  5108  33
>                   ------Sequential Create------ --------Random Create--------
>                   -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>             files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>               128 28027  86 +++++ +++ 20161  49 29618  88 +++++ +++ 14780  43
> monoid,6G,48751,90,77485,27,45263,25,55660,94,238757,49,5107.7,33,128,28027,86,+++++,+++,20161,49,29618,88,+++++,+++,14780,43
>
> Test 4
>
>            total       used       free     shared    buffers     cached
> Mem:          3024        441       2582          0         48         25
> -/+ buffers/cache:        367       2656
> Swap:         1023        594        429
>
> Using uid:0, gid:0.
> Writing with putc()...done
> Writing intelligently...done
> Rewriting...done
> Reading with getc()...done
> Reading intelligently...done
> start 'em...done...done...done...
> Create files in sequential order...done.
> Stat files in sequential order...done.
> Delete files in sequential order...done.
> Create files in random order...done.
> Stat files in random order...done.
> Delete files in random order...done.
> Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
>                   -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
> monoid           6G 44503  83 73894  29 42975  26 56463  95 237173  53  5053  36
>                   ------Sequential Create------ --------Random Create--------
>                   -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>             files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>              1024 18073  73 66151  81  5348  28 18102  65 60213  77  3441  23
> monoid,6G,44503,83,73894,29,42975,26,56463,95,237173,53,5052.9,36,1024,18073,73,66151,81,5348,28,18102,65,60213,77,3441,23
>
>            total       used       free     shared    buffers     cached
> Mem:          3024        734       2289          0        329         42
> -/+ buffers/cache:        362       2661
> Swap:         1023        609        414
>
> Re-adding sda2 and waiting for sync
>
> mdadm: re-added /dev/sda2
> Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5]
> [raid4] [raid10]
> md2 : active raid1 sdb1[0] sda1[1](W)
>     521984 blocks [2/2] [UU]
>     bitmap: 0/64 pages [0KB], 4KB chunk, file: /mnt/ssd-junk/boot-bitmap
>
> md3 : active raid10 sdc2[2] sdd2[1]
>     488102656 blocks super 1.2 64K chunks 2 far-copies [2/2] [UU]
>     bitmap: 1/466 pages [4KB], 512KB chunk, file: /mnt/ssd-junk/extra-bitmap
>
> md1 : active raid1 sda2[1](W) sdb2[0]
>     77352896 blocks [2/2] [UU]
>     bitmap: 29/296 pages [116KB], 128KB chunk, file: /mnt/ssd-junk/root-bitmap
>
> unused devices: <none>
>
> Test 5
>
>            total       used       free     shared    buffers     cached
> Mem:          3024       1374       1649          0        357        621
> -/+ buffers/cache:        396       2627
> Swap:         1023        602        421
>
> Using uid:0, gid:0.
> Writing with putc()...done
> Writing intelligently...done
> Rewriting...done
> Reading with getc()...done
> Reading intelligently...done
> start 'em...done...done...done...
> Create files in sequential order...done.
> Stat files in sequential order...done.
> Delete files in sequential order...done.
> Create files in random order...done.
> Stat files in random order...done.
> Delete files in random order...done.
> Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
>                   -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
> monoid           6G 48580  91 55567  24 40748  22 56803  95 237006  49  4828  29
>                   ------Sequential Create------ --------Random Create--------
>                   -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>             files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>              1024 20332  67 73175  80   783   4 13615  45 58066  74   489   3
> monoid,6G,48580,91,55567,24,40748,22,56803,95,237006,49,4828.2,29,1024,20332,67,73175,80,783,4,13615,45,58066,74,489,3
>
> mdadm: set /dev/sda2 faulty in /dev/md1
> mdadm: hot remove failed for /dev/sda2: Device or resource busy
>
> Test 6
>
>            total       used       free     shared    buffers     cached
> Mem:          3024        720       2303          0        335         40
> -/+ buffers/cache:        345       2678
> Swap:         1023        653        370
>
> Using uid:0, gid:0.
> Writing with putc()...done
> Writing intelligently...done
> Rewriting...done
> Reading with getc()...done
> Reading intelligently...done
> start 'em...done...done...done...
> Create files in sequential order...done.
> Stat files in sequential order...done.
> Delete files in sequential order...done.
> Create files in random order...done.
> Stat files in random order...done.
> Delete files in random order...done.
> Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
>                   -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
> monoid           6G 48966  92 77517  29 43999  24 56863  94 198557  38  3554  16
>                   ------Sequential Create------ --------Random Create--------
>                   -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>             files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>              1024 19273  68 72414  86  6326  31 16499  60 56438  75  2784  15
> monoid,6G,48966,92,77517,29,43999,24,56863,94,198557,38,3553.8,16,1024,19273,68,72414,86,6326,31,16499,60,56438,75,2784,15
>
>            total       used       free     shared    buffers     cached
> Mem:          3024       2174        849          0        545       1261
> -/+ buffers/cache:        366       2657
> Swap:         1023        674        349
>
> mdadm: Cannot open /dev/sda2: Device or resource busy
>

Thanks for providing this information, its quite useful.  I suspect
I'll be trying something like this myself in the near future.

Wil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html