Re: Growing RAID5 SSD Array

[Adam Goryachev <mailinglists@websitemanagers.com.au>]

On 13/03/14 22:58, Stan Hoeppner wrote:
> On 3/12/2014 9:49 PM, Adam Goryachev wrote:
> ...
>>      Number   Major   Minor   RaidDevice State
>>         7       8       33        0      active sync   /dev/sdc1
>>         6       8        1        1      active sync   /dev/sda1
>>         8       8       49        2      active sync   /dev/sdd1
>>         5       8       81        3      active sync   /dev/sdf1
>>         9       8       65        4      active sync   /dev/sde1
> ...
>> /dev/sda	Total_LBAs_Written	845235
>> /dev/sdc	Total_LBAs_Written      851335
>> /dev/sdd	Total_LBAs_Written      804564
>> /dev/sde	Total_LBAs_Written	719767
>> /dev/sdf	Total_LBAs_Written      719982
> ...
>> So the drive with the highest writes 851335 and the drive with the
>> lowest writes 719982 show a big difference. Perhaps I have a problem
>> with the setup/config of my array, or similar?
> This is normal for striped arrays.  If we reorder your write statistics
> table to reflect array device order, we can clearly see the effect of
> partial stripe writes.  These are new file allocations, appends, etc
> that are smaller than stripe width.  Totally normal.  To get these close
> to equal you'd need a chunk size of 16K or smaller.

Would that have a material impact on performance?
While current wear stats (Media Wearout Indicator) are all 98 or higher, 
at some point, would it be reasonable to fail the drive with the lowest 
write count, and then use it to replace the drive with the highest write 
count, repeating twice, so that over the next period of time usage 
should merge toward the average? Given the current wear rate, will 
probably replace all the drives in 5 years, which is well before they 
reach 50% wear anyway.

>> So, I could simply do the following:
>> mdadm --manage /dev/md1 --add /dev/sdb1
>> mdadm --grow /dev/md1 --raid-devices=6
>>
>> Probably also need to remove the bitmap and re-add the bitmap.
> Might want to do
>
> ~$ echo 250000 > /proc/sys/dev/raid/speed_limit_min
> ~$ echo 500000 > /proc/sys/dev/raid/speed_limit_min
>
> That'll bump min resync to 250 MB/s per drive, max 500 MB/s.  IIRC the
> defaults are 1 MB/s and 100 MB/s.

Worked perfectly on one machine, the second machine hung, and basically 
crashed. Almost turned into a disaster, but thankfully having two copies 
over the two machines I managed to get everything sorted. After a 
reboot, the second machine recovered and it grew the array also.

Some of the logs from that time:
Mar 13 23:05:59 san2 kernel: [42511.418380] RAID conf printout:
Mar 13 23:05:59 san2 kernel: [42511.418385]  --- level:5 rd:6 wd:6
Mar 13 23:05:59 san2 kernel: [42511.418388]  disk 0, o:1, dev:sdc1
Mar 13 23:05:59 san2 kernel: [42511.418390]  disk 1, o:1, dev:sde1
Mar 13 23:05:59 san2 kernel: [42511.418392]  disk 2, o:1, dev:sdd1
Mar 13 23:05:59 san2 kernel: [42511.418394]  disk 3, o:1, dev:sdf1
Mar 13 23:05:59 san2 kernel: [42511.418396]  disk 4, o:1, dev:sda1
Mar 13 23:05:59 san2 kernel: [42511.418399]  disk 5, o:1, dev:sdb1
Mar 13 23:05:59 san2 kernel: [42511.418444] md: reshape of RAID array md1
Mar 13 23:05:59 san2 kernel: [42511.418448] md: minimum _guaranteed_  
speed: 1000 KB/sec/disk.
Mar 13 23:05:59 san2 kernel: [42511.418451] md: using maximum available 
idle IO bandwidth (but not more than 200000 KB/sec) for reshape.
Mar 13 23:05:59 san2 kernel: [42511.418493] md: using 128k window, over 
a total of 468847936k.
Mar 13 23:06:00 san2 kernel: [42511.512165] md: md_do_sync() got signal 
... exiting
Mar 13 23:07:01 san2 kernel: [42573.067781] iscsi_trgt: Abort Task (01) 
issued on tid:9 lun:0 by sid:8162774362161664 (Function Complete)
Mar 13 23:07:01 san2 kernel: [42573.067789] iscsi_trgt: Abort Task (01) 
issued on tid:11 lun:0 by sid:7318349599801856 (Function Complete)
Mar 13 23:07:01 san2 kernel: [42573.067797] iscsi_trgt: Abort Task (01) 
issued on tid:12 lun:0 by sid:6473924787110400 (Function Complete)
Mar 13 23:07:01 san2 kernel: [42573.067838] iscsi_trgt: Abort Task (01) 
issued on tid:14 lun:0 by sid:5348025014485504 (Function Complete)
Mar 13 23:07:02 san2 kernel: [42573.237591] iscsi_trgt: Abort Task (01) 
issued on tid:8 lun:0 by sid:4503599899804160 (Function Complete)
Mar 13 23:07:02 san2 kernel: [42573.237600] iscsi_trgt: Abort Task (01) 
issued on tid:2 lun:0 by sid:14918173819994624 (Function Complete)

I probably hit CTRL-C causing the "got signal... exiting" because the 
system wasn't responding. There are a *lot* more iscsi errors and then 
these:
Mar 13 23:09:09 san2 kernel: [42700.645060] INFO: task md1_raid5:314 
blocked for more than 120 seconds.
Mar 13 23:09:09 san2 kernel: [42700.645087] "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 13 23:09:09 san2 kernel: [42700.645117] md1_raid5       D 
ffff880236833780     0   314      2 0x00000000
Mar 13 23:09:09 san2 kernel: [42700.645123]  ffff88022fc53690 
0000000000000046 ffff8801ee330240 ffff88023593e0c0
Mar 13 23:09:09 san2 kernel: [42700.645128]  0000000000013780 
ffff88022d859fd8 ffff88022d859fd8 ffff88022fc53690
Mar 13 23:09:09 san2 kernel: [42700.645133]  ffff8801ee4b85b8 
ffffffff81071011 0000000000000046 ffff8802307aa000
Mar 13 23:09:09 san2 kernel: [42700.645138] Call Trace:
Mar 13 23:09:09 san2 kernel: [42700.645146] [<ffffffff81071011>] ? 
arch_local_irq_save+0x11/0x17
Mar 13 23:09:09 san2 kernel: [42700.645160] [<ffffffffa0111c44>] ? 
check_reshape+0x27b/0x51a [raid456]
Mar 13 23:09:09 san2 kernel: [42700.645165] [<ffffffff8103f6ba>] ? 
try_to_wake_up+0x197/0x197
Mar 13 23:09:09 san2 kernel: [42700.645175] [<ffffffffa0060381>] ? 
md_check_recovery+0x2a5/0x514 [md_mod]
Mar 13 23:09:09 san2 kernel: [42700.645181] [<ffffffffa01156fe>] ? 
raid5d+0x1c/0x483 [raid456]
Mar 13 23:09:09 san2 kernel: [42700.645187] [<ffffffff8134fdc7>] ? 
_raw_spin_unlock_irqrestore+0xe/0xf
Mar 13 23:09:09 san2 kernel: [42700.645192] [<ffffffff8134eedb>] ? 
schedule_timeout+0x2c/0xdb
Mar 13 23:09:09 san2 kernel: [42700.645195] [<ffffffff81071011>] ? 
arch_local_irq_save+0x11/0x17
Mar 13 23:09:09 san2 kernel: [42700.645199] [<ffffffff81071011>] ? 
arch_local_irq_save+0x11/0x17
Mar 13 23:09:09 san2 kernel: [42700.645206] [<ffffffffa005a256>] ? 
md_thread+0x114/0x132 [md_mod]
Mar 13 23:09:09 san2 kernel: [42700.645212] [<ffffffff8105fcd3>] ? 
add_wait_queue+0x3c/0x3c
Mar 13 23:09:09 san2 kernel: [42700.645219] [<ffffffffa005a142>] ? 
md_rdev_init+0xea/0xea [md_mod]
Mar 13 23:09:09 san2 kernel: [42700.645224] [<ffffffff8105f681>] ? 
kthread+0x76/0x7e
Mar 13 23:09:09 san2 kernel: [42700.645229] [<ffffffff81356ef4>] ? 
kernel_thread_helper+0x4/0x10
Mar 13 23:09:09 san2 kernel: [42700.645234] [<ffffffff8105f60b>] ? 
kthread_worker_fn+0x139/0x139
Mar 13 23:09:09 san2 kernel: [42700.645238] [<ffffffff81356ef0>] ? 
gs_change+0x13/0x13
Mar 13 23:11:09 san2 kernel: [42820.250905] INFO: task md1_raid5:314 
blocked for more than 120 seconds.
Mar 13 23:11:09 san2 kernel: [42820.250932] "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 13 23:11:09 san2 kernel: [42820.250961] md1_raid5       D 
ffff880236833780     0   314      2 0x00000000
Mar 13 23:11:09 san2 kernel: [42820.250967]  ffff88022fc53690 
0000000000000046 ffff8801ee330240 ffff88023593e0c0
Mar 13 23:11:09 san2 kernel: [42820.250973]  0000000000013780 
ffff88022d859fd8 ffff88022d859fd8 ffff88022fc53690
Mar 13 23:11:09 san2 kernel: [42820.250978]  ffff8801ee4b85b8 
ffffffff81071011 0000000000000046 ffff8802307aa000
Mar 13 23:11:09 san2 kernel: [42820.250982] Call Trace:
Mar 13 23:11:09 san2 kernel: [42820.250991] [<ffffffff81071011>] ? 
arch_local_irq_save+0x11/0x17
Mar 13 23:11:09 san2 kernel: [42820.251004] [<ffffffffa0111c44>] ? 
check_reshape+0x27b/0x51a [raid456]
Mar 13 23:11:09 san2 kernel: [42820.251009] [<ffffffff8103f6ba>] ? 
try_to_wake_up+0x197/0x197
Mar 13 23:11:09 san2 kernel: [42820.251019] [<ffffffffa0060381>] ? 
md_check_recovery+0x2a5/0x514 [md_mod]
Mar 13 23:11:09 san2 kernel: [42820.251025] [<ffffffffa01156fe>] ? 
raid5d+0x1c/0x483 [raid456]
Mar 13 23:11:09 san2 kernel: [42820.251031] [<ffffffff8134fdc7>] ? 
_raw_spin_unlock_irqrestore+0xe/0xf
Mar 13 23:11:09 san2 kernel: [42820.251035] [<ffffffff8134eedb>] ? 
schedule_timeout+0x2c/0xdb
Mar 13 23:11:09 san2 kernel: [42820.251039] [<ffffffff81071011>] ? 
arch_local_irq_save+0x11/0x17
Mar 13 23:11:09 san2 kernel: [42820.251043] [<ffffffff81071011>] ? 
arch_local_irq_save+0x11/0x17
Mar 13 23:11:09 san2 kernel: [42820.251050] [<ffffffffa005a256>] ? 
md_thread+0x114/0x132 [md_mod]
Mar 13 23:11:09 san2 kernel: [42820.251056] [<ffffffff8105fcd3>] ? 
add_wait_queue+0x3c/0x3c
Mar 13 23:11:09 san2 kernel: [42820.251063] [<ffffffffa005a142>] ? 
md_rdev_init+0xea/0xea [md_mod]
Mar 13 23:11:09 san2 kernel: [42820.251068] [<ffffffff8105f681>] ? 
kthread+0x76/0x7e
Mar 13 23:11:09 san2 kernel: [42820.251073] [<ffffffff81356ef4>] ? 
kernel_thread_helper+0x4/0x10
Mar 13 23:11:09 san2 kernel: [42820.251078] [<ffffffff8105f60b>] ? 
kthread_worker_fn+0x139/0x139
Mar 13 23:11:09 san2 kernel: [42820.251082] [<ffffffff81356ef0>] ? 
gs_change+0x13/0x13

Plus a few more (can provide them if interested), then more iscsi 
errors, and finally I rebooted the machine:
Mar 14 00:55:08 san2 kernel: [    4.415215] md/raid:md1: not clean -- 
starting background reconstruction
Mar 14 00:55:08 san2 kernel: [    4.415216] md/raid:md1: reshape will 
continue
Mar 14 00:55:08 san2 kernel: [    4.415223] md/raid:md1: device sdc1 
operational as raid disk 0
Mar 14 00:55:08 san2 kernel: [    4.415225] md/raid:md1: device sdb1 
operational as raid disk 5
Mar 14 00:55:08 san2 kernel: [    4.415226] md/raid:md1: device sda1 
operational as raid disk 4
Mar 14 00:55:08 san2 kernel: [    4.415227] md/raid:md1: device sdf1 
operational as raid disk 3
Mar 14 00:55:08 san2 kernel: [    4.415228] md/raid:md1: device sdd1 
operational as raid disk 2
Mar 14 00:55:08 san2 kernel: [    4.415230] md/raid:md1: device sde1 
operational as raid disk 1
Mar 14 00:55:08 san2 kernel: [    4.415477] md/raid:md1: allocated 6384kB
Mar 14 00:55:08 san2 kernel: [    4.415491] md/raid:md1: raid level 5 
active with 6 out of 6 devices, algorithm 2
Mar 14 00:55:08 san2 kernel: [    4.415492] RAID conf printout:
Mar 14 00:55:08 san2 kernel: [    4.415493]  --- level:5 rd:6 wd:6
Mar 14 00:55:08 san2 kernel: [    4.415494]  disk 0, o:1, dev:sdc1
Mar 14 00:55:08 san2 kernel: [    4.415495]  disk 1, o:1, dev:sde1
Mar 14 00:55:08 san2 kernel: [    4.415496]  disk 2, o:1, dev:sdd1
Mar 14 00:55:08 san2 kernel: [    4.415497]  disk 3, o:1, dev:sdf1
Mar 14 00:55:08 san2 kernel: [    4.415498]  disk 4, o:1, dev:sda1
Mar 14 00:55:08 san2 kernel: [    4.415499]  disk 5, o:1, dev:sdb1
Mar 14 00:55:08 san2 kernel: [    4.415526] md1: detected capacity 
change from 0 to 1920401145856
Mar 14 00:55:08 san2 kernel: [    4.416733]  md1: unknown partition table

Later after the resync completed I grew the array to make the extra 
space available:
Mar 14 01:37:02 san2 kernel: [ 2514.928987] md: md1: reshape done.
Mar 14 01:37:02 san2 kernel: [ 2514.982394] RAID conf printout:
Mar 14 01:37:02 san2 kernel: [ 2514.982398]  --- level:5 rd:6 wd:6
Mar 14 01:37:02 san2 kernel: [ 2514.982402]  disk 0, o:1, dev:sdc1
Mar 14 01:37:02 san2 kernel: [ 2514.982405]  disk 1, o:1, dev:sde1
Mar 14 01:37:02 san2 kernel: [ 2514.982407]  disk 2, o:1, dev:sdd1
Mar 14 01:37:02 san2 kernel: [ 2514.982410]  disk 3, o:1, dev:sdf1
Mar 14 01:37:02 san2 kernel: [ 2514.982413]  disk 4, o:1, dev:sda1
Mar 14 01:37:02 san2 kernel: [ 2514.982415]  disk 5, o:1, dev:sdb1
Mar 14 01:37:02 san2 kernel: [ 2514.982422] md1: detected capacity 
change from 1920401145856 to 2400501432320
Mar 14 01:37:02 san2 kernel: [ 2514.993988] md: resync of RAID array md1
Mar 14 01:37:02 san2 kernel: [ 2514.993992] md: minimum _guaranteed_  
speed: 300000 KB/sec/disk.
Mar 14 01:37:02 san2 kernel: [ 2514.993995] md: using maximum available 
idle IO bandwidth (but not more than 400000 KB/sec) for resync.
Mar 14 01:37:02 san2 kernel: [ 2514.994041] md: using 128k window, over 
a total of 468847936k.
Mar 14 01:55:16 san2 kernel: [ 3605.141839] md: md1: resync done.
Mar 14 01:55:16 san2 kernel: [ 3605.172547] RAID conf printout:
Mar 14 01:55:16 san2 kernel: [ 3605.172551]  --- level:5 rd:6 wd:6
Mar 14 01:55:16 san2 kernel: [ 3605.172554]  disk 0, o:1, dev:sdc1
Mar 14 01:55:16 san2 kernel: [ 3605.172556]  disk 1, o:1, dev:sde1
Mar 14 01:55:16 san2 kernel: [ 3605.172558]  disk 2, o:1, dev:sdd1
Mar 14 01:55:16 san2 kernel: [ 3605.172560]  disk 3, o:1, dev:sdf1
Mar 14 01:55:16 san2 kernel: [ 3605.172562]  disk 4, o:1, dev:sda1
Mar 14 01:55:16 san2 kernel: [ 3605.172564]  disk 5, o:1, dev:sdb1


This did lead to another observation.... The speed of the resync seemed 
limited by something other than disk IO. It was usually around 250 to 
300MB/s, the maximum achieved was around 420MB/s. I also noticed that 
idle CPU time on one of the cores was relatively low, though I never saw 
it hit 0 (minimum I saw was 12% idle, average around 20%).

So, I'm wondering whether I should consider upgrading the CPU and/or 
motherboard to try and improve peak performance?
Currently I have Intel Xeon E3-1230V2/3.3GHz/8MB 
Cache/4core/8thread/5GTs, my supplier has offered a number of options:
1) Compatible with current motherboard
      Intel Xeon E3-1280V2/3.6GHz/8MB Cache/4core/8thread/5GTs
2)  Intel Xeon E5-2620V2/2.1GHz/15MB Cache/6core/12thread/5GTs
3)  Intel Xeon E5-2630V2/2.6GHz/15MB Cache/6core/12thread/7.2GTs

My understanding is that the RAID5 is single threaded, so will work best 
with a higher speed single core CPU compared to a larger number of cores 
at a lower speed. However, I'm not sure how much "work" is being done 
across the various models. ie, does a E5 CPU do more work even though it 
has a lower clock speed? Does this carry over to the E7 class as well?

Currently I'm looking to replace at least the motherboard with 
http://www.supermicro.com/products/motherboard/Xeon/C202_C204/X9SCM-F.cfm in 
order to get 2 of the PCIe 2.0 8x slots (one for the existing LSI SATA 
controller and one for a dual port 10Gb ethernet card. This will provide 
a 10Gb cross-over connection between the two server, plus replace the 8 
x 1G ports with a single 10Gb port (solving the load balancing across 
the multiple links issue). Finally, this 28 port (4 x 10G + 24 x 1G) 
switch 
http://www.netgear.com.au/business/products/switches/stackable-smart-switches/GS728TXS.aspx# 
should allow the 2 x 10G connections to be connected through to the 8 
servers with 2 x 1G connections each using multipath scsi to setup two 
connections (one on each 1G port) with the same destination (10G port)

Any suggestions/comments would be welcome.

Regards,
Adam

-- 
Adam Goryachev Website Managers www.websitemanagers.com.au