Re: Stuck array after reshape

[NeilBrown <neilb@suse.de>]

[-- Attachment #1: Type: text/plain, Size: 7185 bytes --]

On Tue, 20 May 2014 18:56:17 +0000 Davíð Steinn Geirsson <david@dsg.is> wrote:

> Hi all,
> 
> I tried to reshape an MD RAID array, going from a 4-disk RAID5 to a
> 6-disk RAID6. This seems to have failed and now I'm afraid to turn the
> machine off.
> 
> What I did:
> mdadm --add /dev/md5 /dev/sdh1
> mdadm --add /dev/md5 /dev/sdg1
> mdadm --grow /dev/md5 --backup-file /root/vg_3T_reshape_201405_mdbackup
> --level=6 --raid-devices=6
> 
> The last command returned with no error, the way it usually does.
> However, now everything that tries to access the array hangs:
> mdadm -D /dev/md5 # hangs
> cat /proc/mdstat # hangs
> Trying to read mounted filesystems also hangs.
> 
> The two new drives are on a brand new IBM M1015 (crossflashed to LSI
> 9211). I have not used this controller previously, but before I tried
> the reshape I did write a GPT partition table and successfully read it
> back from the two drives.
> 
> From dmesg around this time:
> [ 1340.951731] md: bind<sdh1>
> [ 1346.150654] scsi_verify_blk_ioctl: 38 callbacks suppressed
> [ 1346.150662] mdadm: sending ioctl 1261 to a partition!
> [ 1346.150669] mdadm: sending ioctl 1261 to a partition!
> [ 1346.155219] mdadm: sending ioctl 1261 to a partition!
> [ 1346.155228] mdadm: sending ioctl 1261 to a partition!
> [ 1346.160528] mdadm: sending ioctl 1261 to a partition!
> [ 1346.160535] mdadm: sending ioctl 1261 to a partition!
> [ 1346.160688] mdadm: sending ioctl 1261 to a partition!
> [ 1346.160694] mdadm: sending ioctl 1261 to a partition!
> [ 1346.160913] mdadm: sending ioctl 1261 to a partition!
> [ 1346.160918] mdadm: sending ioctl 1261 to a partition!
> [ 1346.185864] md: bind<sdg1>
> [ 1370.267086] scsi_verify_blk_ioctl: 38 callbacks suppressed
> [ 1370.267095] mdadm: sending ioctl 1261 to a partition!
> [ 1370.267103] mdadm: sending ioctl 1261 to a partition!
> [ 1461.662068] mdadm: sending ioctl 1261 to a partition!
> [ 1461.662078] mdadm: sending ioctl 1261 to a partition!
> [ 1521.675927] md/raid:md5: device sde1 operational as raid disk 0
> [ 1521.675937] md/raid:md5: device sdd1 operational as raid disk 3
> [ 1521.675943] md/raid:md5: device sda1 operational as raid disk 2
> [ 1521.675949] md/raid:md5: device sdb1 operational as raid disk 1
> [ 1521.677471] md/raid:md5: allocated 5332kB
> [ 1521.692766] md/raid:md5: raid level 6 active with 4 out of 5
> devices, algorithm 18
> [ 1521.692849] RAID conf printout:
> [ 1521.692853]  --- level:6 rd:5 wd:4
> [ 1521.692859]  disk 0, o:1, dev:sde1
> [ 1521.692864]  disk 1, o:1, dev:sdb1
> [ 1521.692869]  disk 2, o:1, dev:sda1
> [ 1521.692873]  disk 3, o:1, dev:sdd1
> [ 1522.801181] RAID conf printout:
> [ 1522.801190]  --- level:6 rd:6 wd:5
> [ 1522.801196]  disk 0, o:1, dev:sde1
> [ 1522.801201]  disk 1, o:1, dev:sdb1
> [ 1522.801205]  disk 2, o:1, dev:sda1
> [ 1522.801210]  disk 3, o:1, dev:sdd1
> [ 1522.801215]  disk 4, o:1, dev:sdg1
> [ 1522.801230] RAID conf printout:
> [ 1522.801234]  --- level:6 rd:6 wd:5
> [ 1522.801239]  disk 0, o:1, dev:sde1
> [ 1522.801243]  disk 1, o:1, dev:sdb1
> [ 1522.801248]  disk 2, o:1, dev:sda1
> [ 1522.801252]  disk 3, o:1, dev:sdd1
> [ 1522.801256]  disk 4, o:1, dev:sdg1
> [ 1522.801261]  disk 5, o:1, dev:sdh1
> [ 1522.801374] md: reshape of RAID array md5
> [ 1522.801379] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
> [ 1522.801384] md: using maximum available idle IO bandwidth (but not
> more than 200000 KB/sec) for reshape.
> [ 1522.801396] md: using 128k window, over a total of 2928578048k.
> [ 1522.802248] mdadm: sending ioctl 1261 to a partition!
> [ 1522.802256] mdadm: sending ioctl 1261 to a partition!
> [ 1522.883851] mdadm: sending ioctl 1261 to a partition!
> [ 1522.883860] mdadm: sending ioctl 1261 to a partition!
> [ 1525.134837] md: md_do_sync() got signal ... exiting

This line is strange.  Something killed the kernel thread which is performing
the reshape.  I cannot imagine what would have done that, but it does seem to
explain why things aren't working.

When you are increasing the size of the array - as you are in this case - you
don't really need a backup file, and even if you give one it will ony be
used for the first few stripes.  So that fact that it no longer exists is
expected.

Based on the --examine output that you subsequently post, the reshape has
started, but didn't get very far.  If you stop and restart the array it
should continue.

Don't remove the new devices as you suggest below.  Just a clean reboot or 
  mdadm -S /dev/md5
  mdadm -A /dev/md5 /dev/sd[abdegh]1

should do the trick.

You data should be completely safe.

NeilBrown



> [ 1681.128046] INFO: task jbd2/dm-3-8:1494 blocked for more than 120
> seconds.
> [ 1681.128129] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 1681.128206] jbd2/dm-3-8     D ffff88007fc13540     0  1494      2
> 0x00000000
> [ 1681.128217]  ffff88007beab0a0 0000000000000046 ffff88005d0b1470
> ffff88007a3208b0
> [ 1681.128227]  0000000000013540 ffff88007c0dffd8 ffff88007c0dffd8
> ffff88007beab0a0
> [ 1681.128236]  059f7b5300000000 ffffffff81065a2f ffff88007bbe4d70
> ffff88007fc13d90
> [ 1681.128245] Call Trace:
> [ 1681.128263]  [<ffffffff81065a2f>] ? timekeeping_get_ns+0xd/0x2a
> [ 1681.128273]  [<ffffffff8111bda1>] ? wait_on_buffer+0x28/0x28
> [ 1681.128283]  [<ffffffff813483e4>] ? io_schedule+0x59/0x71
> [ 1681.128289]  [<ffffffff8111bda7>] ? sleep_on_buffer+0x6/0xa
> [ 1681.128296]  [<ffffffff81348827>] ? __wait_on_bit+0x3e/0x71
> [ 1681.128303]  [<ffffffff813488c9>] ? out_of_line_wait_on_bit+0x6f/0x78
> [ 1681.128310]  [<ffffffff8111bda1>] ? wait_on_buffer+0x28/0x28
> [ 1681.128319]  [<ffffffff8105f575>] ?
> autoremove_wake_function+0x2a/0x2a
> [ 1681.128354]  [<ffffffffa018d9c0>] ?
> jbd2_journal_commit_transaction+0xb9b/0x1057 [jbd2]
> [ 1681.128366]  [<ffffffff8100d02f>] ? load_TLS+0x7/0xa
> [ 1681.128373]  [<ffffffff8100d6a3>] ? __switch_to+0x133/0x258
> [ 1681.128389]  [<ffffffffa01910ae>] ? kjournald2+0xc0/0x20a [jbd2]
> [ 1681.128397]  [<ffffffff8105f54b>] ? add_wait_queue+0x3c/0x3c
> [ 1681.128412]  [<ffffffffa0190fee>] ? commit_timeout+0x5/0x5 [jbd2]
> [ 1681.128420]  [<ffffffff8105ef05>] ? kthread+0x76/0x7e
> [ 1681.128430]  [<ffffffff813505b4>] ? kernel_thread_helper+0x4/0x10
> [ 1681.128438]  [<ffffffff8105ee8f>] ? kthread_worker_fn+0x139/0x139
> [ 1681.128446]  [<ffffffff813505b0>] ? gs_change+0x13/0x13
> [... more hung task warnings from other processes follow ...]
> 
> 
> This machine is running debian wheezy. mdadm version is 3.2.5-1 from
> debian wheezy. Kernel is 3.2.18-1 from wheezy (3.2.0-2-amd64).
> 
> Any help would be much appreciated! Especially if the data is
> recoverable. It's possible that the reshape process never actually got
> started and rebooting the machine without the new disks will make
> everything "just work"... but I don't want to try that just yet, in case
> it prevents future data recovery work.
> 
> Any thoughts? Or more debug info I could provide to diagnose this?
> 
> Best regards,
> Davíð


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]