Stuck array after reshape

["Davíð Steinn Geirsson" <david@dsg.is>]

[-- Attachment #1: Type: text/plain, Size: 6016 bytes --]

Hi all,

I tried to reshape an MD RAID array, going from a 4-disk RAID5 to a
6-disk RAID6. This seems to have failed and now I'm afraid to turn the
machine off.

What I did:
mdadm --add /dev/md5 /dev/sdh1
mdadm --add /dev/md5 /dev/sdg1
mdadm --grow /dev/md5 --backup-file /root/vg_3T_reshape_201405_mdbackup
--level=6 --raid-devices=6

The last command returned with no error, the way it usually does.
However, now everything that tries to access the array hangs:
mdadm -D /dev/md5 # hangs
cat /proc/mdstat # hangs
Trying to read mounted filesystems also hangs.

The two new drives are on a brand new IBM M1015 (crossflashed to LSI
9211). I have not used this controller previously, but before I tried
the reshape I did write a GPT partition table and successfully read it
back from the two drives.

From dmesg around this time:
[ 1340.951731] md: bind<sdh1>
[ 1346.150654] scsi_verify_blk_ioctl: 38 callbacks suppressed
[ 1346.150662] mdadm: sending ioctl 1261 to a partition!
[ 1346.150669] mdadm: sending ioctl 1261 to a partition!
[ 1346.155219] mdadm: sending ioctl 1261 to a partition!
[ 1346.155228] mdadm: sending ioctl 1261 to a partition!
[ 1346.160528] mdadm: sending ioctl 1261 to a partition!
[ 1346.160535] mdadm: sending ioctl 1261 to a partition!
[ 1346.160688] mdadm: sending ioctl 1261 to a partition!
[ 1346.160694] mdadm: sending ioctl 1261 to a partition!
[ 1346.160913] mdadm: sending ioctl 1261 to a partition!
[ 1346.160918] mdadm: sending ioctl 1261 to a partition!
[ 1346.185864] md: bind<sdg1>
[ 1370.267086] scsi_verify_blk_ioctl: 38 callbacks suppressed
[ 1370.267095] mdadm: sending ioctl 1261 to a partition!
[ 1370.267103] mdadm: sending ioctl 1261 to a partition!
[ 1461.662068] mdadm: sending ioctl 1261 to a partition!
[ 1461.662078] mdadm: sending ioctl 1261 to a partition!
[ 1521.675927] md/raid:md5: device sde1 operational as raid disk 0
[ 1521.675937] md/raid:md5: device sdd1 operational as raid disk 3
[ 1521.675943] md/raid:md5: device sda1 operational as raid disk 2
[ 1521.675949] md/raid:md5: device sdb1 operational as raid disk 1
[ 1521.677471] md/raid:md5: allocated 5332kB
[ 1521.692766] md/raid:md5: raid level 6 active with 4 out of 5
devices, algorithm 18
[ 1521.692849] RAID conf printout:
[ 1521.692853]  --- level:6 rd:5 wd:4
[ 1521.692859]  disk 0, o:1, dev:sde1
[ 1521.692864]  disk 1, o:1, dev:sdb1
[ 1521.692869]  disk 2, o:1, dev:sda1
[ 1521.692873]  disk 3, o:1, dev:sdd1
[ 1522.801181] RAID conf printout:
[ 1522.801190]  --- level:6 rd:6 wd:5
[ 1522.801196]  disk 0, o:1, dev:sde1
[ 1522.801201]  disk 1, o:1, dev:sdb1
[ 1522.801205]  disk 2, o:1, dev:sda1
[ 1522.801210]  disk 3, o:1, dev:sdd1
[ 1522.801215]  disk 4, o:1, dev:sdg1
[ 1522.801230] RAID conf printout:
[ 1522.801234]  --- level:6 rd:6 wd:5
[ 1522.801239]  disk 0, o:1, dev:sde1
[ 1522.801243]  disk 1, o:1, dev:sdb1
[ 1522.801248]  disk 2, o:1, dev:sda1
[ 1522.801252]  disk 3, o:1, dev:sdd1
[ 1522.801256]  disk 4, o:1, dev:sdg1
[ 1522.801261]  disk 5, o:1, dev:sdh1
[ 1522.801374] md: reshape of RAID array md5
[ 1522.801379] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[ 1522.801384] md: using maximum available idle IO bandwidth (but not
more than 200000 KB/sec) for reshape.
[ 1522.801396] md: using 128k window, over a total of 2928578048k.
[ 1522.802248] mdadm: sending ioctl 1261 to a partition!
[ 1522.802256] mdadm: sending ioctl 1261 to a partition!
[ 1522.883851] mdadm: sending ioctl 1261 to a partition!
[ 1522.883860] mdadm: sending ioctl 1261 to a partition!
[ 1525.134837] md: md_do_sync() got signal ... exiting
[ 1681.128046] INFO: task jbd2/dm-3-8:1494 blocked for more than 120
seconds.
[ 1681.128129] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1681.128206] jbd2/dm-3-8     D ffff88007fc13540     0  1494      2
0x00000000
[ 1681.128217]  ffff88007beab0a0 0000000000000046 ffff88005d0b1470
ffff88007a3208b0
[ 1681.128227]  0000000000013540 ffff88007c0dffd8 ffff88007c0dffd8
ffff88007beab0a0
[ 1681.128236]  059f7b5300000000 ffffffff81065a2f ffff88007bbe4d70
ffff88007fc13d90
[ 1681.128245] Call Trace:
[ 1681.128263]  [<ffffffff81065a2f>] ? timekeeping_get_ns+0xd/0x2a
[ 1681.128273]  [<ffffffff8111bda1>] ? wait_on_buffer+0x28/0x28
[ 1681.128283]  [<ffffffff813483e4>] ? io_schedule+0x59/0x71
[ 1681.128289]  [<ffffffff8111bda7>] ? sleep_on_buffer+0x6/0xa
[ 1681.128296]  [<ffffffff81348827>] ? __wait_on_bit+0x3e/0x71
[ 1681.128303]  [<ffffffff813488c9>] ? out_of_line_wait_on_bit+0x6f/0x78
[ 1681.128310]  [<ffffffff8111bda1>] ? wait_on_buffer+0x28/0x28
[ 1681.128319]  [<ffffffff8105f575>] ?
autoremove_wake_function+0x2a/0x2a
[ 1681.128354]  [<ffffffffa018d9c0>] ?
jbd2_journal_commit_transaction+0xb9b/0x1057 [jbd2]
[ 1681.128366]  [<ffffffff8100d02f>] ? load_TLS+0x7/0xa
[ 1681.128373]  [<ffffffff8100d6a3>] ? __switch_to+0x133/0x258
[ 1681.128389]  [<ffffffffa01910ae>] ? kjournald2+0xc0/0x20a [jbd2]
[ 1681.128397]  [<ffffffff8105f54b>] ? add_wait_queue+0x3c/0x3c
[ 1681.128412]  [<ffffffffa0190fee>] ? commit_timeout+0x5/0x5 [jbd2]
[ 1681.128420]  [<ffffffff8105ef05>] ? kthread+0x76/0x7e
[ 1681.128430]  [<ffffffff813505b4>] ? kernel_thread_helper+0x4/0x10
[ 1681.128438]  [<ffffffff8105ee8f>] ? kthread_worker_fn+0x139/0x139
[ 1681.128446]  [<ffffffff813505b0>] ? gs_change+0x13/0x13
[... more hung task warnings from other processes follow ...]


This machine is running debian wheezy. mdadm version is 3.2.5-1 from
debian wheezy. Kernel is 3.2.18-1 from wheezy (3.2.0-2-amd64).

Any help would be much appreciated! Especially if the data is
recoverable. It's possible that the reshape process never actually got
started and rebooting the machine without the new disks will make
everything "just work"... but I don't want to try that just yet, in case
it prevents future data recovery work.

Any thoughts? Or more debug info I could provide to diagnose this?

Best regards,
Davíð

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]