Re: RAID5 assemble fails after reboot while reshaping

[NeilBrown <neilb@suse.de>]

[-- Attachment #1: Type: text/plain, Size: 5998 bytes --]

On Sun, 17 May 2015 18:44:01 +0200 Marco Fuckner <marco@die-fuckners.de>
wrote:

> Hi everybody,
> 
> first of all, I'm using mdadm 3.3.2 on linux 4.0.1, all of my disks are
> partitioned with the same geometry.
> 
> I wanted to grow my 4 disk RAID5 array to 7 disks. After adding the
> disks and initiating the grow, the reshape didn't seem to start:
> 
>     md0 : active raid5 sdf1[7] sde1[6] sdd1[5] sdg1[3] sdb1[4] sdh1[1]
> sdc1[0]
>         11720044800 blocks super 1.2 level 5, 256k chunk, algorithm 2
> [7/7] [UUUUUUU]
>         [>....................]  reshape =  0.0% (0/3906681600)
> finish=166847860.0min speed=0K/sec
>         bitmap: 0/30 pages [0KB], 65536KB chunk
> 
> I waited about three hours and checked again:
> 
>     md0 : active raid5 sdf1[7] sde1[6] sdd1[5] sdg1[3] sdb1[4] sdh1[1]
> sdc1[0]
>         11720044800 blocks super 1.2 level 5, 256k chunk, algorithm 2
> [7/7] [UUUUUUU]
>         [>....................]  reshape =  0.0% (0/3906681600)
> finish=9599856140.0min speed=0K/sec
>         bitmap: 0/30 pages [0KB], 65536KB chunk


This looks very much like  the reshape has not done anything at all.
i.e. your data is still exactly where you left it, it is just a case of
getting hold of it.

It's not impossible that running the --assemble with --update=revert-reshape
would work, but I'm far from certain.  If you backed up the first gigabyte of
each device (sd?1) first then it would probably be safe enough to try.

Another option is to add --invalid-backup to the --assemble command.
This has a reasonable chance of allowing the reshape to continue, but also
has a reasonable chance of corrupting the first few megabytes of your array
(the part that it things should be backed up).
If you "make test_stripe" in the mdadm source code, you can use that to extra
the first few megabytes of array data so you could restore it if it gets
corrupted.
Something like
   test_stripe save /root/thing 4 262144 5 2 0 \
   $BIGNUM /dev/sdc1:262144 /dev/sdh1:262144 ......

Check the source to make sure you get the args right.
Make sure the order of the devices and their data_offsets are correct.  check
the "Device Role:" for each and order them by that number.

Another option is to recreate the array as 4-drive RAID5.  Again you need to
make sure the device order and data offsets are correct, along with all the
other data.

I might be able to dig into the code and find out what happened and maybe
offer an "easier" solution, but that won't be for a day or two at least.


NeilBrown



> 
> Unfortunately, I forgot to save the output of the grow command, but it
> exited with 0.
> /mdadm --misc --detail /dev/md0/ didn't show anything suspicious to me:
> 
> /dev/md0:
>     Version : 1.2
>     Creation Time : Sun Nov  9 02:38:25 2014
>     Raid Level : raid5
>     Array Size : 11720044800 (11177.11 GiB 12001.33 GB)
>     Used Dev Size : 3906681600 (3725.70 GiB 4000.44 GB)
>     Raid Devices : 7
>     Total Devices : 7
>     Persistence : Superblock is persistent
> 
>     Intent Bitmap : Internal
> 
>     Update Time : Mon May 11 11:55:07 2015
>     State : clean, reshaping
>     Active Devices : 7
>     Working Devices : 7
>     Failed Devices : 0
>     Spare Devices : 0
> 
>     Layout : left-symmetric
>     Chunk Size : 256K
> 
>     Reshape Status : 0% complete
>     Delta Devices : 3, (4->7)
> 
>     Name : anaNAS:0  (local to host anaNAS)
>     UUID : 33f0604f:46e80f5e:11b1a694:608fd9b3
>     Events : 51839
> 
>     Number   Major   Minor   RaidDevice   State
>         0           8           33        0              active sync  
> /dev/sdc1
>         1           8           113      1              active sync  
> /dev/sdh1
>         3           8           97        2              active sync  
> /dev/sdg1
>         4           8           17        3              active sync  
> /dev/sdb1
>         7           8           81        4              active sync  
> /dev/sdf1
>         6           8           65        5              active sync  
> /dev/sde1
>         5           8           49        6              active sync  
> /dev/sdd1
> 
> As it looked like it wouldn't be ready until long after my death and I
> also wrote a backup file, somehow restarting and continuing afterwards
> seemed reasonable to me.
> The source I was reading suggested running /mdadm /dev/md0 --continue
> --backup-file=$FILE/. Apparently this command was wrong, and I couldn't
> reassamble the array:
> 
> # mdadm --assemble /dev/md0 --verbose /dev/sd[b-h]1
> --backup-file=/root/grow7backup.bak
> 
>     mdadm: looking for devices for /dev/md0
>     mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 4.
>     mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 5.
>     mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 6.
>     mdadm: /dev/sdg1 is identified as a member of /dev/md0, slot 2.
>     mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 3.
>     mdadm: /dev/sdh1 is identified as a member of /dev/md0, slot 1.
>     mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 0.
>     mdadm: :/dev/md0 has an active reshape - checking if critical
> section needs to be restored
>     mdadm: No backup metadata on /root/grow7backup.bak
>     mdadm: No backup metadata on device-4
>     mdadm: No backup metadata on device-5
>     mdadm: No backup metadata on device-6
>     mdadm: Failed to find backup of critical section
>     mdadm: Failed to restore critical section for reshape, sorry.
> 
> I started searching for answers but didn't find anything helpful except
> the hint on the raid.wiki.kernel.org page to send an email here. The
> last sentence from mdadm sounds a bit pessimistic, but I hope someone in
> here can help me. The output of /mdadm --examine /dev/sd[bh]1 /is in the
> attachment.
> 
> Thanks in advance,
> 
> Marco


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]