Re: reboot before reshape from raid 5 to raid 6 (was in state resync=DELAYED). Doesn't assemble anymore.

["Simon SÉHIER" <simon@sehier.fr>]

On 12 oct. 2010 22:46:12, Neil Brown wrote :
> On Tue, 12 Oct 2010 16:27:53 +0200
> 
> Simon S <simon@sehier.fr> wrote:
> > Hi all,
> > 
> > I had a config with 5 disks and 3 raid 5 arrays:
> > 
> > md2 : system root
> > md3 : swap
> > md4 : data
> > 
> > I added a 6th disk with the intention of growing my raid5 into raid6.
> > 
> > The step I used were :
> > 
> > # mdadm /dev/mdX -a  /dev/newdiskX
> > # mdadm -G --level 6 -n 6 /dev/mdX --backup-file /mdXbackup
> > 
> >  (yes, with backup file on root partition md2...)
> 
> Bad idea..  Very bad idea.
> 
> > The md3 array reshaped without any problem.
> > md2 seemed to reshape well until it reaches 50.4%, then the rebuild speed
> > stalled at 14Kb/s.
> 
> This is the expected consequence of that bad idea.  Unfortunately it would
> be hard to reliably get mdadm to complain about that, though I guess the
> common cases are easy to protect against ... added to 'todo' list
> 
> > md4 was still in the state "resync=DELAYED" then.
> > 
> > As the rebuild process seemed hung, I restart the machine ... bad idea.
> 
> Not really, nothing else would have worked.
> 
> > Now mdadm refuses to assemble md2 and md4, and displays this message :
> >   mdadm: Failed to restore critical section for reshape, sorry.
> >   
> >     Possibly you needed to specify the --backup-file
> > 
> > md2 is my linux installation, not very bad if I lose this one.
> > 
> > md4 however contains valuable data.
> > 
> > While md4 was still in the state resync=DELAYED before the shutdown, I
> > expect it should not has been (to much) modified and can be recovered.
> 
> Very true.
> 
> > Any idea on how I could safely do it ?
> > 
> > Should I give a try to the hack "Get 'Grow_restart' to always return 0."
> > mentionned by Neil Brown on 22 april 2010 in this mailing list ?
> 
> That is your best bet.  I plan to make that easier to do in mdadm-3.2 (no
> recompile necessary).
> 
> Before you do, check "mdadm -E /dev/newdiskX" and make sure the "Reshape
> position" is 0.  If it is you should be fine.  I
> 
> It won't be for md2 of course.  So md will quite possible have some
> corruption.  Run fsck on it an it will probably be mostly OK, but there is
> a reasonable chance that some files will be corrupted.  Whether and when
> you will notice is impossible to guess.

Thanks for your answer Neil, 

I recompiled mdadm 3.1.4 with return 0 in the beginning of the function 
Grow_restart (mistake was made with 3.1.2). I have one more question :

I first tried assembling the least valued array, md2. It starts reshaping from 
where it stops, in the first seconds around 1300 K/s, and rapidly above 10K/s.

While my backup file for md4 (the array I care about) was also on md2. Do I 
have to expect a problem assembling md4 with the modified version of mdadm, or 
can I go without worying md2 (rootfs)  isn't assembled ?


> 
> NeilBrown
> 
> > Thank for any advices,
> > 
> > Cheers,
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html