Re: raid10 issues after reorder of boot drives.

[NeilBrown <neilb@suse.de>]

[-- Attachment #1: Type: text/plain, Size: 7807 bytes --]

On Sat, 28 Apr 2012 11:23:18 -0400 likewhoa <likewhoa@weboperative.com> wrote:

> On 04/27/2012 11:51 PM, likewhoa wrote:
> > On 04/27/2012 11:23 PM, NeilBrown wrote:
> >> On Fri, 27 Apr 2012 22:59:48 -0400 likewhoa <likewhoa@weboperative.com> wrote:
> >>
> >>> On 04/27/2012 10:55 PM, NeilBrown wrote:
> >>>> On Fri, 27 Apr 2012 22:37:29 -0400 likewhoa <likewhoa@weboperative.com> wrote:
> >>>>
> >>>>> On 04/27/2012 08:35 PM, likewhoa wrote:
> >>>>>> I am not sure how to proceed now with the output that shows possible
> >>>>>> pairs as it won't allow me to setup all 8 devices on the array but only
> >>>>>> 4. Should I run the array creation with -x4 and set the available spare
> >>>>>> devicesor or just create the array as I can remember which was one pair
> >>>>>> from each controller. i.e /dev/sda3 /dev/sde3 ...?
> >>>>>> --
> >>>>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> >>>>>> the body of a message to majordomo@vger.kernel.org
> >>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>>>> ok I was able to recreate the array with correct order which I took from
> >>>>> my /dev/md0's --details output and was able to decrypt the luks mapping
> >>>>> but XFS didn't open and xfs_repair is currently doing the matrix. I will
> >>>>> keep this posted with updates.
> >>>> I hope the order really is correct.... I wouldn't expect xfs to find problems
> >>>> if it was...
> >>>>
> >>>>> Thanks again Neil.
> >>>>> WRT 3.3.3 should I just go back to 3.3.2 which seemed to run fine and
> >>>>> wait until there is a release of 3.3.3 that has fix?
> >>>> 3.3.4 has the fix and was just released.
> >>>> 3.3.1, 3.3.2 and 3.3.3 all have the bug.  It only triggers on shutdown and
> >>>> even then only occasionally.
> >>>> So I recommend 3.3.4.
> >>>>
> >>>> NeilBrown
> >>> The reason I believe it was correct was that 'cryptsetup luksOpen
> >>> /dev/md1 md1' worked. I really do hope that it was correct too because
> >>> after opening the luks mapping I assume there is no going back.
> >> Opening the luks mapping could just mean that the first few blocks are
> >> correct.  So the first disk is right but others might not be.
> >>
> >> There is going backup unless something has been written to the array.  Once
> >> that happens anything could be corrupted.  So if the xfs check you are  doing
> >> is read-only you could still have room to move.
> >>
> >> With a far=2 array, each first half of each device is mirrored on the second
> >> half.  So you can probably recover the ordering by finding which pairs match.
> >>
> >> The "Used Dev Size" is 902992896 sectors.  Half of that is 451496448
> >> or 231166181376 bytes.
> >>
> >> So to check if two devices are adjacent in the mapping you can try:
> >>
> >>  cmp --ignore-initial=0:231166181376 --bytes=231166181376 first-dev second-dev
> >>
> >> You could possibly use a smaller --bytes= number, at least on the first
> >> attempt.
> >> You a similar 'for' loop to before an use this command and it might tell you
> >> which pairs of devices are consecutive.  From that you should be able to get
> >> the full order.
> >>
> >> NeilBrown
> >>
> > I don't see why xfs_repair would write data unless it actually finds the
> > superblock but I am not sure so I will take my chances since it's still
> > searching for the secondary superblock now.
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> After running the for loop all night which produced this output.
> 
> /dev/sda3 /dev/sdb3 differ: byte 1, line 1
> /dev/sda3 /dev/sdc3 differ: byte 262145, line 2
> /dev/sda3 /dev/sdd3 differ: byte 1, line 1
> /dev/sda3 /dev/sde3 differ: byte 1, line 1
> /dev/sda3 and /dev/sdf3 seem to match
> /dev/sda3 /dev/sdg3 differ: byte 262145, line 2
> /dev/sda3 /dev/sdh3 differ: byte 1, line 1
> /dev/sdb3 /dev/sda3 differ: byte 1, line 1
> /dev/sdb3 /dev/sdc3 differ: byte 1, line 1
> /dev/sdb3 /dev/sdd3 differ: byte 1, line 1
> /dev/sdb3 and /dev/sde3 seem to match
> /dev/sdb3 /dev/sdf3 differ: byte 1, line 1
> /dev/sdb3 /dev/sdg3 differ: byte 1, line 1
> /dev/sdb3 /dev/sdh3 differ: byte 1, line 1
> /dev/sdc3 /dev/sda3 differ: byte 262145, line 2
> /dev/sdc3 /dev/sdb3 differ: byte 1, line 1
> /dev/sdc3 /dev/sdd3 differ: byte 1, line 1
> /dev/sdc3 /dev/sde3 differ: byte 1, line 1
> /dev/sdc3 /dev/sdf3 differ: byte 262145, line 2
> /dev/sdc3 and /dev/sdg3 seem to match
> /dev/sdc3 /dev/sdh3 differ: byte 1, line 1
> /dev/sdd3 /dev/sda3 differ: byte 1, line 1
> /dev/sdd3 /dev/sdb3 differ: byte 1, line 1
> /dev/sdd3 /dev/sdc3 differ: byte 1, line 1
> /dev/sdd3 /dev/sde3 differ: byte 1, line 1
> /dev/sdd3 /dev/sdf3 differ: byte 1, line 1
> /dev/sdd3 /dev/sdg3 differ: byte 1, line 1
> /dev/sdd3 and /dev/sdh3 seem to match
> /dev/sde3 /dev/sda3 differ: byte 262145, line 2
> /dev/sde3 /dev/sdb3 differ: byte 1, line 1
> /dev/sde3 and /dev/sdc3 seem to match
> /dev/sde3 /dev/sdd3 differ: byte 1, line 1
> /dev/sde3 /dev/sdf3 differ: byte 262145, line 2
> /dev/sde3 /dev/sdg3 differ: byte 262145, line 2
> /dev/sde3 /dev/sdh3 differ: byte 1, line 1
> /dev/sdf3 /dev/sda3 differ: byte 1, line 1
> /dev/sdf3 /dev/sdb3 differ: byte 1, line 1
> /dev/sdf3 /dev/sdc3 differ: byte 1, line 1
> /dev/sdf3 and /dev/sdd3 seem to match
> /dev/sdf3 /dev/sde3 differ: byte 1, line 1
> /dev/sdf3 /dev/sdg3 differ: byte 1, line 1
> /dev/sdf3 /dev/sdh3 differ: byte 1, line 1
> /dev/sdg3 and /dev/sda3 seem to match
> /dev/sdg3 /dev/sdb3 differ: byte 1, line 1
> /dev/sdg3 /dev/sdc3 differ: byte 262145, line 2
> /dev/sdg3 /dev/sdd3 differ: byte 1, line 1
> /dev/sdg3 /dev/sde3 differ: byte 1, line 1
> /dev/sdg3 /dev/sdf3 differ: byte 262145, line 2
> /dev/sdg3 /dev/sdh3 differ: byte 1, line 1
> /dev/sdh3 /dev/sda3 differ: byte 1, line 1
> /dev/sdh3 and /dev/sdb3 seem to match
> /dev/sdh3 /dev/sdc3 differ: byte 1, line 1
> /dev/sdh3 /dev/sdd3 differ: byte 1, line 1
> /dev/sdh3 /dev/sde3 differ: byte 1, line 1
> /dev/sdh3 /dev/sdf3 differ: byte 1, line 1
> /dev/sdh3 /dev/sdg3 differ: byte 1, line 1
> 
> I manage recover my luks+xfs with this --create command \o/
> > mdadm --create /dev/md1 --metadata=1.0 -l10 -n8 --chunk=256
> --layout=f2 --assume-clean /dev/sdh3 /dev/sdb3 /dev/sde3 /dev/sdc3
> /dev/sdg3 /dev/sda3 /dev/sdf3 /dev/sdd3

cool!!  I love it when something like that produces a nice clean usable
result.

> 
> Thank you Neil for your assistance you rock! With regards to my
> /etc/mdadm.conf the explicit references to the sd drives was generated
> with 'mdadm -Esv', my question is do you suggest I not even populate
> /etc/mdadm.conf with such output because of change in drives and just
> use 'mdadm -s -A /dev/mdX'? Also after running into this nasty bug I get
> the feeling that I should really keep a copy of all my future 'mdadm
> --create ...' commands handy just for such situations, do you agree?

The output of '-Ds' and '-Es' was only ever meant to be a starting point for
mdadm.conf, not the final content..

I think it is good to populate mdadm.conf, though in simple cases it isn't
essential.
I would just list the UUID for each array:
   ARRAY /dev/md0 uuid=.......
and leave it at that.

Keeping a copy of the output of "mdadm -E" of each device could occasionally
be useful.  You would need to update this copy any time any config change
happened to the array such as a device failure or a spare being added.

Glad it all worked out!

NeilBrown


> 
> Thanks again and have a GREAT weekend.
> Fernando Vasquez a.k.a likewhoa


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]