Re: potentially lost largeish raid5 array..

[NeilBrown <neilb@suse.de>]

[-- Attachment #1: Type: text/plain, Size: 3372 bytes --]

On Thu, 22 Sep 2011 22:49:12 -0600 Thomas Fjellstrom <tfjellstrom@shaw.ca>
wrote:

> On September 22, 2011, NeilBrown wrote:
> > On Thu, 22 Sep 2011 19:50:36 -0600 Thomas Fjellstrom <tfjellstrom@shaw.ca>
> > 
> > wrote:
> > > Hi,
> > > 
> > > I've been struggling with a SAS card recently that has had poor driver
> > > support for a long time, and tonight its decided to kick every drive in
> > > the array one after the other. Now mdstat shows:
> > > 
> > > md1 : active raid5 sdf[0](F) sdh[7](F) sdi[6](F) sdj[5](F) sde[3](F)
> > > sdd[2](F) sdg[1](F)
> > > 
> > >       5860574208 blocks super 1.1 level 5, 512k chunk, algorithm 2 [7/0]
> > > 
> > > [_______]
> > > 
> > >       bitmap: 3/8 pages [12KB], 65536KB chunk
> > > 
> > > Does the fact that I'm using a bitmap save my rear here? Or am I hosed?
> > > If I'm not hosed, is there a way I can recover the array without
> > > rebooting? maybe just a --stop and a --assemble ? If that won't work,
> > > will a reboot be ok?
> > > 
> > > I'd really prefer not to have lost all of my data. Please tell me
> > > (please) that it is possible to recover the array. All but sdi are still
> > > visible in /dev (I may be able to get it back via hotplug maybe, but
> > > it'd get sdk or something).
> > 
> > mdadm --stop /dev/md1
> > 
> > mdadm --examine /dev/sd[fhijedg]
> > mdadm --assemble --verbose /dev/md1 /dev/sd[fhijedg]
> > 
> > Report all output.
> > 
> > NeilBrown
> 
> Hi, thanks for the help. Seems the SAS card/driver is in a funky state at the 
> moment. the --stop worked*. but --examine just gives "no md superblock 
> detected", and dmesg reports io errors for all drives.
> 
> I've just reloaded the driver, and things seem to have come back:

That's good!!


> 
> root@boris:~# mdadm --examine /dev/sd[fhijedg]
....

sd1 has a slightly older event count than the others - Update time is 1:13
older.  So it presumably died first.

> 
> root@boris:~# mdadm --assemble --verbose /dev/md1 /dev/sd[fhijedg]
> mdadm: looking for devices for /dev/md1
> mdadm: /dev/sdd is identified as a member of /dev/md1, slot 2.
> mdadm: /dev/sde is identified as a member of /dev/md1, slot 3.
> mdadm: /dev/sdf is identified as a member of /dev/md1, slot 0.
> mdadm: /dev/sdg is identified as a member of /dev/md1, slot 1.
> mdadm: /dev/sdh is identified as a member of /dev/md1, slot 6.
> mdadm: /dev/sdi is identified as a member of /dev/md1, slot 5.
> mdadm: /dev/sdj is identified as a member of /dev/md1, slot 4.
> mdadm: added /dev/sdg to /dev/md1 as 1
> mdadm: added /dev/sdd to /dev/md1 as 2
> mdadm: added /dev/sde to /dev/md1 as 3
> mdadm: added /dev/sdj to /dev/md1 as 4
> mdadm: added /dev/sdi to /dev/md1 as 5
> mdadm: added /dev/sdh to /dev/md1 as 6
> mdadm: added /dev/sdf to /dev/md1 as 0
> mdadm: /dev/md1 has been started with 6 drives (out of 7).
> 
> 
> Now I guess the question is, how to get that last drive back in? would: 
> 
> mdadm --re-add /dev/md1 /dev/sdi 
> 
> work?
> 

re-add should work, yes.  It will use the bitmap info to only update the
blocks that need updating - presumably not many.
It might be interesting to run
  mdadm -X /dev/sdf

first to see what the bitmap looks like - how many dirty bits and what the
event counts are.

But yes: --re-add should make it all happy.

NeilBrown


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 190 bytes --]