How does kernel decide that a drive is "spare"?

How does kernel decide that a drive is "spare"?

[Dave W]

I have a fileserver with a RAID6 on five 2TB drives.  It appeared to be working
fine before I rebooted it, but now it is complaining

mdadm: /dev/md0 assembled from 2 drives and 3 spares - not enough to start the \
array.

That's strange... I never configured any spare drives.  There should be five
active drives in this array.  I checked /proc/mdstat:

[root@fileserver ~]# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md0 : inactive sdd1[2](S) sdb1[7](S) sdc1[6](S) sdf1[5](S) sde1[3](S)
      9767559680 blocks

[root@fileserver ~]# mdadm -A /dev/md0 -fv /dev/sd[bcdef]1
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 7.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 6.
mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 5.
mdadm: no uptodate device for slot 0 of /dev/md0
mdadm: no uptodate device for slot 1 of /dev/md0
mdadm: added /dev/sde1 to /dev/md0 as 3
mdadm: no uptodate device for slot 4 of /dev/md0
mdadm: added /dev/sdf1 to /dev/md0 as 5
mdadm: added /dev/sdc1 to /dev/md0 as 6
mdadm: added /dev/sdb1 to /dev/md0 as 7
mdadm: added /dev/sdd1 to /dev/md0 as 2
mdadm: /dev/md0 assembled from 2 drives and 3 spares - not enough to start the \
array.

Why are three drives assumed to be spares?  Is there a way for me to recover
this array?

Thanks,
Dave

RE: How does kernel decide that a drive is "spare"?

[Leslie Rhorer]

> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> owner@vger.kernel.org] On Behalf Of Dave W
> Sent: Friday, June 11, 2010 12:19 AM
> To: linux-raid@vger.kernel.org
> Subject: How does kernel decide that a drive is "spare"?
> 
> I have a fileserver with a RAID6 on five 2TB drives.  It appeared to be
> working
> fine before I rebooted it, but now it is complaining
> 
> mdadm: /dev/md0 assembled from 2 drives and 3 spares - not enough to start
> the \
> array.
> 
> That's strange... I never configured any spare drives.  There should be

	Any failed drives are moved to spare status.  Issue the `mdadm -D
/dev/mdX` command, and it will probably show 3 failed drives.

> Why are three drives assumed to be spares?

	It's not assumed.  If was almost surely forced by md / mdadm.

> Is there a way for me to recover this array?

	Probably.  No doubt at least one of the drives probably has enough
info on hand to be able to recover most if not all of the information.  You
should be able to force assemble (-A -f) the array using at least 4 drives,
or perhaps all six.  Once assembled, you should be able to fsck the file
system and then mount it and save any critical data off to a backup.  I'd
find out why the drives are being kicked from the array before I tried bring
the array back into production, though, and replace them, or whatever
component is causing them to be kicked.

Re: How does kernel decide that a drive is "spare"?

[Dave W]

Leslie Rhorer <lrhorer <at> satx.rr.com> writes:

> 	Any failed drives are moved to spare status.  Issue the `mdadm -D
> /dev/mdX` command, and it will probably show 3 failed drives.

You are correct.

> > Why are three drives assumed to be spares?
> 
> 	It's not assumed.  If was almost surely forced by md / mdadm.

Is there any way to get md or mdadm to tell me how it is making this decision? 
If I run "mdadm -Afv /dev/md0 /dev/sd[bcdef]1" I see this in messages:

Jun 11 07:22:10 fileserver kernel: md: md0 stopped.
Jun 11 07:22:10 fileserver kernel: md: unbind<sdd1>
Jun 11 07:22:10 fileserver kernel: md: export_rdev(sdd1)
Jun 11 07:22:10 fileserver kernel: md: unbind<sdb1>
Jun 11 07:22:10 fileserver kernel: md: export_rdev(sdb1)
Jun 11 07:22:10 fileserver kernel: md: unbind<sdc1>
Jun 11 07:22:10 fileserver kernel: md: export_rdev(sdc1)
Jun 11 07:22:10 fileserver kernel: md: unbind<sdf1>
Jun 11 07:22:10 fileserver kernel: md: export_rdev(sdf1)
Jun 11 07:22:10 fileserver kernel: md: unbind<sde1>
Jun 11 07:22:10 fileserver kernel: md: export_rdev(sde1)
Jun 11 07:22:27 fileserver kernel: md: md0 stopped.
Jun 11 07:22:27 fileserver kernel: md: bind<sde1>
Jun 11 07:22:27 fileserver kernel: md: bind<sdf1>
Jun 11 07:22:27 fileserver kernel: md: bind<sdc1>
Jun 11 07:22:27 fileserver kernel: md: bind<sdb1>
Jun 11 07:22:27 fileserver kernel: md: bind<sdd1>

All the ATA and SCSI messages in the log appear normal -- there are no warnings
or errors that I can see.

> No doubt at least one of the drives probably has enough
> info on hand to be able to recover most if not all of the information.  You
> should be able to force assemble (-A -f) the array using at least 4 drives,
> or perhaps all six.

I tried all combinations of three out of the five drives, but at most two
drives ever get used:

# mdadm -Af /dev/md0 /dev/sd[de]1
mdadm: /dev/md0 assembled from 2 drives - not enough to start the array.
# mdadm -Af /dev/md0 /dev/sd[bcd]1
mdadm: /dev/md0 assembled from 1 drive and 2 spares - not enough to start the \
array.
# mdadm -Af /dev/md0 /dev/sd[bce]1
mdadm: /dev/md0 assembled from 1 drive and 2 spares - not enough to start the \
array.
# mdadm -Af /dev/md0 /dev/sd[bcf]1
mdadm: No suitable drives found for /dev/md0
# mdadm -Af /dev/md0 /dev/sd[cde]1
mdadm: /dev/md0 assembled from 2 drives and 1 spare - not enough to start the \
array.
# mdadm -Af /dev/md0 /dev/sd[cdf]1
mdadm: /dev/md0 assembled from 1 drive and 2 spares - not enough to start the \
array.
# mdadm -Af /dev/md0 /dev/sd[def]1
mdadm: /dev/md0 assembled from 2 drives and 1 spare - not enough to start the \
array.


Thanks for the advice,
Dave