/proc/mdstat showing a device missing from array

/proc/mdstat showing a device missing from array

[evoltech]

Hello all,

I just set up a new raid 5 array of 4 750G disks and am having a strange
experience where /proc/mdstat is showing that one device is missing from the
array.  The output from a "--detail" shows that there is some unnamed device
that has been removed from the array and I don't understand why or how to fix
it.  Can someone please shed some light?

kernel version: 2.6.20-1.2312.fc5xen0 (this is a fedora core 5 prebuilt kernel)
mdadm version: mdadm - v2.3.1 - 6 February 2006

[root@storage ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md2 : active raid5 sdg[4] sdf[2] sde[1] sdd[0]
      2197723392 blocks level 5, 64k chunk, algorithm 2 [4/3] [UUU_]

[root@storage ~]# mdadm --query --detail /dev/md2
/dev/md2:
        Version : 00.90.03
  Creation Time : Sat May 12 17:22:30 2007
     Raid Level : raid5
     Array Size : 2197723392 (2095.91 GiB 2250.47 GB)
    Device Size : 732574464 (698.64 GiB 750.16 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 2
    Persistence : Superblock is persistent

    Update Time : Sun May 13 04:07:57 2007
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : 6a0d59e6:e00ec820:d08e477f:3b7ef9bc
         Events : 0.68

    Number   Major   Minor   RaidDevice State
       0       8       48        0      active sync   /dev/sdd
       1       8       64        1      active sync   /dev/sde
       2       8       80        2      active sync   /dev/sdf
       0       0        0        0      removed

       4       8       96        4      active sync   /dev/sdg

Re: /proc/mdstat showing a device missing from array

[Neil Brown]

On Sunday May 13, evoltech@2inches.com wrote:
> 
> 
> Hello all,
> 
> I just set up a new raid 5 array of 4 750G disks and am having a strange
> experience where /proc/mdstat is showing that one device is missing from the
> array.  The output from a "--detail" shows that there is some unnamed device
> that has been removed from the array and I don't understand why or how to fix
> it.  Can someone please shed some light?

A raid5 is always created with one missing device and one spare.  This
is because recovery onto a spare is faster than resync of a brand new
array.

The real problem here is that recovery has not started.
If you 
  mdadm -S /dev/md2
  mdadm -A /dev/md2 /dev/sd[def]
  mdadm /dev/md2 -a /dev/sdg
it will start recovery.

This bug was fixed in mdadm-2.5.2

NeilBrown

Re: /proc/mdstat showing a device missing from array

[evoltech]

Quoting Neil Brown <neilb@suse.de>:

> On Sunday May 13, evoltech@2inches.com wrote:
> > 
> > 
> > Hello all,
> > 
> > I just set up a new raid 5 array of 4 750G disks and am having a strange
> > experience where /proc/mdstat is showing that one device is missing from
> the
> > array.  The output from a "--detail" shows that there is some unnamed
> device
> > that has been removed from the array and I don't understand why or how to
> fix
> > it.  Can someone please shed some light?
> 
> A raid5 is always created with one missing device and one spare.  This
> is because recovery onto a spare is faster than resync of a brand new
> array.

This is unclear to me.  Do you mean that is how mdadm implements raid5 creation
 or do you mean that is how raid5 is designed?  I havn't read about this in any
raid5 documentation or mdadm documentation.  Can you point me in the right
direction?

> 
> The real problem here is that recovery has not started.
> If you 
>   mdadm -S /dev/md2
>   mdadm -A /dev/md2 /dev/sd[def]
>   mdadm /dev/md2 -a /dev/sdg
> it will start recovery.
> 
> This bug was fixed in mdadm-2.5.2

The recovery of the array has started, thanks!

Sincerely,
Dennison Williams

Re: /proc/mdstat showing a device missing from array

[Neil Brown]

On Monday May 14, evoltech@2inches.com wrote:
> Quoting Neil Brown <neilb@suse.de>:
> 
> > 
> > A raid5 is always created with one missing device and one spare.  This
> > is because recovery onto a spare is faster than resync of a brand new
> > array.
> 
> This is unclear to me.  Do you mean that is how mdadm implements raid5 creation
>  or do you mean that is how raid5 is designed?  I havn't read about this in any
> raid5 documentation or mdadm documentation.  Can you point me in the right
> direction?

In the OPTIONS section of mdadm.8, under "For create, build, or
grow:", it says:
       -f, --force
              Insist that mdadm accept the geometry and layout specified with‐
              out  question.   Normally  mdadm  will  not allow creation of an
              array with only one device, and will try to create a raid5 array
              with  one  missing  drive (as this makes the initial resync work
              faster).  With --force, mdadm will not try to be so clever.

This is a feature specific to the md implementation of raid5, not
necessarily general to all raid5 implementations.
When md/raid5 performs a "sync", it assumes that most parity blocks
are correct.  So it simply reads all drives in parallel and check that
the parity block is correct.  When it finds one that isn't (which
should not happen often) it will write the correct data.  This
requires a backward seek and breaks the streaming flow of data off the
drives.

For a new array, it is likely that most if not all parity blocks are
wrong.  The above algorithm will cause every parity block to be first
read, and then written, producing lots of seeks and much slower
throughput.

If you create a new array degraded and add a spare, it will recover
the spare by reading all the good drives in parallel, computing the
missing drive, and writing that purely sequentially.  That goes much
faster.

e.g. on my test machine with 5 SATA drives, I get 42Meg/sec recovery
speed, but only around 30Meg/sec resync on a new array.

> > 
> > This bug was fixed in mdadm-2.5.2
> 
> The recovery of the array has started, thanks!

Excellent.  I have since realised that there is a kernel bug as
well.  I will get that fixed in the next release so that the old mdadm
will also work properly.

NeilBrown