RAID5 in strange state

RAID5 in strange state

[Frank Baumgart]

Dear List,

I use MD RAID 5 since some years and so far had to recover from single
disk failures a few times which was always successful.
Now though, I am puzzled.

Setup:
Some PC with 3x WD 1 TB SATA disk drives set up as RAID 5 using kernel
2.6.27.21 (now); the array ran fine for at least 6 months now.

I check the state of the RAID every few days with looking at
/proc/mdstat manually.
Apparently one drive had been kicked out of the array 4 days ago without
me noticing it.
Root cause seemed to be bad cabling but is not confirmed yet.
Anyway, the disc in question ("sde") reports 23 UDMA_CRC errors,
compared to 0 about 2 weeks ago.
Reading the complete device just now via DD still reports those 23
errors but no new ones.

Well, RAID 5 should survive a single disc failure (again) but after a
reboot (due to non-RAID related reasons) the RAID came up as "md0 stopped".

cat /proc/mdstat

Personalities :
md0 : inactive sdc1[1](S) sdd1[2](S) sde1[0](S)
      2930279424 blocks

unused devices: <none>



What's that?
First, documentation on the web is rather outdated and/or incomplete.
Second, my guess that "(S)" represents a spare is backuped up by the
kernel source.


mdadm --examine [devices] gives consistent reports about the RAID 5
structure as:

          Magic : a92b4efc
        Version : 0.90.00
           UUID : ec4fdb7b:e57733c0:4dc42c07:36d99219
  Creation Time : Wed Dec 24 11:40:29 2008
     Raid Level : raid5
  Used Dev Size : 976759808 (931.51 GiB 1000.20 GB)
     Array Size : 1953519616 (1863.02 GiB 2000.40 GB)
   Raid Devices : 3
  Total Devices : 3
Preferred Minor : 0
...
         Layout : left-symmetric
     Chunk Size : 256K



The state though differs:

sdc1:
    Update Time : Tue Apr  7 20:51:33 2009
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : ccff6a15 - correct
         Events : 177920
...
      Number   Major   Minor   RaidDevice State
this     1       8       33        1      active sync   /dev/sdc1

   0     0       0        0        0      removed
   1     1       8       33        1      active sync   /dev/sdc1
   2     2       8       49        2      active sync   /dev/sdd1



sdd1:
    Update Time : Tue Apr  7 20:51:33 2009
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : ccff6a27 - correct
         Events : 177920

         Layout : left-symmetric
     Chunk Size : 256K

      Number   Major   Minor   RaidDevice State
this     2       8       49        2      active sync   /dev/sdd1

   0     0       0        0        0      removed
   1     1       8       33        1      active sync   /dev/sdc1
   2     2       8       49        2      active sync   /dev/sdd1



sde1:
    Update Time : Fri Apr  3 15:00:31 2009
          State : active
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0
       Checksum : ccf463ec - correct
         Events : 7

         Layout : left-symmetric
     Chunk Size : 256K

      Number   Major   Minor   RaidDevice State
this     0       8       65        0      active sync   /dev/sde1

   0     0       8       65        0      active sync   /dev/sde1
   1     1       8       33        1      active sync   /dev/sdc1
   2     2       8       49        2      active sync   /dev/sdd1



sde is the device that failed once and was kicked out of the array.
The update time reflects that if I interprete that right.
But how can sde1 status claim 3 active and working devices? IMO that's
way off.


Now, my assumption:
I think I should be able to either remove sde temporarily and just
restart the degraded array from sdc1/sdd1.
correct?

My backup is a few days old and I would really like to keep the work on
the RAID done in the meantime.

If the answer is just 2 or 3 mdadm command lines, I am yours :-)

Best regards

Frank Baumgart

Re: RAID5 in strange state

[Goswin von Brederlow]

Frank Baumgart <frank.baumgart@gmx.net> writes:

> Dear List,
>
> I use MD RAID 5 since some years and so far had to recover from single
> disk failures a few times which was always successful.
> Now though, I am puzzled.
>
> Setup:
> Some PC with 3x WD 1 TB SATA disk drives set up as RAID 5 using kernel
> 2.6.27.21 (now); the array ran fine for at least 6 months now.
>
> I check the state of the RAID every few days with looking at
> /proc/mdstat manually.
> Apparently one drive had been kicked out of the array 4 days ago without
> me noticing it.
> Root cause seemed to be bad cabling but is not confirmed yet.
> Anyway, the disc in question ("sde") reports 23 UDMA_CRC errors,
> compared to 0 about 2 weeks ago.
> Reading the complete device just now via DD still reports those 23
> errors but no new ones.
>
> Well, RAID 5 should survive a single disc failure (again) but after a
> reboot (due to non-RAID related reasons) the RAID came up as "md0 stopped".
>
> cat /proc/mdstat
>
> Personalities :
> md0 : inactive sdc1[1](S) sdd1[2](S) sde1[0](S)
>       2930279424 blocks
>
> unused devices: <none>
>
>
>
> What's that?
> First, documentation on the web is rather outdated and/or incomplete.
> Second, my guess that "(S)" represents a spare is backuped up by the
> kernel source.
>
>
> mdadm --examine [devices] gives consistent reports about the RAID 5
> structure as:
>
>           Magic : a92b4efc
>         Version : 0.90.00
>            UUID : ec4fdb7b:e57733c0:4dc42c07:36d99219
>   Creation Time : Wed Dec 24 11:40:29 2008
>      Raid Level : raid5
>   Used Dev Size : 976759808 (931.51 GiB 1000.20 GB)
>      Array Size : 1953519616 (1863.02 GiB 2000.40 GB)
>    Raid Devices : 3
>   Total Devices : 3
> Preferred Minor : 0
> ...
>          Layout : left-symmetric
>      Chunk Size : 256K
>
>
>
> The state though differs:
>
> sdc1:
>     Update Time : Tue Apr  7 20:51:33 2009
>           State : clean
>  Active Devices : 2
> Working Devices : 2
>  Failed Devices : 0
>   Spare Devices : 0
>        Checksum : ccff6a15 - correct
>          Events : 177920
> ...
>       Number   Major   Minor   RaidDevice State
> this     1       8       33        1      active sync   /dev/sdc1
>
>    0     0       0        0        0      removed
>    1     1       8       33        1      active sync   /dev/sdc1
>    2     2       8       49        2      active sync   /dev/sdd1
>
>
>
> sdd1:
>     Update Time : Tue Apr  7 20:51:33 2009
>           State : clean
>  Active Devices : 2
> Working Devices : 2
>  Failed Devices : 0
>   Spare Devices : 0
>        Checksum : ccff6a27 - correct
>          Events : 177920
>
>          Layout : left-symmetric
>      Chunk Size : 256K
>
>       Number   Major   Minor   RaidDevice State
> this     2       8       49        2      active sync   /dev/sdd1
>
>    0     0       0        0        0      removed
>    1     1       8       33        1      active sync   /dev/sdc1
>    2     2       8       49        2      active sync   /dev/sdd1
>
>
>
> sde1:
>     Update Time : Fri Apr  3 15:00:31 2009
>           State : active
>  Active Devices : 3
> Working Devices : 3
>  Failed Devices : 0
>   Spare Devices : 0
>        Checksum : ccf463ec - correct
>          Events : 7
>
>          Layout : left-symmetric
>      Chunk Size : 256K
>
>       Number   Major   Minor   RaidDevice State
> this     0       8       65        0      active sync   /dev/sde1
>
>    0     0       8       65        0      active sync   /dev/sde1
>    1     1       8       33        1      active sync   /dev/sdc1
>    2     2       8       49        2      active sync   /dev/sdd1
>
>
>
> sde is the device that failed once and was kicked out of the array.
> The update time reflects that if I interprete that right.
> But how can sde1 status claim 3 active and working devices? IMO that's
> way off.

Sde gave too many errors and failed. It was kicked out. Now how is md
supposed to update its meta data after it was kicked out?

> Now, my assumption:
> I think I should be able to either remove sde temporarily and just
> restart the degraded array from sdc1/sdd1.
> correct?

Stop the raid and assemble it with just the two reliable disks. For me
that always works automatically. After that add the flaky disk again.

If you fear the disk might flake out again I suggest you add a bitmap
to the raid by runing (works any time the raid is not resyncing)

mdadm --grow --bitmap internal /dev/md0

This will cost you some performance but when a disk fails and you
readd it it will only have to sync regions that have changed and not
the full disk.

You can also remove the bitmap again with

mdadm --grow --bitmap none /dev/md0

at any later time. So I really would do that till you have figured out
if the cable is falky or not.

> My backup is a few days old and I would really like to keep the work on
> the RAID done in the meantime.
>
> If the answer is just 2 or 3 mdadm command lines, I am yours :-)
>
> Best regards
>
> Frank Baumgart

MfG
        Goswin

Re: RAID5 in strange state

[Frank Baumgart]

Goswin von Brederlow schrieb:
> Stop the raid and assemble it with just the two reliable disks. For me
> that always works automatically. After that add the flaky disk again.
>
> If you fear the disk might flake out again I suggest you add a bitmap
> to the raid by runing (works any time the raid is not resyncing)
>
> mdadm --grow --bitmap internal /dev/md0
>
> This will cost you some performance but when a disk fails and you
> readd it it will only have to sync regions that have changed and not
> the full disk.
>
> You can also remove the bitmap again with
>
> mdadm --grow --bitmap none /dev/md0
>
> at any later time. So I really would do that till you have figured out
> if the cable is falky or not.
>   

Adding an internal bitmap to an existing array will not destroy any
data, correct?

mdadm --stop /dev/md0
mdadm --assemble /dev/md0 /dev/sdc1 /dev/sdd1
mdadm /dev/md0 --re-add /dev/sde1

appears to do the trick (still sync'ing...)

Anyway I wonder why the array did not come up automatically as 2 out of
the 3 devices were still working and in sync.
On previous occasions with just the same kind of setup, this always
worked fine out of the box.

Thanks for your help!

Frank

Re: RAID5 in strange state

[David Rees]

On Wed, Apr 8, 2009 at 3:19 PM, Frank Baumgart <frank.baumgart@gmx.net> wrote:
> Adding an internal bitmap to an existing array will not destroy any
> data, correct?

Yes, safe to do at any time (as long as it's not syncing, in which
case it simply won't work).

> Anyway I wonder why the array did not come up automatically as 2 out of
> the 3 devices were still working and in sync.

Good question!  Hard to answer - it sure looks like it should have
assembled the two good drives and had the third drive as stopped.

-Dave

Re: RAID5 in strange state

[Neil Brown]

On Wednesday April 8, frank.baumgart@gmx.net wrote:
> Dear List,
> 
> I use MD RAID 5 since some years and so far had to recover from single
> disk failures a few times which was always successful.

Good to hear!

> Now though, I am puzzled.
> 
> Setup:
> Some PC with 3x WD 1 TB SATA disk drives set up as RAID 5 using kernel
> 2.6.27.21 (now); the array ran fine for at least 6 months now.
> 
> I check the state of the RAID every few days with looking at
> /proc/mdstat manually.

You should set up "mdadm --monitor" to do that for you.
Run
  mdadm --monitor --email=root@myhost --scan
at boot time and
  mdadm --monitor --oneshot --scan --email=root@whatever
as a cron job once a day to nag you about degraded arrays

and you should get email whenever something is amiss.  It doesn't hurt
to also check manually occasionally of course.

> Apparently one drive had been kicked out of the array 4 days ago without
> me noticing it.
> Root cause seemed to be bad cabling but is not confirmed yet.
> Anyway, the disc in question ("sde") reports 23 UDMA_CRC errors,
> compared to 0 about 2 weeks ago.
> Reading the complete device just now via DD still reports those 23
> errors but no new ones.
> 
> Well, RAID 5 should survive a single disc failure (again) but after a
> reboot (due to non-RAID related reasons) the RAID came up as "md0 stopped".
> 
> cat /proc/mdstat
> 
> Personalities :
> md0 : inactive sdc1[1](S) sdd1[2](S) sde1[0](S)
>       2930279424 blocks
> 
> unused devices: <none>
> 
> 
> 
> What's that?

I would need to see kernel logs to be able to guess why.
Presumably it was mdadm which attempted to start the array.
If you can run
  mdadm --assemble -vv /dev/md0 /dev/sd[cde]1
and get useful messages that might help.   Though maybe it is too late
and you have already started the array.


> First, documentation on the web is rather outdated and/or incomplete.
> Second, my guess that "(S)" represents a spare is backuped up by the
> kernel source.

Yes, though when an array is "inactive", everything is considered to
be a spare.

> 
> The state though differs:
> 
> sdc1:
>     Update Time : Tue Apr  7 20:51:33 2009
>           State : clean
             ^^^^^^^^^^^^

The fact that the two devices that are still working think the array
is 'clean' should be enough to start the array.  If they thought it
was dirty (aka 'active'), mdadm would refuse to start the array
because an active degraded array could potentially have corrupted data
and you need to know that...

> sde1:
>           State : active
             ^^^^^^^^^^^^^

sde1 is active, but that is the failed device, so that fact that it is
active shouldn't have an effect... by maybe there is a bug somewhere
and it does.

What versions of mdadm and linux are you using?  I'll see if that
situation could cause a breakage.
> 
> My backup is a few days old and I would really like to keep the work on
> the RAID done in the meantime.
> 
> If the answer is just 2 or 3 mdadm command lines, I am yours :-)

If you haven't got it working already, 

  mdadm -A /dev/md0 -vvv /dev/sd[cde]1
and report the messages produced, then
  mdadm -A --force /dev/md0 -vvv /dev/sd[cd]1
  mdadm /dev/md0 -a /dev/sde1

NeilBrown