Bad drive discovered during raid5 reshape

Bad drive discovered during raid5 reshape

[Kyle Stuart]

Hi,
I bought two new hard drives to expand my raid array today and
unfortunately one of them appears to be bad. The problem didn't arise
until after I attempted to grow the raid array. I was trying to expand
the array from 6 to 8 drives. I added both drives using mdadm --add
/dev/md1 /dev/sdb1 which completed, then mdadm --add /dev/md1 /dev/sdc1
which also completed. I then ran mdadm --grow /dev/md1 --raid-devices=8.
It passed the critical section, then began the grow process.

After a few minutes I started to hear unusual sounds from within the
case. Fearing the worst I tried to cat /proc/mdstat which resulted in no
output so I checked dmesg which showed that /dev/sdb1 was not working
correctly. After several minutes dmesg indicated that mdadm gave up and
the grow process stopped. After googling around I tried the solutions
that seemed most likely to work, including removing the new drives with
mdadm --remove --force /dev/md1 /dev/sd[bc]1 and rebooting after which I
ran mdadm -Af /dev/md1. The grow process restarted then failed almost
immediately. Trying to mount the drive gives me a reiserfs replay
failure and suggests running fsck. I don't dare fsck the array since
I've already messed it up so badly. Is there any way to go back to the
original working 6 disc configuration with minimal data loss? Here's
where I'm at right now, please let me know if I need to include any
additional information.

# uname -a
Linux nas 2.6.22-gentoo-r5 #1 SMP Thu Aug 23 16:59:47 MDT 2007 x86_64
AMD Athlon(tm) 64 Processor 3500+ AuthenticAMD GNU/Linux

# mdadm --version
mdadm - v2.6.2 - 21st May 2007

# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md1 : active raid5 hdb1[0] sdb1[8](F) sda1[5] sdf1[4] sde1[3] sdg1[2]
sdd1[1]
      1220979520 blocks super 0.91 level 5, 64k chunk, algorithm 2 [8/6]
[UUUUUU__]

unused devices: <none>

# mdadm --detail --verbose /dev/md1
/dev/md1:
        Version : 00.91.03
  Creation Time : Sun Apr  8 19:48:01 2007
     Raid Level : raid5
     Array Size : 1220979520 (1164.42 GiB 1250.28 GB)
  Used Dev Size : 244195904 (232.88 GiB 250.06 GB)
   Raid Devices : 8
  Total Devices : 7
Preferred Minor : 1
    Persistence : Superblock is persistent

    Update Time : Mon Oct 29 00:53:21 2007
          State : clean, degraded
 Active Devices : 6
Working Devices : 6
 Failed Devices : 1
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

  Delta Devices : 2, (6->8)

           UUID : 56e7724e:9a5d0949:ff52889f:ac229049
         Events : 0.487460

    Number   Major   Minor   RaidDevice State
       0       3       65        0      active sync   /dev/hdb1
       1       8       49        1      active sync   /dev/sdd1
       2       8       97        2      active sync   /dev/sdg1
       3       8       65        3      active sync   /dev/sde1
       4       8       81        4      active sync   /dev/sdf1
       5       8        1        5      active sync   /dev/sda1
       6       0        0        6      removed
       8       8       17        7      faulty spare rebuilding   /dev/sdb1

#dmesg
<snip>
md: md1 stopped.
md: unbind<hdb1>
md: export_rdev(hdb1)
md: unbind<sdc1>
md: export_rdev(sdc1)
md: unbind<sdb1>
md: export_rdev(sdb1)
md: unbind<sda1>
md: export_rdev(sda1)
md: unbind<sdf1>
md: export_rdev(sdf1)
md: unbind<sde1>
md: export_rdev(sde1)
md: unbind<sdg1>
md: export_rdev(sdg1)
md: unbind<sdd1>
md: export_rdev(sdd1)
md: bind<sdd1>
md: bind<sdg1>
md: bind<sde1>
md: bind<sdf1>
md: bind<sda1>
md: bind<sdb1>
md: bind<sdc1>
md: bind<hdb1>
md: md1 stopped.
md: unbind<hdb1>
md: export_rdev(hdb1)
md: unbind<sdc1>
md: export_rdev(sdc1)
md: unbind<sdb1>
md: export_rdev(sdb1)
md: unbind<sda1>
md: export_rdev(sda1)
md: unbind<sdf1>
md: export_rdev(sdf1)
md: unbind<sde1>
md: export_rdev(sde1)
md: unbind<sdg1>
md: export_rdev(sdg1)
md: unbind<sdd1>
md: export_rdev(sdd1)
md: bind<sdd1>
md: bind<sdg1>
md: bind<sde1>
md: bind<sdf1>
md: bind<sda1>
md: bind<sdb1>
md: bind<sdc1>
md: bind<hdb1>
md: kicking non-fresh sdc1 from array!
md: unbind<sdc1>
md: export_rdev(sdc1)
raid5: reshape will continue
raid5: device hdb1 operational as raid disk 0
raid5: device sdb1 operational as raid disk 7
raid5: device sda1 operational as raid disk 5
raid5: device sdf1 operational as raid disk 4
raid5: device sde1 operational as raid disk 3
raid5: device sdg1 operational as raid disk 2
raid5: device sdd1 operational as raid disk 1
raid5: allocated 8462kB for md1
raid5: raid level 5 set md1 active with 7 out of 8 devices, algorithm 2
RAID5 conf printout:
 --- rd:8 wd:7
 disk 0, o:1, dev:hdb1
 disk 1, o:1, dev:sdd1
 disk 2, o:1, dev:sdg1
 disk 3, o:1, dev:sde1
 disk 4, o:1, dev:sdf1
 disk 5, o:1, dev:sda1
 disk 7, o:1, dev:sdb1
...ok start reshape thread
md: reshape of RAID array md1
md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
md: using maximum available idle IO bandwidth (but not more than 200000
KB/sec) for reshape.
md: using 128k window, over a total of 244195904 blocks.
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x2 frozen
ata2.00: cmd 35/00:00:3f:42:02/00:04:00:00:00/e0 tag 0 cdb 0x0 data
524288 out
         res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
ata2: port is slow to respond, please be patient (Status 0xd8)
ata2: device not ready (errno=-16), forcing hardreset
ata2: hard resetting port
<repeats 4 more times>
ata2: reset failed, giving up
ata2.00: disabled
ata2: EH complete
sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET
driverbyte=DRIVER_OK,SUGGEST_OK
end_request: I/O error, dev sdb, sector 148031
raid5: Disk failure on sdb1, disabling device. Operation continuing on 6
devices
sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET
driverbyte=DRIVER_OK,SUGGEST_OK
end_request: I/O error, dev sdb, sector 149055
sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET
driverbyte=DRIVER_OK,SUGGEST_OK
end_request: I/O error, dev sdb, sector 149439
md: md1: reshape done.

Re: Bad drive discovered during raid5 reshape

[Neil Brown]

On Monday October 29, kstuart@sisna.com wrote:
> Hi,
> I bought two new hard drives to expand my raid array today and
> unfortunately one of them appears to be bad. The problem didn't arise
> until after I attempted to grow the raid array. I was trying to expand
> the array from 6 to 8 drives. I added both drives using mdadm --add
> /dev/md1 /dev/sdb1 which completed, then mdadm --add /dev/md1 /dev/sdc1
> which also completed. I then ran mdadm --grow /dev/md1 --raid-devices=8.
> It passed the critical section, then began the grow process.
> 
> After a few minutes I started to hear unusual sounds from within the
> case. Fearing the worst I tried to cat /proc/mdstat which resulted in no
> output so I checked dmesg which showed that /dev/sdb1 was not working
> correctly. After several minutes dmesg indicated that mdadm gave up and
> the grow process stopped. After googling around I tried the solutions
> that seemed most likely to work, including removing the new drives with
> mdadm --remove --force /dev/md1 /dev/sd[bc]1 and rebooting after which I
> ran mdadm -Af /dev/md1. The grow process restarted then failed almost
> immediately. Trying to mount the drive gives me a reiserfs replay
> failure and suggests running fsck. I don't dare fsck the array since
> I've already messed it up so badly. Is there any way to go back to the
> original working 6 disc configuration with minimal data loss? Here's
> where I'm at right now, please let me know if I need to include any
> additional information.

Looks like you are in real trouble.  Both the drives seem bad in some
way.  If it was just sdc that was failing it would have picked up
after the "-Af", but when it tried, sdb gave errors.

Have two failed devices in a RAID5 is not good!

Your best bet goes like this:

  The reshape has started and got up to some point.  The data
  before that point is spread over 8 drives.  The data after is over
  6.
  We need to restripe the 8drive data back to 6 drives.  This can be
  done with the test_stripe tool that can be built from the mdadm
  source. 

  1/ Find out how far the reshape progressed, by using "mdadm -E" on
     one of the devices.
  2/ use something like
        test_stripe save /some/file 8 $chunksize 5 2 0 $length  /dev/......

     If you get all the args right, this should copy the data from
     the array into /some/file.
     You could possibly do the same thing by assembling the array 
     read-only (set /sys/modules/md_mod/parameters/start_ro to 1)
     and 'dd' from the array.  It might be worth doing both and
     checking you get the same result.

  3/ use something like
        test_stripe restore /some/file 6 ..........
     to restore the data to just 6 devices.

  4/ use "mdadm -C" to create the array a-new on the 6 devices.  Make
     sure the order and the chunksize etc is preserved.

     Once you have done this, the start of the array should (again)
     look like the content of /some/file.  It wouldn't hurt to check.

   Then your data would be as much back together as possible.
   You will probably still need to do an fsck, but I think you did the
   right thing in holding off.  Don't do an fsck until you are sure
   the array is writable.

You can probably do the above without using test_stripe by using dd to
copy of the array before you recreate it, then using dd to put the
same data back.  Using test_stripe as well might give you extra
confidence. 

Feel free to ask questions

NeilBrown

Re: Bad drive discovered during raid5 reshape

[David Greaves]

Neil Brown wrote:
> On Monday October 29, kstuart@sisna.com wrote:
>> Hi,
>> I bought two new hard drives to expand my raid array today and
>> unfortunately one of them appears to be bad. The problem didn't arise

> Looks like you are in real trouble.  Both the drives seem bad in some
> way.  If it was just sdc that was failing it would have picked up
> after the "-Af", but when it tried, sdb gave errors.

Humble enquiry.... :)

I'm not sure that's right?
He *removed* sdb and sdc when the failure occurred so sdc would indeed be non-fresh.

The key question I think is: will md continue to grow an array even if it enters
degraded mode during the grow?
ie grow from a 6 drive array to a 7-of-8 degraded array?

Technically I guess it should be able to.

In which case should he be able to re-add /dev/sdc and allow md to retry the
grow? (possibly losing some data due to the sdc staleness)

David

Re: Bad drive discovered during raid5 reshape

[Neil Brown]

On Tuesday October 30, david@dgreaves.com wrote:
> Neil Brown wrote:
> > On Monday October 29, kstuart@sisna.com wrote:
> >> Hi,
> >> I bought two new hard drives to expand my raid array today and
> >> unfortunately one of them appears to be bad. The problem didn't arise
> 
> > Looks like you are in real trouble.  Both the drives seem bad in some
> > way.  If it was just sdc that was failing it would have picked up
> > after the "-Af", but when it tried, sdb gave errors.
> 
> Humble enquiry.... :)
> 
> I'm not sure that's right?
> He *removed* sdb and sdc when the failure occurred so sdc would indeed be non-fresh.

I'm not sure what point you are making here.
In any case, remove two drives from a raid5 is always a bad thing.
Part of the array was striped over 8 drives by this time.  With only
six still in the array, some data will be missing.

> 
> The key question I think is: will md continue to grow an array even if it enters
> degraded mode during the grow?
> ie grow from a 6 drive array to a 7-of-8 degraded array?
> 
> Technically I guess it should be able to.

Yes, md can grow to a degraded array.  If you get a single failure I
would expect it to abort the growth process, then restart where it
left off (after checking that that made sense).

> 
> In which case should he be able to re-add /dev/sdc and allow md to retry the
> grow? (possibly losing some data due to the sdc staleness)

He only needs one of the two drives in there.  I got the impression
that both sdc and sdb had reported errors.  If not, and sdc really
seems OK, then "--assemble --force" listing all drives except sdb
should make it all work again.

NeilBrown

Re: Bad drive discovered during raid5 reshape

[David Greaves]

Neil Brown wrote:
> On Tuesday October 30, david@dgreaves.com wrote:
>> The key question I think is: will md continue to grow an array even if it enters
>> degraded mode during the grow?
>> ie grow from a 6 drive array to a 7-of-8 degraded array?
> 
> Yes, md can grow to a degraded array.  If you get a single failure I
> would expect it to abort the growth process, then restart where it
> left off (after checking that that made sense).

I read that he aborted it, then removed both drives before giving md a chance to
restart.

He said:
After several minutes dmesg indicated that mdadm gave up and
the grow process stopped. After googling around I tried the solutions
that seemed most likely to work, including removing the new drives with
mdadm --remove --force /dev/md1 /dev/sd[bc]1 and rebooting

and *then* he: "ran mdadm -Af /dev/md1."

>> In which case should he be able to re-add /dev/sdc and allow md to retry the
>> grow? (possibly losing some data due to the sdc staleness)
> 
> He only needs one of the two drives in there.  I got the impression
> that both sdc and sdb had reported errors.  If not, and sdc really
> seems OK, then "--assemble --force" listing all drives except sdb
> should make it all work again.

Kyle - I think you need to clarify this as it may not be too bad. Apologies if I
misread something and sdc is bad too :)

It may be an idea to let us (Neil) know what you've done and if you've done any
writes to any devices before trying this assemble.

David

Re: Bad drive discovered during raid5 reshape

[Kyle Stuart]

David Greaves wrote:
> I read that he aborted it, then removed both drives before giving md a chance to
> restart.
>
> He said:
> After several minutes dmesg indicated that mdadm gave up and
> the grow process stopped. After googling around I tried the solutions
> that seemed most likely to work, including removing the new drives with
> mdadm --remove --force /dev/md1 /dev/sd[bc]1 and rebooting
>
> and *then* he: "ran mdadm -Af /dev/md1."
>   
This is correct. I first removed sdb and sdc then rebooted and ran mdadm
-Af /dev/md1.
>
> Kyle - I think you need to clarify this as it may not be too bad. Apologies if I
> misread something and sdc is bad too :)
>
> It may be an idea to let us (Neil) know what you've done and if you've done any
> writes to any devices before trying this assemble.
>
> David
When I sent the first email I thought only sdb had failed. After digging
into the log files it appears sdc also reported several bad blocks
during the grow. This is what I get for not testing cheap refurbed
drives before trusting them with my data, but hindsight is 20/20.
Fortunately all of the important data is backed up so if I can't recover
anything using Neil's suggestions it's not a total loss.

Thank you both for the help.