Re: Failed RAID-5 with 4 disks

[Tyler <pml@dtbb.net>]

I had a typo in my original email, near the end, where I say "was 
fubar... then I would try the above steps, but force the assemble with 
the original failed disk", I actually meant to say "but force the 
assemble with the newly DD'd copy of the original/first drive that failed."

Tyler.

Tyler wrote:

> My suggestion would be to buy two new drives, and DD (or dd rescue) 
> the two bad drives onto the new drives, and then plug the new drive 
> that has the most recent failure on it (HDG?) in, and try running a 
> forced assemble including the HDG drive, then, in readonly mode, run 
> an fsck to check the file system, and see if it thinks most things are 
> okay.  *IF* it checks out okay (for the most part.. you will probably 
> lose some data), then plug the second new disk in, and add it to the 
> array as a spare, and it would then start a resync of the array.  
> Otherwise, if the fsck found that the entire filesystem was fubar... 
> then I would try the above steps, but force the assemble with the 
> original failed disk.. but depending on how long in between the two 
> failures its been, and if any data was written to the array after the 
> first failure, this is probably not going to be a good thing.. but 
> could still be useful if you were trying to recover specific files 
> that were not touched in between the two failures.
>
> I would also suggest googling raid manual recovery procedures, some 
> info is outdated, but some of it describes what I just described above.
>
> Tyler.
>
> Frank Blendinger wrote:
>
>> Hi,
>>
>> I have a RAID-5 set up with the following raidtab:
>>
>> raiddev /dev/md0
>>        raid-level              5
>>        nr-raid-disks           4
>>        nr-spare-disks          0
>>        persistent-superblock   1
>>        parity-algorithm        left-symmetric
>>        chunk-size              256
>>        device                  /dev/hde
>>        raid-disk               0
>>        device                  /dev/hdg
>>        raid-disk               1
>>        device                  /dev/hdi
>>        raid-disk               2
>>        device                  /dev/hdk
>>        raid-disk               3
>>
>> My hde has failed some time ago, leaving some     hde: dma_intr: 
>> status=0x51 { DriveReady SeekComplete Error }
>>     hde: dma_intr: error=0x84 { DriveStatusError BadCRC }
>> messages in the syslog.
>>
>> I wanted to get sure it really was damaged, so I did a badblocks
>> (read-only) scan on /dev/hde. It actually found a bad sector on the
>> disk.
>>
>>
>> I wanted to take the disk out to get me a new one, but unfortunately my
>> hdg seems to have run into trouble too, now. I also have some
>> SeekComplete/BadCRC errors in my log for that disk, too.
>>
>> Furthermore, i got this:
>>
>> Jul 25 10:35:49 blackbox kernel: ide: failed opcode was: unknown
>> Jul 25 10:35:49 blackbox kernel: hdg: DMA disabled
>> Jul 25 10:35:49 blackbox kernel: PDC202XX: Secondary channel reset.
>> Jul 25 10:35:49 blackbox kernel: PDC202XX: Primary channel reset.
>> Jul 25 10:35:49 blackbox kernel: hde: lost interrupt
>> Jul 25 10:35:49 blackbox kernel: ide3: reset: master: error (0x00?)
>> Jul 25 10:35:49 blackbox kernel: end_request: I/O error, dev hdg, 
>> sector 488396928
>> Jul 25 10:35:49 blackbox kernel: end_request: I/O error, dev hdg, 
>> sector 159368976
>> Jul 25 10:35:49 blackbox kernel: end_request: I/O error, dev hdg, 
>> sector 159368984
>> Jul 25 10:35:49 blackbox kernel: end_request: I/O error, dev hdg, 
>> sector 159368992
>> Jul 25 10:35:49 blackbox kernel: end_request: I/O error, dev hdg, 
>> sector 159369000
>> Jul 25 10:35:49 blackbox kernel: end_request: I/O error, dev hdg, 
>> sector 159369008
>> Jul 25 10:35:49 blackbox kernel: end_request: I/O error, dev hdg, 
>> sector 159369016
>> Jul 25 10:35:49 blackbox kernel: end_request: I/O error, dev hdg, 
>> sector 159369024
>> Jul 25 10:35:49 blackbox kernel: end_request: I/O error, dev hdg, 
>> sector 159369032
>> Jul 25 10:35:49 blackbox kernel: md: write_disk_sb failed for device hdg
>> Jul 25 10:35:49 blackbox kernel: lost page write due to I/O error on md0
>> Jul 25 10:35:49 blackbox kernel: lost page write due to I/O error on md0
>> Jul 25 10:35:49 blackbox kernel: RAID5 conf printout:
>> Jul 25 10:35:49 blackbox kernel:  --- rd:4 wd:2 fd:2
>> Jul 25 10:35:49 blackbox kernel:  disk 0, o:1, dev:hdk
>> Jul 25 10:35:49 blackbox kernel:  disk 1, o:1, dev:hdi
>> Jul 25 10:35:49 blackbox kernel:  disk 2, o:0, dev:hdg
>> Jul 25 10:35:49 blackbox kernel: RAID5 conf printout:
>> Jul 25 10:35:49 blackbox kernel:  --- rd:4 wd:2 fd:2
>> Jul 25 10:35:49 blackbox kernel:  disk 0, o:1, dev:hdk
>> Jul 25 10:35:49 blackbox kernel:  disk 1, o:1, dev:hdi
>> Jul 25 10:35:49 blackbox kernel: lost page write due to I/O error on md0
>>
>>
>> Well, now it seems I have to failed disks in my RAID-5, which of course
>> would be fatal. I am still hoping to somehow rescue the data on the
>> array somehow, but I am not sure what would be the best approach. I 
>> don't
>> want to cause any more damage.
>>
>> When booting my system with all four disks connected, hde and hdg as
>> expected won't get added:
>>
>> Jul 26 18:07:59 blackbox kernel: md: hdg has invalid sb, not importing!
>> Jul 26 18:07:59 blackbox kernel: md: autorun ...
>> Jul 26 18:07:59 blackbox kernel: md: considering hdi ...
>> Jul 26 18:07:59 blackbox kernel: md:  adding hdi ...
>> Jul 26 18:07:59 blackbox kernel: md:  adding hdk ...
>> Jul 26 18:07:59 blackbox kernel: md:  adding hde ...
>> Jul 26 18:07:59 blackbox kernel: md: created md0
>> Jul 26 18:07:59 blackbox kernel: md: bind<hde>
>> Jul 26 18:07:59 blackbox kernel: md: bind<hdk>
>> Jul 26 18:07:59 blackbox kernel: md: bind<hdi>
>> Jul 26 18:07:59 blackbox kernel: md: running: <hdi><hdk><hde>
>> Jul 26 18:07:59 blackbox kernel: md: kicking non-fresh hde from array!
>> Jul 26 18:07:59 blackbox kernel: md: unbind<hde>
>> Jul 26 18:07:59 blackbox kernel: md: export_rdev(hde)
>> Jul 26 18:07:59 blackbox kernel: raid5: device hdi operational as 
>> raid disk 1
>> Jul 26 18:07:59 blackbox kernel: raid5: device hdk operational as 
>> raid disk 0
>> Jul 26 18:07:59 blackbox kernel: RAID5 conf printout:
>> Jul 26 18:07:59 blackbox kernel:  --- rd:4 wd:2 fd:2
>> Jul 26 18:07:59 blackbox kernel:  disk 0, o:1, dev:hdk
>> Jul 26 18:07:59 blackbox kernel:  disk 1, o:1, dev:hdi
>> Jul 26 18:07:59 blackbox kernel: md: do_md_run() returned -22
>> Jul 26 18:07:59 blackbox kernel: md: md0 stopped.
>> Jul 26 18:07:59 blackbox kernel: md: unbind<hdi>
>> Jul 26 18:07:59 blackbox kernel: md: export_rdev(hdi)
>> Jul 26 18:07:59 blackbox kernel: md: unbind<hdk>
>> Jul 26 18:07:59 blackbox kernel: md: export_rdev(hdk)
>> Jul 26 18:07:59 blackbox kernel: md: ... autorun DONE.
>>
>> So hde is not fresh (it has been removed from the array for quite some
>> time now) and hdg has an invalid superblock.
>>
>> Any advice on what I should do now? Should I better try to rebuild the
>> array with hde or with hdg?
>>
>>
>> Greetings,
>> Frank
>> -
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>  
>>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>