I will pay money for the correct RAID recovery instructions

I will pay money for the correct RAID recovery instructions

[Ian Young]

I've been trying to fix a degraded array for a couple of months now
and it's getting frustrating enough that I'm willing to put a bounty
on the correct solution.  The array can start in a degraded state and
the data is accessible, so I know this is possible to fix.  Any
takers?  I'll bet someone could use some beer money or a contribution
to their web hosting costs.

Here's how the system is set up:  There are (6) 3 TB drives.  Each
drive has a BIOS boot partition.  The rest of the space on each drive
is a large GPT partition that is combined in a RAID 10 array.  On top
of the array there are four LVM volumes: /boot, /root, swap, and /srv.

Here's the problem:  /dev/sdf failed.  I replaced it but as it was
resyncing, read errors on /dev/sde kicked the new sdf out and made it
a spare.  The array is now in a precarious degraded state.  All it
would take for the entire array to fail is for /dev/sde to fail, and
it's already showing signs that it will.  I have tried forcing the
array to assemble using /dev/sd[abcde]2 and then forcing it to add
/dev/sdf2.  That still adds sdf2 as a spare.  I've tried "echo check >
/sys/block/md0/md/sync_action" but that finishes immediately and
changes nothing.

Can anyone solve this?  I'd be happy to pay you for your knowledge.

Re: I will pay money for the correct RAID recovery instructions

[Robin Hill]

[-- Attachment #1: Type: text/plain, Size: 2648 bytes --]

On Thu Oct 16, 2014 at 12:59:18pm -0700, Ian Young wrote:

> I've been trying to fix a degraded array for a couple of months now
> and it's getting frustrating enough that I'm willing to put a bounty
> on the correct solution.  The array can start in a degraded state and
> the data is accessible, so I know this is possible to fix.  Any
> takers?  I'll bet someone could use some beer money or a contribution
> to their web hosting costs.
> 
> Here's how the system is set up:  There are (6) 3 TB drives.  Each
> drive has a BIOS boot partition.  The rest of the space on each drive
> is a large GPT partition that is combined in a RAID 10 array.  On top
> of the array there are four LVM volumes: /boot, /root, swap, and /srv.
> 
> Here's the problem:  /dev/sdf failed.  I replaced it but as it was
> resyncing, read errors on /dev/sde kicked the new sdf out and made it
> a spare.  The array is now in a precarious degraded state.  All it
> would take for the entire array to fail is for /dev/sde to fail, and
> it's already showing signs that it will.  I have tried forcing the
> array to assemble using /dev/sd[abcde]2 and then forcing it to add
> /dev/sdf2.  That still adds sdf2 as a spare.  I've tried "echo check >
> /sys/block/md0/md/sync_action" but that finishes immediately and
> changes nothing.
> 
If sdf didn't finish syncing then it's no use adding it to the array as
anything other than a spare. Also, you can't run a check on a degraded
array (as there's nothing to check against), which is why that's
finishing immediately.

If sde is giving a read error during rebuild then the solution is to
stop the array (you'll need to do this via a bootable CD/USB stick I
guess) and use ddrescue to duplicate sde onto a new disk, The
read errors may well mean that some can't be copied (though ddrescue
will try very hard to do so), which may cause file/filesystem corruption
later. You can then reassemble the (degraded) array with the old sda-sdd
and the new sde, then add sdf and wait for the array to recover. You
can then run a fsck on the filesystem to check for any corruption there.
File corruption is a lot trickier to spot - if you have checksums for
the files then that's one way, otherwise you may be able to work out
what files are affected based on the offsets of the missing data (that's
rather beyond the limits of my knowledge though).

HTH,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: I will pay money for the correct RAID recovery instructions

[Ian Young]

Ok, if I can pull this off I owe you a beer.

On Thu, Oct 16, 2014 at 1:22 PM, Robin Hill <robin@robinhill.me.uk> wrote:
> On Thu Oct 16, 2014 at 12:59:18pm -0700, Ian Young wrote:
>
>> I've been trying to fix a degraded array for a couple of months now
>> and it's getting frustrating enough that I'm willing to put a bounty
>> on the correct solution.  The array can start in a degraded state and
>> the data is accessible, so I know this is possible to fix.  Any
>> takers?  I'll bet someone could use some beer money or a contribution
>> to their web hosting costs.
>>
>> Here's how the system is set up:  There are (6) 3 TB drives.  Each
>> drive has a BIOS boot partition.  The rest of the space on each drive
>> is a large GPT partition that is combined in a RAID 10 array.  On top
>> of the array there are four LVM volumes: /boot, /root, swap, and /srv.
>>
>> Here's the problem:  /dev/sdf failed.  I replaced it but as it was
>> resyncing, read errors on /dev/sde kicked the new sdf out and made it
>> a spare.  The array is now in a precarious degraded state.  All it
>> would take for the entire array to fail is for /dev/sde to fail, and
>> it's already showing signs that it will.  I have tried forcing the
>> array to assemble using /dev/sd[abcde]2 and then forcing it to add
>> /dev/sdf2.  That still adds sdf2 as a spare.  I've tried "echo check >
>> /sys/block/md0/md/sync_action" but that finishes immediately and
>> changes nothing.
>>
> If sdf didn't finish syncing then it's no use adding it to the array as
> anything other than a spare. Also, you can't run a check on a degraded
> array (as there's nothing to check against), which is why that's
> finishing immediately.
>
> If sde is giving a read error during rebuild then the solution is to
> stop the array (you'll need to do this via a bootable CD/USB stick I
> guess) and use ddrescue to duplicate sde onto a new disk, The
> read errors may well mean that some can't be copied (though ddrescue
> will try very hard to do so), which may cause file/filesystem corruption
> later. You can then reassemble the (degraded) array with the old sda-sdd
> and the new sde, then add sdf and wait for the array to recover. You
> can then run a fsck on the filesystem to check for any corruption there.
> File corruption is a lot trickier to spot - if you have checksums for
> the files then that's one way, otherwise you may be able to work out
> what files are affected based on the offsets of the missing data (that's
> rather beyond the limits of my knowledge though).
>
> HTH,
>     Robin
> --
>      ___
>     ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
>    / / )      | Little Jim says ....                            |
>   // !!       |      "He fallen in de water !!"                 |

Re: I will pay money for the correct RAID recovery instructions

[John Stoffel]

Ian,

It would also help if you posted the details of your setup using:

cat /proc/partitions
cat /proc/mdstat

mdadm -D /dev/md#
 - for each of the devices above.

mdadm -E /dev/sd<drive><#>
 - for each disk or partition in the array from above.



But the suggestions to ddrescue the going bad drive onto a new disk is
a good one.  On my debian system, I would do the following:

  sudo apt-get install gddrescue
  ddrescue /dev/sde /dev/sdf /var/tmp/ddrecue-sde.log

and see how that goes.  

Good luck,
John

Fwd: I will pay money for the correct RAID recovery instructions

[Another Sillyname]

Sorry to state the obvious but......

To restore the degraded array would (based on the info you've posted)
likely take longer then temporarily moving the data to a different set
of drives.

As time seems to be a major consideration here (likely/possible
failure of sde) then surely the optimal strategy has to be to get the
data off first, then look at the rebuilding the degraded array?

Just my 2c

On 17 October 2014 15:05, John Stoffel <john@stoffel.org> wrote:
>
>
> Ian,
>
> It would also help if you posted the details of your setup using:
>
> cat /proc/partitions
> cat /proc/mdstat
>
> mdadm -D /dev/md#
>  - for each of the devices above.
>
> mdadm -E /dev/sd<drive><#>
>  - for each disk or partition in the array from above.
>
>
>
> But the suggestions to ddrescue the going bad drive onto a new disk is
> a good one.  On my debian system, I would do the following:
>
>   sudo apt-get install gddrescue
>   ddrescue /dev/sde /dev/sdf /var/tmp/ddrecue-sde.log
>
> and see how that goes.
>
> Good luck,
> John
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fwd: I will pay money for the correct RAID recovery instructions

[John Stoffel]

Another> Sorry to state the obvious but......  To restore the degraded
Another> array would (based on the info you've posted) likely take
Another> longer then temporarily moving the data to a different set of
Another> drives.

How do you figure this?  He hasn't posted any useful data on his array
at all.  We know it's six 3Tb disks in RAID10, so that's at most 9Tb
of data.  Moving 3Tb (bad disk ddrescue'd onto new disk) is only 3Tb
to move.  Then the resync will of course take time.  

But if he doesn't have the spare disk space, it's probably the only way.

Another> As time seems to be a major consideration here (likely/possible
Another> failure of sde) then surely the optimal strategy has to be to get the
Another> data off first, then look at the rebuilding the degraded array?

Another> Just my 2c

Another> On 17 October 2014 15:05, John Stoffel <john@stoffel.org> wrote:
>> 
>> 
>> Ian,
>> 
>> It would also help if you posted the details of your setup using:
>> 
>> cat /proc/partitions
>> cat /proc/mdstat
>> 
>> mdadm -D /dev/md#
>> - for each of the devices above.
>> 
>> mdadm -E /dev/sd<drive><#>
>> - for each disk or partition in the array from above.
>> 
>> 
>> 
>> But the suggestions to ddrescue the going bad drive onto a new disk is
>> a good one.  On my debian system, I would do the following:
>> 
>> sudo apt-get install gddrescue
>> ddrescue /dev/sde /dev/sdf /var/tmp/ddrecue-sde.log
>> 
>> and see how that goes.
>> 
>> Good luck,
>> John
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
Another> --
Another> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
Another> the body of a message to majordomo@vger.kernel.org
Another> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fwd: I will pay money for the correct RAID recovery instructions

[Ian Young]

Robin's solution worked.  I used ddrescue to copy sde to a new disk,
then put the new disk in place of sde, and started the array in
degraded mode.  Now I just need to get a replacement disk for sdf and
the array should be all set.

I believe I owe Robin a pint.  Got Paypal or Google Wallet?

On Fri, Oct 17, 2014 at 9:14 AM, John Stoffel <john@stoffel.org> wrote:
>
> Another> Sorry to state the obvious but......  To restore the degraded
> Another> array would (based on the info you've posted) likely take
> Another> longer then temporarily moving the data to a different set of
> Another> drives.
>
> How do you figure this?  He hasn't posted any useful data on his array
> at all.  We know it's six 3Tb disks in RAID10, so that's at most 9Tb
> of data.  Moving 3Tb (bad disk ddrescue'd onto new disk) is only 3Tb
> to move.  Then the resync will of course take time.
>
> But if he doesn't have the spare disk space, it's probably the only way.
>
> Another> As time seems to be a major consideration here (likely/possible
> Another> failure of sde) then surely the optimal strategy has to be to get the
> Another> data off first, then look at the rebuilding the degraded array?
>
> Another> Just my 2c
>
> Another> On 17 October 2014 15:05, John Stoffel <john@stoffel.org> wrote:
>>>
>>>
>>> Ian,
>>>
>>> It would also help if you posted the details of your setup using:
>>>
>>> cat /proc/partitions
>>> cat /proc/mdstat
>>>
>>> mdadm -D /dev/md#
>>> - for each of the devices above.
>>>
>>> mdadm -E /dev/sd<drive><#>
>>> - for each disk or partition in the array from above.
>>>
>>>
>>>
>>> But the suggestions to ddrescue the going bad drive onto a new disk is
>>> a good one.  On my debian system, I would do the following:
>>>
>>> sudo apt-get install gddrescue
>>> ddrescue /dev/sde /dev/sdf /var/tmp/ddrecue-sde.log
>>>
>>> and see how that goes.
>>>
>>> Good luck,
>>> John
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Another> --
> Another> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> Another> the body of a message to majordomo@vger.kernel.org
> Another> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html