Repaired the sectors of a drive, how do I get the md to assemble and start degraded?

Repaired the sectors of a drive, how do I get the md to assemble and start degraded?

[Jeff Wiegley]

I have a storage box that I'm attempting to get data off of.

The box was pulled by another because mdadm shut the md device
off due to hard drive errors. and because the OS harddive (separate)
had a motor failure. I replace the OS drive and install a fresh
copy of Ubuntu server 14.04.

When I brought the machine up I thought we were in luck because
it auto assembled and started the Raid 6 array were all the data is.
But it showed that 2 of the 15 drives were off line... [UUUUUU_UUUUU_UU]
something like that. So I mounted the fs and began a tar backup to
another storage location overnight.

During the night it failed a third drive (which is probably what
brought it down in the first place.) I have been able to repair the
drive with Seagate's SeaTools. Now I want to plug the drive back in
and bring the array up in the 2-drive degraded state.

How do I do this? how do I specify a command to take these 15 drives,
mark two of them failed and force the assembly and start of the
remaining 13 good drives (though one has probably been marked as failed
in the past?)

- Jeff

Re: Repaired the sectors of a drive, how do I get the md to assemble and start degraded?

[Mikael Abrahamsson]

On Thu, 24 Apr 2014, Jeff Wiegley wrote:

> How do I do this? how do I specify a command to take these 15 drives, 
> mark two of them failed and force the assembly and start of the 
> remaining 13 good drives (though one has probably been marked as failed 
> in the past?)

Try to use --assemble with all working drives (not the failed ones). If it 
doesn't work, add --force. Do not use --create --assume-clean unless you 
really really know what you're doing. If you go down that route, first 
make sure you do mdadm --examine on all devices so you know what the 
superblocks contained before you erase them (which is what --create will 
do).

Did you try to re-add the failed drives initially? If you had been using a 
bitmap, they might have come online and successfully added, and then you 
could have run md repair and avoided data loss altogether.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Repaired the sectors of a drive, how do I get the md to assemble and start degraded?

[Jeff Wiegley]

I don't want to simply re-add the failed drives as I believe
they will start re-syncing won't they? I don't want their data
lost and overwritten. I want the drive to be treated like it
never failed in the first place.

I might have some filesystem corruption but not as much as I
will if the entire drive is resynced.

I also cannot repair the two other dead drives. So I need this
drive treated as is so that array can come up degraded. Then I
can get what data I can off it before replacing all drives
and probably starting fresh.

- Jeff

On 4/24/2014 10:27 AM, Mikael Abrahamsson wrote:
> On Thu, 24 Apr 2014, Jeff Wiegley wrote:
>
>> How do I do this? how do I specify a command to take these 15 drives,
>> mark two of them failed and force the assembly and start of the
>> remaining 13 good drives (though one has probably been marked as failed
>> in the past?)
>
> Try to use --assemble with all working drives (not the failed ones). If it
> doesn't work, add --force. Do not use --create --assume-clean unless you
> really really know what you're doing. If you go down that route, first
> make sure you do mdadm --examine on all devices so you know what the
> superblocks contained before you erase them (which is what --create will
> do).
>
> Did you try to re-add the failed drives initially? If you had been using a
> bitmap, they might have come online and successfully added, and then you
> could have run md repair and avoided data loss altogether.
>

Re: Repaired the sectors of a drive, how do I get the md to assemble and start degraded?

[Mikael Abrahamsson]

On Thu, 24 Apr 2014, Jeff Wiegley wrote:

> I don't want to simply re-add the failed drives as I believe
> they will start re-syncing won't they? I don't want their data
> lost and overwritten. I want the drive to be treated like it
> never failed in the first place.
>
> I might have some filesystem corruption but not as much as I
> will if the entire drive is resynced.
>
> I also cannot repair the two other dead drives. So I need this
> drive treated as is so that array can come up degraded. Then I
> can get what data I can off it before replacing all drives
> and probably starting fresh.

What do you mean by "repair"?

Well, anyway, if you --assemble --force with all parity drives gone, no 
resync will be done.

As long as you do not use --create, no "bad" information will be synced 
even if you use the previously failed drives. If their even count is way 
off, then --assemble --force might give you a lot more corruption.

But my original point was that if you have a RAID6 with bitmap and two 
drives are kicked out, but they are not dead, it's better to re-add them 
back in, let things re-sync. You then run a "repair" on the volume so you 
try to make sure that any UNC read errors are repaired by md.

Right now, you have no parity and any other UNC sectors will have to be 
written to in order for you to get your data.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Repaired the sectors of a drive, how do I get the md to assemble and start degraded?

[Jeff Wiegley]

Thank you. this one I figured out. The drives are throwing
sector faults that can be automatically reallocated. If you
take such a drive and run is through the SeaTools long fix
it won't "repair" your drive and you will lose the data in
the bad sectors. but you can make the drive usable again
because the fix program will mark the sectors bad and map
those sectors to extra sectors on the drive used for
reallocation.  I have two drives that have been dead a long
time and so are massively out of sync.

I had a third drive die while backing up the array so I could
repair it. this took the entire array offline. But I could
take the recently failed drive which is 100% synced up (just
failed) because I wasn't doing writes during the backup.

I ran that drive through SeaTools to hide/remap the bad sectors.

Then I wanted to know how to force the assembly of the drive
to come up and ignore that this drive had been marked as
out of date.

the answer was to --assemble --force /dev/sd[abcefghiklmno]1
(notice I left out d and j which were the two old very out of
sync drives.)

It therefore thought I had 12 of the 14 drives available.
(it commented that it had no volumes from two of the expected
slots) but it came up anyways.

I then did xfs_check followed by xfs_repair to make sure my
fs wasn't too corrupt and then proceeded to pull all my data
off it successfully. There was some stuff tossed into lost+found
by the repair tool and I'm sure there is some missing or
corrupt files by I'm also sure I recovered 99.99999% of everything
I had on the storage.

Thank you.

- Jeff

Now I just need fix the offset/size/arrangement problem of my
other home machine which is turning out to be very hard :(

On 4/24/2014 8:31 PM, Mikael Abrahamsson wrote:
> On Thu, 24 Apr 2014, Jeff Wiegley wrote:
>
>> I don't want to simply re-add the failed drives as I believe
>> they will start re-syncing won't they? I don't want their data
>> lost and overwritten. I want the drive to be treated like it
>> never failed in the first place.
>>
>> I might have some filesystem corruption but not as much as I
>> will if the entire drive is resynced.
>>
>> I also cannot repair the two other dead drives. So I need this
>> drive treated as is so that array can come up degraded. Then I
>> can get what data I can off it before replacing all drives
>> and probably starting fresh.
>
> What do you mean by "repair"?
>
> Well, anyway, if you --assemble --force with all parity drives gone, no
> resync will be done.
>
> As long as you do not use --create, no "bad" information will be synced
> even if you use the previously failed drives. If their even count is way
> off, then --assemble --force might give you a lot more corruption.
>
> But my original point was that if you have a RAID6 with bitmap and two
> drives are kicked out, but they are not dead, it's better to re-add them
> back in, let things re-sync. You then run a "repair" on the volume so you
> try to make sure that any UNC read errors are repaired by md.
>
> Right now, you have no parity and any other UNC sectors will have to be
> written to in order for you to get your data.
>