Re: Requesting assistance recovering RAID-5 array

Phil Turmel
2020-04-01T04:45:10Z

Hi Daniel,

On 3/31/20 11:39 PM, Daniel Jones wrote:
> Hello Phil, et al.,
>
> Phil, after reading through your email I have some questions.
>
>> The array had bad block logging turned on. We won't re-enable this
>> mis-feature. It is default, so you must turn it off in your --create.
>
> Am I able to turn off during --create? The man page for mdadm on my
> system (mdadm - v4.1 - 2018-10-01) suggests that --update=no-bbl can
> be used for --assemble and --manage but doesn't list it for --create.

Uhm, self compile and see what you get. In these situations, relying on
a potentially buggy system mdadm is not recommended. But if still not
available in create, fix it afterwards. You definitely do not want this.

>> However, the one we know appears to be the typical you'd get from
>> one reshape after a modern default creation (262144).
>
>> You'll have to test four combinations: all at 261124 plus one at a
>> time at 262144.
>
> I'm confused by the offsets. The one remaining superblock I have
> reports "Data Offset : 261120 sectors". Your email mentions 261124
> and 262144. I don't understand how these three values are related?

Yeah, doing math in one's head quickly sometimes yields a fail. 262144
is 128MB in sectors. Minus 1024 sectors (your chunk size) yields
261120. /:

> I think it is most likely that my one existing superblock with 261120
> is one of the original three drives and not the fourth drive that was
> added later. (Based on the position in drive bay).
>
> So possible offsets (I'm still not clear on this) could be:
>
> a) all 261120

Yes.

> b) all 261124

No.

> c) all 262144

No.

> d) three at 261120, one at 262144

Yes.

> e) three at 261120, one at 261124
> f) three at 261124, one at 261120
> g) three at 261124, one at 262144

No, no, and no.

> h) three at 262144, one at 261120

Extremely unlikely. Not in my recommended combinations to check.

> i) three at 262144, one at 261124

No.

> ( this ignores the combinations of not knowing which drive gets the
> oddball offset )
> ( this also ignores for now the offsets of 259076 and 260100 mentioned below )
>
>> 2) Member order for the other drives. Three drives taken three at a
>> time is six combinations.
>>
>> 3) Identity of the first drive kicked out. (Or do we know?) If not
>> known, there's four more combinations: whether to leave out or one of
>> three left out.
>
> Can I make any tentative conclusions from this information:
>
> Device Role : Active device 1
> Array State : .AAA ('A' == active, '.' == missing, 'R' == replacing)

This device will always be listed as the 2nd member in all of your
--create commands, and always with the offset of 261120 - 2048.

> I know /dev/sde is the device that didn't initially respond to BIOS
> and suspect it is the "missing" drive from my superblock.

That eliminates the combinations of (3). Section (2) becomes three
drives taken two at a time (since you don't know which device role
/dev/sde had). But that is still six combinations.

> I know that /dev/sdc is the drive with a working superblock that
> reports itself as "Active device 1".

Right, as above.

> I don't know how mdadm counts things (starting at 0 or starting at 1,
> left to right or right to left, including or excluding the missing
> drive).

Active devices start with zero.

> Would it be reasonable for a first guess that:
>
> .AAA = sde sdd sdc sdb (assuming the order is missing, active 0,
> active 1, active 2) ?

No. Order is always active devices 0-3, with one of those replaced (in
order) with "missing".

> Procedure questions:
>
> If I understand all the above, attempted recovery is going to be along
> the lines of:
>
> mdadm --create /dev/md0 --force --assume-clean --readonly
> --data-offset=261120 --chunk=512K --level=5 --raid-devices=4 missing
> /dev/sdd /dev/sdc /dev/sdb
> fsck -n /dev/md0

Yes, but with the order above, and with --data-offset=variable when
mixing them.

> Subject to:
> Don't know if --force is desirable in this case?

Not applicable to --create.

> Might need to try different offsets from above. Don't know how to set
> offsets if they are different per drive.

man page.

> Should I start with guessing "missing" for 1 or should I start with all 4?
> Might need to try all device orders.
>
>> Start by creating partitions on all devices, preferably at 2048 sectors.
>> (Should be the default offered.) Use data offsets 259076 and 260100
>> instead of 261124 and 262144.
>
> If I understand, this is an alternative to recreating the whole-disk
> mdadm containing one partition. Instead it would involve creating new
> partition tables on each physical drive, creating one partition per
> table, writing superblocks to the new /dev/sd[bcde]1 with offsets
> adjusted by either 2044 or 2048 sectors, and then doing the fsck on
> the assembled RAID.

Yes, 2048.

> I think the advantage proposed here is that it prevents this
> "automated superblock overwrite" from happening again if/when I try
> the motherboard upgrade, but the risk I'm not comfortable with yet is
> going beyond "do the minimum to get it working again". Although it
> isn't practical for me to do a dd full backup of these drives, if I
> can get the array mounted again I can copy off the most important data
> before doing a grander repartitioning.

It's virtually impossible to correct at any time other than create, so
do it now. The "minimum" is a rather brutal situation. Fix it right.

> Can you advise on any of the above?
>
> Thanks,
> DJ
>

Phil