Reversing parity recalculation on out-of-order RAID 6 array

Reversing parity recalculation on out-of-order RAID 6 array

[Steven Reddie]

Hi,

Continuing from the "mdadm --create parameter ordering" thread, I now
want to reverse the effects of recreating the array with the disks
specified in the incorrect order.

Are there existing tools that will help with this?

I've had a quick play around with nbd and I'm confident that I can
write an nbd client that will effectively reorder the underlying disks
and recalculate the parity blocks to present 5 pseudo devices that
look like the original disks. I could then attempt to start an array
via those nbd devices and mount/check the filesystem. This approach
would also allow me to ensure that access to the physical disks is
readonly, and implement copy-on-write if necessary to support this.
I'm kind of hoping that all of this might have already been done, so
any pointers would be appreciated.

[BTW, I also expect that I'll have to mess with metadata and
superblocks, but first need to solve the problem above.]

Thanks,

Steven

Re: Reversing parity recalculation on out-of-order RAID 6 array

[Mikael Abrahamsson]

On Sat, 18 Jul 2015, Steven Reddie wrote:

> I'm kind of hoping that all of this might have already been done, so
> any pointers would be appreciated.

https://raid.wiki.kernel.org/index.php/RAID_Recovery
https://raid.wiki.kernel.org/index.php/Permute_array.pl

There have historically been lots of threads here with people having the 
same problems, so if you search the archives for things like 
"assume-clean", "order", "mdadm", "create", "superblock", you'll find 
multiple examples of people actually successfully getting out of the 
problem you're also having.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Reversing parity recalculation on out-of-order RAID 6 array

[Steven Reddie]

Hi Mikael,

I did look through the archives and also at that recovery wiki and
permute script, but I think my case is different, even though I'm sure
others must have also hit it before. My problem isn't that I don't
know the order of the disks in the array, but that the data in the
array has been scrambled. As an example, if a given extent in the
array was ordered across the disks as so:

    A B C P Q

where:
    P = A ^ B ^ C
    Q = A ^ mul(B) ^ mul(mul(C))

Then when my array was reconfigured in the incorrect order (say, with
disk 2 moved to the end), the layout was reordered to became:

    A C P Q B

Then when the array was recreated in this state, the parity
information was recalculated such that the data on disk became:

    A C P P' Q'

where:
    P' = A ^ C ^ P
    Q' = A ^ mul(C) ^ mul(mul(P))

Thus, the new parity information is now a function of two out of three
of the original data chunks (A and C) and one of the original parity
chunks (P), with data chunk B having been overwritten.

I need to reverse these disk writes across then entire length of the
array, after which I can reassemble the array in the correct order.

Thanks,

Steven

On Sat, Jul 18, 2015 at 3:33 PM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Sat, 18 Jul 2015, Steven Reddie wrote:
>
>> I'm kind of hoping that all of this might have already been done, so
>> any pointers would be appreciated.
>
>
> https://raid.wiki.kernel.org/index.php/RAID_Recovery
> https://raid.wiki.kernel.org/index.php/Permute_array.pl
>
> There have historically been lots of threads here with people having the
> same problems, so if you search the archives for things like "assume-clean",
> "order", "mdadm", "create", "superblock", you'll find multiple examples of
> people actually successfully getting out of the problem you're also having.
>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Reversing parity recalculation on out-of-order RAID 6 array

[Mikael Abrahamsson]

On Sat, 18 Jul 2015, Steven Reddie wrote:

> I need to reverse these disk writes across then entire length of the 
> array, after which I can reassemble the array in the correct order.

If you did not use --assume-clean, and had the wrong parity order and the 
array was "resynced", I don't see how you did not irrevocably overwrote 
important information and you now can't reverse this.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Reversing parity recalculation on out-of-order RAID 6 array

[Steven Reddie]

It's the same as if I had lost two disks from the array, which RAID 6
can recover. The difference is that for each stripe (right
terminology?) of the array, the overwrites didn't occur on the same
two disks due to the "Q A B C P" layout rotating from stripe to stripe
(each 64KB * 5 in my case). Therefore, for any given stripe, which
disks were overwritten is a function of the current incorrect ordering
of the disks in the array and the offset of the stripe (which tells
which rotation of "Q A B C P" is used). Including knowledge of the
original correct ordering of the disks then allows mapping to which
two components of the original data have been lost (ie. two of Q, A,
B, C, or P). Then it is a simple matter of recalculating the missing
components from the survived components.

I'm now almost done with an nbd-based translator that does this
reversal on the fly (ie. takes /dev/sdg[a-e]5 as input and presents
the translated versions as /dev/nbd[0-4]). Since I wanted to do the
remapping only once per stripe, I haven't used the nbd server and
client code as I'd have to run an instance of each for each of the 5
disks in my array. I've got my own server and client code that
interfaces with the nbd kernel module in the same way, but provides
access to all disks through the one instance.

I don't want to risk messing with any of the MD or LVM metadata on the
Synology, so once the translator is finished I will try assembling an
array from these devices across the network on another machine and
check/mount the filesystem.

I was hoping that there might be a tried and tested tool already
available for this, but if not I'll continue with mine.

Steven

On Sat, Jul 18, 2015 at 9:43 PM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Sat, 18 Jul 2015, Steven Reddie wrote:
>
>> I need to reverse these disk writes across then entire length of the
>> array, after which I can reassemble the array in the correct order.
>
>
> If you did not use --assume-clean, and had the wrong parity order and the
> array was "resynced", I don't see how you did not irrevocably overwrote
> important information and you now can't reverse this.
>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se