What parameters to mdadm, to re-create md device with payload starting at 0x22000 position on backing storage?

What parameters to mdadm, to re-create md device with payload starting at 0x22000 position on backing storage?

[Adam Ryczkowski]

I try to recover from mdadm raid disaster, which happened when moving 
from ubuntu server 10.04 to 12.04. I know the correct order of devices 
from dmesg log, but given this information, I still cannot access the data.

The superblocks look messy; the mdadm --examine for each disk is on 
http://askubuntu.com/questions/186666/mdadm-fails-after-10-04-12-04-upgrade

By inspecting the raw contents of backing storage, I found the beginning 
of my data (the LUKS container in my case) starts at position 0x22000 
relative to the beginning of the first partition in the raid, .i.e. 
/dev/sdb6.

Question: What is the combination of options issued to "mdadm --create" 
to re-create mdadm that starts with the given offset? Bitmap size?

The relevant information from syslog when the system was healthy are 
pasted here: http://pastebin.com/9KKaXXiU

P.S. This question, like the previous one, is a copy from my question on 
Serverfault.com: 
http://serverfault.com/questions/427683/what-parameters-to-mdadm-to-re-create-md-device-with-payload-starting-at-0x2200

Yours,

Adam Ryczkowski
Skype:sisteczko

Re: What parameters to mdadm, to re-create md device with payload starting at 0x22000 position on backing storage?

[Robin Hill]

[-- Attachment #1: Type: text/plain, Size: 2785 bytes --]

On Mon Sep 17, 2012 at 09:00:14AM +0200, Adam Ryczkowski wrote:

> I try to recover from mdadm raid disaster, which happened when moving 
> from ubuntu server 10.04 to 12.04. I know the correct order of devices 
> from dmesg log, but given this information, I still cannot access the data.
> 
> The superblocks look messy; the mdadm --examine for each disk is on 
> http://askubuntu.com/questions/186666/mdadm-fails-after-10-04-12-04-upgrade
> 
> By inspecting the raw contents of backing storage, I found the beginning 
> of my data (the LUKS container in my case) starts at position 0x22000 
> relative to the beginning of the first partition in the raid, .i.e. 
> /dev/sdb6.
> 
> Question: What is the combination of options issued to "mdadm --create" 
> to re-create mdadm that starts with the given offset? Bitmap size?
> 
> The relevant information from syslog when the system was healthy are 
> pasted here: http://pastebin.com/9KKaXXiU
> 
Looks like you've been hit by the bug mentioned here:
    http://neil.brown.name/blog/20120615073245

You've got most of the required data to recreate the array. It's just
the chunk size missing I think. This is probably 512K as the arrays are
using 1.2 superblocks, but it may be 64K if you were using an old mdadm
version.

Your data offset is 272 sectors, which means you'll need to use an older
mdadm version to re-create the array. Newer versions use 2048 sectors,
but it looks like version 3.0 uses 272, so grab a copy of that if
possible.

Assuming your drives are numbered in the same order as previously, the
create commands will be:
    mdadm -C /dev/md5 -e 1.2 -l 5 -n 5 /dev/sd{b,f,c,e,d}6 \
        --assume-clean --uuid=a3945c40:73237cd6:3d61998e:8f773d03
and
    mdadm -C /dev/md6 -e 1.2 -l 6 -n 5 /dev/sd{b,f,c,e,d}5 \
        --assume-clean --uuid=2a8fd2df:3ad53552:e8b80ecf:a46d93a2

Make sure you use the curly brace format to list the disks rather than
square brackets - that'll ensure the order is maintained. Also make sure
that --assume-clean is used to prevent rebuilding from kicking off, and
losing data.

After recreating the array, run a "fsck -n" on the array to check
whether the details are correct. If not, stop the array and try
recreating it again, adding "-c 64" to set the chunk size to 64K.

Once you've got them up and fsck shows clean, you can re-add the bitmaps
using "mdadm -G /dev/mdX --bitmap=internal".

Do read through Neil's blog post first as there may be something I've
missed.

HTH and good luck,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: What parameters to mdadm, to re-create md device with payload starting at 0x22000 position on backing storage?

[Adam Ryczkowski]

On 2012-09-17 19:49, Robin Hill wrote:

> Your data offset is 272 sectors, which means you'll need to use an older
> mdadm version to re-create the array. Newer versions use 2048 sectors,
> but it looks like version 3.0 uses 272, so grab a copy of that if
> possible.
Does it mean, that with the current raid setup I need to stick to the 
mdadm 3.0, or is there way to convert the arrays to the new layout so I 
can use a recent mdadm? Re-creating the arrays from scratch would be a 
very long and complex enterprise.

Adam

Re: What parameters to mdadm, to re-create md device with payload starting at 0x22000 position on backing storage?

[Robin Hill]

[-- Attachment #1: Type: text/plain, Size: 1390 bytes --]

On Tue Sep 18, 2012 at 08:53:25AM +0200, Adam Ryczkowski wrote:

> On 2012-09-17 19:49, Robin Hill wrote:
> 
> > Your data offset is 272 sectors, which means you'll need to use an older
> > mdadm version to re-create the array. Newer versions use 2048 sectors,
> > but it looks like version 3.0 uses 272, so grab a copy of that if
> > possible.
> Does it mean, that with the current raid setup I need to stick to the 
> mdadm 3.0, or is there way to convert the arrays to the new layout so I 
> can use a recent mdadm? Re-creating the arrays from scratch would be a 
> very long and complex enterprise.
> 
You only need 3.0 to do the create. Once that's done, you can use any
newer version.

The only way I know of to update the offset location in an existing
array would be to fail the disks one at a time, zeroing the superblocks,
then adding them back into the array using a new mdadm. When it's added
back in, it'll use the new offset location. If you have a spare disk and
a new enough kernel version, you can use the pre-emptive replacement
option, which will prevent the need to run without full parity
available.

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]