mdadm --create parameter ordering

mdadm --create parameter ordering

[Steven Reddie]

Hi,

Given the following command:

    mdadm -Cf /dev/md3 -e1.2 -n5 -l6 /dev/sdga5 /dev/sdgb5 /dev/sdgc5
/dev/sdgd5 /dev/sdge5 -u<uuid>

Is the order that the disks are specified on the command-line significant?

Somebody suggested this command to put the disks back in the correct
order (though I believe he was mistaken about the correct order). It
seems to me that if the order on the command-line is significant that
this would have caused parity to get recalculated (based on the
[incorrect] order of the disks) and the new parity to be written over
actual data blocks. I see no evidence that this has occurred, which
leads me to believe that the actual ordering within the array is
detected and everything proceeds as it should, in which case issuing
the command above likely was a very time-consuming no-op.

Also, it seems that I should be able to issue such a command without
the full scan of the disks. Is that what the "--run" option is for? If
so, does "--readonly" protect me from harm that improper use of
"--run" may cause?

Thanks,

Steven

Re: mdadm --create parameter ordering

[Robin Hill]

[-- Attachment #1: Type: text/plain, Size: 2960 bytes --]

On Mon Jul 13, 2015 at 08:26:48PM +1000, Steven Reddie wrote:

> Hi,
> 
> Given the following command:
> 
>     mdadm -Cf /dev/md3 -e1.2 -n5 -l6 /dev/sdga5 /dev/sdgb5 /dev/sdgc5
> /dev/sdgd5 /dev/sdge5 -u<uuid>
> 
> Is the order that the disks are specified on the command-line significant?
> 
Yes, it specifies the disks' position within the array.

> Somebody suggested this command to put the disks back in the correct
> order (though I believe he was mistaken about the correct order). It
> seems to me that if the order on the command-line is significant that
> this would have caused parity to get recalculated (based on the
> [incorrect] order of the disks) and the new parity to be written over
> actual data blocks. I see no evidence that this has occurred, which
> leads me to believe that the actual ordering within the array is
> detected and everything proceeds as it should, in which case issuing
> the command above likely was a very time-consuming no-op.
> 
You're creating a new array, so there is no "actual ordering" to detect
- any old array will be ignored. If you are re-creating the array in
order to fix an issue (which you should only do as a very last resort,
and only after seeking help and advice here) then an incorrect order
will definitely cause new parity information to be written over data
blocks (which is why you should always leave "missing" disks, or use
--assume-clean,  to prevent parity calculation until after you've
verified the order is correct and the data is readable), If the
filesystem shows as readable (and error free) then the order specified
would appear to be correct.

> Also, it seems that I should be able to issue such a command without
> the full scan of the disks. Is that what the "--run" option is for? If
> so, does "--readonly" protect me from harm that improper use of
> "--run" may cause?
> 
The --run option is to start an array which has been assembled but not
started. The udev auto-assemble rule, for example, will add disks into
the array as they're discovered but only start it once the array is
complete (or as complete as it was the last time it was assembled), so
in the case of a missing disk, a manual --run command may be required.

I'm not entirely sure what the --readonly option will do in the case of
a create command. Possibly just the metadata will be written and the
parity calculation will be paused.

The --assume-clean option will prevent parity calculation (as it assumes
that the parity on disk is already correct), which may be what you're
after. It should only be used it you're absolutely certain of the data
though, as it can lead to large-scale corruption otherwise.

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: mdadm --create parameter ordering

[Phil Turmel]

On 07/13/2015 06:59 AM, Robin Hill wrote:

[snip /]

Let me elaborate on one of Robin's comments:

> The --assume-clean option will prevent parity calculation (as it assumes
> that the parity on disk is already correct), which may be what you're
> after. It should only be used it you're absolutely certain of the data
> though, as it can lead to large-scale corruption otherwise.

It is not enough to be certain of the order to use --create
--assume-clean on existing data.  You must also be certain of superblock
type, chunk size, data offset, and layout option.

Data loss due to unwise use of --create has a history of bad outcomes.
It destroys the previous metadata on array members, metadata that is
almost always needed to recover damaged arrays.  The --create option
should *never* be used on an array with important data without first
asking for advice on this list.

Phil

Re: mdadm --create parameter ordering

[Steven Reddie]

Thanks Robin and Phil, I've verified that the parity information did
in fact get recalculated and written to different disks, though I'm
confident that I can still recover by reversing this. I expect that I
could manually reconstruct the metadata and superblocks if I needed to
but before I began I took a dd backup of the first 1GB of each
physical disk, each partition, and the MD device, and I also have the
/etc/space/*.xml files, so I expect I have enough to restore these.
I'll start a new thread to ask about the best way to proceed.

Thanks,

Steven


On Mon, Jul 13, 2015 at 11:17 PM, Phil Turmel <philip@turmel.org> wrote:
> On 07/13/2015 06:59 AM, Robin Hill wrote:
>
> [snip /]
>
> Let me elaborate on one of Robin's comments:
>
>> The --assume-clean option will prevent parity calculation (as it assumes
>> that the parity on disk is already correct), which may be what you're
>> after. It should only be used it you're absolutely certain of the data
>> though, as it can lead to large-scale corruption otherwise.
>
> It is not enough to be certain of the order to use --create
> --assume-clean on existing data.  You must also be certain of superblock
> type, chunk size, data offset, and layout option.
>
> Data loss due to unwise use of --create has a history of bad outcomes.
> It destroys the previous metadata on array members, metadata that is
> almost always needed to recover damaged arrays.  The --create option
> should *never* be used on an array with important data without first
> asking for advice on this list.
>
> Phil