Recreate raid 10 array

Recreate raid 10 array

[LCID Fire]

On my raid10 array (4 drives) I've had 2 drives (the same model) which 
got disconnected by the kernel (almost at the same time). It seems like 
both were one raid1 part so one half of the raid0 is missing.
Afterwards I tried to readd the drives again (my bad) so now I'm stuck 
with 1 half of the raid0 part being present and valid and the other (2 
drives) marked as spare.
The thing I'd like to do is:
- Take 2 new drives (different manufacturers)
- Clone one of the valid and one of the spare drives to the new drives
- Try to reassemble the array.
Problem is - is this even possible?
How do I tell mdadm to not care about the spare state?
Does the kernel write bogus data to the still valid drives if the other 
2 fail?

Would be great if someone could enlighten me ;)

P.S.: Does someone know how to easily report the different sata errors 
which the kernel encounters?

Re: Recreate raid 10 array

[Goswin von Brederlow]

LCID Fire <lcid-fire@gmx.net> writes:

> On my raid10 array (4 drives) I've had 2 drives (the same model) which
> got disconnected by the kernel (almost at the same time). It seems
> like both were one raid1 part so one half of the raid0 is missing.
> Afterwards I tried to readd the drives again (my bad) so now I'm stuck
> with 1 half of the raid0 part being present and valid and the other (2
> drives) marked as spare.
> The thing I'd like to do is:
> - Take 2 new drives (different manufacturers)
> - Clone one of the valid and one of the spare drives to the new drives
> - Try to reassemble the array.
> Problem is - is this even possible?
> How do I tell mdadm to not care about the spare state?
> Does the kernel write bogus data to the still valid drives if the
> other 2 fail?
>
> Would be great if someone could enlighten me ;)
>
> P.S.: Does someone know how to easily report the different sata errors
> which the kernel encounters?

mdadm --create --assume-clean -l 10 -n 4 /dev/mdX /dev/copied_disk_1 /dev/copied_disk2 missing missing

You need to match the create parameters exactly with the ones you
initially used (near/offset/farcopies? stripe size? ...) and the order
of devices is relevant so you might have to shuffle the disk
arguments. So just try different orders till the result can be mounted
or fscked. With the wrong options the mount/fsck could screw up the
data but then you copy the disk again for the next try. It should be
reasonably obvious when mount/fsck goes wrong as it should find tons
of errors. Mostly I would expect mount/fsck to just fail with the
wrong mdadm args though.

MfG
        Goswin

Re: Recreate raid 10 array

[Bill Davidsen]

Goswin von Brederlow wrote:
> LCID Fire <lcid-fire@gmx.net> writes:
>
>   
>> On my raid10 array (4 drives) I've had 2 drives (the same model) which
>> got disconnected by the kernel (almost at the same time). It seems
>> like both were one raid1 part so one half of the raid0 is missing.
>> Afterwards I tried to readd the drives again (my bad) so now I'm stuck
>> with 1 half of the raid0 part being present and valid and the other (2
>> drives) marked as spare.
>> The thing I'd like to do is:
>> - Take 2 new drives (different manufacturers)
>> - Clone one of the valid and one of the spare drives to the new drives
>> - Try to reassemble the array.
>> Problem is - is this even possible?
>> How do I tell mdadm to not care about the spare state?
>> Does the kernel write bogus data to the still valid drives if the
>> other 2 fail?
>>
>> Would be great if someone could enlighten me ;)
>>
>> P.S.: Does someone know how to easily report the different sata errors
>> which the kernel encounters?
>>     
>
> mdadm --create --assume-clean -l 10 -n 4 /dev/mdX /dev/copied_disk_1 /dev/copied_disk2 missing missing
>
> You need to match the create parameters exactly with the ones you
> initially used (near/offset/farcopies? stripe size? ...) and the order
> of devices is relevant so you might have to shuffle the disk
> arguments. So just try different orders till the result can be mounted
> or fscked. With the wrong options the mount/fsck could screw up the
> data but then you copy the disk again for the next try. It should be
> reasonably obvious when mount/fsck goes wrong as it should find tons
> of errors. Mostly I would expect mount/fsck to just fail with the
> wrong mdadm args though.
>   

May I say that this makes a great case for saving the contents of some 
files to a safe place when the system is up and running right.? Maybe 
all of /etc, and at least a "tree /sys" and /proc/mdstat would be 
useful, preferably on something readable like a CD or USB flash drive, 
so you have a chance of reading it if you can't boot.

Of course a rescue flash drive is pretty useful as well, so that's 
probably the way to go.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc

"You are disgraced professional losers. And by the way, give us our money back."
    - Representative Earl Pomeroy,  Democrat of North Dakota
on the A.I.G. executives who were paid bonuses  after a federal bailout.

Re: Recreate raid 10 array

[Andrew Burgess]

On Wed, 2009-04-08 at 17:47 -0400, Bill Davidsen wrote:
> Goswin von Brederlow wrote:
> > mdadm --create --assume-clean -l 10 -n 4 /dev/mdX /dev/copied_disk_1 /dev/copied_disk2 missing missing
> >
> > You need to match the create parameters exactly with the ones you
> > initially used (near/offset/farcopies? stripe size? ...) and the order
> > of devices is relevant so you might have to shuffle the disk
> > arguments. So just try different orders till the result can be mounted
> > or fscked. With the wrong options the mount/fsck could screw up the
> > data but then you copy the disk again for the next try. It should be
> > reasonably obvious when mount/fsck goes wrong as it should find tons
> > of errors. Mostly I would expect mount/fsck to just fail with the
> > wrong mdadm args though.

Most fscks can be told to run read-only so they won't write to the
device and also interactive so they ask before writing so you should be
able to avoid recopying. The ext3 journal recovery violates at least one
of these IIRC (or used to) so if it's ext3 find an option to tell it to
ignore the journal.

> May I say that this makes a great case for saving the contents of some 
> files to a safe place when the system is up and running right.? Maybe 
> all of /etc, and at least a "tree /sys" and /proc/mdstat would be 
> useful, preferably on something readable like a CD or USB flash drive, 
> so you have a chance of reading it if you can't boot.
> 
> Of course a rescue flash drive is pretty useful as well, so that's 
> probably the way to go.

It's a good idea

It also seems like mdadm could be enhanced to figure stuff like this out
given intact device superblocks (I suggest --wild-ass-guess as the
option name)

Re: Recreate raid 10 array

[Goswin von Brederlow]

Andrew Burgess <aab@cichlid.com> writes:

> It also seems like mdadm could be enhanced to figure stuff like this out
> given intact device superblocks (I suggest --wild-ass-guess as the
> option name)

Like (an imaginary)

mdadm --recreate --assume-clean /dev/mdX /dev/disk1 /dev/disk2

figuring out all the right parameters, number of missing disks and
positions of the existing ones and then writing a new supberlock? The
ultimate --assemble --force option.

MfG
        Goswin

Re: Recreate raid 10 array

[LCID Fire]

First off the good news: I'm currently running on my raid10 again - with 
only little data loss.

Andrew Burgess wrote:
> On Wed, 2009-04-08 at 17:47 -0400, Bill Davidsen wrote:
>> Goswin von Brederlow wrote:
>>> mdadm --create --assume-clean -l 10 -n 4 /dev/mdX /dev/copied_disk_1 /dev/copied_disk2 missing missing
>>>
>>> You need to match the create parameters exactly with the ones you
>>> initially used (near/offset/farcopies? stripe size? ...) and the order
>>> of devices is relevant so you might have to shuffle the disk
>>> arguments. So just try different orders till the result can be mounted
>>> or fscked. With the wrong options the mount/fsck could screw up the
>>> data but then you copy the disk again for the next try. It should be
>>> reasonably obvious when mount/fsck goes wrong as it should find tons
>>> of errors. Mostly I would expect mount/fsck to just fail with the
>>> wrong mdadm args though.
> 
> Most fscks can be told to run read-only so they won't write to the
> device and also interactive so they ask before writing so you should be
> able to avoid recopying. The ext3 journal recovery violates at least one
> of these IIRC (or used to) so if it's ext3 find an option to tell it to
> ignore the journal.
Too late. The journal recovery did complain quite a bit and I didn't 
know better than to have it fix the things it liked to fix.
As a result it shows the problem with many apps using sqlite these days 
- it's not very good when the database file is corrupted.

>> May I say that this makes a great case for saving the contents of some 
>> files to a safe place when the system is up and running right.? Maybe 
>> all of /etc, and at least a "tree /sys" and /proc/mdstat would be 
>> useful, preferably on something readable like a CD or USB flash drive, 
>> so you have a chance of reading it if you can't boot.
>>
>> Of course a rescue flash drive is pretty useful as well, so that's 
>> probably the way to go.
Quite frankly I don't really care about / - as long as my /home is safe 
- because I can setup my machine again - but losing my work means losing 
far more time.

> It also seems like mdadm could be enhanced to figure stuff like this out
> given intact device superblocks (I suggest --wild-ass-guess as the
> option name)
That would be great (not that I'm eager to run into that again).

As a note I did a binary comparison between the raid1 stuff and got 
quite shocked. The corrupted one had around 1.000.000 byte difference - 
something I would expect - but even the valid mirror had around 20.0000 
bytes difference - which I can't explain to myself this easily.

Anyway - thanks guys for the great help.

Re: Recreate raid 10 array

[Andrew Burgess]

On Thu, 2009-04-09 at 00:14 +0200, LCID Fire wrote:

> > Most fscks can be told to run read-only so they won't write to the
> > device and also interactive so they ask before writing so you should be
> > able to avoid recopying. The ext3 journal recovery violates at least one
> > of these IIRC (or used to) so if it's ext3 find an option to tell it to
> > ignore the journal.

> Too late. The journal recovery did complain quite a bit and I didn't 
> know better than to have it fix the things it liked to fix.

Sorry to hear that. Actually our advice should have been to assemble the
array read-only, then all manner of fs bugs/features could not write to
it.

If anyone ever implements the --wild-ass-guess/--recreate mdadm option
perhaps that should automatically set read-only?

Re: Recreate raid 10 array

[Goswin von Brederlow]

Andrew Burgess <aab@cichlid.com> writes:

> On Thu, 2009-04-09 at 00:14 +0200, LCID Fire wrote:
>
>> > Most fscks can be told to run read-only so they won't write to the
>> > device and also interactive so they ask before writing so you should be
>> > able to avoid recopying. The ext3 journal recovery violates at least one
>> > of these IIRC (or used to) so if it's ext3 find an option to tell it to
>> > ignore the journal.
>
>> Too late. The journal recovery did complain quite a bit and I didn't 
>> know better than to have it fix the things it liked to fix.
>
> Sorry to hear that. Actually our advice should have been to assemble the
> array read-only, then all manner of fs bugs/features could not write to
> it.

But then you couldn't mount it at all since the journal replay needs
to happen in ext3.

> If anyone ever implements the --wild-ass-guess/--recreate mdadm option
> perhaps that should automatically set read-only?

MfG
        Goswin

Re: Recreate raid 10 array

[Bill Davidsen]

LCID Fire wrote:
> First off the good news: I'm currently running on my raid10 again - 
> with only little data loss.
>
> Andrew Burgess wrote:
>> On Wed, 2009-04-08 at 17:47 -0400, Bill Davidsen wrote:
>>> Goswin von Brederlow wrote:
>>>> mdadm --create --assume-clean -l 10 -n 4 /dev/mdX 
>>>> /dev/copied_disk_1 /dev/copied_disk2 missing missing
>>>>
>>>> You need to match the create parameters exactly with the ones you
>>>> initially used (near/offset/farcopies? stripe size? ...) and the order
>>>> of devices is relevant so you might have to shuffle the disk
>>>> arguments. So just try different orders till the result can be mounted
>>>> or fscked. With the wrong options the mount/fsck could screw up the
>>>> data but then you copy the disk again for the next try. It should be
>>>> reasonably obvious when mount/fsck goes wrong as it should find tons
>>>> of errors. Mostly I would expect mount/fsck to just fail with the
>>>> wrong mdadm args though.
>>
>> Most fscks can be told to run read-only so they won't write to the
>> device and also interactive so they ask before writing so you should be
>> able to avoid recopying. The ext3 journal recovery violates at least one
>> of these IIRC (or used to) so if it's ext3 find an option to tell it to
>> ignore the journal.
> Too late. The journal recovery did complain quite a bit and I didn't 
> know better than to have it fix the things it liked to fix.
> As a result it shows the problem with many apps using sqlite these 
> days - it's not very good when the database file is corrupted.
>
>>> May I say that this makes a great case for saving the contents of 
>>> some files to a safe place when the system is up and running right.? 
>>> Maybe all of /etc, and at least a "tree /sys" and /proc/mdstat would 
>>> be useful, preferably on something readable like a CD or USB flash 
>>> drive, so you have a chance of reading it if you can't boot.
>>>
>>> Of course a rescue flash drive is pretty useful as well, so that's 
>>> probably the way to go.
> Quite frankly I don't really care about / - as long as my /home is 
> safe - because I can setup my machine again - but losing my work means 
> losing far more time.
>
>> It also seems like mdadm could be enhanced to figure stuff like this out
>> given intact device superblocks (I suggest --wild-ass-guess as the
>> option name)
> That would be great (not that I'm eager to run into that again).
>
> As a note I did a binary comparison between the raid1 stuff and got 
> quite shocked. The corrupted one had around 1.000.000 byte difference 
> - something I would expect - but even the valid mirror had around 
> 20.0000 bytes difference - which I can't explain to myself this easily.

My personal explanation for mismatch on raid1 swap is this: because 
software raid isn't real raid, you don't get a mirror by writing the 
same page once only to two different drives by sending the data to the 
controller and letting that happen. What you get is the kernel sending 
the same page to two drives without locking the page so it can't change 
as it's being written. That's a gross over-simplification, but I think 
it addresses the heart of the matter. To prevent that would require a 
flag which causes COW until the page had been moved to the controller 
for the last time.

The alternate I've seen proposed is that somewhere between writing 
copies the page is deallocated so the second (or Nth for more than two 
way mirror) write is abandoned.

In any case this appears to be because the page in memory changes 
between write and/or not all writes are mirrored.

The interesting thing is that some systems seem to have lots of 
mishatches and some almost always none.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc

"You are disgraced professional losers. And by the way, give us our money back."
    - Representative Earl Pomeroy,  Democrat of North Dakota
on the A.I.G. executives who were paid bonuses  after a federal bailout.

Re: Recreate raid 10 array

[LCID Fire]

LCID Fire wrote:
> First off the good news: I'm currently running on my raid10 again - 
> with only little data loss.
And now also some bad news again :(
Appearently something is wrong.
mount /dev/md0 /mnt
complains
mount: unknown filesystem type 'linux_raid_member'

On older kernels it works when stating ext3 as the filesystem - but with 
recent 2.6.28/mdadm it doesn't anymore.

Re: Recreate raid 10 array

[LCID Fire]

LCID Fire wrote:
> And now also some bad news again :(
> Appearently something is wrong.
> mount /dev/md0 /mnt
> complains
> mount: unknown filesystem type 'linux_raid_member'
> 
> On older kernels it works when stating ext3 as the filesystem - but with 
> recent 2.6.28/mdadm it doesn't anymore.
Seems like I accidentally added /dev/md0 to another (temporarily) raid 
device. After removing the superblock it works fine now.