Cannot auto assemble a raid1 array on boot

Cannot auto assemble a raid1 array on boot

[Jivko Sabev]

Greetings,

I have a RAID1 array setup of the following

/dev/md0 [linear raid array consisting of two 500GB SATA drives]
/dev/md1  [RAID1 array consisting of /dev/md0 and one 1TB SATA drive]

here is my /proc/mdstat

--------

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md1 : active raid1 md0[0] sde1[2]
      976639296 blocks super 1.2 [2/2] [UU]

md0 : active linear sdb1[0] sdc1[1]
      976770537 blocks super 1.2 0k rounding

unused devices: <none>


--------

However, on every reboot, the md1 array is in degraded mode and I get
dumped to a intramfs shell. I can then assemble the said array - i.e.
mdadm --assemble /dev/md1 /dev/md0 /dev/sde1 and everything is fine.
Is it possible to have such an array auto assembled and how?

Many thanks,

Jivko

Re: Cannot auto assemble a raid1 array on boot

[Adam Goryachev]

On 18/10/12 13:35, Jivko Sabev wrote:
> Greetings,
>
> I have a RAID1 array setup of the following
>
> /dev/md0 [linear raid array consisting of two 500GB SATA drives]
> /dev/md1  [RAID1 array consisting of /dev/md0 and one 1TB SATA drive]
>
> here is my /proc/mdstat
>
> --------
>
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
> [raid4] [raid10]
> md1 : active raid1 md0[0] sde1[2]
>       976639296 blocks super 1.2 [2/2] [UU]
>
> md0 : active linear sdb1[0] sdc1[1]
>       976770537 blocks super 1.2 0k rounding
>
> unused devices: <none>
>
>
> --------
>
> However, on every reboot, the md1 array is in degraded mode and I get
> dumped to a intramfs shell. I can then assemble the said array - i.e.
> mdadm --assemble /dev/md1 /dev/md0 /dev/sde1 and everything is fine.
> Is it possible to have such an array auto assembled and how?
>

Maybe you need to update the mdadm.conf within the initrd image ?

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

Re: Cannot auto assemble a raid1 array on boot

[Jivko Sabev]

Hi,

The mdadm.conf in the initrd image contains the correct devices. I.e.
the contents of mdadm.conf in initrd are the output of

mdadm --detail --scan

ARRAY /dev/md0 metadata=1.2 name=mercury:0
UUID=60ea870e:029dcf99:eaae356e:f1c12085
ARRAY /dev/md1 metadata=1.2 name=mercury:1
UUID=d89a52ed:0247f2e8:5edf5d09:21e7fa48

However, the problem remains. That is when booting, the system dumps
into initramfs shell with the raid array in an inactive state. I have
to manually stop the array and then reassemble.

mdadm --manage --stop /dev/md1
mdadm --assemble /dev/md1 /dev/sde1 /dev/md0


At the point, I am able to continue booting and everything is fine after.

Here are the contents of /proc/mdstat from the initrd shell before
reassembling the array.

md1 : inactive sde1[2](S)
      976639672 blocks super 1.2

md0 : active linear sdb1[0] sdc1[1]
      976770537 blocks super 1.2 0k rounding

unused devices: <none>

It just seems to me that it is not possible to mix md devices and sata
devices for new arrays.

Regards,

Jivko

On Wed, Oct 17, 2012 at 9:14 PM, Adam Goryachev
<mailinglists@websitemanagers.com.au> wrote:
> On 18/10/12 13:35, Jivko Sabev wrote:
>> Greetings,
>>
>> I have a RAID1 array setup of the following
>>
>> /dev/md0 [linear raid array consisting of two 500GB SATA drives]
>> /dev/md1  [RAID1 array consisting of /dev/md0 and one 1TB SATA drive]
>>
>> here is my /proc/mdstat
>>
>> --------
>>
>> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
>> [raid4] [raid10]
>> md1 : active raid1 md0[0] sde1[2]
>>       976639296 blocks super 1.2 [2/2] [UU]
>>
>> md0 : active linear sdb1[0] sdc1[1]
>>       976770537 blocks super 1.2 0k rounding
>>
>> unused devices: <none>
>>
>>
>> --------
>>
>> However, on every reboot, the md1 array is in degraded mode and I get
>> dumped to a intramfs shell. I can then assemble the said array - i.e.
>> mdadm --assemble /dev/md1 /dev/md0 /dev/sde1 and everything is fine.
>> Is it possible to have such an array auto assembled and how?
>>
>
> Maybe you need to update the mdadm.conf within the initrd image ?
>
> Regards,
> Adam
>
> --
> Adam Goryachev
> Website Managers
> www.websitemanagers.com.au
>

Re: Cannot auto assemble a raid1 array on boot

[Adam Goryachev]

On 19/10/12 02:19, Jivko Sabev wrote:
> Hi,
> 
> The mdadm.conf in the initrd image contains the correct devices. I.e.
> the contents of mdadm.conf in initrd are the output of
> 
> mdadm --detail --scan
> 
> ARRAY /dev/md0 metadata=1.2 name=mercury:0
> UUID=60ea870e:029dcf99:eaae356e:f1c12085
> ARRAY /dev/md1 metadata=1.2 name=mercury:1
> UUID=d89a52ed:0247f2e8:5edf5d09:21e7fa48
> 
> Here are the contents of /proc/mdstat from the initrd shell before
> reassembling the array.
> 
> md1 : inactive sde1[2](S)
>       976639672 blocks super 1.2
> 
> md0 : active linear sdb1[0] sdc1[1]
>       976770537 blocks super 1.2 0k rounding
> 
> unused devices: <none>
> 
> It just seems to me that it is not possible to mix md devices and sata
> devices for new arrays.

I know it is possible because I have done this before ...

Try adding the device names to the mdadm.conf....

An alternative would be to create a partition table on /dev/md0 of type
fd, this way it should be handled properly by the rest of the MD auto
assemble code.

The other option might be to use a different superblock version between
the two arrays...

Other than that, I'm not too sure, maybe someone else could comment?
Perhaps you could provide the logs generated during bootup in relation
to the MD discovery etc

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

Re: Cannot auto assemble a raid1 array on boot

[Jivko Sabev]

Hi,

Thank you for the suggestions. I have tried explicitly listing the
devices comprising the /dev/md1 array in mdadm.conf as follows:

[snip]
 # definitions of existing MD arrays
ARRAY /dev/md0 metadata=1.2 UUID=60ea870e:029dcf99:eaae356e:f1c12085
name=mercury:0
ARRAY /dev/md1 devices=/dev/md0,/dev/sde1
[snip]

but that has yielded no new results. Same issue of the /dev/md1 array
not being present at boot - dumped into initramfs shell. Below is the
relevent info from "dmesg" in the initramfs shell.

[snip]
[    2.934840] md: bind<sdc1>
[    2.936390] md: linear personality registered for level -1
[    2.936588] bio: create slab <bio-1> at 1
[    2.936698] md0: detected capacity change from 0 to 1000213030400
[    2.939927]  md0: unknown partition table
[    2.944653] md: bind<sde1>
[    4.181978] md: multipath personality registered for level -4
[    4.183187] md: raid0 personality registered for level 0
[    4.184479] md: raid1 personality registered for level 1
[    4.185683] async_tx: api initialized (async)
[    4.252024] raid6: int64x1   2581 MB/s
[    4.320019] raid6: int64x2   3726 MB/s
[    4.388028] raid6: int64x4   2713 MB/s
[    4.456033] raid6: int64x8   2406 MB/s
[    4.524023] raid6: sse2x1    3911 MB/s
[    4.592011] raid6: sse2x2    6382 MB/s
[    4.660027] raid6: sse2x4    7641 MB/s
[    4.660048] raid6: using algorithm sse2x4 (7641 MB/s)
[    4.660321] xor: automatically using best checksumming function: generic_sse
[    4.680016]    generic_sse: 12316.000 MB/sec
[    4.680040] xor: using function: generic_sse (12316.000 MB/sec)
[    4.680555] md: raid6 personality registered for level 6
[    4.680584] md: raid5 personality registered for level 5
[    4.680611] md: raid4 personality registered for level 4
[    4.684297] md: raid10 personality registered for level 10
[snip]


Looking at the log, /dev/md1 is not even mentioned. My suspicion is
that because /dev/md0 is of "unknown partition type" it cannot be used
for assembly.

Following the leads on the other suggestings, created a partion on md0
of type FD (Linux Raid auto detect). Reassembled the array, updated
the initrd image but the problem remains. That is the RAID array is
not auto-assembled on boot. Here is the dmesg log

[snip]
[    1.958921] md: bind<sdb1>
[    2.470635] md: bind<sdc1>
[    2.471928] md: linear personality registered for level -1
[    2.875391] md0: detected capacity change from 0 to 1000213030400
[    2.878472]  md0: p1
[    2.945002] md: bind<sde1>
[    4.397985] md: multipath personality registered for level -4
[    4.399191] md: raid0 personality registered for level 0
[    4.400490] md: raid1 personality registered for level 1
[    4.896552] md: raid6 personality registered for level 6
[    4.896580] md: raid5 personality registered for level 5
[    4.896607] md: raid4 personality registered for level 4
[    4.900306] md: raid10 personality registered for level 10
[   57.097141] md: md1 stopped.
[   57.097238] md: unbind<sde1>
[   57.100051] md: export_rdev(sde1)
[   75.147783] md: md1 stopped.
[   75.148555] md: bind<sde1>
[   75.148909] md: bind<md0>
[   75.149880] md/raid1:md1: active with 2 out of 2 mirrors
[   75.150003] md1: detected capacity change from 0 to 1000078639104
[   75.170616]  md1: unknown partition table
[snip]

Same problem. The system is in initramfs rescue shell. Array manually
reassembled after which everything works.

It is worth mentioning that /dev/md1 is listed in the /etc/crypttab
and is encrypted with dm-crypt luks extension.

Any one out there with similar setup?

Thanks again for you time.

Jivko

On Thu, Oct 18, 2012 at 5:11 PM, Adam Goryachev
<mailinglists@websitemanagers.com.au> wrote:
> On 19/10/12 02:19, Jivko Sabev wrote:
>> Hi,
>>
>> The mdadm.conf in the initrd image contains the correct devices. I.e.
>> the contents of mdadm.conf in initrd are the output of
>>
>> mdadm --detail --scan
>>
>> ARRAY /dev/md0 metadata=1.2 name=mercury:0
>> UUID=60ea870e:029dcf99:eaae356e:f1c12085
>> ARRAY /dev/md1 metadata=1.2 name=mercury:1
>> UUID=d89a52ed:0247f2e8:5edf5d09:21e7fa48
>>
>> Here are the contents of /proc/mdstat from the initrd shell before
>> reassembling the array.
>>
>> md1 : inactive sde1[2](S)
>>       976639672 blocks super 1.2
>>
>> md0 : active linear sdb1[0] sdc1[1]
>>       976770537 blocks super 1.2 0k rounding
>>
>> unused devices: <none>
>>
>> It just seems to me that it is not possible to mix md devices and sata
>> devices for new arrays.
>
> I know it is possible because I have done this before ...
>
> Try adding the device names to the mdadm.conf....
>
> An alternative would be to create a partition table on /dev/md0 of type
> fd, this way it should be handled properly by the rest of the MD auto
> assemble code.
>
> The other option might be to use a different superblock version between
> the two arrays...
>
> Other than that, I'm not too sure, maybe someone else could comment?
> Perhaps you could provide the logs generated during bootup in relation
> to the MD discovery etc
>
> Regards,
> Adam
>
> --
> Adam Goryachev
> Website Managers
> www.websitemanagers.com.au

Re: Cannot auto assemble a raid1 array on boot

[Adam Goryachev]

On 24/10/12 02:43, Jivko Sabev wrote:
> Hi,
> 
> Thank you for the suggestions. I have tried explicitly listing the
> devices comprising the /dev/md1 array in mdadm.conf as follows:
> 
> [snip]
>  # definitions of existing MD arrays
> ARRAY /dev/md0 metadata=1.2 UUID=60ea870e:029dcf99:eaae356e:f1c12085
> name=mercury:0
> ARRAY /dev/md1 devices=/dev/md0,/dev/sde1
> [snip]
> 
> but that has yielded no new results. Same issue of the /dev/md1 array
> not being present at boot - dumped into initramfs shell. Below is the
> relevent info from "dmesg" in the initramfs shell.
> 
> [snip]

> Following the leads on the other suggestings, created a partion on md0
> of type FD (Linux Raid auto detect). Reassembled the array, updated
> the initrd image but the problem remains. That is the RAID array is
> not auto-assembled on boot. Here is the dmesg log
> 
> [snip]
> [    1.958921] md: bind<sdb1>
> [    2.470635] md: bind<sdc1>
> [    2.471928] md: linear personality registered for level -1
> [    2.875391] md0: detected capacity change from 0 to 1000213030400
> [    2.878472]  md0: p1
> [    2.945002] md: bind<sde1>
> [    4.397985] md: multipath personality registered for level -4
> [    4.399191] md: raid0 personality registered for level 0
> [    4.400490] md: raid1 personality registered for level 1
> [    4.896552] md: raid6 personality registered for level 6
> [    4.896580] md: raid5 personality registered for level 5
> [    4.896607] md: raid4 personality registered for level 4
> [    4.900306] md: raid10 personality registered for level 10
> [   57.097141] md: md1 stopped.
> [   57.097238] md: unbind<sde1>
> [   57.100051] md: export_rdev(sde1)
> [   75.147783] md: md1 stopped.
> [   75.148555] md: bind<sde1>
> [   75.148909] md: bind<md0>
> [   75.149880] md/raid1:md1: active with 2 out of 2 mirrors
> [   75.150003] md1: detected capacity change from 0 to 1000078639104
> [   75.170616]  md1: unknown partition table
> [snip]
> 
> Same problem. The system is in initramfs rescue shell. Array manually
> reassembled after which everything works.

Please show /etc/mdadm/mdadm.conf from within the initramfs (without the
comments).

I forget what I mentioned I had before, but I have:
md1 - 2 disk RAID1 sdc1 sde1
md2 - 2 disk RAID1 sdb1 sdd1
md3 - 2 md linear md1 md2

This is working for me...

BTW, my DEVICES line is
DEVICE partitions
If this matches your DEVICES line, please also send the content of
/proc/partitions within your initramfs before you manually make any
changes/do anything.

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

Re: Cannot auto assemble a raid1 array on boot

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2502 bytes --]

On Thu, 18 Oct 2012 09:19:58 -0600 Jivko Sabev <jsabev@nicmus.com> wrote:

> Hi,
> 
> The mdadm.conf in the initrd image contains the correct devices. I.e.
> the contents of mdadm.conf in initrd are the output of
> 
> mdadm --detail --scan
> 
> ARRAY /dev/md0 metadata=1.2 name=mercury:0
> UUID=60ea870e:029dcf99:eaae356e:f1c12085
> ARRAY /dev/md1 metadata=1.2 name=mercury:1
> UUID=d89a52ed:0247f2e8:5edf5d09:21e7fa48
> 
> However, the problem remains. That is when booting, the system dumps
> into initramfs shell with the raid array in an inactive state. I have
> to manually stop the array and then reassemble.
> 
> mdadm --manage --stop /dev/md1
> mdadm --assemble /dev/md1 /dev/sde1 /dev/md0
> 
> 
> At the point, I am able to continue booting and everything is fine after.
> 
> Here are the contents of /proc/mdstat from the initrd shell before
> reassembling the array.
> 
> md1 : inactive sde1[2](S)
>       976639672 blocks super 1.2
> 
> md0 : active linear sdb1[0] sdc1[1]
>       976770537 blocks super 1.2 0k rounding

There are two problems here.

Firstly, the fact that the array doesn't assemble completely should not cause
the boot to fail.  A degraded raid1 is perfectly sufficient for booting.

What is happening is that the initrd is relying on udev to assemble the array
by passing each new device to "mdadm --incremental $DEVNAME".
This will assemble the array as soon as all devices are present, but not
before.   If a device failed before shutdown that will be recorded in the
metadata and "mdadm --incremental" will not wait for it.  If it disappears
during reboot, mdadm will still expect it.

To deal with this issue, the initrd should run
  mdadm --incremental --scan --run

which means "look for all arrays that are being incrementally assembled, and
start them".
This should be called after running "udevadm settle" and before mounting the
root filesystem.

However fixing this won't fix your problem, it will just change it.

The udev rules files which is calling "mdadm --incremental" does so
on /dev/sdb1 /dev/sdc1 and /dev/sde1, but apparently not on /dev/md0.

If at the initrd shell prompt you run
  mdadm -I /dev/md0

it should finish assembling md1 for you.  For some reason udev isn't doing
that.

Have a look in /lib/udev/rules.d or /etc/udev/rules.d for a file that runs
"mdadm --incremental" or "mdadm -I" and see how it works.
Maybe post it.

BTW what distro are you using?

NeilBrown


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Cannot auto assemble a raid1 array on boot

[Jivko Sabev]

Hi,

Thanks for the insightful information. I am running ubuntu 12.04.1 LTS
(desktop version). As per your suggestion, I have run

mdadm -I /dev/md0

from the initramfs and indeed /dev/md1 gets assembled.

Your suggestions have definitely started me on the right path. I have
done some further googling and apparently the problem I am
experiencing is somewhat common in 12.04. I am in the process of
modifying some of the init scripts to make my array assemble
correctly. I have tried the workaround described here

http://utcc.utoronto.ca/~cks/space/blog/linux/Ubuntu1204SoftwareRaidFail

but it did not solve the problem for me.

I will post my solution once it is done.

Regards,

Jivko


Re



On Tue, Oct 23, 2012 at 7:33 PM, NeilBrown <neilb@suse.de> wrote:
> On Thu, 18 Oct 2012 09:19:58 -0600 Jivko Sabev <jsabev@nicmus.com> wrote:
>
>> Hi,
>>
>> The mdadm.conf in the initrd image contains the correct devices. I.e.
>> the contents of mdadm.conf in initrd are the output of
>>
>> mdadm --detail --scan
>>
>> ARRAY /dev/md0 metadata=1.2 name=mercury:0
>> UUID=60ea870e:029dcf99:eaae356e:f1c12085
>> ARRAY /dev/md1 metadata=1.2 name=mercury:1
>> UUID=d89a52ed:0247f2e8:5edf5d09:21e7fa48
>>
>> However, the problem remains. That is when booting, the system dumps
>> into initramfs shell with the raid array in an inactive state. I have
>> to manually stop the array and then reassemble.
>>
>> mdadm --manage --stop /dev/md1
>> mdadm --assemble /dev/md1 /dev/sde1 /dev/md0
>>
>>
>> At the point, I am able to continue booting and everything is fine after.
>>
>> Here are the contents of /proc/mdstat from the initrd shell before
>> reassembling the array.
>>
>> md1 : inactive sde1[2](S)
>>       976639672 blocks super 1.2
>>
>> md0 : active linear sdb1[0] sdc1[1]
>>       976770537 blocks super 1.2 0k rounding
>
> There are two problems here.
>
> Firstly, the fact that the array doesn't assemble completely should not cause
> the boot to fail.  A degraded raid1 is perfectly sufficient for booting.
>
> What is happening is that the initrd is relying on udev to assemble the array
> by passing each new device to "mdadm --incremental $DEVNAME".
> This will assemble the array as soon as all devices are present, but not
> before.   If a device failed before shutdown that will be recorded in the
> metadata and "mdadm --incremental" will not wait for it.  If it disappears
> during reboot, mdadm will still expect it.
>
> To deal with this issue, the initrd should run
>   mdadm --incremental --scan --run
>
> which means "look for all arrays that are being incrementally assembled, and
> start them".
> This should be called after running "udevadm settle" and before mounting the
> root filesystem.
>
> However fixing this won't fix your problem, it will just change it.
>
> The udev rules files which is calling "mdadm --incremental" does so
> on /dev/sdb1 /dev/sdc1 and /dev/sde1, but apparently not on /dev/md0.
>
> If at the initrd shell prompt you run
>   mdadm -I /dev/md0
>
> it should finish assembling md1 for you.  For some reason udev isn't doing
> that.
>
> Have a look in /lib/udev/rules.d or /etc/udev/rules.d for a file that runs
> "mdadm --incremental" or "mdadm -I" and see how it works.
> Maybe post it.
>
> BTW what distro are you using?
>
> NeilBrown
>

Re: Cannot auto assemble a raid1 array on boot

[John Robinson]

On 18/10/2012 03:35, Jivko Sabev wrote:
> Greetings,
>
> I have a RAID1 array setup of the following
>
> /dev/md0 [linear raid array consisting of two 500GB SATA drives]
> /dev/md1  [RAID1 array consisting of /dev/md0 and one 1TB SATA drive]
>
> here is my /proc/mdstat
>
> --------
>
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
> [raid4] [raid10]
> md1 : active raid1 md0[0] sde1[2]
>        976639296 blocks super 1.2 [2/2] [UU]
>
> md0 : active linear sdb1[0] sdc1[1]
>        976770537 blocks super 1.2 0k rounding
>
> unused devices: <none>
>
>
> --------
>
> However, on every reboot, the md1 array is in degraded mode and I get
> dumped to a intramfs shell. I can then assemble the said array - i.e.
> mdadm --assemble /dev/md1 /dev/md0 /dev/sde1 and everything is fine.
> Is it possible to have such an array auto assembled and how?

I'm wondering why the slots for md1 are [0] and [2], and what happened 
to [1]?

It's possible you may need your mdadm.conf to mention
   DEVICE /dev/sd* /dev/md*

or something like that to make it work.

Also, your shutdown process perhaps needs to stop md1 and then md0 in 
that order, and your startup process needs to start md0 and then md1 in 
that order, i.e. you may need additional mdadm invocations to start and 
stop one array on top of another, and I'm not sure what distro, init 
scripts, udev scripts etc you have and whether they will do it all for you.

Cheers,

John.

-- 
John Robinson, yuiop IT services
0131 557 9577 / 07771 784 058
46/12 Broughton Road, Edinburgh EH7 4EE