Automatically start two-level mdadm RAID arrays (i.e. RAID 60) on boot?

Automatically start two-level mdadm RAID arrays (i.e. RAID 60) on boot?

[Sean Caron]

Hi all,

My site is very happily using MD RAID arrays constructed in two
levels. That is to say, we do RAID 60 by constructing an mdadm RAID 0
out of a number of mdadm RAID 6 strings.

These work great but one thing we have found and been living with is
that, when the system boots, mdadm will automatically start all the
RAID 6 strings, but it will not automatically start the top level RAID
0 container. The machine pauses on boot ("Hit S to skip, M to manually
resolve") trying to mount the RAID because the device is not
available, and a sys admin needs to manually go to the console and
start the top level RAID 0 to continue the boot.

In an attempt to reduce a little bit of toil and make it easier for
staff to reboot machines for service, I wanted to ask the list if
there was some way to get mdadm to automatically start the top-level
RAID 0 containers without operator intervention in this type of
scenario? I'm not sure if it's just an issue of inserting a little
delay somewhere between starting all the RAID 6 strings and attempting
to start the RAID 0, or if there is some configuration parameter I
have missed.

Thanks!

Sean

Re: Automatically start two-level mdadm RAID arrays (i.e. RAID 60) on boot?

[Roman Mamedov]

On Thu, 15 Feb 2018 15:41:37 -0500
Sean Caron <scaron@umich.edu> wrote:

> Hi all,
> 
> My site is very happily using MD RAID arrays constructed in two
> levels. That is to say, we do RAID 60 by constructing an mdadm RAID 0
> out of a number of mdadm RAID 6 strings.
> 
> These work great but one thing we have found and been living with is
> that, when the system boots, mdadm will automatically start all the
> RAID 6 strings, but it will not automatically start the top level RAID
> 0 container. The machine pauses on boot ("Hit S to skip, M to manually
> resolve") trying to mount the RAID because the device is not
> available, and a sys admin needs to manually go to the console and
> start the top level RAID 0 to continue the boot.

I had RAID5/6 with some members being RAID0s. IIRC it was a simple matter of
listing them in the correct order of initialization in mdadm.conf. Not sure if
this was strictly required, just that I did so right away (expecting that it
would be), and everything simply worked.

On the other hand, these days you likely have systemd, and it never misses its
chance to make people's lives much more interesting than that.

-- 
With respect,
Roman

Re: Automatically start two-level mdadm RAID arrays (i.e. RAID 60) on boot?

[Phil Turmel]

On 02/15/2018 04:02 PM, Roman Mamedov wrote:
> On Thu, 15 Feb 2018 15:41:37 -0500
> Sean Caron <scaron@umich.edu> wrote:
> 
>> Hi all,
>>
>> My site is very happily using MD RAID arrays constructed in two
>> levels. That is to say, we do RAID 60 by constructing an mdadm RAID 0
>> out of a number of mdadm RAID 6 strings.
>>
>> These work great but one thing we have found and been living with is
>> that, when the system boots, mdadm will automatically start all the
>> RAID 6 strings, but it will not automatically start the top level RAID
>> 0 container. The machine pauses on boot ("Hit S to skip, M to manually
>> resolve") trying to mount the RAID because the device is not
>> available, and a sys admin needs to manually go to the console and
>> start the top level RAID 0 to continue the boot.
> 
> I had RAID5/6 with some members being RAID0s. IIRC it was a simple matter of
> listing them in the correct order of initialization in mdadm.conf. Not sure if
> this was strictly required, just that I did so right away (expecting that it
> would be), and everything simply worked.
> 
> On the other hand, these days you likely have systemd, and it never misses its
> chance to make people's lives much more interesting than that.
> 

No, it sounds like an old distro that isn't using udev and incremental
assembly.  I don't know of a solution.

Phil

Re: Automatically start two-level mdadm RAID arrays (i.e. RAID 60) on boot?

[Brad Campbell]

On 16/02/18 05:51, Phil Turmel wrote:
> 
> No, it sounds like an old distro that isn't using udev and incremental
> assembly.  I don't know of a solution.

I use several of those (old distros that is). In the initramfs mdadm 
waits for the devices to become available, the starts the arrays in 
order as listed in the mdadm.conf. I've never had an issue failing to 
start a stacked array and I don't use incremental assembly.

You do need to make sure they are listed in the correct order in the 
config file, but even mdadm --detail --scan seems to spit out the arrays 
in the correct order (or maybe I just get lucky).

Brad

Re: Automatically start two-level mdadm RAID arrays (i.e. RAID 60) on boot?

[Sean Caron]

On Thu, Feb 15, 2018 at 8:41 PM, Brad Campbell
<lists2009@fnarfbargle.com> wrote:
> On 16/02/18 05:51, Phil Turmel wrote:
>>
>>
>> No, it sounds like an old distro that isn't using udev and incremental
>> assembly.  I don't know of a solution.
>
>
> I use several of those (old distros that is). In the initramfs mdadm waits
> for the devices to become available, the starts the arrays in order as
> listed in the mdadm.conf. I've never had an issue failing to start a stacked
> array and I don't use incremental assembly.
>
> You do need to make sure they are listed in the correct order in the config
> file, but even mdadm --detail --scan seems to spit out the arrays in the
> correct order (or maybe I just get lucky).
>
> Brad
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hi all,

Thanks for the responses. I should say I'm on Ubuntu 14.04 LTS right
now, mdadm 3.2.5ubuntu4.4, with a 3.13 Ubuntu kernel, but I've
observed this behavior since we started using nested mdadm arrays at
this site back in 2011 on kernel 2.6.X and some long-since-forgotten
older version of mdadm on Ubuntu 12. It's been a little while and my
experience is not as extensive as with Ubuntu but IIRC, I have had the
same problem on CentOS 6 and CentOS 7 with stock kernels on those
distros.

As it is now, we just populate /etc/mdadm/mdadm.conf ARRAY lines with
the output of mdadm --examine --scan after building all the RAID 6
strings and the top level RAID 0 container. This gives the result
where the RAID 0 containers are the last lines in the file, after all
the RAID 6 strings. For example, on a machine with RAID 6 strings
md0...4 and RAID 0 string md5 (containing md0...md4), we have
mdadm.conf contents like:

ARRAY /dev/md/3 metadata=1.2 UUID=4a3cf66a:68cf425a:e139582e:c5c2f782
name=nodename:3
ARRAY /dev/md/2 metadata=1.2 UUID=84def8a0:d866133b:fbb473f5:72230b88
name=nodename:2
ARRAY /dev/md/1 metadata=1.2 UUID=3f3a1b81:c38a008a:331eebd9:22fceef4
name=nodename:1
ARRAY /dev/md/0 metadata=1.2 UUID=02d98ae0:e03f1424:8fc7c8d6:9f8d65ef
name=nodename:0
ARRAY /dev/md/4 metadata=1.2 UUID=b0079313:e6837042:53743e3e:c0cbfc17
name=nodename:4
ARRAY /dev/md/5 metadata=1.2 UUID=daffa41a:5198c27c:e76bd140:95054619
name=nodename:5

It works in the sense that, all RAID 6 strings start automatically on
boot, and the RAID 0 string starts with no problems given manual
operator intervention, but the RAID 0 string will not start
automatically on boot after all the RAID 6 strings have come up. This
has been persistent behavior across many kernel revisions, mdadm
revisions, udev revisions and so on.

It's interesting that everyone has reported nested mdadm arrays
starting automatically. Is there something special in mdadm.conf to
enable this? I was starting to suspect mdadm only does a single pass
when trying to auto-start arrays, but everyone's reports make me
wonder if I'm just not configuring something quite right.

Best,

Sean

Re: Automatically start two-level mdadm RAID arrays (i.e. RAID 60) on boot?

[Brad Campbell]

On 16/02/18 10:17, Sean Caron wrote:

> It's interesting that everyone has reported nested mdadm arrays
> starting automatically. Is there something special in mdadm.conf to
> enable this? I was starting to suspect mdadm only does a single pass
> when trying to auto-start arrays, but everyone's reports make me
> wonder if I'm just not configuring something quite right.

I've just checked onne of my Ubuntu 14 VMs and it still uses 
initramfs-utils. Try this :

Boot with break=top on the kernel cmd line. That'll drop you into the 
initramfs at the point where all your modules should be loaded but the 
scripts in /scripts/local-top/ haven't been executed yet. You can then 
run them manually (in the order specified in the ORDER file). Run the 
mdadm script with sh -x mdadm and watch what it does.

I'm using a Devuan Jessie (so basically a Debian Jessie without systemd) 
system on my test box.

test:/# mdadm --version
mdadm - v3.3.2 - 21st August 2014

Here's the current mdadm.conf :

test:/# cat /etc/mdadm/mdadm.conf
ARRAY /dev/md/1 metadata=1.0 name=unset:1 
UUID=ab86019d:ffa93771:2091569f:726abaae
ARRAY /dev/md3 metadata=1.2 name=test:3 
UUID=dbbca7b5:327751b1:895f8f11:443f6ecb
ARRAY /dev/md4 metadata=1.2 name=test:4 
UUID=1153a2e1:dc24b66d:45e319b3:333b6285
ARRAY /dev/md/2 metadata=1.2 name=unset:2 
UUID=84c30044:f3e879cd:6e492af6:f192b576
MAILADDR root

test:/# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md2 : active raid1 md4[2](W) nvme0n1[3]
       243542016 blocks super 1.2 [2/2] [UU]
       bitmap: 1/2 pages [4KB], 65536KB chunk

md4 : active raid0 sdf[0]
       249927680 blocks super 1.2 512k chunks

md3 : active raid6 sdg[0] sdm[8] sdc[7] sde[6] sdi[5] sdk[4] sdl[3] 
sdj[2] sdh[1]
       13673684416 blocks super 1.2 level 6, 64k chunk, algorithm 2 
[9/9] [UUUUUUUUU]
       bitmap: 0/15 pages [0KB], 65536KB chunk

md1 : active raid1 sdb1[2] sda1[3]
       524224 blocks super 1.0 [2/2] [UU]


So md2 is a RAID1 made from md4 (a RAID0) and an nvme drive.
I will say over the years I've made some pretty pathological test 
configurations with multiple stacks as far as 4 md devices deep and I've 
never had an issue with them starting in sequence.

Trace your boot sequence and bring the box up by hand to find out why 
it's not auto-starting.

Brad

Re: Automatically start two-level mdadm RAID arrays (i.e. RAID 60) on boot?

[Brad Campbell]

On 16/02/18 11:53, Brad Campbell wrote:
> On 16/02/18 10:17, Sean Caron wrote:
> 
>> It's interesting that everyone has reported nested mdadm arrays
>> starting automatically. Is there something special in mdadm.conf to
>> enable this? I was starting to suspect mdadm only does a single pass
>> when trying to auto-start arrays, but everyone's reports make me
>> wonder if I'm just not configuring something quite right.
> 
> I've just checked onne of my Ubuntu 14 VMs and it still uses 
> initramfs-utils. Try this :
> 
> Boot with break=top on the kernel cmd line. That'll drop you into the 
> initramfs at the point where all your modules should be loaded but the 
> scripts in /scripts/local-top/ haven't been executed yet. You can then 
> run them manually (in the order specified in the ORDER file). Run the 
> mdadm script with sh -x mdadm and watch what it does.

Actually, before you try that make sure /etc/default/mdadm has :

INITRDSTART='all'

in it.

Brad

Re: Automatically start two-level mdadm RAID arrays (i.e. RAID 60) on boot?

[Wolfgang Denk]

Dear Sean,

In message <CAA43vkW82nJXmwo7=HAhO+shx_Obak_tm7Vy0EKWW+HWW6brnw@mail.gmail.com> you wrote:
>
> As it is now, we just populate /etc/mdadm/mdadm.conf ARRAY lines with
> the output of mdadm --examine --scan after building all the RAID 6
> strings and the top level RAID 0 container. This gives the result
> where the RAID 0 containers are the last lines in the file, after all
> the RAID 6 strings. For example, on a machine with RAID 6 strings
> md0...4 and RAID 0 string md5 (containing md0...md4), we have
> mdadm.conf contents like:

Just in case it is not clear (you didn't mention you did): after
generating /etc/mdadm/mdadm.conf you _must_ also rebuild the
initramfs image for your running kernel, so the new mdadm.conf gets
included there, where it is actually needed.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
I don't know if it's what you want, but it's what you get.  :-)
                      - Larry Wall in <10502@jpl-devvax.JPL.NASA.GOV>

Re: Automatically start two-level mdadm RAID arrays (i.e. RAID 60) on boot?

[Sean Caron]

On Fri, Feb 16, 2018 at 6:00 AM, Wolfgang Denk <wd@denx.de> wrote:
> Dear Sean,
>
> In message <CAA43vkW82nJXmwo7=HAhO+shx_Obak_tm7Vy0EKWW+HWW6brnw@mail.gmail.com> you wrote:
>>
>> As it is now, we just populate /etc/mdadm/mdadm.conf ARRAY lines with
>> the output of mdadm --examine --scan after building all the RAID 6
>> strings and the top level RAID 0 container. This gives the result
>> where the RAID 0 containers are the last lines in the file, after all
>> the RAID 6 strings. For example, on a machine with RAID 6 strings
>> md0...4 and RAID 0 string md5 (containing md0...md4), we have
>> mdadm.conf contents like:
>
> Just in case it is not clear (you didn't mention you did): after
> generating /etc/mdadm/mdadm.conf you _must_ also rebuild the
> initramfs image for your running kernel, so the new mdadm.conf gets
> included there, where it is actually needed.
>
> Best regards,
>
> Wolfgang Denk
>

We always run update-initramfs -c -v -k "<kernelver>" after generating
the mdadm.conf file as part of our build procedure. The Ubuntu package
manager also kicks off a triggered update-initramfs every time a new
version of linux-image-generic is installed, so it's included on
subsequent kernel updates. I feel like none of the arrays would start
if mdadm.conf were not included in the initrd?

Best,

Sean