Array over arrays - how to autoassemble correctly?

Array over arrays - how to autoassemble correctly?

[Kristleifur Daðason]

[-- Attachment #1: Type: text/plain, Size: 742 bytes --]

Hi all,

I'm running a RAID-60 array with mdadm. There are two RAID6 arrays,
/dev/md/d0 and d1, and then a RAID0 "meta-array" over that,
/dev/md/tank.

What happens on boot is that the d0 and d1 arrays are assembled, but
the tank array isn't - Ubuntu drops me to a root shell because it
couldn't find the tank partition for mounting from fstab. However, I
just have to do a simple 'mdadm --assemble --scan', and the tank array
pops up instantly and everything is fine.

How do I instruct mdadm to wait for d0 and d1 and do kind of a second
assembly pass to construct tank over them?

/proc/mdstat, output of 'mdadm --detail --scan' and
/etc/mdadm/mdadm.conf are included. I hope that's enough background
information.

Thanks!

-- Kristleifur

[-- Attachment #2: proc.mdstat.txt --]
[-- Type: text/plain, Size: 501 bytes --]

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md127 : active raid0 md_d0[0] md_d1[1]
      11720314368 blocks super 1.1 256k chunks

md_d0 : active raid6 sdh1[2] sdg1[0] sdm1[4] sdk1[5] sdj1[3] sdi1[1]
      5860157440 blocks super 1.1 level 6, 256k chunk, algorithm 2 [6/6] [UUUUUU]

md_d1 : active raid6 sdf1[1] sde1[0] sdl1[3] sdd1[6] sdc1[5] sdb1[4]
      5860157440 blocks super 1.1 level 6, 256k chunk, algorithm 2 [6/6] [UUUUUU]

unused devices: <none>


[-- Attachment #3: mdadm.detail.scan.txt --]
[-- Type: text/plain, Size: 263 bytes --]

ARRAY /dev/md/d1 metadata=1.01 name=mamma:d1 UUID=da547022:042a6f68:d5fe251e:5e89f263
ARRAY /dev/md/d0 metadata=1.01 name=mamma:d0 UUID=7f98e926:206a8ebe:59fc8d7b:4644a883
ARRAY /dev/md/tank metadata=1.01 name=mamma:tank UUID=6f98a7a8:d7b50181:587c27d1:59d6d137


[-- Attachment #4: mdadm.conf.txt --]
[-- Type: text/plain, Size: 892 bytes --]

# mdadm.conf
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions /dev/md/d0 /dev/md/d1

# auto-create devices with Debian standard permissions
# CREATE owner=root group=disk mode=0660 auto=mdp

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# definitions of existing MD arrays

ARRAY /dev/md/d0 metadata=1.1 name=mamma:d0 UUID=7f98e926:206a8ebe:59fc8d7b:4644a883 level=raid6 num-devices=6 # container=6f98a7a8:d7b50181:587c27d1:59d6d137

ARRAY /dev/md/d1 metadata=1.1 name=mamma:d1 UUID=da547022:042a6f68:d5fe251e:5e89f263 level=raid6 num-devices=6 # container=6f98a7a8:d7b50181:587c27d1:59d6d137

ARRAY /dev/md/tank metadata=1.1 name=mamma:tank UUID=6f98a7a8:d7b50181:587c27d1:59d6d137 level=raid0 num-devices=2 devices=/dev/md/d0,/dev/md/d1

Re: Array over arrays - how to autoassemble correctly?

[Luca Berra]

On Wed, Jan 27, 2010 at 06:32:57PM +0000, Kristleifur Daðason wrote:
># mdadm.conf
>#
>
># by default, scan all partitions (/proc/partitions) for MD superblocks.
># alternatively, specify devices to scan, using wildcards if desired.
>DEVICE partitions /dev/md/d0 /dev/md/d1
>
># auto-create devices with Debian standard permissions
># CREATE owner=root group=disk mode=0660 auto=mdp
>
># automatically tag new arrays as belonging to the local system
>HOMEHOST <system>
>
># definitions of existing MD arrays
>
>ARRAY /dev/md/d0 metadata=1.1 name=mamma:d0 UUID=7f98e926:206a8ebe:59fc8d7b:4644a883 level=raid6 num-devices=6 # container=6f98a7a8:d7b50181:587c27d1:59d6d137
>
>ARRAY /dev/md/d1 metadata=1.1 name=mamma:d1 UUID=da547022:042a6f68:d5fe251e:5e89f263 level=raid6 num-devices=6 # container=6f98a7a8:d7b50181:587c27d1:59d6d137
>
>ARRAY /dev/md/tank metadata=1.1 name=mamma:tank UUID=6f98a7a8:d7b50181:587c27d1:59d6d137 level=raid0 num-devices=2 devices=/dev/md/d0,/dev/md/d1
>

this kind of setup should be working without any issue on recent mdadm
try removing
DEVICE partitions /dev/md/d0 /dev/md/d1
and the "devices=/dev/md/d0,/dev/md/d1" from mdadm.conf
and let it discover things on boot by itself

L.


-- 
Luca Berra -- bluca@comedia.it
         Communication Media & Services S.r.l.
  /"\
  \ /     ASCII RIBBON CAMPAIGN
   X        AGAINST HTML MAIL
  / \
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Array over arrays - how to autoassemble correctly?

[Kristleifur Daðason]

On Wed, Jan 27, 2010 at 7:41 PM, Luca Berra <bluca@comedia.it> wrote:
> On Wed, Jan 27, 2010 at 06:32:57PM +0000, Kristleifur Dađason wrote:
>>
>> # mdadm.conf
>> #
>>
>> # by default, scan all partitions (/proc/partitions) for MD superblocks.
>> # alternatively, specify devices to scan, using wildcards if desired.
>> DEVICE partitions /dev/md/d0 /dev/md/d1
>>
>> # auto-create devices with Debian standard permissions
>> # CREATE owner=root group=disk mode=0660 auto=mdp
>>
>> # automatically tag new arrays as belonging to the local system
>> HOMEHOST <system>
>>
>> # definitions of existing MD arrays
>>
>> ARRAY /dev/md/d0 metadata=1.1 name=mamma:d0
>> UUID=7f98e926:206a8ebe:59fc8d7b:4644a883 level=raid6 num-devices=6 #
>> container=6f98a7a8:d7b50181:587c27d1:59d6d137
>>
>> ARRAY /dev/md/d1 metadata=1.1 name=mamma:d1
>> UUID=da547022:042a6f68:d5fe251e:5e89f263 level=raid6 num-devices=6 #
>> container=6f98a7a8:d7b50181:587c27d1:59d6d137
>>
>> ARRAY /dev/md/tank metadata=1.1 name=mamma:tank
>> UUID=6f98a7a8:d7b50181:587c27d1:59d6d137 level=raid0 num-devices=2
>> devices=/dev/md/d0,/dev/md/d1
>>
>
> this kind of setup should be working without any issue on recent mdadm
> try removing
> DEVICE partitions /dev/md/d0 /dev/md/d1
> and the "devices=/dev/md/d0,/dev/md/d1" from mdadm.conf
> and let it discover things on boot by itself
>
> L.
>

Many thanks for the reply!

However, that's how the mdadm.conf looked like until today - adding
the DEVICE and devices stuff is something I put in to get things to
autoassemble themselves fully, as I thought 'man mdadm.conf' indicated
it.

Still, I'll certianly try - with and without - and double-checking
that my initramfs is OK.

Any further input most well appreciated.

-- Kristleifur
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Array over arrays - how to autoassemble correctly?

[Goswin von Brederlow]

Kristleifur Daðason <kristleifur@gmail.com> writes:

> On Wed, Jan 27, 2010 at 7:41 PM, Luca Berra <bluca@comedia.it> wrote:
>> On Wed, Jan 27, 2010 at 06:32:57PM +0000, Kristleifur Dađason wrote:
>>>
>>> # mdadm.conf
>>> #
>>>
>>> # by default, scan all partitions (/proc/partitions) for MD superblocks.
>>> # alternatively, specify devices to scan, using wildcards if desired.
>>> DEVICE partitions /dev/md/d0 /dev/md/d1
>>>
>>> # auto-create devices with Debian standard permissions
>>> # CREATE owner=root group=disk mode=0660 auto=mdp
>>>
>>> # automatically tag new arrays as belonging to the local system
>>> HOMEHOST <system>
>>>
>>> # definitions of existing MD arrays
>>>
>>> ARRAY /dev/md/d0 metadata=1.1 name=mamma:d0
>>> UUID=7f98e926:206a8ebe:59fc8d7b:4644a883 level=raid6 num-devices=6 #
>>> container=6f98a7a8:d7b50181:587c27d1:59d6d137
>>>
>>> ARRAY /dev/md/d1 metadata=1.1 name=mamma:d1
>>> UUID=da547022:042a6f68:d5fe251e:5e89f263 level=raid6 num-devices=6 #
>>> container=6f98a7a8:d7b50181:587c27d1:59d6d137
>>>
>>> ARRAY /dev/md/tank metadata=1.1 name=mamma:tank
>>> UUID=6f98a7a8:d7b50181:587c27d1:59d6d137 level=raid0 num-devices=2
>>> devices=/dev/md/d0,/dev/md/d1
>>>
>>
>> this kind of setup should be working without any issue on recent mdadm
>> try removing
>> DEVICE partitions /dev/md/d0 /dev/md/d1
>> and the "devices=/dev/md/d0,/dev/md/d1" from mdadm.conf
>> and let it discover things on boot by itself
>>
>> L.
>>
>
> Many thanks for the reply!
>
> However, that's how the mdadm.conf looked like until today - adding
> the DEVICE and devices stuff is something I put in to get things to
> autoassemble themselves fully, as I thought 'man mdadm.conf' indicated
> it.
>
> Still, I'll certianly try - with and without - and double-checking
> that my initramfs is OK.
>
> Any further input most well appreciated.

Alternative add a scriplet into the initramfs config that calls the
mdadm assembling a second time.

MfG
        Goswin
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Array over arrays - how to autoassemble correctly?

[Luca Berra]

On Wed, Jan 27, 2010 at 06:32:57PM +0000, Kristleifur Daðason wrote:
>Hi all,
>
>I'm running a RAID-60 array with mdadm. There are two RAID6 arrays,
>/dev/md/d0 and d1, and then a RAID0 "meta-array" over that,
>/dev/md/tank.
>
>What happens on boot is that the d0 and d1 arrays are assembled, but
>the tank array isn't - Ubuntu drops me to a root shell because it
>couldn't find the tank partition for mounting from fstab. However, I
>just have to do a simple 'mdadm --assemble --scan', and the tank array
>pops up instantly and everything is fine.
>
>How do I instruct mdadm to wait for d0 and d1 and do kind of a second
>assembly pass to construct tank over them?
>
>/proc/mdstat, output of 'mdadm --detail --scan' and
>/etc/mdadm/mdadm.conf are included. I hope that's enough background
>information.

I tested it again, "mdadm --assemble --scan" is able to start stacked
arrays correctly either with a configuration similar to yours or with no
configuration at all !!!
Maybe it is due to how ubuntu tries to assemble your array, i would
guess through udev rules.
Maybe the rules are triggered for real devices (/dev/sd*) and not for
/dev/md*, but im not running ubuntu, so i have no way to know.

Regards,
L.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html