Constructing RAID 1+0 w/ LVM2/DM

Constructing RAID 1+0 w/ LVM2/DM

[Alan D. Brunelle]

I'm attempting to create a RAID 1+0 environment using only LVM/DM and 
running into a strange problem: I can't use the underlying mirrored 
devices. I was able to create a RAID 0+1 environment easy enough, so I'm 
perplexed by this.

I have 24 devices (12 each behind an FC RAID array), I first pvcreate those:

/sbin/pvcreate -f -y /dev/sdu
/sbin/pvcreate -f -y /dev/sdw
/sbin/pvcreate -f -y /dev/sdx
/sbin/pvcreate -f -y /dev/sdz
/sbin/pvcreate -f -y /dev/sdab
/sbin/pvcreate -f -y /dev/sdad
/sbin/pvcreate -f -y /dev/sdaf
/sbin/pvcreate -f -y /dev/sdah
/sbin/pvcreate -f -y /dev/sdaj
/sbin/pvcreate -f -y /dev/sdal
/sbin/pvcreate -f -y /dev/sdao
/sbin/pvcreate -f -y /dev/sdap
/sbin/pvcreate -f -y /dev/sds
/sbin/pvcreate -f -y /dev/sdt
/sbin/pvcreate -f -y /dev/sdv
/sbin/pvcreate -f -y /dev/sdy
/sbin/pvcreate -f -y /dev/sdaa
/sbin/pvcreate -f -y /dev/sdac
/sbin/pvcreate -f -y /dev/sdae
/sbin/pvcreate -f -y /dev/sdag
/sbin/pvcreate -f -y /dev/sdai
/sbin/pvcreate -f -y /dev/sdak
/sbin/pvcreate -f -y /dev/sdam
/sbin/pvcreate -f -y /dev/sdan

I then mirror pairs of those:

/sbin/vgcreate vg00 /dev/sdu /dev/sds
/sbin/lvcreate -m 1 --corelog --nosync -n lv00 -L 65G vg00
/sbin/vgcreate vg01 /dev/sdw /dev/sdt
/sbin/lvcreate -m 1 --corelog --nosync -n lv01 -L 65G vg01
/sbin/vgcreate vg02 /dev/sdx /dev/sdv
/sbin/lvcreate -m 1 --corelog --nosync -n lv02 -L 65G vg02
/sbin/vgcreate vg03 /dev/sdz /dev/sdy
/sbin/lvcreate -m 1 --corelog --nosync -n lv03 -L 65G vg03
/sbin/vgcreate vg04 /dev/sdab /dev/sdaa
/sbin/lvcreate -m 1 --corelog --nosync -n lv04 -L 65G vg04
/sbin/vgcreate vg05 /dev/sdad /dev/sdac
/sbin/lvcreate -m 1 --corelog --nosync -n lv05 -L 65G vg05
/sbin/vgcreate vg06 /dev/sdaf /dev/sdae
/sbin/lvcreate -m 1 --corelog --nosync -n lv06 -L 65G vg06
/sbin/vgcreate vg07 /dev/sdah /dev/sdag
/sbin/lvcreate -m 1 --corelog --nosync -n lv07 -L 65G vg07
/sbin/vgcreate vg08 /dev/sdaj /dev/sdai
/sbin/lvcreate -m 1 --corelog --nosync -n lv08 -L 65G vg08
/sbin/vgcreate vg09 /dev/sdal /dev/sdak
/sbin/lvcreate -m 1 --corelog --nosync -n lv09 -L 65G vg09
/sbin/vgcreate vg10 /dev/sdao /dev/sdam
/sbin/lvcreate -m 1 --corelog --nosync -n lv10 -L 65G vg10
/sbin/vgcreate vg11 /dev/sdap /dev/sdan
/sbin/lvcreate -m 1 --corelog --nosync -n lv11 -L 65G vg11

Now, I'd like to stripe over those mirrors, and would expect that the 
first step would be to pvcreate the new lv's  (lv00-lv11), before doing so:

# lvs
  LV   VG   Attr   LSize  Origin Snap%  Move Log Copy%
  lv00 vg00 Mwi-a- 65.00G                        100.00
  lv01 vg01 Mwi-a- 65.00G                        100.00
  lv02 vg02 Mwi-a- 65.00G                        100.00
  lv03 vg03 Mwi-a- 65.00G                        100.00
  lv04 vg04 Mwi-a- 65.00G                        100.00
  lv05 vg05 Mwi-a- 65.00G                        100.00
  lv06 vg06 Mwi-a- 65.00G                        100.00
  lv07 vg07 Mwi-a- 65.00G                        100.00
  lv08 vg08 Mwi-a- 65.00G                        100.00
  lv09 vg09 Mwi-a- 65.00G                        100.00

# vgs
  VG   #PV #LV #SN Attr   VSize   VFree
  vg00   2   3   0 wz--n- 135.66G 5.66G
  vg01   2   3   0 wz--n- 135.66G 5.66G
  vg02   2   3   0 wz--n- 135.66G 5.66G
  vg03   2   3   0 wz--n- 135.66G 5.66G
  vg04   2   3   0 wz--n- 135.66G 5.66G
  vg05   2   3   0 wz--n- 135.66G 5.66G
  vg06   2   3   0 wz--n- 135.66G 5.66G
  vg07   2   3   0 wz--n- 135.66G 5.66G
  vg08   2   3   0 wz--n- 135.66G 5.66G
  vg09   2   3   0 wz--n- 135.66G 5.66G

But then:

# pvcreate -ff -y -v /dev/vg00/lv00
    Wiping cache of LVM-capable devices
    Wiping internal VG cache
  Device /dev/vg00/lv00 not found (or ignored by filtering).

I did run w/ -vvv, and extracted this pertinent looking stuff:

    Wiping internal VG cache
      Locking /var/lock/lvm/P_orphans WB
        dm status   (253:2) OF   [16384]
        /dev/dm-2: Skipping: Suspended dm device
  Device /dev/vg00/lv00 not found (or ignored by filtering).
      Unlocking /var/lock/lvm/P_orphans

I don't understand why the device is suspended - due to the --nosync the 
copy% is at 100, and the device read/writes just fine. I'm using:

# pvcreate --version
  LVM version:     2.02.29 (2007-12-05)
  Library version: 1.02.24 (2007-12-20)
  Driver version:  4.12.0

and 2.6.24-rc6 kernel.

As noted previously, I could invert the operation, and things worked 
fine (RAID 0+1 - mirror 2 stripes constructed from 12 devices each).

Any help would be much appreciated, thanks,
Alan

Re: Constructing RAID 1+0 w/ LVM2/DM

[Alasdair G Kergon]

On Thu, Jan 24, 2008 at 09:16:02AM -0500, Alan D. Brunelle wrote:
>        /dev/dm-2: Skipping: Suspended dm device

That code was never completed.  It applies a stronger test than
necessary and assumes no mirrors are safe to use, even though some
would be.

Alasdair
-- 
agk@redhat.com

Re: Constructing RAID 1+0 w/ LVM2/DM

[Alan D. Brunelle]

Alasdair G Kergon wrote:
> On Thu, Jan 24, 2008 at 09:16:02AM -0500, Alan D. Brunelle wrote:
>>        /dev/dm-2: Skipping: Suspended dm device
>
> That code was never completed.  It applies a stronger test than
> necessary and assumes no mirrors are safe to use, even though some
> would be.
>
> Alasdair
So, does that mean that one can not construct a RAID 1+0 device using 
LVM2/DM in such a manner? Is there some way to override this behavior? 
Is it the 'ignore_suspended_devices() or !device_is_usable() call in the 
following causing the problem:

        /* Skip suspended devices */
        if (MAJOR(dev->dev) == _device_mapper_major &&
            ignore_suspended_devices() && !device_is_usable(dev->dev)) {
                log_debug("%s: Skipping: Suspended dm device", name);
                return 0;
        }

Thanks for the answer though...
Alan

Re: Constructing RAID 1+0 w/ LVM2/DM

[Alasdair G Kergon]

On Thu, Jan 24, 2008 at 09:41:18AM -0500, Alan D. Brunelle wrote:
>            ignore_suspended_devices() && !device_is_usable(dev->dev)) {

If you know what you're doing and using upstream kernels without dmeventd, drop
the mirror test from device_is_usable().

(Are all those corelogs just while testing?)

Alasdair
-- 
agk@redhat.com

Re: Constructing RAID 1+0 w/ LVM2/DM

[Alan D. Brunelle]

Alasdair G Kergon wrote:
> On Thu, Jan 24, 2008 at 09:41:18AM -0500, Alan D. Brunelle wrote:
>   
>>            ignore_suspended_devices() && !device_is_usable(dev->dev)) {
>>     
>
> If you know what you're doing and using upstream kernels without dmeventd, drop
> the mirror test from device_is_usable().
>   

Thanks Alasdair!

> (Are all those corelogs just while testing?)
>   

Most assuredly so...

Alan

> Alasdair
>