From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx2.redhat.com (mx2.redhat.com [10.255.15.25]) by int-mx2.corp.redhat.com (8.13.1/8.13.1) with ESMTP id kBSNOx5E010543 for ; Thu, 28 Dec 2006 18:24:59 -0500 Received: from wx-out-0506.google.com (wx-out-0506.google.com [66.249.82.239]) by mx2.redhat.com (8.12.11.20060308/8.12.11) with ESMTP id kBSNOw0H018013 for ; Thu, 28 Dec 2006 18:24:58 -0500 Received: by wx-out-0506.google.com with SMTP id s11so5830082wxc for ; Thu, 28 Dec 2006 15:24:58 -0800 (PST) Message-ID: <859a78260612281524u56385ab6x71a29e45edf4dfbe@mail.gmail.com> Date: Thu, 28 Dec 2006 17:24:57 -0600 From: "Matt P" Subject: Re: [linux-lvm] Mirror between different SAN fabrics In-Reply-To: <45943EDA.5080209@nrel.colostate.edu> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_73706_20085825.1167348297830" References: <2421805.1167306770765.JavaMail.mailer@post.ch> <45940A96.5040006@nrel.colostate.edu> <859a78260612281130m761fce9rf32240b523471608@mail.gmail.com> <45943EDA.5080209@nrel.colostate.edu> Reply-To: LVM general discussion and development List-Id: LVM general discussion and development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , List-Id: To: LVM general discussion and development ------=_Part_73706_20085825.1167348297830 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline That's exactly the steps I used to test it out. The failure I encountered was at the point of adding the StripeLVs to the MirrorVG. I don't recall the error I got but I seem to recall it being the same error as if the PV hadn't been pvcreated. Although when I went through the process again just now, without any errors... It's quite likely I mistyped or skipped a step during my first attempt... I've gone through the whole process 4 times now from raw disk to a mirrored LV without getting the error again. It looks as though this could work and isn't nearly as crazy as my initial configuration. I wonder now if anything wonky will happen if we were to lose a disk or in the original post if we lost one of the fabrics... On 12/28/06, Ty! Boyack wrote: > > Actually, I'm quite surprised that this approach mangles the lvm data. > It seems that when you do a pvcreate on a block device, LVM should (and > I think, does) write the lvm metadata in a region of that device, and > then never let anything touch that metadata. This way, if you do a 'dd > if=/dev/zeros of=' it blanks out the device, but the metadata > is intact. > > So, if you do a 'pvcreate' on an LV, it should contain a second copy of > the metadata, unique and independent from the first copy on the original > block device. My tests on this has worked fine (although my tests have > been for building two VGs that have striped volumes across the member > disks, and then a VG that creates a mirrored LV of the striped volumes, > so no multipathing is involved). I'm wondering if we can compare notes > to see if I'm doing something that makes it look like it's working -- I > don't want to be quietly destroying my lvm data and not knowing it!!! > > I'm doing (roughly, block devices are a bit made-up): > > # prepare the physical volumes > pvcreate /dev/sda > pvcreate /dev/sdb > pvcreate /dev/sdc > pvcreate /dev/sdd > pvcreate /dev/sde > > # Create volume groups to contain uniquely striped volumes > vgcreate Stripe1VG /dev/sda /dev/sdb > vgcreate Stripe2VG /dev/sdc /dev/sdd > > # Create the striped volumes > lvcreate -i 2 -n Stripe1LV -L 1G Stripe1VG > lvcreate -i 2 -n Stripe2LV -L 1G Stripe2VG > > # Make the striped volumes into PVs > pvcreate /dev/Stripe1VG/Stripe1LV > pvcreate /dev/Stripe2VG/Stripe2LV > > # Create the volume group for mirrored volumes > vgcreate MirrorVG /dev/Stripe1VG/Stripe1LV /dev/Stripe2VG/Stripe2LV > /dev/sde > # (Had to use three PVs to have the mirror log in place) > > # Create the mirrored volume > lvcreate -m 1 -n Mirror1LV -L 500M MirrorVG > > # Filesystem, test, etc. this will be GFS eventually, but testing with > ext3 for now. > mke2fs -j -i16384 -v /dev/MirrorVG/Mirror1LV > mkdir /mnt/mirror1lv > mount /dev/MirrorVG/Mirror1LV /mnt/mirror1 > > > > Is that about your procedure as well? When does the lvm data get mangled? > > (Sorry if this is going off topic - but if this is solvable it might > actually answer the original question...) > > -Ty! > > > > > Matt P wrote: > > This is basically the "messy" way I mentioned in my reply above. I > > found if you pvcreate the LV device, you end up mangling the lvm data > > (this probably comes as little surprise) and it breaks down after > > that. So, I ended up using losetup and an "image file", one for/on > > each fabric. Then did pvcreate on each loop device, and made a new VG > > containing both PVs and created the LV with mirroring.... It > > worked.... I did no performance, stability, or failure testing... > > > -- > -===========================- > Ty! Boyack > NREL Unix Network Manager > ty@nrel.colostate.edu > (970) 491-1186 > -===========================- > > _______________________________________________ > linux-lvm mailing list > linux-lvm@redhat.com > https://www.redhat.com/mailman/listinfo/linux-lvm > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ > ------=_Part_73706_20085825.1167348297830 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline That's exactly the steps I used to test it out. The failure I encountered was at the point of adding the StripeLVs to the MirrorVG. I don't recall the error I got but I seem to recall it being the same error as if the PV hadn't been pvcreated. Although when I went through the process again just now, without any errors... It's quite likely I mistyped or skipped a step during my first attempt...

I've gone through the whole process 4 times now from raw disk to a mirrored LV without getting the error again.

It looks as though this could work and isn't nearly as crazy as my initial configuration. I wonder now if anything wonky will happen if we were to lose a disk or in the original post if we lost one of the fabrics...

On 12/28/06, Ty! Boyack <ty@nrel.colostate.edu> wrote:
Actually, I'm quite surprised that this approach mangles the lvm data.
It seems that when you do a pvcreate on a block device, LVM should (and
I think, does) write the lvm metadata in a region of that device, and
then never let anything touch that metadata.  This way, if you do a 'dd
if=/dev/zeros of=<PV-DEVICE>' it blanks out the device, but the metadata
is intact.

So, if you do a 'pvcreate' on an LV, it should contain a second copy of
the metadata, unique and independent from the first copy on the original
block device.  My tests on this has worked fine (although my tests have
been for building two VGs that have striped volumes across the member
disks, and then a VG that creates a mirrored LV of the striped volumes,
so no multipathing is involved).  I'm wondering if we can compare notes
to see if I'm doing something that makes it look like it's working -- I
don't want to be quietly destroying my lvm data and not knowing it!!!

I'm doing (roughly, block devices are a bit made-up):

# prepare the physical volumes
pvcreate /dev/sda
pvcreate /dev/sdb
pvcreate /dev/sdc
pvcreate /dev/sdd
pvcreate /dev/sde

# Create volume groups to contain uniquely striped volumes
vgcreate Stripe1VG /dev/sda /dev/sdb
vgcreate Stripe2VG /dev/sdc /dev/sdd

# Create the striped volumes
lvcreate -i 2 -n Stripe1LV -L 1G Stripe1VG
lvcreate -i 2 -n Stripe2LV -L 1G Stripe2VG

# Make the striped volumes into PVs
pvcreate /dev/Stripe1VG/Stripe1LV
pvcreate /dev/Stripe2VG/Stripe2LV

# Create the volume group for mirrored volumes
vgcreate MirrorVG /dev/Stripe1VG/Stripe1LV /dev/Stripe2VG/Stripe2LV /dev/sde
# (Had to use three PVs to have the mirror log in place)

# Create the mirrored volume
lvcreate -m 1 -n Mirror1LV -L 500M MirrorVG

# Filesystem, test, etc.  this will be GFS eventually, but testing with
ext3 for now.
mke2fs -j -i16384 -v /dev/MirrorVG/Mirror1LV
mkdir /mnt/mirror1lv
mount /dev/MirrorVG/Mirror1LV /mnt/mirror1



Is that about your procedure as well?  When does the lvm data get mangled?

(Sorry if this is going off topic - but if this is solvable it might
actually answer the original question...)

-Ty!




Matt P wrote:
> This is basically the "messy" way I mentioned in my reply above. I
> found if you pvcreate the  LV device, you end up mangling the lvm data
> (this probably comes as little surprise) and it breaks down after
> that. So, I ended up using losetup and an "image file", one for/on
> each fabric. Then did pvcreate on each loop device, and made a new VG
> containing both PVs and created the LV with mirroring.... It
> worked.... I did no performance, stability, or failure testing...


--
-===========================-
  Ty! Boyack
  NREL Unix Network Manager
  ty@nrel.colostate.edu
  (970) 491-1186
-===========================-

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

------=_Part_73706_20085825.1167348297830--