Use LVM on /dev/mapper/diskname

Use LVM on /dev/mapper/diskname

[S. J. van Harmelen]

Hi,

I have a couple of LUN's that are handled by the multipath driver. This
works great, but now I want to use LVM so I can take snapshots of one
LUN to another LUN.

Can someone tell me how to do this?

When I do pvcreate /dev/mapper/diskname I get an error that the disk is
already part of an volume group. But I sertainly did not configure that.

Can it be that the multipath driver of devmapper did that?

Sander

Re: Use LVM on /dev/mapper/diskname

[Tore Anderson]

* S. J. van Harmelen

> I have a couple of LUN's that are handled by the multipath driver.
> This works great, but now I want to use LVM so I can take snapshots
> of one LUN to another LUN.
> 
> Can someone tell me how to do this?
> 
> When I do pvcreate /dev/mapper/diskname I get an error that the disk
> is already part of an volume group. But I sertainly did not configure
> that.
> 
> Can it be that the multipath driver of devmapper did that?

If you have a PV signature on the volume, LVM might have used one of the
paths instead of using the multipath'ed device under /dev/mapper.  You
should be able to check this by running the command "pvs" - it should
list all the detected PVs on your system.

If this was the problem, you can avoid it by instructing LVM to not scan
SCSI devices directly, by adding a line in /etc/lvm/lvm.conf like this:

device {
  filter = [ "a|^/dev/mapper/.*|", "r|.*|" ]
}

This will make LVM only look at devices in /dev/mapper/ as possible PV
candidates, ignoring all other devices.  Beware if you have a PV on the
internal drives though, you might want to have something like this
instead in that case (if that PV is on /dev/sda2, for instance):

  filter = [ "a|^/dev/mapper/.*|", "a|^/dev/sda2$|", "r|.*|" ]

Make sure lvm.conf makes its way into the initramfs if you start the LVM
stuff there.  I know Ubuntu doesn't include this file by default, at least.

Regards
-- 
Tore Anderson

Re: Use LVM on /dev/mapper/diskname and iSCSI

[S. J. van Harmelen]

Thanks for your help!

I executed the pvs command and it did find a few partitions. I use iSCSI
to share the LUN's to a XenSource server, and XenSource creates the PV's
and LV's in order to provision VM's. Didn't think of that, but that's on
the disk of course. 

Is there any way to configure snapshots for these already created LVM
partitions? Or do I have to whipe the disk clean (rather not do that,
but if I have to it can be done), then create the pv's and lv's and then
share the lv's true iSCSI?

In that case XenSource will again create pv's and lv's in the shared lv.
Ain't that a problem?

Sander



On Mon, 2007-11-19 at 15:06 +0100, Tore Anderson wrote:
> * S. J. van Harmelen
> 
> > I have a couple of LUN's that are handled by the multipath driver.
> > This works great, but now I want to use LVM so I can take snapshots
> > of one LUN to another LUN.
> > 
> > Can someone tell me how to do this?
> > 
> > When I do pvcreate /dev/mapper/diskname I get an error that the disk
> > is already part of an volume group. But I sertainly did not configure
> > that.
> > 
> > Can it be that the multipath driver of devmapper did that?
> 
> If you have a PV signature on the volume, LVM might have used one of the
> paths instead of using the multipath'ed device under /dev/mapper.  You
> should be able to check this by running the command "pvs" - it should
> list all the detected PVs on your system.
> 
> If this was the problem, you can avoid it by instructing LVM to not scan
> SCSI devices directly, by adding a line in /etc/lvm/lvm.conf like this:
> 
> device {
>   filter = [ "a|^/dev/mapper/.*|", "r|.*|" ]
> }
> 
> This will make LVM only look at devices in /dev/mapper/ as possible PV
> candidates, ignoring all other devices.  Beware if you have a PV on the
> internal drives though, you might want to have something like this
> instead in that case (if that PV is on /dev/sda2, for instance):
> 
>   filter = [ "a|^/dev/mapper/.*|", "a|^/dev/sda2$|", "r|.*|" ]
> 
> Make sure lvm.conf makes its way into the initramfs if you start the LVM
> stuff there.  I know Ubuntu doesn't include this file by default, at least.
> 
> Regards

Re: Use LVM on /dev/mapper/diskname and iSCSI

[Tore Anderson]

* S. J. van Harmelen

> Thanks for your help!
> 
> I executed the pvs command and it did find a few partitions. I use
> iSCSI to share the LUN's to a XenSource server, and XenSource creates
> the PV's and LV's in order to provision VM's. Didn't think of that,
> but that's on the disk of course.
> 
> Is there any way to configure snapshots for these already created LVM
> partitions? Or do I have to whipe the disk clean (rather not do
> that, but if I have to it can be done), then create the pv's and lv's
> and then share the lv's true iSCSI?
> 
> In that case XenSource will again create pv's and lv's in the shared
> lv. Ain't that a problem?

You really need a filter line in /etc/lvm/lvm.conf, to keep LVM from 
claiming 1) individual paths to a dm-multipath controlled LUN, and 2) 
LUNs that are PVs used by the LVM implementation in your domUs.

You don't want LVM to be claiming the paths directly, because you'd no 
longer have any redundancy - it will override dm-multipath.  You also 
don't want the LVM implementation in your dom0 to be using the same 
block device as LVM in your domU as physical volumes, as LVM is designed 
to have exclusive access to its PVs (you might get around this with 
CLVM, though).

Do I understand correctly if you have one LUN that contains your domO 
(having its file systems in LVs), and several LUNs that contain your 
domUs, which again are PVs that are supposed to be used by your domU 
kernels?

If so you will want to keep your dom0 from claiming the domU PVs.  Use a 
filter line that accepts only the multipath'ed volume that contains the 
dom0 installation, and drop the rest.  For instance:

filter = [ "a|^/dev/mapper/domzero$", "r|.*" ]

...assuming your dom0 data volume is called "domzero" in multipath.conf.

Regards
-- 
Tore Anderson

Re: Use LVM on /dev/mapper/diskname and iSCSI

[S. J. van Harmelen]

Tore,

Please take a look at my reply to Malaha. This is also a reply to your
message...

Hope you have the time to read it.

Thanks,

Sander


On Mon, 2007-11-19 at 19:43 +0100, Tore Anderson wrote:
> * S. J. van Harmelen
> 
> > Thanks for your help!
> > 
> > I executed the pvs command and it did find a few partitions. I use
> > iSCSI to share the LUN's to a XenSource server, and XenSource creates
> > the PV's and LV's in order to provision VM's. Didn't think of that,
> > but that's on the disk of course.
> > 
> > Is there any way to configure snapshots for these already created LVM
> > partitions? Or do I have to whipe the disk clean (rather not do
> > that, but if I have to it can be done), then create the pv's and lv's
> > and then share the lv's true iSCSI?
> > 
> > In that case XenSource will again create pv's and lv's in the shared
> > lv. Ain't that a problem?
> 
> You really need a filter line in /etc/lvm/lvm.conf, to keep LVM from 
> claiming 1) individual paths to a dm-multipath controlled LUN, and 2) 
> LUNs that are PVs used by the LVM implementation in your domUs.
> 
> You don't want LVM to be claiming the paths directly, because you'd no 
> longer have any redundancy - it will override dm-multipath.  You also 
> don't want the LVM implementation in your dom0 to be using the same 
> block device as LVM in your domU as physical volumes, as LVM is designed 
> to have exclusive access to its PVs (you might get around this with 
> CLVM, though).
> 
> Do I understand correctly if you have one LUN that contains your domO 
> (having its file systems in LVs), and several LUNs that contain your 
> domUs, which again are PVs that are supposed to be used by your domU 
> kernels?
> 
> If so you will want to keep your dom0 from claiming the domU PVs.  Use a 
> filter line that accepts only the multipath'ed volume that contains the 
> dom0 installation, and drop the rest.  For instance:
> 
> filter = [ "a|^/dev/mapper/domzero$", "r|.*" ]
> 
> ...assuming your dom0 data volume is called "domzero" in multipath.conf.
> 
> Regards

Re: Use LVM on /dev/mapper/diskname and iSCSI

[Chip Coldwell]

On Mon, 19 Nov 2007, S. J. van Harmelen wrote:

> Tore,
> 
> Please take a look at my reply to Malaha. This is also a reply to your
> message...
> 
> Hope you have the time to read it.
> 
> Thanks,
> 
> Sander
> 
> 
> On Mon, 2007-11-19 at 19:43 +0100, Tore Anderson wrote:
> > * S. J. van Harmelen
> > 
> > > Thanks for your help!
> > > 
> > > I executed the pvs command and it did find a few partitions. I use
> > > iSCSI to share the LUN's to a XenSource server, and XenSource creates
> > > the PV's and LV's in order to provision VM's. Didn't think of that,
> > > but that's on the disk of course.


Sander: As a matter of netiquette, please don't top-post.

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

Chip

-- 
Charles M. "Chip" Coldwell
Senior Software Engineer
Red Hat, Inc
978-392-2426

Re: Use LVM on /dev/mapper/diskname and iSCSI

[S. J. van Harmelen]

On Tue, 2007-11-20 at 09:14 -0500, Chip Coldwell wrote:
> On Mon, 19 Nov 2007, S. J. van Harmelen wrote:
> 
> > Tore,
> > 
> > Please take a look at my reply to Malaha. This is also a reply to your
> > message...
> > 
> > Hope you have the time to read it.
> > 
> > Thanks,
> > 
> > Sander
> > 
> > 
> > On Mon, 2007-11-19 at 19:43 +0100, Tore Anderson wrote:
> > > * S. J. van Harmelen
> > > 
> > > > Thanks for your help!
> > > > 
> > > > I executed the pvs command and it did find a few partitions. I use
> > > > iSCSI to share the LUN's to a XenSource server, and XenSource creates
> > > > the PV's and LV's in order to provision VM's. Didn't think of that,
> > > > but that's on the disk of course.
> 
> 
> Sander: As a matter of netiquette, please don't top-post.
> 
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?
> A: Top-posting.
> Q: What is the most annoying thing in e-mail?

Point made... Will do... :)

Sander

Re: Use LVM on /dev/mapper/diskname and iSCSI

[malahal]

S. J. van Harmelen [svh@dds.nl] wrote:
> Thanks for your help!
> 
> I executed the pvs command and it did find a few partitions. I use iSCSI
> to share the LUN's to a XenSource server, and XenSource creates the PV's
> and LV's in order to provision VM's. Didn't think of that, but that's on
> the disk of course. 
> 
> Is there any way to configure snapshots for these already created LVM
> partitions?

If I understand, you exported some LUNs to XenSource server. So you
should NOT be using those LUNs at this machine, right? You should be
able to create snapshots at the XenSource server though.

Re: Use LVM on /dev/mapper/diskname and iSCSI

[S. J. van Harmelen]

On Mon, 2007-11-19 at 10:50 -0800, malahal@us.ibm.com wrote:
> S. J. van Harmelen [svh@dds.nl] wrote:
> > Thanks for your help!
> > 
> > I executed the pvs command and it did find a few partitions. I use iSCSI
> > to share the LUN's to a XenSource server, and XenSource creates the PV's
> > and LV's in order to provision VM's. Didn't think of that, but that's on
> > the disk of course. 
> > 
> > Is there any way to configure snapshots for these already created LVM
> > partitions?
> 
> If I understand, you exported some LUNs to XenSource server. So you
> should NOT be using those LUNs at this machine, right? You should be
> able to create snapshots at the XenSource server though.

That is correct. Sorry if I did not give enough information to
understand the problem at first, but what your saying now is correct.

I have a storage server with an MD3000 attached to it. This server runs
the multipath-tools to create redundant paths:

root@storage:~# multipath -ll
<snip>
xen (360019b9000d7e11000004485473faa94) dm-0 DELL    ,MD3000        
[size=200G][features=0][hwhandler=0]
\_ round-robin 0 [prio=3][active]
 \_ 1:0:0:3 sde 8:64  [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 1:0:1:3 sdi 8:128 [active][ghost]
<snip>

So the path that I use for sharing the LUN with the XenSource server
is /dev/mapper/xen:

root@storage:~# cat /etc/ietd.conf
<snip>
Target iqn.2007-11.local.xillan:storage.xen.disk1
          Lun 0 Path=/dev/mapper/xen,Type=blockio
<snip>

On the XenSource server I use a shared iSCSI storage so multiple servers
can access the same data so I can do live migrations. When I add the
storage XenSource creates a PV and then when I create a VM, it creates
LV's for each virtual harddisk.

So far so good I guess, as the XenSource machine uses the multipathed
drive, right?

Now I would like to take snapshots of the whole PV so I can make
backups. My thoughts where to make a PV on /dev/mapper/xen that spans
the whole disk, and then create a single LV on it. 

If I then take the path to that single LV, say /dev/volumegroup/disk1 as
target for iSCSI, then the XenSsource server should only see that LV as
the shared iSCSI repository.

But then XenSource wil create a PV and a couple of LV's in the existing
LV (created on the storage server and shared true iSCSI). Could that
create any problems or can I just do that?

In this way, I can take the snapshots on the storage server and I don't
have to create a second iSCSI target for the XenSource server, And I
don't have to make extensive use of the network (it can then be done on
one machine directly connected to both LUN's).

Sorry for the long post, but hope you give me some insight on this.

Thanks,

Sander

Re: Use LVM on /dev/mapper/diskname and iSCSI

[malahal]

S. J. van Harmelen [svh@dds.nl] wrote:
> 
> On the XenSource server I use a shared iSCSI storage so multiple servers
> can access the same data so I can do live migrations. When I add the
> storage XenSource creates a PV and then when I create a VM, it creates
> LV's for each virtual harddisk.
> 
> So far so good I guess, as the XenSource machine uses the multipathed
> drive, right?

Should.
 
> Now I would like to take snapshots of the whole PV so I can make
> backups. My thoughts where to make a PV on /dev/mapper/xen that spans
> the whole disk, and then create a single LV on it. 
> 
> If I then take the path to that single LV, say /dev/volumegroup/disk1 as
> target for iSCSI, then the XenSsource server should only see that LV as
> the shared iSCSI repository.
> 
> But then XenSource wil create a PV and a couple of LV's in the existing
> LV (created on the storage server and shared true iSCSI). Could that
> create any problems or can I just do that?

I don't see any problem. You should be able to do that, but then I am
not an expert. Give it a try.

Re: Use LVM on /dev/mapper/diskname and iSCSI

[S. J. van Harmelen]

On Mon, 2007-11-19 at 14:36 -0800, malahal@us.ibm.com wrote:
> S. J. van Harmelen [svh@dds.nl] wrote:
> > 
> > On the XenSource server I use a shared iSCSI storage so multiple servers
> > can access the same data so I can do live migrations. When I add the
> > storage XenSource creates a PV and then when I create a VM, it creates
> > LV's for each virtual harddisk.
> > 
> > So far so good I guess, as the XenSource machine uses the multipathed
> > drive, right?
> 
> Should.
>  
> > Now I would like to take snapshots of the whole PV so I can make
> > backups. My thoughts where to make a PV on /dev/mapper/xen that spans
> > the whole disk, and then create a single LV on it. 
> > 
> > If I then take the path to that single LV, say /dev/volumegroup/disk1 as
> > target for iSCSI, then the XenSsource server should only see that LV as
> > the shared iSCSI repository.
> > 
> > But then XenSource wil create a PV and a couple of LV's in the existing
> > LV (created on the storage server and shared true iSCSI). Could that
> > create any problems or can I just do that?
> 
> I don't see any problem. You should be able to do that, but then I am
> not an expert. Give it a try.

Thanks, I will give it a try then!

Sander

Re: Use LVM on /dev/mapper/diskname and iSCSI

[Tore Anderson]

* S. J. van Harmelen

> I have a storage server with an MD3000 attached to it. This server
> runs the multipath-tools to create redundant paths:
> 
> root@storage:~# multipath -ll
> <snip>
> xen (360019b9000d7e11000004485473faa94) dm-0 DELL    ,MD3000        
> [size=200G][features=0][hwhandler=0]
> \_ round-robin 0 [prio=3][active]
>  \_ 1:0:0:3 sde 8:64  [active][ready]
> \_ round-robin 0 [prio=0][enabled]
>  \_ 1:0:1:3 sdi 8:128 [active][ghost]
> <snip>

I think you need hwhandler = "1 rdac" for the MD3000 to be able to fail
over properly.  At least a colleague that have played with it told me so.

> On the XenSource server I use a shared iSCSI storage so multiple
> servers can access the same data so I can do live migrations. When I
> add the storage XenSource creates a PV and then when I create a VM,
> it creates LV's for each virtual harddisk.
> 
> So far so good I guess, as the XenSource machine uses the multipathed
> drive, right?

You absolutely need to keep LVM (in dom0) from using /dev/sde or
/dev/sdi as the PV (instead of /dev/mapper/xen).  You can accomplish
this with a filter line in lvm.conf:

filter = [ "a|^/dev/mapper/xen$|", "r|.*|" ]

Which means that a device node that is exactly «/dev/mapper/xen» is
considered a valid PV candidate, all other devices are rejected.

From the email you sent with earlier pvscan output you got /dev/dm-0 as
the active PV, with pvs you got /dev/sde (and with both you got I/O
errors to the passive paths).  It can't be relied upon that you'll get
dm-0 (which is absolutely necessary if you want dm-multipath to serve
any purpose) in the future unless you force LVM to only look for PVs in
the multipath'ed block devices.

> Now I would like to take snapshots of the whole PV so I can make 
> backups. My thoughts where to make a PV on /dev/mapper/xen that spans
> the whole disk, and then create a single LV on it.

You can only take snapshots of LVs, not PVs.  And you would have to
create a VG first with /dev/mapper/xen as a PV (you can't add LVs
directly on PVs).  But I think you've got the basic idea correct.  :-)

> If I then take the path to that single LV, say /dev/volumegroup/disk1
> as target for iSCSI, then the XenSsource server should only see that
> LV as the shared iSCSI repository.
> 
> But then XenSource wil create a PV and a couple of LV's in the
> existing LV (created on the storage server and shared true iSCSI).
> Could that create any problems or can I just do that?

I see no reason why that wouldn't work.  Try it and see?  Inside the
domU the Linux kernel won't know that its available block devices are
really mapped to another LVM implementation that runs in dom0.  LVM is
probably clever enough to disregard that LV-containing-a-PV as a valid
PV candidate, but if not the filter line in lvm.conf should handle it.

Regards
-- 
Tore Anderson

Re: Use LVM on /dev/mapper/diskname and iSCSI

[S. J. van Harmelen]

On Tue, 2007-11-20 at 08:36 +0100, Tore Anderson wrote:
> * S. J. van Harmelen
> 
> > I have a storage server with an MD3000 attached to it. This server
> > runs the multipath-tools to create redundant paths:
> > 
> > root@storage:~# multipath -ll
> > <snip>
> > xen (360019b9000d7e11000004485473faa94) dm-0 DELL    ,MD3000        
> > [size=200G][features=0][hwhandler=0]
> > \_ round-robin 0 [prio=3][active]
> >  \_ 1:0:0:3 sde 8:64  [active][ready]
> > \_ round-robin 0 [prio=0][enabled]
> >  \_ 1:0:1:3 sdi 8:128 [active][ghost]
> > <snip>
> 
> I think you need hwhandler = "1 rdac" for the MD3000 to be able to fail
> over properly.  At least a colleague that have played with it told me so.

You are correct about that, but I did have some issues with the new
2.6.23.x kernels. So for testing I went back to a 2.6.22.x kernel which
does not have the RDAC hardware handler.

When I do use a 2.6.23.x kernel with the RDAC hardware handler I get the
following error: multipathd[6895]: segfault at
000000000000000a rip 00002b251f24394f rsp 00007fff8bfaa730 error 4 

Besides that it seems to work fine, but I do not know if this can be
disgarded?

> 
> > On the XenSource server I use a shared iSCSI storage so multiple
> > servers can access the same data so I can do live migrations. When I
> > add the storage XenSource creates a PV and then when I create a VM,
> > it creates LV's for each virtual harddisk.
> > 
> > So far so good I guess, as the XenSource machine uses the multipathed
> > drive, right?
> 
> You absolutely need to keep LVM (in dom0) from using /dev/sde or
> /dev/sdi as the PV (instead of /dev/mapper/xen).  You can accomplish
> this with a filter line in lvm.conf:
> 
> filter = [ "a|^/dev/mapper/xen$|", "r|.*|" ]
> 
> Which means that a device node that is exactly «/dev/mapper/xen» is
> considered a valid PV candidate, all other devices are rejected.

Will add this line to lvm.conf. Thanks!

> 
> >From the email you sent with earlier pvscan output you got /dev/dm-0 as
> the active PV, with pvs you got /dev/sde (and with both you got I/O
> errors to the passive paths).  It can't be relied upon that you'll get
> dm-0 (which is absolutely necessary if you want dm-multipath to serve
> any purpose) in the future unless you force LVM to only look for PVs in
> the multipath'ed block devices.
> 
> > Now I would like to take snapshots of the whole PV so I can make 
> > backups. My thoughts where to make a PV on /dev/mapper/xen that spans
> > the whole disk, and then create a single LV on it.
> 
> You can only take snapshots of LVs, not PVs.  And you would have to
> create a VG first with /dev/mapper/xen as a PV (you can't add LVs
> directly on PVs).  But I think you've got the basic idea correct.  :-)

Indeed, I understand :)

> 
> > If I then take the path to that single LV, say /dev/volumegroup/disk1
> > as target for iSCSI, then the XenSsource server should only see that
> > LV as the shared iSCSI repository.
> > 
> > But then XenSource wil create a PV and a couple of LV's in the
> > existing LV (created on the storage server and shared true iSCSI).
> > Could that create any problems or can I just do that?
> 
> I see no reason why that wouldn't work.  Try it and see?  Inside the
> domU the Linux kernel won't know that its available block devices are
> really mapped to another LVM implementation that runs in dom0.  LVM is
> probably clever enough to disregard that LV-containing-a-PV as a valid
> PV candidate, but if not the filter line in lvm.conf should handle it.

Oke, I will give it a try then.

Thanks,

Sander

Re: Use LVM on /dev/mapper/diskname and iSCSI

[Tore Anderson]

* S. J. van Harmelen

> When I do use a 2.6.23.x kernel with the RDAC hardware handler I get the
> following error: multipathd[6895]: segfault at
> 000000000000000a rip 00002b251f24394f rsp 00007fff8bfaa730 error 4 

This is indicative of a bug in multipathd.  It might or might not be
serious, but I guess Christophe Varoqui would be interested in seeing a
backtrace of the segfault in any case...

Try running multipathd (make sure it's not stripped of debugging
symbols) with core dumps enabled ("ulimit -c unlimited"), and get a
backtrace ("gdb /sbin/multipathd core" - then run "bt full").

Regards
-- 
Tore Anderson

Possible bug in multipathd (getting a segfault)

[S. J. van Harmelen]

On Tue, 2007-11-20 at 10:49 +0100, Tore Anderson wrote:
> * S. J. van Harmelen
> 
> > When I do use a 2.6.23.x kernel with the RDAC hardware handler I get the
> > following error: multipathd[6895]: segfault at
> > 000000000000000a rip 00002b251f24394f rsp 00007fff8bfaa730 error 4 
> 
> This is indicative of a bug in multipathd.  It might or might not be
> serious, but I guess Christophe Varoqui would be interested in seeing a
> backtrace of the segfault in any case...
> 
> Try running multipathd (make sure it's not stripped of debugging
> symbols) with core dumps enabled ("ulimit -c unlimited"), and get a
> backtrace ("gdb /sbin/multipathd core" - then run "bt full").

I just changed the subject, because this is realy another issue then
what we where talking about before.

Could you specify the steps I have to take to create the backtrace a bit
more as I have not done this before.

I run a debian etch server with a custom compiled 2.6.23.1 kernel when I
get the error.

Sander

Re: Possible bug in multipathd (getting a segfault)

[Tore Anderson]

* S. J. van Harmelen

> Could you specify the steps I have to take to create the backtrace a
> bit more as I have not done this before.
> 
> I run a debian etch server with a custom compiled 2.6.23.1 kernel
> when I get the error.

Debian strips all their binaries by default, so you need to recompile
multipath-tools and reinstall it:

$ export DEB_BUILD_OPTIONS=nostrip
$ apt-get build-dep multipath-tools
$ apt-get --compile source multipath-tools
$ dpkg -i multipath-tools*deb

Then start multipathd from a shell that has core dumps enabled (make
sure it's not already running):

$ ulimit -c unlimited
$ multipathd

Wait for a segfault to happen, and you should have gotten a file named
"core" in your current directory (or maybe in the root directory since
multipathd chdir()s there, I'm not sure) that contains the backtrace.
We need to extract it with GDB, though, so run:

$ gdb /sbin/multipathd /path/to/core
(gdb) bt full

...and post the resulting backtrace here.  Hopefully a developer (not
me, unfortunately) will be able to make sense of it.

Another thing you might want to try is to use your 2.6.23.1 with the
same configuration as your 2.6.22.x (ie. don't use hwhandler rdac), and
see if the segfaults still occur.

Regards
-- 
Tore Anderson

Re: Possible bug in multipathd (getting a segfault)

[S. J. van Harmelen]

On Tue, 2007-11-20 at 11:21 +0100, Tore Anderson wrote:
> * S. J. van Harmelen
> 
> > Could you specify the steps I have to take to create the backtrace a
> > bit more as I have not done this before.
> > 
> > I run a debian etch server with a custom compiled 2.6.23.1 kernel
> > when I get the error.
> 
> Debian strips all their binaries by default, so you need to recompile
> multipath-tools and reinstall it:
> 
> $ export DEB_BUILD_OPTIONS=nostrip
> $ apt-get build-dep multipath-tools
> $ apt-get --compile source multipath-tools
> $ dpkg -i multipath-tools*deb

I guess I don't have to do that as I compiled the multipath-tools
myself?

wget http://christophe.varoqui.free.fr/.../multipath-tools-0.4.8.tar.bz2
tar jxf multipath-tools-0.4.8.tar.bz2 
cd multipath-tools-0.4.8/
make
make install

I did compile them when running the 2.6.22.2 kernel though. Should I
recompile them when running the 2.6.23.1 kernel, or doesn't that matter?

> 
> Then start multipathd from a shell that has core dumps enabled (make
> sure it's not already running):
> 
> $ ulimit -c unlimited
> $ multipathd
> 
> Wait for a segfault to happen, and you should have gotten a file named
> "core" in your current directory (or maybe in the root directory since
> multipathd chdir()s there, I'm not sure) that contains the backtrace.
> We need to extract it with GDB, though, so run:
> 
> $ gdb /sbin/multipathd /path/to/core
> (gdb) bt full

Oke, the segfault happens right at the start (and only at the start), so
that shouldn't be a problem. But I have to stop the running mulipathd
and that seems like a problem. This machine is running in production :)

So that will have to wait until I get a maintance window I guess...

> 
> ...and post the resulting backtrace here.  Hopefully a developer (not
> me, unfortunately) will be able to make sense of it.
> 
> Another thing you might want to try is to use your 2.6.23.1 with the
> same configuration as your 2.6.22.x (ie. don't use hwhandler rdac), and
> see if the segfaults still occur.
> 
> Regards

Re: Possible bug in multipathd (getting a segfault)

[Tore Anderson]

* S. J. van Harmelen

> I guess I don't have to do that as I compiled the multipath-tools 
> myself?

Then you should have the debugging symbols in place, yes.

> I did compile them when running the 2.6.22.2 kernel though. Should I 
> recompile them when running the 2.6.23.1 kernel, or doesn't that
> matter?

Doesn't matter.

> Oke, the segfault happens right at the start (and only at the start),
> so that shouldn't be a problem. But I have to stop the running
> mulipathd and that seems like a problem. This machine is running in
> production :)
> 
> So that will have to wait until I get a maintance window I guess...

Stopping multipathd isn't problematic.  It is only responsible for
periodically testing paths to pre-emptively fail non-active paths or
re-instate paths that has been failed by the kernel's dm-multipath layer.

If everything is running stable with all paths alive and well,
restarting multipathd is nothing to worry about.

Regards
-- 
Tore Anderson

Re: Possible bug in multipathd (getting a segfault)

[S. J. van Harmelen]

On Tue, 2007-11-20 at 12:15 +0100, Tore Anderson wrote:
> * S. J. van Harmelen
> 
> > I guess I don't have to do that as I compiled the multipath-tools 
> > myself?
> 
> Then you should have the debugging symbols in place, yes.
> 
> > I did compile them when running the 2.6.22.2 kernel though. Should I 
> > recompile them when running the 2.6.23.1 kernel, or doesn't that
> > matter?
> 
> Doesn't matter.
> 
> > Oke, the segfault happens right at the start (and only at the start),
> > so that shouldn't be a problem. But I have to stop the running
> > mulipathd and that seems like a problem. This machine is running in
> > production :)
> > 
> > So that will have to wait until I get a maintance window I guess...
> 
> Stopping multipathd isn't problematic.  It is only responsible for
> periodically testing paths to pre-emptively fail non-active paths or
> re-instate paths that has been failed by the kernel's dm-multipath layer.
> 
> If everything is running stable with all paths alive and well,
> restarting multipathd is nothing to worry about.

Oke, then I will give this a try this evening and post the results here
for de developers...

Thanks!

> 
> Regards

Re: Possible bug in multipathd (getting a segfault)

[S. J. van Harmelen]

On Tue, 2007-11-20 at 12:15 +0100, Tore Anderson wrote:
> * S. J. van Harmelen
> 
> > I guess I don't have to do that as I compiled the multipath-tools 
> > myself?
> 
> Then you should have the debugging symbols in place, yes.
> 
> > I did compile them when running the 2.6.22.2 kernel though. Should I 
> > recompile them when running the 2.6.23.1 kernel, or doesn't that
> > matter?
> 
> Doesn't matter.
> 
> > Oke, the segfault happens right at the start (and only at the start),
> > so that shouldn't be a problem. But I have to stop the running
> > mulipathd and that seems like a problem. This machine is running in
> > production :)
> > 
> > So that will have to wait until I get a maintance window I guess...
> 
> Stopping multipathd isn't problematic.  It is only responsible for
> periodically testing paths to pre-emptively fail non-active paths or
> re-instate paths that has been failed by the kernel's dm-multipath layer.
> 
> If everything is running stable with all paths alive and well,
> restarting multipathd is nothing to worry about.

Oeps... How stupid of me! I'm running the 2.6.22.2 kernel now, so I will
have to reboot to get kernel 2.6.23.1 running.

So I will have to wait a few days for this...

Thanks again :)

> 
> Regards

Re: Possible bug in multipathd (getting a segfault)

[S. J. van Harmelen]

On Tue, 2007-11-20 at 12:15 +0100, Tore Anderson wrote:
> * S. J. van Harmelen
> 
> > I guess I don't have to do that as I compiled the multipath-tools 
> > myself?
> 
> Then you should have the debugging symbols in place, yes.
> 
> > I did compile them when running the 2.6.22.2 kernel though. Should I 
> > recompile them when running the 2.6.23.1 kernel, or doesn't that
> > matter?
> 
> Doesn't matter.
> 
> > Oke, the segfault happens right at the start (and only at the start),
> > so that shouldn't be a problem. But I have to stop the running
> > mulipathd and that seems like a problem. This machine is running in
> > production :)
> > 
> > So that will have to wait until I get a maintance window I guess...
> 
> Stopping multipathd isn't problematic.  It is only responsible for
> periodically testing paths to pre-emptively fail non-active paths or
> re-instate paths that has been failed by the kernel's dm-multipath layer.
> 
> If everything is running stable with all paths alive and well,
> restarting multipathd is nothing to worry about.

I tried it, but it doesn't give me a core dump. It only shows one line
in syslog for each time that I start multipathd. Multipathd keeps
running with no noticeable errors though.

I'm running the system with a 2.6.23.8 kernel and it shows this error
one time per start of multipathd:

kernel: multipathd[2303]: segfault at 000000000000000a rip
00002baa8765f94f rsp 00007fff23b8e370 error 4
kernel: multipathd[3742]: segfault at 000000000000000a rip
00002b09ff82394f rsp 00007fffab9ca130 error 4
kernel: multipathd[4103]: segfault at 000000000000000a rip
00002adefdbe894f rsp 00007fffad604d90 error 4

Also I noticed that I get the "DELL     ,Universal Xpor" in my multipath
-ll output. Didn't get that before. Is that something to look at?

root@storage:~# multipath -ll
360019b9000d7e1100000000000000000dm-5 DELL    ,Universal Xpor
[size=20M][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][enabled]
 \_ 1:0:0:31 sdf 8:80  [active][ready]
backup (360019b9000d7e1100000444b46be82c4) dm-2 DELL    ,MD3000        
[size=200G][features=0][hwhandler=1 rdac]
\_ round-robin 0 [prio=3][active]
 \_ 1:0:1:2  sdi 8:128 [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 1:0:0:2  sdd 8:48  [active][ghost]
xen (360019b9000d7e11000004485473faa94) dm-1 DELL    ,MD3000        
[size=200G][features=0][hwhandler=1 rdac]
\_ round-robin 0 [prio=3][enabled]
 \_ 1:0:0:3  sde 8:64  [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 1:0:1:3  sdj 8:144 [active][ghost]
360019b9000d7ddab0000000000000000dm-0 DELL    ,Universal Xpor
[size=20M][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][enabled]
 \_ 1:0:1:31 sdk 8:160 [active][ready]
vmware (360019b9000d7e1100000444846be81b6) dm-4 DELL    ,MD3000        
[size=200G][features=0][hwhandler=1 rdac]
\_ round-robin 0 [prio=3][enabled]
 \_ 1:0:0:0  sdb 8:16  [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 1:0:1:0  sdg 8:96  [active][ghost]
data (360019b9000d7ddab0000493846be81e5) dm-3 DELL    ,MD3000        
[size=200G][features=0][hwhandler=1 rdac]
\_ round-robin 0 [prio=3][active]
 \_ 1:0:1:1  sdh 8:112 [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 1:0:0:1  sdc 8:32  [active][ghost]

Regards,

Sander

Re: Use LVM on /dev/mapper/diskname and iSCSI

[S. J. van Harmelen]

P.S. pvs only scanned the real disks and not the /dev/mapper/diskname
disks. So some of the scan's where rejected because the path to that
path was the redudant path.

But I assume that the /dev/mapper/diskname will have the exact same lvm
info written on it, ass it is in fact the real disk (/dev/sdi for
example)?

Sander


On Mon, 2007-11-19 at 15:06 +0100, Tore Anderson wrote:
> * S. J. van Harmelen
> 
> > I have a couple of LUN's that are handled by the multipath driver.
> > This works great, but now I want to use LVM so I can take snapshots
> > of one LUN to another LUN.
> > 
> > Can someone tell me how to do this?
> > 
> > When I do pvcreate /dev/mapper/diskname I get an error that the disk
> > is already part of an volume group. But I sertainly did not configure
> > that.
> > 
> > Can it be that the multipath driver of devmapper did that?
> 
> If you have a PV signature on the volume, LVM might have used one of the
> paths instead of using the multipath'ed device under /dev/mapper.  You
> should be able to check this by running the command "pvs" - it should
> list all the detected PVs on your system.
> 
> If this was the problem, you can avoid it by instructing LVM to not scan
> SCSI devices directly, by adding a line in /etc/lvm/lvm.conf like this:
> 
> device {
>   filter = [ "a|^/dev/mapper/.*|", "r|.*|" ]
> }
> 
> This will make LVM only look at devices in /dev/mapper/ as possible PV
> candidates, ignoring all other devices.  Beware if you have a PV on the
> internal drives though, you might want to have something like this
> instead in that case (if that PV is on /dev/sda2, for instance):
> 
>   filter = [ "a|^/dev/mapper/.*|", "a|^/dev/sda2$|", "r|.*|" ]
> 
> Make sure lvm.conf makes its way into the initramfs if you start the LVM
> stuff there.  I know Ubuntu doesn't include this file by default, at least.
> 
> Regards

Re: Use LVM on /dev/mapper/diskname

[malahal]

Tore Anderson [tore@linpro.no] wrote:
> * S. J. van Harmelen
> 
> > I have a couple of LUN's that are handled by the multipath driver.
> > This works great, but now I want to use LVM so I can take snapshots
> > of one LUN to another LUN.
> > 
> > Can someone tell me how to do this?
> > 
> > When I do pvcreate /dev/mapper/diskname I get an error that the disk
> > is already part of an volume group. But I sertainly did not configure
> > that.
> > 
> > Can it be that the multipath driver of devmapper did that?
> 
> If you have a PV signature on the volume, LVM might have used one of the
> paths instead of using the multipath'ed device under /dev/mapper.  You
> should be able to check this by running the command "pvs" - it should
> list all the detected PVs on your system.

If LVM uses single pathed disks like /dev/sda, then multipath can't use
the same path to make a multipathed device, right? My guess is that
/dev/mapper/multipath-name is indeed part of the LVM in your case. I
would like to know what happened otherwise. Thank you.

--Malahal.

Re: Use LVM on /dev/mapper/diskname

[S. J. van Harmelen]

On Mon, 2007-11-19 at 09:22 -0800, malahal@us.ibm.com wrote:
> Tore Anderson [tore@linpro.no] wrote:
> > * S. J. van Harmelen
> > 
> > > I have a couple of LUN's that are handled by the multipath driver.
> > > This works great, but now I want to use LVM so I can take snapshots
> > > of one LUN to another LUN.
> > > 
> > > Can someone tell me how to do this?
> > > 
> > > When I do pvcreate /dev/mapper/diskname I get an error that the disk
> > > is already part of an volume group. But I sertainly did not configure
> > > that.
> > > 
> > > Can it be that the multipath driver of devmapper did that?
> > 
> > If you have a PV signature on the volume, LVM might have used one of the
> > paths instead of using the multipath'ed device under /dev/mapper.  You
> > should be able to check this by running the command "pvs" - it should
> > list all the detected PVs on your system.
> 
> If LVM uses single pathed disks like /dev/sda, then multipath can't use
> the same path to make a multipathed device, right? My guess is that
> /dev/mapper/multipath-name is indeed part of the LVM in your case. I
> would like to know what happened otherwise. Thank you.

I already send this in a reply to the list, but will mail it to you
directly in case you missed it:

I executed the pvs command and it did find a few partitions. I use iSCSI
to share the LUN's to a XenSource server (I use the /dev/mapper/diskname
as sharing point for iSCSI), and XenSource creates the PV's
and LV's in order to provision VM's. Didn't think of that, but that's on
the disk of course.

Here's the output of pvs:

root@storage:~# pvs
  /dev/sdc: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdd: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdf: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdi: read failed after 0 of 4096 at 0: Input/output error
  PV         VG                                                 Fmt
Attr PSize   PFree  
  /dev/sde   VG_XenStorage-755d6649-e770-6f01-5e17-fd7f8dcd8651 lvm2 a-
199.99G 185.49G

The four disks that give a "read failed" error are the redundant disk,
so this is no problem.

Now is there any way to configure snapshots for these already created
LVM partitions? Or do I have to whipe the disk clean (rather not do
that, but if I have to it can be done), then create the pv's and lv's
and then share the lv's true iSCSI?

In that case XenSource will again create pv's and lv's in the shared lv.
Ain't that a problem?

Sander

> 
> --Malahal.
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: Use LVM on /dev/mapper/diskname

[S. J. van Harmelen]

P.S. When I run pvscan I get this output:

root@storage:~# pvscan
  /dev/sdc: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 214748299264: Input/output
error
  /dev/sdc: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdd: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdd: read failed after 0 of 4096 at 214748299264: Input/output
error
  /dev/sdd: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdf: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdf: read failed after 0 of 4096 at 214748299264: Input/output
error
  /dev/sdf: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdi: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdi: read failed after 0 of 4096 at 214748299264: Input/output
error
  /dev/sdi: read failed after 0 of 4096 at 0: Input/output error
  PV /dev/dm-0   VG VG_XenStorage-755d6649-e770-6f01-5e17-fd7f8dcd8651
lvm2 [199.99 GB / 185.49 GB free]
  Total: 1 [199.99 GB] / in use: 1 [199.99 GB] / in no VG: 0 [0   ]

So here it does see the dm-0? Please advice...

Sander


On Mon, 2007-11-19 at 09:22 -0800, malahal@us.ibm.com wrote:
> Tore Anderson [tore@linpro.no] wrote:
> > * S. J. van Harmelen
> > 
> > > I have a couple of LUN's that are handled by the multipath driver.
> > > This works great, but now I want to use LVM so I can take snapshots
> > > of one LUN to another LUN.
> > > 
> > > Can someone tell me how to do this?
> > > 
> > > When I do pvcreate /dev/mapper/diskname I get an error that the disk
> > > is already part of an volume group. But I sertainly did not configure
> > > that.
> > > 
> > > Can it be that the multipath driver of devmapper did that?
> > 
> > If you have a PV signature on the volume, LVM might have used one of the
> > paths instead of using the multipath'ed device under /dev/mapper.  You
> > should be able to check this by running the command "pvs" - it should
> > list all the detected PVs on your system.
> 
> If LVM uses single pathed disks like /dev/sda, then multipath can't use
> the same path to make a multipathed device, right? My guess is that
> /dev/mapper/multipath-name is indeed part of the LVM in your case. I
> would like to know what happened otherwise. Thank you.
> 
> --Malahal.
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel