* Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]
@ 2013-03-15 16:31 David Teigland
2013-03-15 17:46 ` Vladislav Bogdanov
0 siblings, 1 reply; 20+ messages in thread
From: David Teigland @ 2013-03-15 16:31 UTC (permalink / raw)
To: linux-lvm; +Cc: Vladislav Bogdanov, Andreas Pflug
> I need to convert lock on a remote node during last stage of ver3
> migration in libvirt/qemu
Hi, I wrote and maintain the dlm and have more recently written a new
disk-based lock manager called sanlock, https://fedorahosted.org/sanlock/
which operates with only shared storage among nodes (no networking or
other cluster manager.)
sanlock was written to allow RHEV to manage leases for vm images on shared
storage, including the ability to migrate leases among hosts (which is the
most complicated part, as you've mentioned above.) sanlock plugs into the
libvirt locking api, which also supports file locks (usable on local or
nfs file systems.) (Search for "virtlockd".)
Trying to use/control vm locks via lvm commands is not a good way to solve
the vm management/migration problem at the present time (but see below).
Instead, I'd suggest doing the locking via the libvirt locking api which
was designed for this purpose. As I mentioned, libvirt supports both
sanlock and file locks, but another option is to write a new libvirt
locking plug-in for dlm/corosync. This would be the best way to use dlm
locks to protect vm images on shared storage; I've been hoping someone
would do this for some time.
Incidentally, I've recently started a new project which is to replace
clvmd with a new "lvmlockd". I'm designing lvmlockd to support both dlm
and sanlock on the back side (transparent to lvm itself). With sanlock,
you will not need a dlm/corosync cluster to have the benefits of locking
vgs and lvs on shared storage. This project is requiring a lot of
preliminary work in the lvm code, because the clvmd approach reaches
deeply into lvm itself. Relating back to virt environments, lvmlockd will
give you direct control of the lock types, modes and objects in each
command. This will hopefully make it much easier to use lvm locks in a
controlled and programatic way to solve problems like vm management.
So, in preparation for lvmlockd, you should not assume that lvm commands
will operate in a dlm/corosync environment. Any new options or
capabilities should be considered more generally. Also, the concept of
lvm commands executing remote operations in a cluster was a poor design
choice for clvm. lvmlockd will probably not support this notion.
Executing remote commands should be done at a higher layer.
Dave
^ permalink raw reply [flat|nested] 20+ messages in thread* Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED] 2013-03-15 16:31 [linux-lvm] LVM snapshot with Clustered VG [SOLVED] David Teigland @ 2013-03-15 17:46 ` Vladislav Bogdanov 2013-03-15 18:38 ` David Teigland 0 siblings, 1 reply; 20+ messages in thread From: Vladislav Bogdanov @ 2013-03-15 17:46 UTC (permalink / raw) To: David Teigland; +Cc: Andreas Pflug, linux-lvm 15.03.2013 19:31, David Teigland wrote: >> I need to convert lock on a remote node during last stage of ver3 >> migration in libvirt/qemu > > Hi, I wrote and maintain the dlm and have more recently written a new > disk-based lock manager called sanlock, https://fedorahosted.org/sanlock/ > which operates with only shared storage among nodes (no networking or > other cluster manager.) > > sanlock was written to allow RHEV to manage leases for vm images on shared > storage, including the ability to migrate leases among hosts (which is the > most complicated part, as you've mentioned above.) sanlock plugs into the > libvirt locking api, which also supports file locks (usable on local or > nfs file systems.) (Search for "virtlockd".) Yes, I know, please see below. > > Trying to use/control vm locks via lvm commands is not a good way to solve > the vm management/migration problem at the present time (but see below). > Instead, I'd suggest doing the locking via the libvirt locking api which > was designed for this purpose. As I mentioned, libvirt supports both > sanlock and file locks, but another option is to write a new libvirt > locking plug-in for dlm/corosync. This would be the best way to use dlm > locks to protect vm images on shared storage; I've been hoping someone > would do this for some time. I almost agree. That's why I developed one more locking mech for libvirt which works and solves what I need. I also thought about dlm one (may be the next step :) ), but I need lvm to activate volumes, because I do not like current libvirt's "-aln everywhere" approach with VLM, so I decided to use clvm-based locking driver (with my additions to LVM). So, I made new "clvm2" pool type (subtype of a "logical" together with "lvm2") which does not activate volumes at all by itself, and "clvm" locking driver. Of course, that was not so easy, because all current locking drivers assume that disk device used by VM is always accessible (file on NFS f.e.), and that is not true for clvm with exclusive activation. But, I really made it all work, so I expect this to be a very strong alternative to all other locking mechs in libvirt (but only for LVM volumes). Local (prevent LV from being opened on the same node) part is not yet fully baked, only a PoC, but that can be easily replaced with some flock-based implementation (not released yet, but I saw some words about it is being written by Daniel) or even allow chaining of another driver (sanlock on a local storage?) for local (per-node) locking. Please keep an eye on libvirt list, I want to send my work there if my ideas are accepted here on lvm list, there is strong dependency on them. It is very hard to inject ideas simultaneously to related projects, but I'll try. > > Incidentally, I've recently started a new project which is to replace > clvmd with a new "lvmlockd". I'm designing lvmlockd to support both dlm > and sanlock on the back side (transparent to lvm itself). With sanlock, > you will not need a dlm/corosync cluster to have the benefits of locking > vgs and lvs on shared storage. This project is requiring a lot of > preliminary work in the lvm code, because the clvmd approach reaches > deeply into lvm itself. Relating back to virt environments, lvmlockd will > give you direct control of the lock types, modes and objects in each > command. This will hopefully make it much easier to use lvm locks in a > controlled and programatic way to solve problems like vm management. Anyways, I decided to go with pacemaker, so what I have now fully fits my needs (and I have it now, not with EL7). Of course, I fully agree that clvm is just a big hack from the today's point of view and something need to change here. Unfortunately I do not have a power and fundings to fully redesign clustered LVM (I do what I do for projects which did not give me a one dollar yet), so I decided to just solve my task, being not intrusive. And I like a state I have it all now in. Your idea is clean, and it reminds me idea of moving quorum to corosync, so please keep going. ;) > > So, in preparation for lvmlockd, you should not assume that lvm commands > will operate in a dlm/corosync environment. Any new options or > capabilities should be considered more generally. Also, the concept of > lvm commands executing remote operations in a cluster was a poor design > choice for clvm. lvmlockd will probably not support this notion. > Executing remote commands should be done at a higher layer. It all takes soooo long, but I need it all to be done yesterday ;) Anyways, please feel free to add me to CC of lvmlockd discussions. Vladislav ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED] 2013-03-15 17:46 ` Vladislav Bogdanov @ 2013-03-15 18:38 ` David Teigland 2013-03-16 11:00 ` Vladislav Bogdanov 0 siblings, 1 reply; 20+ messages in thread From: David Teigland @ 2013-03-15 18:38 UTC (permalink / raw) To: Vladislav Bogdanov; +Cc: Andreas Pflug, linux-lvm On Fri, Mar 15, 2013 at 08:46:53PM +0300, Vladislav Bogdanov wrote: > I also thought about dlm one (may be the next step :) ), Let me know if you decide to start work on this and I can probably lend some help. > Of course, I fully agree that clvm is just a big hack from the today's > point of view and something need to change here. Since you're open to hacking components together, here's another possible hack: you could create a gfs2 file system on the shared storage, and use file locks on gfs2 files. The gfs2 files could be the actual vm images, but they do not have to be. You could still use lv's for vms directly, and create empty files on gfs2 representing each lv. The virtlockd file locks would be acquired on the empty gfs2 files, representing the lvs. This is another indirect way of using dlm locks, since the gfs2 file locks are passed to the dlm. Dave ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED] 2013-03-15 18:38 ` David Teigland @ 2013-03-16 11:00 ` Vladislav Bogdanov 0 siblings, 0 replies; 20+ messages in thread From: Vladislav Bogdanov @ 2013-03-16 11:00 UTC (permalink / raw) To: David Teigland; +Cc: Andreas Pflug, linux-lvm 15.03.2013 21:38, David Teigland wrote: > On Fri, Mar 15, 2013 at 08:46:53PM +0300, Vladislav Bogdanov wrote: >> I also thought about dlm one (may be the next step :) ), > > Let me know if you decide to start work on this and I can probably lend > some help. > >> Of course, I fully agree that clvm is just a big hack from the today's >> point of view and something need to change here. > > Since you're open to hacking components together, here's another possible > hack: you could create a gfs2 file system on the shared storage, and use > file locks on gfs2 files. The gfs2 files could be the actual vm images, > but they do not have to be. You could still use lv's for vms directly, > and create empty files on gfs2 representing each lv. The virtlockd file > locks would be acquired on the empty gfs2 files, representing the lvs. > This is another indirect way of using dlm locks, since the gfs2 file locks > are passed to the dlm. That would not solve issue with exclusive LV activation which is required to take snapshots (and just for more safety), but prevents migration. Vladislav ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG @ 2013-03-01 11:28 Andreas Pflug 2013-03-01 15:41 ` Vladislav Bogdanov 0 siblings, 1 reply; 20+ messages in thread From: Andreas Pflug @ 2013-03-01 11:28 UTC (permalink / raw) To: LVM general discussion and development, bubble Hi Vladislav, I tried the patch you mentioned in https://www.redhat.com/archives/linux-lvm/2013-January/msg00006.html , but apparently it doesn't work for me. On a locally active LV, I still get ./lvm lvchange -aey -vvvv vg/locktest ... Activating logical volume "locktest" exclusively #activate/dev_manager.c:284 Getting device info for vg-locktest [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN] #ioctl/libdm-iface.c:1724 dm info LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN NF [16384] (*1) #locking/cluster_locking.c:568 Lock held for oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN, node 7400a8c0 : CR #activate/activate.c:1067 vg/locktest is active #locking/cluster_locking.c:508 Locking LV oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN EX (LV|NONBLOCK|CLUSTER|LOCAL) (0xdd) #locking/cluster_locking.c:395 Error locking on node 7400a8c0: Device or resource busy Can you give me a hint what to try further? Regards Andreas ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG 2013-03-01 11:28 [linux-lvm] LVM snapshot with Clustered VG Andreas Pflug @ 2013-03-01 15:41 ` Vladislav Bogdanov 2013-03-06 7:40 ` Andreas Pflug 0 siblings, 1 reply; 20+ messages in thread From: Vladislav Bogdanov @ 2013-03-01 15:41 UTC (permalink / raw) To: Andreas Pflug, LVM general discussion and development Sorry, forgot to reply to list from the first attempt. Andreas Pflug <pgadmin@pse-consulting.de> wrote: >Hi Vladislav, > >I tried the patch you mentioned in >https://www.redhat.com/archives/linux-lvm/2013-January/msg00006.html , >but apparently it doesn't work for me. On a locally active LV, I still >get > >./lvm lvchange -aey -vvvv vg/locktest > >... >Activating logical volume "locktest" exclusively >#activate/dev_manager.c:284 Getting device info for vg-locktest > >[LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN] >#ioctl/libdm-iface.c:1724 dm info >LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN >NF [16384] (*1) >#locking/cluster_locking.c:568 Lock held for >oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN, node >7400a8c0 : CR >#activate/activate.c:1067 vg/locktest is active >#locking/cluster_locking.c:508 Locking LV >oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN EX >(LV|NONBLOCK|CLUSTER|LOCAL) (0xdd) >#locking/cluster_locking.c:395 Error locking on node 7400a8c0: Device > >or resource busy > >Can you give me a hint what to try further? > >Regards >Andreas Hi Andreas, Lock convertion is only enabled if you pass --force flag. Also, to upgrade local lock to exclusive one, you need to ensure IIRC that no more node holds local lock. Vladislav ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG 2013-03-01 15:41 ` Vladislav Bogdanov @ 2013-03-06 7:40 ` Andreas Pflug 2013-03-06 7:58 ` Vladislav Bogdanov 0 siblings, 1 reply; 20+ messages in thread From: Andreas Pflug @ 2013-03-06 7:40 UTC (permalink / raw) To: Vladislav Bogdanov; +Cc: LVM general discussion and development Am 01.03.13 16:41, schrieb Vladislav Bogdanov: > Sorry, forgot to reply to list from the first attempt. > > Andreas Pflug <pgadmin@pse-consulting.de> wrote: > >> Hi Vladislav, >> >> I tried the patch you mentioned in >> https://www.redhat.com/archives/linux-lvm/2013-January/msg00006.html , >> but apparently it doesn't work for me. On a locally active LV, I still >> get >> >> ./lvm lvchange -aey -vvvv vg/locktest >> >> ... >> Activating logical volume "locktest" exclusively >> #activate/dev_manager.c:284 Getting device info for vg-locktest >> >> [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN] >> #ioctl/libdm-iface.c:1724 dm info >> LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN >> NF [16384] (*1) >> #locking/cluster_locking.c:568 Lock held for >> oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN, node >> 7400a8c0 : CR >> #activate/activate.c:1067 vg/locktest is active >> #locking/cluster_locking.c:508 Locking LV >> oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN EX >> (LV|NONBLOCK|CLUSTER|LOCAL) (0xdd) >> #locking/cluster_locking.c:395 Error locking on node 7400a8c0: Device >> >> or resource busy >> >> Can you give me a hint what to try further? >> >> Regards >> Andreas > Hi Andreas, > Lock convertion is only enabled if you pass --force flag. > Also, to upgrade local lock to exclusive one, you need to ensure IIRC that no more node holds local lock. Hm, tried that as well: tools/lvm lvchange --force -aey -vvvv vg/locktest --force changes the error from "resource busy" to "invalid argument": #lvchange.c:243 Activating logical volume "locktest" exclusively (forced) #activate/dev_manager.c:284 Getting device info for vg-locktest [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN] #ioctl/libdm-iface.c:1724 dm info LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN NF [16384] (*1) #locking/cluster_locking.c:568 Lock held for oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN, node 7400a8c0 : CR #activate/activate.c:1067 vg/locktest is active #activate/dev_manager.c:284 Getting device info for vg-locktest [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN] #ioctl/libdm-iface.c:1724 dm info LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN NF [16384] (*1) #activate/activate.c:1067 vg/locktest is active #activate/dev_manager.c:284 Getting device info for vg-locktest [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN] #ioctl/libdm-iface.c:1724 dm info LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN NF [16384] (*1) #locking/cluster_locking.c:568 Lock held for oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN, node 7400a8c0 : CR #activate/activate.c:1067 vg/locktest is active #locking/cluster_locking.c:508 Locking LV oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN EX (LV|NONBLOCK|CLUSTER|LOCAL|CONVERT) (0x40dd) #locking/cluster_locking.c:395 Error locking on node 7400a8c0: Invalid argument Regards Andreas ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG 2013-03-06 7:40 ` Andreas Pflug @ 2013-03-06 7:58 ` Vladislav Bogdanov 2013-03-06 9:15 ` Andreas Pflug 0 siblings, 1 reply; 20+ messages in thread From: Vladislav Bogdanov @ 2013-03-06 7:58 UTC (permalink / raw) To: Andreas Pflug; +Cc: LVM general discussion and development [-- Attachment #1: Type: text/plain, Size: 4135 bytes --] 06.03.2013 10:40, Andreas Pflug wrote: > Am 01.03.13 16:41, schrieb Vladislav Bogdanov: >> Sorry, forgot to reply to list from the first attempt. >> >> Andreas Pflug <pgadmin@pse-consulting.de> wrote: >> >>> Hi Vladislav, >>> >>> I tried the patch you mentioned in >>> https://www.redhat.com/archives/linux-lvm/2013-January/msg00006.html , >>> but apparently it doesn't work for me. On a locally active LV, I still >>> get >>> >>> ./lvm lvchange -aey -vvvv vg/locktest >>> >>> ... >>> Activating logical volume "locktest" exclusively >>> #activate/dev_manager.c:284 Getting device info for vg-locktest >>> >>> [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN] >>> #ioctl/libdm-iface.c:1724 dm info >>> LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN >>> NF [16384] (*1) >>> #locking/cluster_locking.c:568 Lock held for >>> oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN, node >>> 7400a8c0 : CR >>> #activate/activate.c:1067 vg/locktest is active >>> #locking/cluster_locking.c:508 Locking LV >>> oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN EX >>> (LV|NONBLOCK|CLUSTER|LOCAL) (0xdd) >>> #locking/cluster_locking.c:395 Error locking on node 7400a8c0: Device >>> >>> or resource busy >>> >>> Can you give me a hint what to try further? >>> >>> Regards >>> Andreas >> Hi Andreas, >> Lock convertion is only enabled if you pass --force flag. >> Also, to upgrade local lock to exclusive one, you need to ensure IIRC >> that no more node holds local lock. > > Hm, tried that as well: > > tools/lvm lvchange --force -aey -vvvv vg/locktest > > --force changes the error from "resource busy" to "invalid argument": Is volume active on other nodes at that time? And do you run clvmd from that build tree as well? Also, can you please try attached patch (on top of that one you have)? I polished conversion a bit more, denying -an if volume is ex-locked somewhere and other fixes to logic. This patch also allows locking (activation) to be performed on remote nodes. I only tested this with corosync2 (which is set up in a way latest pacemaker - post-1.1.8 git master - needs, nodes has additional 'name' value in nodelist, please see http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-name.html). I did not have a time to split this all into git commits suitable for sending upstream, so it all is a little bit messy yet. > > #lvchange.c:243 Activating logical volume "locktest" exclusively > (forced) > #activate/dev_manager.c:284 Getting device info for vg-locktest > [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN] > #ioctl/libdm-iface.c:1724 dm info > LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN > NF [16384] (*1) > #locking/cluster_locking.c:568 Lock held for > oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN, node > 7400a8c0 : CR > #activate/activate.c:1067 vg/locktest is active > #activate/dev_manager.c:284 Getting device info for vg-locktest > [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN] > #ioctl/libdm-iface.c:1724 dm info > LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN > NF [16384] (*1) > #activate/activate.c:1067 vg/locktest is active > #activate/dev_manager.c:284 Getting device info for vg-locktest > [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN] > #ioctl/libdm-iface.c:1724 dm info > LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN > NF [16384] (*1) > #locking/cluster_locking.c:568 Lock held for > oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN, node > 7400a8c0 : CR > #activate/activate.c:1067 vg/locktest is active > #locking/cluster_locking.c:508 Locking LV > oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN EX > (LV|NONBLOCK|CLUSTER|LOCAL|CONVERT) (0x40dd) > #locking/cluster_locking.c:395 Error locking on node 7400a8c0: Invalid > argument > > > Regards > Andreas [-- Attachment #2: LVM2.2.02.98-remote-activation.patch --] [-- Type: text/x-patch, Size: 26826 bytes --] diff -urNp LVM2.2.02.98.orig/daemons/clvmd/clvmd.c LVM2.2.02.98/daemons/clvmd/clvmd.c --- LVM2.2.02.98.orig/daemons/clvmd/clvmd.c 2012-12-28 13:32:10.000000000 +0000 +++ LVM2.2.02.98/daemons/clvmd/clvmd.c 2013-03-05 10:59:47.874192528 +0000 @@ -1293,7 +1293,7 @@ static int read_from_local_sock(struct l } /* Check the node name for validity */ - if (inheader->node[0] && clops->csid_from_name(csid, inheader->node)) { + if (inheader->node[0] && clops->csid_from_name(csid, inheader->node) <= 0) { /* Error, node is not in the cluster */ struct clvm_header reply = { .cmd = CLVMD_CMD_REPLY, @@ -1436,7 +1436,7 @@ static int distribute_command(struct loc /* Do it on a single node */ char csid[MAX_CSID_LEN]; - if (clops->csid_from_name(csid, inheader->node)) { + if (clops->csid_from_name(csid, inheader->node) <= 0) { /* This has already been checked so should not happen */ return 0; } else { @@ -1448,6 +1448,8 @@ static int distribute_command(struct loc /* Are we the requested node ?? */ if (memcmp(csid, our_csid, max_csid_len) == 0) { DEBUGLOG("Doing command on local node only\n"); + inheader->flags &= ~CLVMD_FLAG_REMOTE; + inheader->flags |= CLVMD_FLAG_LOCAL; add_to_lvmqueue(thisfd, inheader, len, NULL); } else { DEBUGLOG("Sending message to single node: %s\n", @@ -1497,7 +1499,7 @@ static void process_remote_command(struc clvmd-command.c */ if (msg->cmd == CLVMD_CMD_VERSION) { int version_nums[3]; - char node[256]; + char node[max_cluster_member_name_len]; memcpy(version_nums, msg->args, sizeof(version_nums)); diff -urNp LVM2.2.02.98.orig/daemons/clvmd/clvmd-corosync.c LVM2.2.02.98/daemons/clvmd/clvmd-corosync.c --- LVM2.2.02.98.orig/daemons/clvmd/clvmd-corosync.c 2012-12-28 13:32:10.000000000 +0000 +++ LVM2.2.02.98/daemons/clvmd/clvmd-corosync.c 2013-03-05 10:59:47.874192528 +0000 @@ -72,6 +72,9 @@ static struct local_client *cluster_clie /* Corosync handles */ static cpg_handle_t cpg_handle; static quorum_handle_t quorum_handle; +#if defined HAVE_COROSYNC_CMAP_H +static cmap_handle_t cmap_handle = 0; +#endif /* DLM Handle */ static dlm_lshandle_t *lockspace; @@ -298,6 +301,17 @@ static int _init_cluster(void) return cs_to_errno(err); } +#if defined HAVE_COROSYNC_CMAP_H + err = cmap_initialize(&cmap_handle); + if (err != CS_OK) { + syslog(LOG_ERR, "Cannot initialise Corosync CMAP service: %d", + err); + DEBUGLOG("Cannot initialise Corosync CMAP service: %d", err); + cpg_finalize(cpg_handle); + return cs_to_errno(err); + } +#endif + #ifdef QUORUM_SET err = quorum_initialize(&quorum_handle, &quorum_callbacks, @@ -306,6 +320,10 @@ static int _init_cluster(void) if (quorum_type != QUORUM_SET) { syslog(LOG_ERR, "Corosync quorum service is not configured"); DEBUGLOG("Corosync quorum service is not configured"); +#if defined HAVE_COROSYNC_CMAP_H + cmap_finalize(cmap_handle); +#endif + cpg_finalize(cpg_handle); return EINVAL; } #else @@ -316,6 +334,10 @@ static int _init_cluster(void) if (err != CS_OK) { syslog(LOG_ERR, "Cannot initialise Corosync quorum service: %d", err); +#if defined HAVE_COROSYNC_CMAP_H + cmap_finalize(cmap_handle); +#endif + cpg_finalize(cpg_handle); DEBUGLOG("Cannot initialise Corosync quorum service: %d", err); return cs_to_errno(err); } @@ -326,6 +348,10 @@ static int _init_cluster(void) lockspace = dlm_create_lockspace(LOCKSPACE_NAME, 0600); if (!lockspace) { syslog(LOG_ERR, "Unable to create DLM lockspace for CLVM: %m"); +#if defined HAVE_COROSYNC_CMAP_H + cmap_finalize(cmap_handle); +#endif + cpg_finalize(cpg_handle); return -1; } DEBUGLOG("Created DLM lockspace for CLVMD.\n"); @@ -340,6 +366,9 @@ static int _init_cluster(void) cpg_group_name.length = strlen((char *)cpg_group_name.value); err = cpg_join(cpg_handle, &cpg_group_name); if (err != CS_OK) { +#if defined HAVE_COROSYNC_CMAP_H + cmap_finalize(cmap_handle); +#endif cpg_finalize(cpg_handle); quorum_finalize(quorum_handle); dlm_release_lockspace(LOCKSPACE_NAME, lockspace, 1); @@ -351,6 +380,9 @@ static int _init_cluster(void) err = cpg_local_get(cpg_handle, &our_nodeid); if (err != CS_OK) { +#if defined HAVE_COROSYNC_CMAP_H + cmap_finalize(cmap_handle); +#endif cpg_finalize(cpg_handle); quorum_finalize(quorum_handle); dlm_release_lockspace(LOCKSPACE_NAME, lockspace, 1); @@ -372,6 +404,9 @@ static void _cluster_closedown(void) dlm_release_lockspace(LOCKSPACE_NAME, lockspace, 1); cpg_finalize(cpg_handle); quorum_finalize(quorum_handle); +#if defined HAVE_COROSYNC_CMAP_H + cmap_finalize(cmap_handle); +#endif } static void _get_our_csid(char *csid) @@ -380,17 +415,56 @@ static void _get_our_csid(char *csid) } /* Corosync doesn't really have nmode names so we - just use the node ID in hex instead */ + just use the node ID in hex instead. + The only case when we have name is when nodelist is configured for + pacemaker the way described in pacemaker docs (has nodelist.node.X.name) */ static int _csid_from_name(char *csid, const char *name) { - int nodeid; + int nodeid = 0; struct node_info *ninfo; - if (sscanf(name, "%x", &nodeid) == 1) { - ninfo = dm_hash_lookup_binary(node_hash, csid, COROSYNC_CSID_LEN); - if (ninfo) +#if defined HAVE_COROSYNC_CMAP_H + int idx = 0; + char key[128]; + + while (1) { + cs_error_t rc; + char *n_name = NULL; + + snprintf(key, 128, "nodelist.node.%d.name", idx); + rc = cmap_get_string(cmap_handle, key, &n_name); + + if (rc != CS_OK) { + snprintf(key, 128, "nodelist.node.%d.ring0_ipaddr", idx); + rc = cmap_get_string(cmap_handle, key, &n_name); + + if (rc != CS_OK) { + break; + } + } + if (strcmp(name, n_name) == 0) { + free(n_name); + snprintf(key, 128, "nodelist.node.%d.nodeid", idx); + rc = cmap_get_uint32(cmap_handle, key, (uint32_t *)&nodeid); + if (rc != CS_OK) { + break; + } + + } else { + free(n_name); + } + idx++; + } +#endif + + if (nodeid || sscanf(name, "%d", &nodeid) == 1) { + ninfo = dm_hash_lookup_binary(node_hash, (char *)&nodeid, COROSYNC_CSID_LEN); + if (ninfo) { + memcpy(csid, &nodeid, COROSYNC_CSID_LEN); return nodeid; + } } + *csid = 0; return -1; } @@ -398,14 +472,64 @@ static int _name_from_csid(const char *c { struct node_info *ninfo; +#if defined HAVE_COROSYNC_CMAP_H + int nodeid = 0; + int idx = 0; + char key[128]; + + memcpy(&nodeid, csid, COROSYNC_CSID_LEN); + + *name = '\0'; + + while (1) { + cs_error_t rc; + char *n_name = NULL; + uint32_t id; + + snprintf(key, 128, "nodelist.node.%d.nodeid", idx); + rc = cmap_get_uint32(cmap_handle, key, &id); + if (rc != CS_OK) { + break; + } + if (id == nodeid) { + snprintf(key, 128, "nodelist.node.%d.name", idx); + rc = cmap_get_string(cmap_handle, key, &n_name); + if (rc != CS_OK || !n_name) { + int octet; + snprintf(key, 128, "nodelist.node.%d.ring0_ipaddr", idx); + rc = cmap_get_string(cmap_handle, key, &n_name); + + if (rc != CS_OK || !n_name) { + break; + } + if (sscanf(n_name, "%d.%d.%d.%d", &octet, &octet, &octet, &octet) == 4 || strstr(n_name, ":") != NULL) { + /* ring0_ipaddr contains IP address */ + free(n_name); + n_name = NULL; + break; + } + } + snprintf(name, COROSYNC_MAX_CLUSTER_MEMBER_NAME_LEN, "%s", n_name); + free(n_name); + break; + } + + idx++; + } +#endif + ninfo = dm_hash_lookup_binary(node_hash, csid, COROSYNC_CSID_LEN); if (!ninfo) { - sprintf(name, "UNKNOWN %s", print_corosync_csid(csid)); + if (! *name) + snprintf(name, COROSYNC_MAX_CLUSTER_MEMBER_NAME_LEN, "UNKNOWN %s", print_corosync_csid(csid)); + return -1; } - sprintf(name, "%x", ninfo->nodeid); + if (! *name) + snprintf(name, COROSYNC_MAX_CLUSTER_MEMBER_NAME_LEN, "%d", ninfo->nodeid); + return 0; } @@ -629,18 +753,12 @@ out: static int _get_cluster_name(char *buf, int buflen) { - cmap_handle_t cmap_handle = 0; int result; char *name = NULL; /* This is a default in case everything else fails */ strncpy(buf, "Corosync", buflen); - /* Look for a cluster name in cmap */ - result = cmap_initialize(&cmap_handle); - if (result != CS_OK) - return 0; - result = cmap_get_string(cmap_handle, "totem.cluster_name", &name); if (result != CS_OK) goto out; @@ -651,7 +769,6 @@ static int _get_cluster_name(char *buf, out: if (name) free(name); - cmap_finalize(cmap_handle); return 0; } diff -urNp LVM2.2.02.98.orig/lib/activate/activate.c LVM2.2.02.98/lib/activate/activate.c --- LVM2.2.02.98.orig/lib/activate/activate.c 2012-10-15 14:24:58.000000000 +0000 +++ LVM2.2.02.98/lib/activate/activate.c 2013-03-05 10:59:47.876252746 +0000 @@ -270,7 +270,7 @@ int lv_is_active_exclusive_locally(const { return 0; } -int lv_is_active_exclusive_remotely(const struct logical_volume *lv) +int lv_is_active_exclusive_remotely(const struct logical_volume *lv, const char *node) { return 0; } @@ -1014,7 +1014,7 @@ int lvs_in_vg_opened(const struct volume * Returns: 0 or 1 */ static int _lv_is_active(const struct logical_volume *lv, - int *locally, int *exclusive) + int *locally, int *exclusive, const char *node) { int r, l, e; /* remote, local, and exclusive */ @@ -1033,7 +1033,7 @@ static int _lv_is_active(const struct lo if (l && !exclusive) goto out; - if ((r = remote_lock_held(lv->lvid.s, &e)) >= 0) + if ((r = remote_lock_held(lv->lvid.s, &e, node)) >= 0) goto out; /* @@ -1071,34 +1071,40 @@ out: int lv_is_active(const struct logical_volume *lv) { - return _lv_is_active(lv, NULL, NULL); + return _lv_is_active(lv, NULL, NULL, NULL); +} + +int lv_is_active_remotely(const struct logical_volume *lv, const char *node) +{ + int l; + return _lv_is_active(lv, &l, NULL, node) && !l; } int lv_is_active_but_not_locally(const struct logical_volume *lv) { int l; - return _lv_is_active(lv, &l, NULL) && !l; + return _lv_is_active(lv, &l, NULL, NULL) && !l; } int lv_is_active_exclusive(const struct logical_volume *lv) { int e; - return _lv_is_active(lv, NULL, &e) && e; + return _lv_is_active(lv, NULL, &e, NULL) && e; } int lv_is_active_exclusive_locally(const struct logical_volume *lv) { int l, e; - return _lv_is_active(lv, &l, &e) && l && e; + return _lv_is_active(lv, &l, &e, NULL) && l && e; } -int lv_is_active_exclusive_remotely(const struct logical_volume *lv) +int lv_is_active_exclusive_remotely(const struct logical_volume *lv, const char *node) { int l, e; - return _lv_is_active(lv, &l, &e) && !l && e; + return _lv_is_active(lv, &l, &e, node) && !l && e; } #ifdef DMEVENTD diff -urNp LVM2.2.02.98.orig/lib/activate/activate.h LVM2.2.02.98/lib/activate/activate.h --- LVM2.2.02.98.orig/lib/activate/activate.h 2012-10-15 14:24:58.000000000 +0000 +++ LVM2.2.02.98/lib/activate/activate.h 2013-03-05 10:59:47.878201236 +0000 @@ -130,10 +130,11 @@ int lvs_in_vg_activated(const struct vol int lvs_in_vg_opened(const struct volume_group *vg); int lv_is_active(const struct logical_volume *lv); +int lv_is_active_remotely(const struct logical_volume *lv, const char *node); int lv_is_active_but_not_locally(const struct logical_volume *lv); int lv_is_active_exclusive(const struct logical_volume *lv); int lv_is_active_exclusive_locally(const struct logical_volume *lv); -int lv_is_active_exclusive_remotely(const struct logical_volume *lv); +int lv_is_active_exclusive_remotely(const struct logical_volume *lv, const char *node); int lv_has_target_type(struct dm_pool *mem, struct logical_volume *lv, const char *layer, const char *target_type); diff -urNp LVM2.2.02.98.orig/lib/commands/toolcontext.h LVM2.2.02.98/lib/commands/toolcontext.h --- LVM2.2.02.98.orig/lib/commands/toolcontext.h 2012-10-15 14:24:58.000000000 +0000 +++ LVM2.2.02.98/lib/commands/toolcontext.h 2013-03-05 10:59:47.879245750 +0000 @@ -112,6 +112,7 @@ struct cmd_context { char dev_dir[PATH_MAX]; char proc_dir[PATH_MAX]; char sysfs_dir[PATH_MAX]; /* FIXME Use global value instead. */ + const char *node; }; /* diff -urNp LVM2.2.02.98.orig/lib/locking/cluster_locking.c LVM2.2.02.98/lib/locking/cluster_locking.c --- LVM2.2.02.98.orig/lib/locking/cluster_locking.c 2012-12-28 13:32:10.000000000 +0000 +++ LVM2.2.02.98/lib/locking/cluster_locking.c 2013-03-05 10:59:47.881190506 +0000 @@ -190,8 +190,10 @@ static void _build_header(struct clvm_he } else if (!strcmp(node, NODE_REMOTE)) { head->node[0] = '\0'; head->flags = CLVMD_FLAG_REMOTE; - } else - strcpy(head->node, node); + } else { + memcpy(head->node, node, strlen(node) + 1); + head->flags = CLVMD_FLAG_REMOTE; + } } /* @@ -315,6 +317,9 @@ static int _lock_for_cluster(struct cmd_ assert(name); + if (node == NULL) { + node = ""; + } len = strlen(name) + 3; args = alloca(len); strcpy(args + 2, name); @@ -374,7 +379,7 @@ static int _lock_for_cluster(struct cmd_ !(flags & LCK_CLUSTER_VG))) node = NODE_LOCAL; else if (flags & LCK_REMOTE) - node = NODE_REMOTE; + node = cmd->node ? cmd->node : NODE_REMOTE; } status = _cluster_request(clvmd_cmd, node, args, len, @@ -527,16 +532,18 @@ static int decode_lock_type(const char * } #ifdef CLUSTER_LOCKING_INTERNAL -static int _query_resource(const char *resource, int *mode) +static int _query_resource(const char *resource, int *mode, const char *node) #else -int query_resource(const char *resource, int *mode) +int query_resource(const char *resource, int *mode, const char *node) #endif { int i, status, len, num_responses, saved_errno; - const char *node = ""; char *args; lvm_response_t *response = NULL; + if (node == NULL) { + node = ""; + } saved_errno = errno; len = strlen(resource) + 3; args = alloca(len); diff -urNp LVM2.2.02.98.orig/lib/locking/external_locking.c LVM2.2.02.98/lib/locking/external_locking.c --- LVM2.2.02.98.orig/lib/locking/external_locking.c 2012-10-15 14:24:58.000000000 +0000 +++ LVM2.2.02.98/lib/locking/external_locking.c 2013-03-05 10:59:47.882313739 +0000 @@ -28,7 +28,7 @@ static int (*_lock_fn) (struct cmd_conte uint32_t flags) = NULL; static int (*_init_fn) (int type, struct dm_config_tree * cft, uint32_t *flags) = NULL; -static int (*_lock_query_fn) (const char *resource, int *mode) = NULL; +static int (*_lock_query_fn) (const char *resource, int *mode, const char *node) = NULL; static int _lock_resource(struct cmd_context *cmd, const char *resource, uint32_t flags) diff -urNp LVM2.2.02.98.orig/lib/locking/locking.c LVM2.2.02.98/lib/locking/locking.c --- LVM2.2.02.98.orig/lib/locking/locking.c 2012-12-28 13:32:10.000000000 +0000 +++ LVM2.2.02.98/lib/locking/locking.c 2013-03-06 06:42:59.043441030 +0000 @@ -544,10 +544,12 @@ int deactivate_lv(struct cmd_context *cm { if (vg_is_clustered(lv->vg)) { if (lv_is_active_exclusive(lv) && ! lv_is_active_exclusive_locally(lv)) { + log_error("Failed to deactivate %s: exclusively activated on a remote node. Use -aln --node Node.", lv->name); return_0; } } lock_lv_vol(cmd, lv, LCK_LV_DEACTIVATE); + return 1; } /* @@ -561,20 +563,29 @@ int activate_lv_excl(struct cmd_context if (!vg_is_clustered(lv->vg)) return activate_lv_excl_local(cmd, lv); - if (lv_is_active_exclusive_locally(lv)) - return 1; + if (!cmd->node) { + if (lv_is_active_exclusive_locally(lv)) + return 1; - if (!activate_lv_excl_local(cmd, lv)) - return_0; + if (!activate_lv_excl_local(cmd, lv)) + return_0; - if (lv_is_active_exclusive(lv)) - return 1; - /* FIXME Deal with error return codes. */ - if (activate_lv_excl_remote(cmd, lv)) - stack; + if (lv_is_active_exclusive(lv)) + return 1; + } else { + if (lv_is_active_exclusive_remotely(lv, cmd->node)) + return 1; + + if (!activate_lv_excl_remote(cmd, lv)) + return_0; + + + if (lv_is_active_exclusive(lv)) + return 1; + } - return lv_is_active_exclusive(lv); + return_0; } int activate_lv_excl_force(struct cmd_context *cmd, struct logical_volume *lv) @@ -583,20 +594,29 @@ int activate_lv_excl_force(struct cmd_co if (!vg_is_clustered(lv->vg)) return activate_lv_excl_local(cmd, lv); - if (lv_is_active_exclusive_locally(lv)) - return 1; + if (!cmd->node) { + if (lv_is_active_exclusive_locally(lv)) + return 1; - if (!activate_lv_excl_local_force(cmd, lv)) - return_0; + if (!activate_lv_excl_local_force(cmd, lv)) + return_0; - if (lv_is_active_exclusive(lv)) - return 1; - /* FIXME Deal with error return codes. */ - if (activate_lv_excl_remote_force(cmd, lv)) - stack; + if (lv_is_active_exclusive(lv)) + return 1; + } else { + if (lv_is_active_exclusive_remotely(lv, cmd->node)) + return 1; + + if (!activate_lv_excl_remote_force(cmd, lv)) + return_0; + + + if (lv_is_active_exclusive(lv)) + return 1; + } - return lv_is_active_exclusive(lv); + return_0; } /* Lock a list of LVs */ @@ -635,7 +655,7 @@ int locking_is_clustered(void) return (_locking.flags & LCK_CLUSTERED) ? 1 : 0; } -int remote_lock_held(const char *vol, int *exclusive) +int remote_lock_held(const char *vol, int *exclusive, const char *node) { int mode = LCK_NULL; @@ -648,7 +668,7 @@ int remote_lock_held(const char *vol, in /* * If an error occured, expect that volume is active */ - if (!_locking.query_resource(vol, &mode)) { + if (!_locking.query_resource(vol, &mode, node)) { stack; return 1; } diff -urNp LVM2.2.02.98.orig/lib/locking/locking.h LVM2.2.02.98/lib/locking/locking.h --- LVM2.2.02.98.orig/lib/locking/locking.h 2012-12-28 13:32:10.000000000 +0000 +++ LVM2.2.02.98/lib/locking/locking.h 2013-03-05 10:59:47.885191648 +0000 @@ -25,7 +25,7 @@ void reset_locking(void); int vg_write_lock_held(void); int locking_is_clustered(void); -int remote_lock_held(const char *vol, int *exclusive); +int remote_lock_held(const char *vol, int *exclusive, const char *node); /* * LCK_VG: @@ -186,7 +186,7 @@ int check_lvm1_vg_inactive(struct cmd_co #define activate_lv_excl_local_force(cmd, lv) \ lock_lv_vol(cmd, lv, LCK_LV_EXCLUSIVE | LCK_HOLD | LCK_LOCAL | (lv_is_active(lv) && ! lv_is_active_exclusive(lv) ? LCK_TRY_CONVERT : 0)) #define activate_lv_excl_remote_force(cmd, lv) \ - lock_lv_vol(cmd, lv, LCK_LV_EXCLUSIVE | LCK_HOLD | LCK_REMOTE | (lv_is_active(lv) && ! lv_is_active_exclusive(lv) ? LCK_TRY_CONVERT : 0)) + lock_lv_vol(cmd, lv, LCK_LV_EXCLUSIVE | LCK_HOLD | LCK_REMOTE | (lv_is_active_remotely(lv, cmd->node) && ! lv_is_active_exclusive(lv) ? LCK_TRY_CONVERT : 0)) struct logical_volume; int activate_lv_excl(struct cmd_context *cmd, struct logical_volume *lv); @@ -197,8 +197,14 @@ int deactivate_lv(struct cmd_context *cm lock_lv_vol(cmd, lv, LCK_LV_ACTIVATE | LCK_HOLD | LCK_LOCAL) #define activate_lv_local_force(cmd, lv) \ lock_lv_vol(cmd, lv, LCK_LV_ACTIVATE | LCK_HOLD | LCK_LOCAL | (lv_is_active_exclusive_locally(lv) ? LCK_TRY_CONVERT : 0)) +#define activate_lv_remote(cmd, lv) \ + lock_lv_vol(cmd, lv, LCK_LV_ACTIVATE | LCK_HOLD | LCK_REMOTE) +#define activate_lv_remote_force(cmd, lv) \ + lock_lv_vol(cmd, lv, LCK_LV_ACTIVATE | LCK_HOLD | LCK_REMOTE | (lv_is_active_exclusive_remotely(lv, cmd->node) ? LCK_TRY_CONVERT : 0)) #define deactivate_lv_local(cmd, lv) \ lock_lv_vol(cmd, lv, LCK_LV_DEACTIVATE | LCK_LOCAL) +#define deactivate_lv_remote(cmd, lv) \ + lock_lv_vol(cmd, lv, LCK_LV_DEACTIVATE | LCK_REMOTE) #define drop_cached_metadata(vg) \ lock_vol((vg)->cmd, (vg)->name, LCK_VG_DROP_CACHE) #define remote_commit_cached_metadata(vg) \ diff -urNp LVM2.2.02.98.orig/lib/locking/locking_types.h LVM2.2.02.98/lib/locking/locking_types.h --- LVM2.2.02.98.orig/lib/locking/locking_types.h 2012-10-15 14:24:58.000000000 +0000 +++ LVM2.2.02.98/lib/locking/locking_types.h 2013-03-05 10:59:47.886265186 +0000 @@ -18,7 +18,7 @@ typedef int (*lock_resource_fn) (struct cmd_context * cmd, const char *resource, uint32_t flags); -typedef int (*query_resource_fn) (const char *resource, int *mode); +typedef int (*query_resource_fn) (const char *resource, int *mode, const char *node); typedef void (*fin_lock_fn) (void); typedef void (*reset_lock_fn) (void); diff -urNp LVM2.2.02.98.orig/tools/args.h LVM2.2.02.98/tools/args.h --- LVM2.2.02.98.orig/tools/args.h 2012-10-15 14:24:59.000000000 +0000 +++ LVM2.2.02.98/tools/args.h 2013-03-05 10:59:47.888190332 +0000 @@ -76,6 +76,7 @@ arg(discards_ARG, '\0', "discards", disc arg(stripes_long_ARG, '\0', "stripes", int_arg, 0) arg(sysinit_ARG, '\0', "sysinit", NULL, 0) arg(thinpool_ARG, '\0', "thinpool", string_arg, 0) +arg(node_ARG, '\0', "node", string_arg, 0) /* Allow some variations */ arg(resizable_ARG, '\0', "resizable", yes_no_arg, 0) diff -urNp LVM2.2.02.98.orig/tools/commands.h LVM2.2.02.98/tools/commands.h --- LVM2.2.02.98.orig/tools/commands.h 2012-10-15 14:24:59.000000000 +0000 +++ LVM2.2.02.98/tools/commands.h 2013-03-05 10:59:47.889258249 +0000 @@ -87,6 +87,7 @@ xx(lvchange, "\t[-y|--yes]\n" "\t[--version]\n" "\t[-Z|--zero {y|n}]\n" + "\t[--node Node ]\n" "\tLogicalVolume[Path] [LogicalVolume[Path]...]\n", alloc_ARG, autobackup_ARG, activate_ARG, available_ARG, contiguous_ARG, @@ -94,7 +95,7 @@ xx(lvchange, major_ARG, minor_ARG, monitor_ARG, noudevsync_ARG, partial_ARG, permission_ARG, persistent_ARG, poll_ARG, readahead_ARG, resync_ARG, refresh_ARG, addtag_ARG, deltag_ARG, sysinit_ARG, test_ARG, yes_ARG, - zero_ARG) + zero_ARG, node_ARG) xx(lvconvert, "Change logical volume layout", diff -urNp LVM2.2.02.98.orig/tools/lvchange.c LVM2.2.02.98/tools/lvchange.c --- LVM2.2.02.98.orig/tools/lvchange.c 2012-12-28 13:32:10.000000000 +0000 +++ LVM2.2.02.98/tools/lvchange.c 2013-03-05 10:59:47.891443499 +0000 @@ -226,11 +226,22 @@ static int _lvchange_activate(struct cmd } if (activate == CHANGE_ALN) { - log_verbose("Deactivating logical volume \"%s\" locally", - lv->name); - if (!deactivate_lv_local(cmd, lv)) - return_0; + if (cmd->node) { + log_verbose("Deactivating logical volume \"%s\" locally on node %s", + lv->name, cmd->node); + if (!deactivate_lv_remote(cmd, lv)) + return_0; + } else { + log_verbose("Deactivating logical volume \"%s\" locally", + lv->name); + if (!deactivate_lv_local(cmd, lv)) + return_0; + } } else if (activate == CHANGE_AN) { + if (cmd->node) { + log_error("Use -aln to deactivate volume on a remote node"); + return_0; + } log_verbose("Deactivating logical volume \"%s\"", lv->name); if (!deactivate_lv(cmd, lv)) return_0; @@ -239,29 +250,55 @@ static int _lvchange_activate(struct cmd lv_is_origin(lv) || lv_is_thin_type(lv)) { if (arg_count(cmd, force_ARG)) { - log_verbose("Activating logical volume \"%s\" " - "exclusively (forced)", lv->name); + if (cmd->node) + log_verbose("Activating logical volume \"%s\" " + "exclusively on node %s (forced)", lv->name, cmd->node); + else + log_verbose("Activating logical volume \"%s\" " + "exclusively (forced)", lv->name); if (!activate_lv_excl_force(cmd, lv)) return_0; } else { - log_verbose("Activating logical volume \"%s\" " - "exclusively", lv->name); + if (cmd->node) + log_verbose("Activating logical volume \"%s\" " + "exclusively on node %s", lv->name, cmd->node); + else + log_verbose("Activating logical volume \"%s\" " + "exclusively", lv->name); if (!activate_lv_excl(cmd, lv)) return_0; } } else if (activate == CHANGE_ALY) { - if (arg_count(cmd, force_ARG)) { - log_verbose("Activating logical volume \"%s\" locally (forced)", - lv->name); - if (!activate_lv_local_force(cmd, lv)) - return_0; + if (cmd->node) { + if (arg_count(cmd, force_ARG)) { + log_verbose("Activating logical volume \"%s\" locally on node %s (forced)", + lv->name, cmd->node); + if (!activate_lv_remote_force(cmd, lv)) + return_0; + } else { + log_verbose("Activating logical volume \"%s\" locally on node %s", + lv->name, cmd->node); + if (!activate_lv_remote(cmd, lv)) + return_0; + } } else { - log_verbose("Activating logical volume \"%s\" locally", - lv->name); - if (!activate_lv_local(cmd, lv)) - return_0; + if (arg_count(cmd, force_ARG)) { + log_verbose("Activating logical volume \"%s\" locally (forced)", + lv->name); + if (!activate_lv_local_force(cmd, lv)) + return_0; + } else { + log_verbose("Activating logical volume \"%s\" locally", + lv->name); + if (!activate_lv_local(cmd, lv)) + return_0; + } } } else { + if (cmd->node) { + log_error("Use -aly or -aey to activate volume on a remote node"); + return_0; + } log_verbose("Activating logical volume \"%s\"", lv->name); if (!activate_lv(cmd, lv)) @@ -808,6 +845,25 @@ static int lvchange_single(struct cmd_co return ECMD_FAILED; } + switch (arg_count(cmd, node_ARG)) { + case 1: + if (!arg_count(cmd, activate_ARG)) { + log_error("--node argument may be used only with -a"); + return ECMD_FAILED; + } + if (!vg_is_clustered(lv->vg)) { + log_error("--node argument may be used only with clustered volume groups"); + return ECMD_FAILED; + } + cmd->node = arg_str_value(cmd, node_ARG, NULL); + break; + case 0: + break; + default: + log_error("Only one --node argument is supported"); + return ECMD_FAILED; + break; + } /* * FIXME: DEFAULT_BACKGROUND_POLLING should be "unspecified". * If --poll is explicitly provided use it; otherwise polling diff -urNp LVM2.2.02.98.orig/tools/pvmove.c LVM2.2.02.98/tools/pvmove.c --- LVM2.2.02.98.orig/tools/pvmove.c 2012-10-15 14:24:59.000000000 +0000 +++ LVM2.2.02.98/tools/pvmove.c 2013-03-05 10:59:47.893194162 +0000 @@ -240,7 +240,7 @@ static struct logical_volume *_set_up_pv } if (vg_is_clustered(vg) && - lv_is_active_exclusive_remotely(lv)) { + lv_is_active_exclusive_remotely(lv, NULL)) { lv_skipped = 1; log_print_unless_silent("Skipping LV %s which is activated " "exclusively on remote node.", lv->name); diff -urNp LVM2.2.02.98.orig/tools/vgchange.c LVM2.2.02.98/tools/vgchange.c --- LVM2.2.02.98.orig/tools/vgchange.c 2012-10-15 14:24:59.000000000 +0000 +++ LVM2.2.02.98/tools/vgchange.c 2013-03-05 10:59:47.894198836 +0000 @@ -125,7 +125,7 @@ static int _activate_lvs_in_vg(struct cm * If the LV is active exclusive remotely, * then ignore it here */ - if (lv_is_active_exclusive_remotely(lv)) { + if (lv_is_active_exclusive_remotely(lv, NULL)) { log_verbose("%s/%s is exclusively active on" " a remote node", vg->name, lv->name); continue; ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG 2013-03-06 7:58 ` Vladislav Bogdanov @ 2013-03-06 9:15 ` Andreas Pflug 2013-03-06 9:35 ` Vladislav Bogdanov 0 siblings, 1 reply; 20+ messages in thread From: Andreas Pflug @ 2013-03-06 9:15 UTC (permalink / raw) To: Vladislav Bogdanov; +Cc: LVM general discussion and development Am 06.03.13 08:58, schrieb Vladislav Bogdanov: > 06.03.2013 10:40, Andreas Pflug wrote: >> Am 01.03.13 16:41, schrieb Vladislav Bogdanovr? >> >>> Hi Andreas, >>> Lock convertion is only enabled if you pass --force flag. >>> Also, to upgrade local lock to exclusive one, you need to ensure IIRC >>> that no more node holds local lock. >> Hm, tried that as well: >> >> tools/lvm lvchange --force -aey -vvvv vg/locktest >> >> --force changes the error from "resource busy" to "invalid argument": > Is volume active on other nodes at that time? I made sure it's not active on other nodes: lvchange -an vg/locktest ; lvchange -aly vg/locktest > And do you run clvmd from that build tree as well? > > Also, can you please try attached patch (on top of that one you have)? I > polished conversion a bit more, denying -an if volume is ex-locked > somewhere and other fixes to logic. I tried that additional patch. I'm running this test versions on my test node only (including clvmd), the other nodes are still running clvmd 2.2.95 (I guess this shouldn't matter since all are inactive). Same result: #lvchange.c:258 Activating logical volume "locktest" exclusively (forced) #activate/dev_manager.c:284 Getting device info for vg-locktest [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN] #ioctl/libdm-iface.c:1724 dm info LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN NF [16384] (*1) #activate/activate.c:1067 vg/locktest is active #activate/dev_manager.c:284 Getting device info for vg-locktest [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN] #ioctl/libdm-iface.c:1724 dm info LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN NF [16384] (*1) #activate/activate.c:1067 vg/locktest is active #activate/dev_manager.c:284 Getting device info for vg-locktest [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN] #ioctl/libdm-iface.c:1724 dm info LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN NF [16384] (*1) #activate/activate.c:1067 vg/locktest is active #locking/cluster_locking.c:513 Locking LV oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN EX (LV|NONBLOCK|CLUSTER|LOCAL|CONVERT) (0x40dd) #locking/cluster_locking.c:400 Error locking on node 7400a8c0: Invalid argument > This patch also allows locking (activation) to be performed on remote > nodes. I only tested this with corosync2 (which is set up in a way > latest pacemaker - post-1.1.8 git master - needs, nodes has additional > 'name' value in nodelist, please see > http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-name.html). I'm running corosync 1.4.2 (debian wheezy). Regards, Andreas ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG 2013-03-06 9:15 ` Andreas Pflug @ 2013-03-06 9:35 ` Vladislav Bogdanov 2013-03-06 9:59 ` Andreas Pflug 0 siblings, 1 reply; 20+ messages in thread From: Vladislav Bogdanov @ 2013-03-06 9:35 UTC (permalink / raw) To: Andreas Pflug; +Cc: LVM general discussion and development 06.03.2013 12:15, Andreas Pflug wrote: > Am 06.03.13 08:58, schrieb Vladislav Bogdanov: >> 06.03.2013 10:40, Andreas Pflug wrote: >>> Am 01.03.13 16:41, schrieb Vladislav Bogdanovr? >>> >>>> Hi Andreas, >>>> Lock convertion is only enabled if you pass --force flag. >>>> Also, to upgrade local lock to exclusive one, you need to ensure IIRC >>>> that no more node holds local lock. >>> Hm, tried that as well: >>> >>> tools/lvm lvchange --force -aey -vvvv vg/locktest >>> >>> --force changes the error from "resource busy" to "invalid argument": >> Is volume active on other nodes at that time? > > I made sure it's not active on other nodes: lvchange -an vg/locktest ; > lvchange -aly vg/locktest >> And do you run clvmd from that build tree as well? >> >> Also, can you please try attached patch (on top of that one you have)? I >> polished conversion a bit more, denying -an if volume is ex-locked >> somewhere and other fixes to logic. > I tried that additional patch. I'm running this test versions on my test > node only (including clvmd), the other nodes are still running clvmd > 2.2.95 (I guess this shouldn't matter since all are inactive). Same result: I believe this matters, because error you see is received from a remote node. Is node with ID 7400a8c0 local? > > #lvchange.c:258 Activating logical volume "locktest" exclusively > (forced) > #activate/dev_manager.c:284 Getting device info for vg-locktest > [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN] > #ioctl/libdm-iface.c:1724 dm info > LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN > NF [16384] (*1) > #activate/activate.c:1067 vg/locktest is active > #activate/dev_manager.c:284 Getting device info for vg-locktest > [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN] > #ioctl/libdm-iface.c:1724 dm info > LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN > NF [16384] (*1) > #activate/activate.c:1067 vg/locktest is active > #activate/dev_manager.c:284 Getting device info for vg-locktest > [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN] > #ioctl/libdm-iface.c:1724 dm info > LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN > NF [16384] (*1) > #activate/activate.c:1067 vg/locktest is active > #locking/cluster_locking.c:513 Locking LV > oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN EX > (LV|NONBLOCK|CLUSTER|LOCAL|CONVERT) (0x40dd) > #locking/cluster_locking.c:400 Error locking on node 7400a8c0: Invalid > argument > > >> This patch also allows locking (activation) to be performed on remote >> nodes. I only tested this with corosync2 (which is set up in a way >> latest pacemaker - post-1.1.8 git master - needs, nodes has additional >> 'name' value in nodelist, please see >> http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-name.html). >> > > I'm running corosync 1.4.2 (debian wheezy). Which cluster manager interface does clvmd detect? corosync or openais? You should use former, openais one is(was) using LCK service which is very unstable. > > Regards, > Andreas ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG 2013-03-06 9:35 ` Vladislav Bogdanov @ 2013-03-06 9:59 ` Andreas Pflug 2013-03-06 11:20 ` Vladislav Bogdanov 0 siblings, 1 reply; 20+ messages in thread From: Andreas Pflug @ 2013-03-06 9:59 UTC (permalink / raw) To: Vladislav Bogdanov; +Cc: LVM general discussion and development Am 06.03.13 10:35, schrieb Vladislav Bogdanov: > 06.03.2013 12:15, Andreas Pflug wrote: >> I made sure it's not active on other nodes: lvchange -an vg/locktest ; >> lvchange -aly vg/locktest >>> And do you run clvmd from that build tree as well? >>> >>> Also, can you please try attached patch (on top of that one you have)? I >>> polished conversion a bit more, denying -an if volume is ex-locked >>> somewhere and other fixes to logic. >> I tried that additional patch. I'm running this test versions on my test >> node only (including clvmd), the other nodes are still running clvmd >> 2.2.95 (I guess this shouldn't matter since all are inactive). Same result: > I believe this matters, because error you see is received from a remote > node. Is node with ID 7400a8c0 local? Yes, that's the test node. Hm, not funny if I have to upgrade all nodes on the production system... I'm a little surprised that remote inactive nodes need to be aware of that force-exclusive stuff. > > I'm running corosync 1.4.2 (debian wheezy). > Which cluster manager interface does clvmd detect? corosync or openais? > You should use former, openais one is(was) using LCK service which is > very unstable. It's using openais. I'm not too happy about the stability, so maybe I'd switch to corosync now. Could this be a reason for the x-lock failure as well? Regards, Andreas ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG 2013-03-06 9:59 ` Andreas Pflug @ 2013-03-06 11:20 ` Vladislav Bogdanov 2013-03-06 12:17 ` Andreas Pflug 0 siblings, 1 reply; 20+ messages in thread From: Vladislav Bogdanov @ 2013-03-06 11:20 UTC (permalink / raw) To: Andreas Pflug; +Cc: LVM general discussion and development 06.03.2013 12:59, Andreas Pflug wrote: > Am 06.03.13 10:35, schrieb Vladislav Bogdanov: >> 06.03.2013 12:15, Andreas Pflug wrote: >>> I made sure it's not active on other nodes: lvchange -an vg/locktest ; >>> lvchange -aly vg/locktest >>>> And do you run clvmd from that build tree as well? >>>> >>>> Also, can you please try attached patch (on top of that one you >>>> have)? I >>>> polished conversion a bit more, denying -an if volume is ex-locked >>>> somewhere and other fixes to logic. >>> I tried that additional patch. I'm running this test versions on my test >>> node only (including clvmd), the other nodes are still running clvmd >>> 2.2.95 (I guess this shouldn't matter since all are inactive). Same >>> result: >> I believe this matters, because error you see is received from a remote >> node. Is node with ID 7400a8c0 local? > Yes, that's the test node. But you receive "Invalid argument" error from the cluster layer, so is should go from a local clvmd if you are correct about node IDs. You may run clvmd in a debugging mode as well to catch what goes wrong. > Hm, not funny if I have to upgrade all nodes on the production system... > I'm a little surprised that remote inactive nodes need to be aware of > that force-exclusive stuff. You probably should try to test that on a test cluster first (as I do) although I'm almost sure it works correctly in setup I described. >> >> I'm running corosync 1.4.2 (debian wheezy). >> Which cluster manager interface does clvmd detect? corosync or openais? >> You should use former, openais one is(was) using LCK service which is >> very unstable. > It's using openais. I'm not too happy about the stability, so maybe I'd > switch to corosync now. That could be the problem btw. I did neither test nor look at openais module implementation in clvmd, because I had plenty problems with it (actually with LCK under it) in the past, so I even forced to use corosync (CPG) + dlm instead of detected openais (CPG+LCK) for older systems (look at -I switch of clvmd). And, openais is deprecated upstream, so I do not see any reason to use it. Even gfs_controld (which is probably the only well-known user of openais services) actually does not strictly require it, at least I was able to port it to pure CPG+DLM with dlm4 on top of corosync2, which is not compatible with openais plugins. Also you may need quorum patch found in this list, it does its job well. > Could this be a reason for the x-lock failure as well? Can you please reword? Vladislav ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG 2013-03-06 11:20 ` Vladislav Bogdanov @ 2013-03-06 12:17 ` Andreas Pflug 2013-03-06 13:28 ` Vladislav Bogdanov 0 siblings, 1 reply; 20+ messages in thread From: Andreas Pflug @ 2013-03-06 12:17 UTC (permalink / raw) To: Vladislav Bogdanov; +Cc: LVM general discussion and development Am 06.03.13 12:20, schrieb Vladislav Bogdanov: >>> I'm running corosync 1.4.2 (debian wheezy). >>> Which cluster manager interface does clvmd detect? corosync or openais? >>> You should use former, openais one is(was) using LCK service which is >>> very unstable. >> It's using openais. I'm not too happy about the stability, so maybe I'd >> switch to corosync now. > That could be the problem btw. I did neither test nor look at openais > module implementation in clvmd, because I had plenty problems with it > (actually with LCK under it) in the past, so I even forced to use > corosync (CPG) + dlm instead of detected openais (CPG+LCK) for older > systems (look at -I switch of clvmd). > > And, openais is deprecated upstream, so I do not see any reason to use > it. Even gfs_controld (which is probably the only well-known user of > openais services) actually does not strictly require it, at least I was > able to port it to pure CPG+DLM with dlm4 on top of corosync2, which is > not compatible with openais plugins. > > Also you may need quorum patch found in this list, it does its job well. > >> Could this be a reason for the x-lock failure as well? You just answered the quirky question :-) Unfortunately, corosync/dlm don't work for me as expected. When starting clvmd -I corosync (with dlm kernel module loaded), creating the dlm lockspace "clvmd" fails, with dlm: no local IP address has been set dlm: cannot start dlm lowcomms -107 I haven't found any hint what might go wrong on the machine (checked already hostname resolves correctly to its ip address via /etc/hosts; corosync uses that network too). Regards Andreas ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG 2013-03-06 12:17 ` Andreas Pflug @ 2013-03-06 13:28 ` Vladislav Bogdanov 2013-03-13 15:14 ` [linux-lvm] LVM snapshot with Clustered VG [SOLVED] Andreas Pflug 0 siblings, 1 reply; 20+ messages in thread From: Vladislav Bogdanov @ 2013-03-06 13:28 UTC (permalink / raw) To: Andreas Pflug; +Cc: LVM general discussion and development 06.03.2013 15:17, Andreas Pflug wrote: > Am 06.03.13 12:20, schrieb Vladislav Bogdanov: >>>> I'm running corosync 1.4.2 (debian wheezy). >>>> Which cluster manager interface does clvmd detect? corosync or openais? >>>> You should use former, openais one is(was) using LCK service which is >>>> very unstable. >>> It's using openais. I'm not too happy about the stability, so maybe I'd >>> switch to corosync now. >> That could be the problem btw. I did neither test nor look at openais >> module implementation in clvmd, because I had plenty problems with it >> (actually with LCK under it) in the past, so I even forced to use >> corosync (CPG) + dlm instead of detected openais (CPG+LCK) for older >> systems (look at -I switch of clvmd). >> >> And, openais is deprecated upstream, so I do not see any reason to use >> it. Even gfs_controld (which is probably the only well-known user of >> openais services) actually does not strictly require it, at least I was >> able to port it to pure CPG+DLM with dlm4 on top of corosync2, which is >> not compatible with openais plugins. >> >> Also you may need quorum patch found in this list, it does its job well. >> >>> Could this be a reason for the x-lock failure as well? > You just answered the quirky question :-) > > Unfortunately, corosync/dlm don't work for me as expected. When starting > clvmd -I corosync (with dlm kernel module loaded), creating the dlm > lockspace "clvmd" fails, with > > dlm: no local IP address has been set > dlm: cannot start dlm lowcomms -107 > You need to have dlm_controld running on all nodes. And that is not trivial with corosync1. You need to either use cman or use deprecated dlm_controld.pcmk which was removed from a cluster (cman) package after 3.0.17. Latter does not work well without heavy patching (http://www.mail-archive.com/pacemaker@oss.clusterlabs.org/msg09959.html , http://www.mail-archive.com/pacemaker@oss.clusterlabs.org/msg11123.html and http://oss.clusterlabs.org/pipermail/pacemaker/2012-February/013073.html). Even after that there are some problems with it. Latest versions (4.x) of dlm (it was split from cman) work fine with corosync2 and pacemaker 1.1.6+. But I doubt it can ever be compiled with corosync1. Yes, it cannot, because it requires quorum which appeared in corosync2. You may look at thread http://oss.clusterlabs.org/pipermail/pacemaker/2012-October/015826.html for some info. That's all why I moved to corosync2. You can search through pacemaker list archives for dlm-related messages from me to get full picture. You may also look at slides I presented at LVEE winter 2013: http://lvee.org/uploads/image_upload/file/267/Linux_Clusters_at_LVEE_2013.pdf > I haven't found any hint what might go wrong on the machine (checked > already hostname resolves correctly to its ip address via /etc/hosts; > corosync uses that network too). > > Regards > Andreas ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED] 2013-03-06 13:28 ` Vladislav Bogdanov @ 2013-03-13 15:14 ` Andreas Pflug 2013-03-13 16:53 ` Vladislav Bogdanov 0 siblings, 1 reply; 20+ messages in thread From: Andreas Pflug @ 2013-03-13 15:14 UTC (permalink / raw) To: Vladislav Bogdanov; +Cc: LVM general discussion and development Hi Vladislav, I finally succeeded making all working as expected! What I did: - compile corosync 2.3.0 (http://corosync.org/download/corosync-2.3.0.tar.gz) - compile libqb 0.14.4 (https://fedorahosted.org/releases/q/u/quarterback/libqb-0.14.4.tar.gz) - compile dlm_4.0.1 (http://git.fedorahosted.org/cgit/dlm.git/snapshot/dlm-4.0.1.tar.gz) - Compiled LVM 2.2.98 (http://git.fedorahosted.org/cgit/lvm2.git/snapshot/lvm2-2_02_98.tar.gz) with your PREVIOUS patch ( https://www.redhat.com/archives/linux-lvm/2013-January/msg00006.html ) I made a 2-node test cluster, both accessing a test iscsi target, all daemons load fine. Everything works as expected! Snapshotting is possible after -aey --force explicit exclusive locking, lvresize works as well. Previously, I tried your latest patch. I had a plethiora of errors, starting with lvcreate failing: #metadata/lv_manip.c:4137 Clearing start of logical volume "testvol2" #device/dev-cache.c:599 /dev/san2/testvol2: stat failed: No such file or directory #metadata/lv_manip.c:4140 /dev/san2/testvol2: not found: device not cleared #metadata/lv_manip.c:4620 Aborting. Failed to wipe start of new LV. I need to add -Z n to have it created. After that, most activations failed, with error code 5. I went back to vanilla 2.2.98, all ok except exclusive locking, applied the initial locking patch and voila! All done. Thanks for your support! Andreas ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED] 2013-03-13 15:14 ` [linux-lvm] LVM snapshot with Clustered VG [SOLVED] Andreas Pflug @ 2013-03-13 16:53 ` Vladislav Bogdanov 2013-03-13 17:37 ` Andreas Pflug 0 siblings, 1 reply; 20+ messages in thread From: Vladislav Bogdanov @ 2013-03-13 16:53 UTC (permalink / raw) To: Andreas Pflug; +Cc: LVM general discussion and development 13.03.2013 18:14, Andreas Pflug wrote: > Hi Vladislav, > > I finally succeeded making all working as expected! > > What I did: > - compile corosync 2.3.0 > (http://corosync.org/download/corosync-2.3.0.tar.gz) > - compile libqb 0.14.4 > (https://fedorahosted.org/releases/q/u/quarterback/libqb-0.14.4.tar.gz) > - compile dlm_4.0.1 > (http://git.fedorahosted.org/cgit/dlm.git/snapshot/dlm-4.0.1.tar.gz) > - Compiled LVM 2.2.98 > (http://git.fedorahosted.org/cgit/lvm2.git/snapshot/lvm2-2_02_98.tar.gz) > with your PREVIOUS patch ( > https://www.redhat.com/archives/linux-lvm/2013-January/msg00006.html ) > > I made a 2-node test cluster, both accessing a test iscsi target, all > daemons load fine. > Everything works as expected! Snapshotting is possible after -aey > --force explicit exclusive locking, lvresize works as well. > > Previously, I tried your latest patch. I had a plethiora of errors, > starting with lvcreate failing: > > #metadata/lv_manip.c:4137 Clearing start of logical volume "testvol2" > #device/dev-cache.c:599 /dev/san2/testvol2: stat failed: No such > file or directory > #metadata/lv_manip.c:4140 /dev/san2/testvol2: not found: device not > cleared > #metadata/lv_manip.c:4620 Aborting. Failed to wipe start of new LV. Yep, I know about this, I reworked that patch a lot, and now it does every thing I wanted without regressions. I will try to post updated version in the next few days when I manage to apply it to a git tree. > > I need to add -Z n to have it created. After that, most activations > failed, with error code 5. I went back to vanilla 2.2.98, all ok except > exclusive locking, applied the initial locking patch and voila! All done. > > Thanks for your support! > Andreas > > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED] 2013-03-13 16:53 ` Vladislav Bogdanov @ 2013-03-13 17:37 ` Andreas Pflug 2013-03-13 18:30 ` Vladislav Bogdanov 0 siblings, 1 reply; 20+ messages in thread From: Andreas Pflug @ 2013-03-13 17:37 UTC (permalink / raw) To: Vladislav Bogdanov; +Cc: LVM general discussion and development Am 13.03.13 17:53, schrieb Vladislav Bogdanov: > > #metadata/lv_manip.c:4137 Clearing start of logical volume "testvol2" > #device/dev-cache.c:599 /dev/san2/testvol2: stat failed: No such > file or directory > #metadata/lv_manip.c:4140 /dev/san2/testvol2: not found: device not > cleared > #metadata/lv_manip.c:4620 Aborting. Failed to wipe start of new LV. > Yep, I know about this, I reworked that patch a lot, and now it does > every thing I wanted without regressions. > I will try to post updated version in the next few days when I manage to > apply it to a git tree. Is there a way to find out if a LV is locked exclusively? lvs displaying -e-- instead of -a-- would be nice. Seems not even lvdisplay knows about exclusive locking. Regards, Andreas ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED] 2013-03-13 17:37 ` Andreas Pflug @ 2013-03-13 18:30 ` Vladislav Bogdanov 2013-03-14 21:57 ` Andreas Pflug 0 siblings, 1 reply; 20+ messages in thread From: Vladislav Bogdanov @ 2013-03-13 18:30 UTC (permalink / raw) To: Andreas Pflug; +Cc: LVM general discussion and development 13.03.2013 20:37, Andreas Pflug wrote: > Am 13.03.13 17:53, schrieb Vladislav Bogdanov: >> >> #metadata/lv_manip.c:4137 Clearing start of logical volume "testvol2" >> #device/dev-cache.c:599 /dev/san2/testvol2: stat failed: No such >> file or directory >> #metadata/lv_manip.c:4140 /dev/san2/testvol2: not found: device not >> cleared >> #metadata/lv_manip.c:4620 Aborting. Failed to wipe start of new LV. >> Yep, I know about this, I reworked that patch a lot, and now it does >> every thing I wanted without regressions. >> I will try to post updated version in the next few days when I manage to >> apply it to a git tree. > > Is there a way to find out if a LV is locked exclusively? lvs displaying > -e-- instead of -a-- would be nice. Seems not even lvdisplay knows about > exclusive locking. That would break other tools which rely on their output. F.e. cluster resource agents of libvirt (yes, it runs lvm tools rather then using API, which is not yet complete btw). As I also need to obtain this information, I think about writing simple tool (f.e. clvm_tool) which would display needed info. As a workaround you can run lvchange -aly without force parameter. If it succeeds, the volume is locked in a shared mode, otherwise it is locked exclusively. > > Regards, > Andreas ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED] 2013-03-13 18:30 ` Vladislav Bogdanov @ 2013-03-14 21:57 ` Andreas Pflug 2013-03-15 9:00 ` Zdenek Kabelac 0 siblings, 1 reply; 20+ messages in thread From: Andreas Pflug @ 2013-03-14 21:57 UTC (permalink / raw) To: Vladislav Bogdanov; +Cc: LVM general discussion and development On 03/13/13 19:30, Vladislav Bogdanov wrote: > >> Is there a way to find out if a LV is locked exclusively? lvs displaying >> -e-- instead of -a-- would be nice. Seems not even lvdisplay knows about >> exclusive locking. > That would break other tools which rely on their output. F.e. cluster > resource agents of libvirt (yes, it runs lvm tools rather then using > API, which is not yet complete btw). As I also need to obtain this > information, I think about writing simple tool (f.e. clvm_tool) which > would display needed info. > > As a workaround you can run lvchange -aly without force parameter. If it > succeeds, the volume is locked in a shared mode, otherwise it is locked > exclusively. Hm, thats one ugly workaround... How about a clvmd option, something like -l to list all locks and exit. Regards, Andreas ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED] 2013-03-14 21:57 ` Andreas Pflug @ 2013-03-15 9:00 ` Zdenek Kabelac 2013-03-15 9:29 ` Vladislav Bogdanov 0 siblings, 1 reply; 20+ messages in thread From: Zdenek Kabelac @ 2013-03-15 9:00 UTC (permalink / raw) To: LVM general discussion and development; +Cc: Vladislav Bogdanov, Andreas Pflug Dne 14.3.2013 22:57, Andreas Pflug napsal(a): > On 03/13/13 19:30, Vladislav Bogdanov wrote: >> >>> Is there a way to find out if a LV is locked exclusively? lvs displaying >>> -e-- instead of -a-- would be nice. Seems not even lvdisplay knows about >>> exclusive locking. >> That would break other tools which rely on their output. F.e. cluster >> resource agents of libvirt (yes, it runs lvm tools rather then using >> API, which is not yet complete btw). As I also need to obtain this >> information, I think about writing simple tool (f.e. clvm_tool) which >> would display needed info. >> >> As a workaround you can run lvchange -aly without force parameter. If it >> succeeds, the volume is locked in a shared mode, otherwise it is locked >> exclusively. > > Hm, thats one ugly workaround... > How about a clvmd option, something like -l to list all locks and exit. > I think - the extension to 'lvs' command could be relatively simple (adding a new column) You may query for exclusive/local activation on the node. (So you cannot just tell on which other node is the device active, but you could print about these states: active exclusive local active exclusive active local active Zdenek ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED] 2013-03-15 9:00 ` Zdenek Kabelac @ 2013-03-15 9:29 ` Vladislav Bogdanov 2013-03-15 9:37 ` Zdenek Kabelac 0 siblings, 1 reply; 20+ messages in thread From: Vladislav Bogdanov @ 2013-03-15 9:29 UTC (permalink / raw) To: Zdenek Kabelac; +Cc: Andreas Pflug, LVM general discussion and development 15.03.2013 12:00, Zdenek Kabelac wrote: > Dne 14.3.2013 22:57, Andreas Pflug napsal(a): >> On 03/13/13 19:30, Vladislav Bogdanov wrote: >>> >>>> Is there a way to find out if a LV is locked exclusively? lvs >>>> displaying >>>> -e-- instead of -a-- would be nice. Seems not even lvdisplay knows >>>> about >>>> exclusive locking. >>> That would break other tools which rely on their output. F.e. cluster >>> resource agents of libvirt (yes, it runs lvm tools rather then using >>> API, which is not yet complete btw). As I also need to obtain this >>> information, I think about writing simple tool (f.e. clvm_tool) which >>> would display needed info. >>> >>> As a workaround you can run lvchange -aly without force parameter. If it >>> succeeds, the volume is locked in a shared mode, otherwise it is locked >>> exclusively. >> >> Hm, thats one ugly workaround... >> How about a clvmd option, something like -l to list all locks and exit. >> > > > I think - the extension to 'lvs' command could be relatively simple > (adding a new column) Yes, that's correct. > > You may query for exclusive/local activation on the node. > (So you cannot just tell on which other node is the device active, > but you could print about these states: > > active exclusive local > active exclusive > active local > active You also may poll all know nodes, but that is a hack too. That's why I prefer to have this as a separate tool (with dlm_tool-like params and output) which lists node IDs and lock mode. Unfortunately do not have power to write it now. Are core LVM devels interested in these two features: lock conversion and managing remote node locks? If yes, then I can (hopefully) prepare git patches next week. Vladislav ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED] 2013-03-15 9:29 ` Vladislav Bogdanov @ 2013-03-15 9:37 ` Zdenek Kabelac 2013-03-15 12:53 ` Vladislav Bogdanov 0 siblings, 1 reply; 20+ messages in thread From: Zdenek Kabelac @ 2013-03-15 9:37 UTC (permalink / raw) To: Vladislav Bogdanov; +Cc: Andreas Pflug, LVM general discussion and development Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a): > 15.03.2013 12:00, Zdenek Kabelac wrote: >> Dne 14.3.2013 22:57, Andreas Pflug napsal(a): >>> On 03/13/13 19:30, Vladislav Bogdanov wrote: >>>> >>>>> Is there a way to find out if a LV is locked exclusively? lvs >>>>> displaying >>>>> -e-- instead of -a-- would be nice. Seems not even lvdisplay knows >>>>> about >>>>> exclusive locking. >>>> That would break other tools which rely on their output. F.e. cluster >>>> resource agents of libvirt (yes, it runs lvm tools rather then using >>>> API, which is not yet complete btw). As I also need to obtain this >>>> information, I think about writing simple tool (f.e. clvm_tool) which >>>> would display needed info. >>>> >>>> As a workaround you can run lvchange -aly without force parameter. If it >>>> succeeds, the volume is locked in a shared mode, otherwise it is locked >>>> exclusively. >>> >>> Hm, thats one ugly workaround... >>> How about a clvmd option, something like -l to list all locks and exit. >>> >> >> >> I think - the extension to 'lvs' command could be relatively simple >> (adding a new column) > > Yes, that's correct. > >> >> You may query for exclusive/local activation on the node. >> (So you cannot just tell on which other node is the device active, >> but you could print about these states: >> >> active exclusive local >> active exclusive >> active local >> active > > You also may poll all know nodes, but that is a hack too. > > That's why I prefer to have this as a separate tool (with dlm_tool-like > params and output) which lists node IDs and lock mode. Unfortunately do > not have power to write it now. > > Are core LVM devels interested in these two features: lock conversion > and managing remote node locks? If yes, then I can (hopefully) prepare > git patches next week. I'm not quite sure what do you mean by 'managing remote node locks' ? Current login behind lvm command is - You could activate LVs with the above syntax [ael] (there is a tag support - so you could exclusively activate LV on remote node in via some configuration tags) And you want to 'upgrade' remote locks to something else ? What would be the use-case you could not resolve with current command line args? Is that supported by dlm (since lvm locks are mapped to dlm)? How would you resolve error path fallbacks ? Also I believe the clvmd protocol is out of free bits for extension, so how the protocol would look like ? Zdenek ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED] 2013-03-15 9:37 ` Zdenek Kabelac @ 2013-03-15 12:53 ` Vladislav Bogdanov 2013-03-15 13:11 ` Vladislav Bogdanov 2013-03-15 13:32 ` Zdenek Kabelac 0 siblings, 2 replies; 20+ messages in thread From: Vladislav Bogdanov @ 2013-03-15 12:53 UTC (permalink / raw) To: LVM general discussion and development; +Cc: Andreas Pflug, Zdenek Kabelac 15.03.2013 12:37, Zdenek Kabelac wrote: > Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a): >> 15.03.2013 12:00, Zdenek Kabelac wrote: >>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a): >>>> On 03/13/13 19:30, Vladislav Bogdanov wrote: >>>>> >>>>>> Is there a way to find out if a LV is locked exclusively? lvs >>>>>> displaying >>>>>> -e-- instead of -a-- would be nice. Seems not even lvdisplay knows >>>>>> about >>>>>> exclusive locking. >>>>> That would break other tools which rely on their output. F.e. cluster >>>>> resource agents of libvirt (yes, it runs lvm tools rather then using >>>>> API, which is not yet complete btw). As I also need to obtain this >>>>> information, I think about writing simple tool (f.e. clvm_tool) which >>>>> would display needed info. >>>>> >>>>> As a workaround you can run lvchange -aly without force parameter. >>>>> If it >>>>> succeeds, the volume is locked in a shared mode, otherwise it is >>>>> locked >>>>> exclusively. >>>> >>>> Hm, thats one ugly workaround... >>>> How about a clvmd option, something like -l to list all locks and exit. >>>> >>> >>> >>> I think - the extension to 'lvs' command could be relatively simple >>> (adding a new column) >> >> Yes, that's correct. >> >>> >>> You may query for exclusive/local activation on the node. >>> (So you cannot just tell on which other node is the device active, >>> but you could print about these states: >>> >>> active exclusive local >>> active exclusive >>> active local >>> active >> >> You also may poll all know nodes, but that is a hack too. >> >> That's why I prefer to have this as a separate tool (with dlm_tool-like >> params and output) which lists node IDs and lock mode. Unfortunately do >> not have power to write it now. >> >> Are core LVM devels interested in these two features: lock conversion >> and managing remote node locks? If yes, then I can (hopefully) prepare >> git patches next week. > > > I'm not quite sure what do you mean by 'managing remote node locks' ? Activation/deactivation of LVs on a different corosync cluster node, specified by its node name (with pacemaker-like method to determine that name). Also conversion of locks on that node. > > Current login behind lvm command is - > > You could activate LVs with the above syntax [ael] > (there is a tag support - so you could exclusively activate LV on remote > node in via some configuration tags) Could you please explain this - I do not see anything relevant in man pages. > > And you want to 'upgrade' remote locks to something else ? Yes, shared-to-exclusive end vice verse. > > What would be the use-case you could not resolve with current command > line args? I need to convert lock on a remote node during last stage of ver3 migration in libvirt/qemu. That is a "confirm" stage, which runs on a "source" node, during which "old" vm is killed and disk is released. So, I first ("begin" stage) convert lock from exclusive to shared on a source node, then obtain shared lock on a target node (during "prepare" stage, when receiving qemu instance is started), then migrate vm between two processes which have LV opened, and then release shared lock on a source node ("confirm" stage, after source qemu is killed). There is no other events on a destination node in ver3 migration protocol, so I'm unable to convert lock to exclusive there after migration is finished. So I do that from a source node, after it released lock. > > Is that supported by dlm (since lvm locks are mapped to dlm)? Command just sent to a specific clvm instance and performed there. > How would you resolve error path fallbacks ? Could you please tell what exactly do you mean? If dlm on a remote node is unable to perform requested operation, then error is returned to a initiator. > Also I believe the clvmd protocol is out of free bits for extension, > so how the protocol would look like ? It contains 'node' field (I assume it was never actually used before) and with some fixes that works. Vladislav ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED] 2013-03-15 12:53 ` Vladislav Bogdanov @ 2013-03-15 13:11 ` Vladislav Bogdanov 2013-03-15 13:32 ` Zdenek Kabelac 1 sibling, 0 replies; 20+ messages in thread From: Vladislav Bogdanov @ 2013-03-15 13:11 UTC (permalink / raw) To: LVM general discussion and development; +Cc: Andreas Pflug, Zdenek Kabelac 15.03.2013 15:53, Vladislav Bogdanov wrote: > 15.03.2013 12:37, Zdenek Kabelac wrote: >> Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a): >>> 15.03.2013 12:00, Zdenek Kabelac wrote: >>>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a): >>>>> On 03/13/13 19:30, Vladislav Bogdanov wrote: >>>>>> >>>>>>> Is there a way to find out if a LV is locked exclusively? lvs >>>>>>> displaying >>>>>>> -e-- instead of -a-- would be nice. Seems not even lvdisplay knows >>>>>>> about >>>>>>> exclusive locking. >>>>>> That would break other tools which rely on their output. F.e. cluster >>>>>> resource agents of libvirt (yes, it runs lvm tools rather then using >>>>>> API, which is not yet complete btw). As I also need to obtain this >>>>>> information, I think about writing simple tool (f.e. clvm_tool) which >>>>>> would display needed info. >>>>>> >>>>>> As a workaround you can run lvchange -aly without force parameter. >>>>>> If it >>>>>> succeeds, the volume is locked in a shared mode, otherwise it is >>>>>> locked >>>>>> exclusively. >>>>> >>>>> Hm, thats one ugly workaround... >>>>> How about a clvmd option, something like -l to list all locks and exit. >>>>> >>>> >>>> >>>> I think - the extension to 'lvs' command could be relatively simple >>>> (adding a new column) >>> >>> Yes, that's correct. >>> >>>> >>>> You may query for exclusive/local activation on the node. >>>> (So you cannot just tell on which other node is the device active, >>>> but you could print about these states: >>>> >>>> active exclusive local >>>> active exclusive >>>> active local >>>> active >>> >>> You also may poll all know nodes, but that is a hack too. >>> >>> That's why I prefer to have this as a separate tool (with dlm_tool-like >>> params and output) which lists node IDs and lock mode. Unfortunately do >>> not have power to write it now. >>> >>> Are core LVM devels interested in these two features: lock conversion >>> and managing remote node locks? If yes, then I can (hopefully) prepare >>> git patches next week. >> >> >> I'm not quite sure what do you mean by 'managing remote node locks' ? > > Activation/deactivation of LVs on a different corosync cluster node, > specified by its node name (with pacemaker-like method to determine that > name). > Also conversion of locks on that node. > >> >> Current login behind lvm command is - >> >> You could activate LVs with the above syntax [ael] >> (there is a tag support - so you could exclusively activate LV on remote >> node in via some configuration tags) > > Could you please explain this - I do not see anything relevant in man pages. > >> >> And you want to 'upgrade' remote locks to something else ? > > Yes, shared-to-exclusive end vice verse. > >> >> What would be the use-case you could not resolve with current command >> line args? > > I need to convert lock on a remote node during last stage of ver3 > migration in libvirt/qemu. That is a "confirm" stage, which runs on a > "source" node, during which "old" vm is killed and disk is released. > So, I first ("begin" stage) convert lock from exclusive to shared on a > source node, then obtain shared lock on a target node (during "prepare" > stage, when receiving qemu instance is started), then migrate vm between > two processes which have LV opened, and then release shared lock on a > source node ("confirm" stage, after source qemu is killed). > > There is no other events on a destination node in ver3 migration > protocol, so I'm unable to convert lock to exclusive there after > migration is finished. So I do that from a source node, after it > released lock. > >> >> Is that supported by dlm (since lvm locks are mapped to dlm)? > Command just sent to a specific clvm instance and performed there. > >> How would you resolve error path fallbacks ? > > Could you please tell what exactly do you mean? > If dlm on a remote node is unable to perform requested operation, then > error is returned to a initiator. > >> Also I believe the clvmd protocol is out of free bits for extension, >> so how the protocol would look like ? > It contains 'node' field (I assume it was never actually used before) > and with some fixes that works. Fix to myself. clvm client API has that field. And there is no need for changes in on-wire protocol - corosync module already allows to send a message to a specific csid (node). > > Vladislav > > _______________________________________________ > linux-lvm mailing list > linux-lvm@redhat.com > https://www.redhat.com/mailman/listinfo/linux-lvm > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED] 2013-03-15 12:53 ` Vladislav Bogdanov 2013-03-15 13:11 ` Vladislav Bogdanov @ 2013-03-15 13:32 ` Zdenek Kabelac 2013-03-15 14:51 ` Vladislav Bogdanov 1 sibling, 1 reply; 20+ messages in thread From: Zdenek Kabelac @ 2013-03-15 13:32 UTC (permalink / raw) To: Vladislav Bogdanov; +Cc: Andreas Pflug, LVM general discussion and development Dne 15.3.2013 13:53, Vladislav Bogdanov napsal(a): > 15.03.2013 12:37, Zdenek Kabelac wrote: >> Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a): >>> 15.03.2013 12:00, Zdenek Kabelac wrote: >>>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a): >>>>> On 03/13/13 19:30, Vladislav Bogdanov wrote: >>>>>> >>> You could activate LVs with the above syntax [ael] >> (there is a tag support - so you could exclusively activate LV on remote >> node in via some configuration tags) > > Could you please explain this - I do not see anything relevant in man pages. Let's say - you have 3 nodes A, B, C - each have a TAG_A, TAG_B, TAG_C, then on node A you may exclusively activate LV which has TAG_B - this will try to exclusively active LV on the node which has it configured in lvm.conf (see the volume_list= []) > >> >> And you want to 'upgrade' remote locks to something else ? > > Yes, shared-to-exclusive end vice verse. So how do you convert the lock from shared to exclusive without unlock (if I get it right - you keep the ConcurrentRead lock - and you want to take Exlusive - to make change state from 'active' to 'active exlusive') https://en.wikipedia.org/wiki/Distributed_lock_manager Clvmd 'communicates' via these locks. So the proper algorithm needs to be there for ending with some proper state after locks changes (and sorry I'm not a dlm expert here) >> >> What would be the use-case you could not resolve with current command >> line args? > > I need to convert lock on a remote node during last stage of ver3 > migration in libvirt/qemu. That is a "confirm" stage, which runs on a > "source" node, during which "old" vm is killed and disk is released. > So, I first ("begin" stage) convert lock from exclusive to shared on a > source node, then obtain shared lock on a target node (during "prepare" Which most probably works only your environment - since you do not try to 'break' the logic - but it's probably not a generic concept - i.e. in this controlled environment you may live probably happily even with local activation of LVs - since you always know what the tool is doing. > There is no other events on a destination node in ver3 migration > protocol, so I'm unable to convert lock to exclusive there after > migration is finished. So I do that from a source node, after it > released lock. > >> >> Is that supported by dlm (since lvm locks are mapped to dlm)? > Command just sent to a specific clvm instance and performed there. As said - the 'lock' is the thing which controls the activation state. So faking it on the software level may possible lead to inconsistency between the dlm and clvmd view of the lock state. Zdenek ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED] 2013-03-15 13:32 ` Zdenek Kabelac @ 2013-03-15 14:51 ` Vladislav Bogdanov 2013-03-15 15:02 ` Zdenek Kabelac 0 siblings, 1 reply; 20+ messages in thread From: Vladislav Bogdanov @ 2013-03-15 14:51 UTC (permalink / raw) To: Zdenek Kabelac; +Cc: Andreas Pflug, LVM general discussion and development 15.03.2013 16:32, Zdenek Kabelac wrote: > Dne 15.3.2013 13:53, Vladislav Bogdanov napsal(a): >> 15.03.2013 12:37, Zdenek Kabelac wrote: >>> Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a): >>>> 15.03.2013 12:00, Zdenek Kabelac wrote: >>>>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a): >>>>>> On 03/13/13 19:30, Vladislav Bogdanov wrote: >>>>>>> >>>> You could activate LVs with the above syntax [ael] >>> (there is a tag support - so you could exclusively activate LV on remote >>> node in via some configuration tags) >> >> Could you please explain this - I do not see anything relevant in man >> pages. > > Let's say - you have 3 nodes A, B, C - each have a TAG_A, TAG_B, TAG_C, > then on node A you may exclusively activate LV which has TAG_B - this > will try to exclusively active LV on the node which has it configured > in lvm.conf (see the volume_list= []) Aha, if I understand correctly this is absolutely not what I need. I want all this to be fully dynamic without any "config-editing voodoo". > >> >>> >>> And you want to 'upgrade' remote locks to something else ? >> >> Yes, shared-to-exclusive end vice verse. > > So how do you convert the lock from shared to exclusive without unlock > (if I get it right - you keep the ConcurrentRead lock - and you want to > take Exlusive - to make change state from 'active' to 'active exlusive') > https://en.wikipedia.org/wiki/Distributed_lock_manager I just pass LCKF_CONVERT to dlm_controld if requested and needed. And that is dlm's task to either satisfy conversion or to refuse it. > > Clvmd 'communicates' via these locks. Not exactly true. clvmd does cluster communications with corosync, which implements virtual synchrony, so all cluster nodes receive messages in the same order. At the bottom, clvmd uses libdlm to communicate with dlm_controld and request it to lock/unlock. dlm_controld instances use corosync for membership and locally manages in-kernel dlm counter-part, which uses tcp/sctp mesh-like connections to communicate. So request from one clvmd instance goes to another and goes to kernel from there, and then it is distributed to other nodes. Actually that does not matter where does it hits kernel space if it supports delegation of locks to remote nodes, but I suspect it doesn't. But if it doesn't support such thing, then the only option to manage lock on a remote node is to request that's node dlm instance to do the locking job. > So the proper algorithm needs to be there for ending with some proper > state after locks changes (and sorry I'm not a dlm expert here) That is what actually happens. There is just no difference between running (to upgrade local lock to exclusive on node <node>. ssh <node> lvchange -aey --force VG/LV or lvchange -aey --node <node> --force VG/LV The same command, it is just sent via different channels. Again, I just send locking request to a remote clvmd instance through corosync. It asks its local dlm to convert (acquire, release) lock and returns its answer back. After dlm answers, operation is either performed, and then OK is send back to a initiator, or refused, and the error is sent back. > > >>> >>> What would be the use-case you could not resolve with current command >>> line args? >> >> I need to convert lock on a remote node during last stage of ver3 >> migration in libvirt/qemu. That is a "confirm" stage, which runs on a >> "source" node, during which "old" vm is killed and disk is released. >> So, I first ("begin" stage) convert lock from exclusive to shared on a >> source node, then obtain shared lock on a target node (during "prepare" > > Which most probably works only your environment - since you do not try to > 'break' the logic - but it's probably not a generic concept - i.e. > in this controlled environment you may live probably happily even with > local activation of LVs - since you always know what the tool is doing. I prefer to do not allow anybody else to open an LV at all (and to be able to take snapshots of it when needed on a node where it is exclusively locked). That's why I implemented a lock-manager driver for libvirt, which controls inter-node locks with clvm exclusive activation, and then controls inner open attempts with local locking mechanism. While libvirt already has two lok drivers, neither of fully satisfy me because or shared storage requirement. And, I want to take snapshots, that's why I need exclusive activation. I plan to send libvirt patches to the libvirt list if lvm part is accepted (otherwise it makes no sense to do that and I will need to maintain everything out-of-the-tree). > >> There is no other events on a destination node in ver3 migration >> protocol, so I'm unable to convert lock to exclusive there after >> migration is finished. So I do that from a source node, after it >> released lock. >> >>> >>> Is that supported by dlm (since lvm locks are mapped to dlm)? >> Command just sent to a specific clvm instance and performed there. > > As said - the 'lock' is the thing which controls the activation state. > So faking it on the software level may possible lead to inconsistency > between the dlm and clvmd view of the lock state. No faking. Just a remote management of the same lock. Vladislav ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED] 2013-03-15 14:51 ` Vladislav Bogdanov @ 2013-03-15 15:02 ` Zdenek Kabelac 2013-03-15 15:36 ` Vladislav Bogdanov 0 siblings, 1 reply; 20+ messages in thread From: Zdenek Kabelac @ 2013-03-15 15:02 UTC (permalink / raw) To: Vladislav Bogdanov; +Cc: Andreas Pflug, LVM general discussion and development Dne 15.3.2013 15:51, Vladislav Bogdanov napsal(a): > 15.03.2013 16:32, Zdenek Kabelac wrote: >> Dne 15.3.2013 13:53, Vladislav Bogdanov napsal(a): >>> 15.03.2013 12:37, Zdenek Kabelac wrote: >>>> Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a): >>>>> 15.03.2013 12:00, Zdenek Kabelac wrote: >>>>>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a): >>>>>>> On 03/13/13 19:30, Vladislav Bogdanov wrote: >>>>>>>> >>>>> You could activate LVs with the above syntax [ael] >>>> (there is a tag support - so you could exclusively activate LV on remote >>>> node in via some configuration tags) >>> >>> Could you please explain this - I do not see anything relevant in man >>> pages. >> >> Let's say - you have 3 nodes A, B, C - each have a TAG_A, TAG_B, TAG_C, >> then on node A you may exclusively activate LV which has TAG_B - this >> will try to exclusively active LV on the node which has it configured >> in lvm.conf (see the volume_list= []) > > Aha, if I understand correctly this is absolutely not what I need. > I want all this to be fully dynamic without any "config-editing voodoo". > >> >>> >>>> >>>> And you want to 'upgrade' remote locks to something else ? >>> >>> Yes, shared-to-exclusive end vice verse. >> >> So how do you convert the lock from shared to exclusive without unlock >> (if I get it right - you keep the ConcurrentRead lock - and you want to >> take Exlusive - to make change state from 'active' to 'active exlusive') >> https://en.wikipedia.org/wiki/Distributed_lock_manager > > I just pass LCKF_CONVERT to dlm_controld if requested and needed. And > that is dlm's task to either satisfy conversion or to refuse it. > So to understand myself better this thing - the dlm sends 'unlock' requests to all other nodes except the one which should be converted to exclusive mode and send exclusive lock to the prefered node? >> >> Clvmd 'communicates' via these locks. > > Not exactly true. > > clvmd does cluster communications with corosync, which implements > virtual synchrony, so all cluster nodes receive messages in the same order. > At the bottom, clvmd uses libdlm to communicate with dlm_controld and > request it to lock/unlock. > dlm_controld instances use corosync for membership and locally manages > in-kernel dlm counter-part, which uses tcp/sctp mesh-like connections to > communicate. > So request from one clvmd instance goes to another and goes to kernel > from there, and then it is distributed to other nodes. Actually that > does not matter where does it hits kernel space if it supports > delegation of locks to remote nodes, but I suspect it doesn't. But if it > doesn't support such thing, then the only option to manage lock on a > remote node is to request that's node dlm instance to do the locking job. > >> So the proper algorithm needs to be there for ending with some proper >> state after locks changes (and sorry I'm not a dlm expert here) > > That is what actually happens. > There is just no difference between running (to upgrade local lock to > exclusive on node <node>. > > ssh <node> lvchange -aey --force VG/LV > > or > > lvchange -aey --node <node> --force VG/LV --node is exactly what the tag is for - each node may have it's tag. lvm doesn't work with cluster nodes. The question is - could be the code transformed to use this logic ? I guess you need to know dlm node name here right ? > The same command, it is just sent via different channels. > > Again, I just send locking request to a remote clvmd instance through > corosync. > It asks its local dlm to convert (acquire, release) lock and returns its > answer back. After dlm answers, operation is either performed, and then > OK is send back to a initiator, or refused, and the error is sent back. >>> There is no other events on a destination node in ver3 migration >>> protocol, so I'm unable to convert lock to exclusive there after >>> migration is finished. So I do that from a source node, after it >>> released lock. >>> >>>> >>>> Is that supported by dlm (since lvm locks are mapped to dlm)? >>> Command just sent to a specific clvm instance and performed there. >> >> As said - the 'lock' is the thing which controls the activation state. >> So faking it on the software level may possible lead to inconsistency >> between the dlm and clvmd view of the lock state. > > No faking. Just a remote management of the same lock. Could you repost patches against git ? With some usage examples ? Zdenek ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED] 2013-03-15 15:02 ` Zdenek Kabelac @ 2013-03-15 15:36 ` Vladislav Bogdanov 2013-03-15 15:55 ` Zdenek Kabelac 0 siblings, 1 reply; 20+ messages in thread From: Vladislav Bogdanov @ 2013-03-15 15:36 UTC (permalink / raw) To: Zdenek Kabelac; +Cc: Andreas Pflug, LVM general discussion and development 15.03.2013 18:02, Zdenek Kabelac wrote: > Dne 15.3.2013 15:51, Vladislav Bogdanov napsal(a): >> 15.03.2013 16:32, Zdenek Kabelac wrote: >>> Dne 15.3.2013 13:53, Vladislav Bogdanov napsal(a): >>>> 15.03.2013 12:37, Zdenek Kabelac wrote: >>>>> Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a): >>>>>> 15.03.2013 12:00, Zdenek Kabelac wrote: >>>>>>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a): >>>>>>>> On 03/13/13 19:30, Vladislav Bogdanov wrote: >>>>>>>>> >>>>>> You could activate LVs with the above syntax [ael] >>>>> (there is a tag support - so you could exclusively activate LV on >>>>> remote >>>>> node in via some configuration tags) >>>> >>>> Could you please explain this - I do not see anything relevant in man >>>> pages. >>> >>> Let's say - you have 3 nodes A, B, C - each have a TAG_A, TAG_B, TAG_C, >>> then on node A you may exclusively activate LV which has TAG_B - this >>> will try to exclusively active LV on the node which has it configured >>> in lvm.conf (see the volume_list= []) >> >> Aha, if I understand correctly this is absolutely not what I need. >> I want all this to be fully dynamic without any "config-editing voodoo". >> >>> >>>> >>>>> >>>>> And you want to 'upgrade' remote locks to something else ? >>>> >>>> Yes, shared-to-exclusive end vice verse. >>> >>> So how do you convert the lock from shared to exclusive without unlock >>> (if I get it right - you keep the ConcurrentRead lock - and you want to >>> take Exlusive - to make change state from 'active' to 'active >>> exlusive') >>> https://en.wikipedia.org/wiki/Distributed_lock_manager >> >> I just pass LCKF_CONVERT to dlm_controld if requested and needed. And >> that is dlm's task to either satisfy conversion or to refuse it. >> > > So to understand myself better this thing - > > the dlm sends 'unlock' requests to all other nodes except the one which > should be converted to exclusive mode and send exclusive lock to the > prefered node? No. clvmd sends request to a remote clvmd to upgrade or acquire or release the lock. That remote instance asks local dlm to do the job. dlm either says OK or says ERROR. It does not do anything except that. It LV is locked on a remote node, be it shared or exclusive lock, dlm says ERROR if exclusive lock (or conversion to it) is requested. My patches also allow "-an --force" to release shared locks on other nodes. Exclusive lock may be released or downgraded only on node which holds it (or with --node <node>). > >>> >>> Clvmd 'communicates' via these locks. >> >> Not exactly true. >> >> clvmd does cluster communications with corosync, which implements >> virtual synchrony, so all cluster nodes receive messages in the same >> order. >> At the bottom, clvmd uses libdlm to communicate with dlm_controld and >> request it to lock/unlock. >> dlm_controld instances use corosync for membership and locally manages >> in-kernel dlm counter-part, which uses tcp/sctp mesh-like connections to >> communicate. >> So request from one clvmd instance goes to another and goes to kernel >> from there, and then it is distributed to other nodes. Actually that >> does not matter where does it hits kernel space if it supports >> delegation of locks to remote nodes, but I suspect it doesn't. But if it >> doesn't support such thing, then the only option to manage lock on a >> remote node is to request that's node dlm instance to do the locking job. >> >>> So the proper algorithm needs to be there for ending with some proper >>> state after locks changes (and sorry I'm not a dlm expert here) >> >> That is what actually happens. >> There is just no difference between running (to upgrade local lock to >> exclusive on node <node>. >> >> ssh <node> lvchange -aey --force VG/LV >> >> or >> >> lvchange -aey --node <node> --force VG/LV > > > --node is exactly what the tag is for - each node may have it's tag. > lvm doesn't work with cluster nodes. But corosync and dlm operate node IDs, and pacemaker operates node names and IDs. None of them use tags. > > The question is - could be the code transformed to use this logic ? > I guess you need to know dlm node name here right ? Node IDs are obtained from corosync membership list, and may be used for that. If corosync is configured with nodelist in a way pacemaker wants it (http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-name.html), then node names may be used too. > > >> The same command, it is just sent via different channels. >> >> Again, I just send locking request to a remote clvmd instance through >> corosync. >> It asks its local dlm to convert (acquire, release) lock and returns its >> answer back. After dlm answers, operation is either performed, and then >> OK is send back to a initiator, or refused, and the error is sent back. > > >>>> There is no other events on a destination node in ver3 migration >>>> protocol, so I'm unable to convert lock to exclusive there after >>>> migration is finished. So I do that from a source node, after it >>>> released lock. >>>> >>>>> >>>>> Is that supported by dlm (since lvm locks are mapped to dlm)? >>>> Command just sent to a specific clvm instance and performed there. >>> >>> As said - the 'lock' is the thing which controls the activation state. >>> So faking it on the software level may possible lead to inconsistency >>> between the dlm and clvmd view of the lock state. >> >> No faking. Just a remote management of the same lock. > > Could you repost patches against git ? I plan that for the next week. > With some usage examples ? Yes, if you give me an "example of example" ;) Vladislav ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED] 2013-03-15 15:36 ` Vladislav Bogdanov @ 2013-03-15 15:55 ` Zdenek Kabelac 2013-03-15 17:16 ` Vladislav Bogdanov 0 siblings, 1 reply; 20+ messages in thread From: Zdenek Kabelac @ 2013-03-15 15:55 UTC (permalink / raw) To: Vladislav Bogdanov; +Cc: Andreas Pflug, LVM general discussion and development Dne 15.3.2013 16:36, Vladislav Bogdanov napsal(a): > 15.03.2013 18:02, Zdenek Kabelac wrote: >> Dne 15.3.2013 15:51, Vladislav Bogdanov napsal(a): >>> 15.03.2013 16:32, Zdenek Kabelac wrote: >>>> Dne 15.3.2013 13:53, Vladislav Bogdanov napsal(a): >>>>> 15.03.2013 12:37, Zdenek Kabelac wrote: >>>>>> Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a): >>>>>>> 15.03.2013 12:00, Zdenek Kabelac wrote: >>>>>>>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a): >>>>>>>>> On 03/13/13 19:30, Vladislav Bogdanov wrote: >>>>>>>>>> >>>>>>> You could activate LVs with the above syntax [ael] >>>>>> (there is a tag support - so you could exclusively activate LV on >>>>>> remote >>>>>> node in via some configuration tags) >>>>> >>>>> Could you please explain this - I do not see anything relevant in man >>>>> pages. >>>> >>>> Let's say - you have 3 nodes A, B, C - each have a TAG_A, TAG_B, TAG_C, >>>> then on node A you may exclusively activate LV which has TAG_B - this >>>> will try to exclusively active LV on the node which has it configured >>>> in lvm.conf (see the volume_list= []) >>> >>> Aha, if I understand correctly this is absolutely not what I need. >>> I want all this to be fully dynamic without any "config-editing voodoo". >>> >>>> >>>>> >>>>>> >>>>>> And you want to 'upgrade' remote locks to something else ? >>>>> >>>>> Yes, shared-to-exclusive end vice verse. >>>> >>>> So how do you convert the lock from shared to exclusive without unlock >>>> (if I get it right - you keep the ConcurrentRead lock - and you want to >>>> take Exlusive - to make change state from 'active' to 'active >>>> exlusive') >>>> https://en.wikipedia.org/wiki/Distributed_lock_manager >>> >>> I just pass LCKF_CONVERT to dlm_controld if requested and needed. And >>> that is dlm's task to either satisfy conversion or to refuse it. >>> >> >> So to understand myself better this thing - >> >> the dlm sends 'unlock' requests to all other nodes except the one which >> should be converted to exclusive mode and send exclusive lock to the >> prefered node? > > No. > clvmd sends request to a remote clvmd to upgrade or acquire or release > the lock. > That remote instance asks local dlm to do the job. dlm either says OK or > says ERROR. > It does not do anything except that. > It LV is locked on a remote node, be it shared or exclusive lock, dlm > says ERROR if exclusive lock (or conversion to it) is requested. > > My patches also allow "-an --force" to release shared locks on other > nodes. Exclusive lock may be released or downgraded only on node which > holds it (or with --node <node>). > >> >>>> >>>> Clvmd 'communicates' via these locks. >>> >>> Not exactly true. >>> >>> clvmd does cluster communications with corosync, which implements >>> virtual synchrony, so all cluster nodes receive messages in the same >>> order. >>> At the bottom, clvmd uses libdlm to communicate with dlm_controld and >>> request it to lock/unlock. >>> dlm_controld instances use corosync for membership and locally manages >>> in-kernel dlm counter-part, which uses tcp/sctp mesh-like connections to >>> communicate. >>> So request from one clvmd instance goes to another and goes to kernel >>> from there, and then it is distributed to other nodes. Actually that >>> does not matter where does it hits kernel space if it supports >>> delegation of locks to remote nodes, but I suspect it doesn't. But if it >>> doesn't support such thing, then the only option to manage lock on a >>> remote node is to request that's node dlm instance to do the locking job. >>> >>>> So the proper algorithm needs to be there for ending with some proper >>>> state after locks changes (and sorry I'm not a dlm expert here) >>> >>> That is what actually happens. >>> There is just no difference between running (to upgrade local lock to >>> exclusive on node <node>. >>> >>> ssh <node> lvchange -aey --force VG/LV >>> >>> or >>> >>> lvchange -aey --node <node> --force VG/LV >> >> >> --node is exactly what the tag is for - each node may have it's tag. >> lvm doesn't work with cluster nodes. > > But corosync and dlm operate node IDs, and pacemaker operates node names > and IDs. None of them use tags. > >> >> The question is - could be the code transformed to use this logic ? >> I guess you need to know dlm node name here right ? > > Node IDs are obtained from corosync membership list, and may be used for > that. If corosync is configured with nodelist in a way pacemaker wants > it > (http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-name.html), > then node names may be used too. clvmd knows the dlm node name - but lvm command should reference things via tags. There will be probably no way to add '--node' option into lvm command. Can you think about using 'tags'? So your machines will have configured tags (be it machine name) and instead of the node - you would use 'tag' ? Zdenek ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED] 2013-03-15 15:55 ` Zdenek Kabelac @ 2013-03-15 17:16 ` Vladislav Bogdanov 0 siblings, 0 replies; 20+ messages in thread From: Vladislav Bogdanov @ 2013-03-15 17:16 UTC (permalink / raw) To: Zdenek Kabelac; +Cc: Andreas Pflug, LVM general discussion and development 15.03.2013 18:55, Zdenek Kabelac пишет: > Dne 15.3.2013 16:36, Vladislav Bogdanov napsal(a): >> 15.03.2013 18:02, Zdenek Kabelac wrote: >>> Dne 15.3.2013 15:51, Vladislav Bogdanov napsal(a): >>>> 15.03.2013 16:32, Zdenek Kabelac wrote: >>>>> Dne 15.3.2013 13:53, Vladislav Bogdanov napsal(a): >>>>>> 15.03.2013 12:37, Zdenek Kabelac wrote: >>>>>>> Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a): >>>>>>>> 15.03.2013 12:00, Zdenek Kabelac wrote: >>>>>>>>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a): >>>>>>>>>> On 03/13/13 19:30, Vladislav Bogdanov wrote: >>>>>>>>>>> >>>>>>>> You could activate LVs with the above syntax [ael] >>>>>>> (there is a tag support - so you could exclusively activate LV on >>>>>>> remote >>>>>>> node in via some configuration tags) >>>>>> >>>>>> Could you please explain this - I do not see anything relevant in man >>>>>> pages. >>>>> >>>>> Let's say - you have 3 nodes A, B, C - each have a TAG_A, TAG_B, >>>>> TAG_C, >>>>> then on node A you may exclusively activate LV which has TAG_B - this >>>>> will try to exclusively active LV on the node which has it configured >>>>> in lvm.conf (see the volume_list= []) >>>> >>>> Aha, if I understand correctly this is absolutely not what I need. >>>> I want all this to be fully dynamic without any "config-editing >>>> voodoo". >>>> >>>>> >>>>>> >>>>>>> >>>>>>> And you want to 'upgrade' remote locks to something else ? >>>>>> >>>>>> Yes, shared-to-exclusive end vice verse. >>>>> >>>>> So how do you convert the lock from shared to exclusive without >>>>> unlock >>>>> (if I get it right - you keep the ConcurrentRead lock - and you >>>>> want to >>>>> take Exlusive - to make change state from 'active' to 'active >>>>> exlusive') >>>>> https://en.wikipedia.org/wiki/Distributed_lock_manager >>>> >>>> I just pass LCKF_CONVERT to dlm_controld if requested and needed. And >>>> that is dlm's task to either satisfy conversion or to refuse it. >>>> >>> >>> So to understand myself better this thing - >>> >>> the dlm sends 'unlock' requests to all other nodes except the one which >>> should be converted to exclusive mode and send exclusive lock to the >>> prefered node? >> >> No. >> clvmd sends request to a remote clvmd to upgrade or acquire or release >> the lock. >> That remote instance asks local dlm to do the job. dlm either says OK or >> says ERROR. >> It does not do anything except that. >> It LV is locked on a remote node, be it shared or exclusive lock, dlm >> says ERROR if exclusive lock (or conversion to it) is requested. >> >> My patches also allow "-an --force" to release shared locks on other >> nodes. Exclusive lock may be released or downgraded only on node which >> holds it (or with --node <node>). >> >>> >>>>> >>>>> Clvmd 'communicates' via these locks. >>>> >>>> Not exactly true. >>>> >>>> clvmd does cluster communications with corosync, which implements >>>> virtual synchrony, so all cluster nodes receive messages in the same >>>> order. >>>> At the bottom, clvmd uses libdlm to communicate with dlm_controld and >>>> request it to lock/unlock. >>>> dlm_controld instances use corosync for membership and locally manages >>>> in-kernel dlm counter-part, which uses tcp/sctp mesh-like >>>> connections to >>>> communicate. >>>> So request from one clvmd instance goes to another and goes to kernel >>>> from there, and then it is distributed to other nodes. Actually that >>>> does not matter where does it hits kernel space if it supports >>>> delegation of locks to remote nodes, but I suspect it doesn't. But >>>> if it >>>> doesn't support such thing, then the only option to manage lock on a >>>> remote node is to request that's node dlm instance to do the locking >>>> job. >>>> >>>>> So the proper algorithm needs to be there for ending with some proper >>>>> state after locks changes (and sorry I'm not a dlm expert here) >>>> >>>> That is what actually happens. >>>> There is just no difference between running (to upgrade local lock to >>>> exclusive on node <node>. >>>> >>>> ssh <node> lvchange -aey --force VG/LV >>>> >>>> or >>>> >>>> lvchange -aey --node <node> --force VG/LV >>> >>> >>> --node is exactly what the tag is for - each node may have it's tag. >>> lvm doesn't work with cluster nodes. >> >> But corosync and dlm operate node IDs, and pacemaker operates node names >> and IDs. None of them use tags. >> >>> >>> The question is - could be the code transformed to use this logic ? >>> I guess you need to know dlm node name here right ? >> >> Node IDs are obtained from corosync membership list, and may be used for >> that. If corosync is configured with nodelist in a way pacemaker wants >> it >> (http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-name.html), >> >> then node names may be used too. > > clvmd knows the dlm node name - but lvm command should reference things > via tags. There will be probably no way to add '--node' option into lvm > command. > > Can you think about using 'tags'? > > So your machines will have configured tags (be it machine name) and > instead of the node - you would use 'tag' ? So, --tag should be used instead on --node? I'd better make --node a alias for --tag for exactly this case if that is possible... But... For those with cluster background --node is a native term while --tag is... alien one... So, I'd better leave it as is. > > Zdenek > ^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2013-03-16 11:00 UTC | newest] Thread overview: 20+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-03-15 16:31 [linux-lvm] LVM snapshot with Clustered VG [SOLVED] David Teigland 2013-03-15 17:46 ` Vladislav Bogdanov 2013-03-15 18:38 ` David Teigland 2013-03-16 11:00 ` Vladislav Bogdanov -- strict thread matches above, loose matches on Subject: below -- 2013-03-01 11:28 [linux-lvm] LVM snapshot with Clustered VG Andreas Pflug 2013-03-01 15:41 ` Vladislav Bogdanov 2013-03-06 7:40 ` Andreas Pflug 2013-03-06 7:58 ` Vladislav Bogdanov 2013-03-06 9:15 ` Andreas Pflug 2013-03-06 9:35 ` Vladislav Bogdanov 2013-03-06 9:59 ` Andreas Pflug 2013-03-06 11:20 ` Vladislav Bogdanov 2013-03-06 12:17 ` Andreas Pflug 2013-03-06 13:28 ` Vladislav Bogdanov 2013-03-13 15:14 ` [linux-lvm] LVM snapshot with Clustered VG [SOLVED] Andreas Pflug 2013-03-13 16:53 ` Vladislav Bogdanov 2013-03-13 17:37 ` Andreas Pflug 2013-03-13 18:30 ` Vladislav Bogdanov 2013-03-14 21:57 ` Andreas Pflug 2013-03-15 9:00 ` Zdenek Kabelac 2013-03-15 9:29 ` Vladislav Bogdanov 2013-03-15 9:37 ` Zdenek Kabelac 2013-03-15 12:53 ` Vladislav Bogdanov 2013-03-15 13:11 ` Vladislav Bogdanov 2013-03-15 13:32 ` Zdenek Kabelac 2013-03-15 14:51 ` Vladislav Bogdanov 2013-03-15 15:02 ` Zdenek Kabelac 2013-03-15 15:36 ` Vladislav Bogdanov 2013-03-15 15:55 ` Zdenek Kabelac 2013-03-15 17:16 ` Vladislav Bogdanov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).