Re: [linux-lvm] LVM snapshot with Clustered VG

Re: [linux-lvm] LVM snapshot with Clustered VG

[Andreas Pflug]

Hi Vladislav,

I tried the patch you mentioned in 
https://www.redhat.com/archives/linux-lvm/2013-January/msg00006.html , 
but apparently it doesn't work for me. On a locally active LV, I still get

./lvm lvchange -aey -vvvv vg/locktest

...
Activating logical volume "locktest" exclusively
#activate/dev_manager.c:284         Getting device info for vg-locktest 
[LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
#ioctl/libdm-iface.c:1724         dm info 
LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN 
NF   [16384] (*1)
#locking/cluster_locking.c:568         Lock held for 
oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN, node 
7400a8c0 : CR
#activate/activate.c:1067       vg/locktest is active
#locking/cluster_locking.c:508       Locking LV 
oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN EX 
(LV|NONBLOCK|CLUSTER|LOCAL) (0xdd)
#locking/cluster_locking.c:395   Error locking on node 7400a8c0: Device 
or resource busy

Can you give me a hint what to try further?

Regards
Andreas

Re: [linux-lvm] LVM snapshot with Clustered VG

[Vladislav Bogdanov]

Sorry, forgot to reply to list from the first attempt.

Andreas Pflug <pgadmin@pse-consulting.de> wrote:

>Hi Vladislav,
>
>I tried the patch you mentioned in 
>https://www.redhat.com/archives/linux-lvm/2013-January/msg00006.html , 
>but apparently it doesn't work for me. On a locally active LV, I still
>get
>
>./lvm lvchange -aey -vvvv vg/locktest
>
>...
>Activating logical volume "locktest" exclusively
>#activate/dev_manager.c:284         Getting device info for vg-locktest
>
>[LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
>#ioctl/libdm-iface.c:1724         dm info 
>LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN 
>NF   [16384] (*1)
>#locking/cluster_locking.c:568         Lock held for 
>oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN, node 
>7400a8c0 : CR
>#activate/activate.c:1067       vg/locktest is active
>#locking/cluster_locking.c:508       Locking LV 
>oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN EX 
>(LV|NONBLOCK|CLUSTER|LOCAL) (0xdd)
>#locking/cluster_locking.c:395   Error locking on node 7400a8c0: Device
>
>or resource busy
>
>Can you give me a hint what to try further?
>
>Regards
>Andreas

Hi Andreas,
Lock convertion is only enabled if you pass --force flag.
Also, to upgrade local lock to exclusive one, you need to ensure IIRC that no more node holds local lock.

Vladislav

Re: [linux-lvm] LVM snapshot with Clustered VG

[Andreas Pflug]

Am 01.03.13 16:41, schrieb Vladislav Bogdanov:
> Sorry, forgot to reply to list from the first attempt.
>
> Andreas Pflug <pgadmin@pse-consulting.de> wrote:
>
>> Hi Vladislav,
>>
>> I tried the patch you mentioned in
>> https://www.redhat.com/archives/linux-lvm/2013-January/msg00006.html ,
>> but apparently it doesn't work for me. On a locally active LV, I still
>> get
>>
>> ./lvm lvchange -aey -vvvv vg/locktest
>>
>> ...
>> Activating logical volume "locktest" exclusively
>> #activate/dev_manager.c:284         Getting device info for vg-locktest
>>
>> [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
>> #ioctl/libdm-iface.c:1724         dm info
>> LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN
>> NF   [16384] (*1)
>> #locking/cluster_locking.c:568         Lock held for
>> oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN, node
>> 7400a8c0 : CR
>> #activate/activate.c:1067       vg/locktest is active
>> #locking/cluster_locking.c:508       Locking LV
>> oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN EX
>> (LV|NONBLOCK|CLUSTER|LOCAL) (0xdd)
>> #locking/cluster_locking.c:395   Error locking on node 7400a8c0: Device
>>
>> or resource busy
>>
>> Can you give me a hint what to try further?
>>
>> Regards
>> Andreas
> Hi Andreas,
> Lock convertion is only enabled if you pass --force flag.
> Also, to upgrade local lock to exclusive one, you need to ensure IIRC that no more node holds local lock.

Hm, tried that as well:

tools/lvm lvchange --force -aey  -vvvv vg/locktest

--force changes the error from "resource busy" to "invalid argument":

#lvchange.c:243     Activating logical volume "locktest" exclusively 
(forced)
#activate/dev_manager.c:284         Getting device info for vg-locktest 
[LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
#ioctl/libdm-iface.c:1724         dm info 
LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN 
NF   [16384] (*1)
#locking/cluster_locking.c:568         Lock held for 
oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN, node 
7400a8c0 : CR
#activate/activate.c:1067       vg/locktest is active
#activate/dev_manager.c:284         Getting device info for vg-locktest 
[LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
#ioctl/libdm-iface.c:1724         dm info 
LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN 
NF   [16384] (*1)
#activate/activate.c:1067       vg/locktest is active
#activate/dev_manager.c:284         Getting device info for vg-locktest 
[LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
#ioctl/libdm-iface.c:1724         dm info 
LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN 
NF   [16384] (*1)
#locking/cluster_locking.c:568         Lock held for 
oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN, node 
7400a8c0 : CR
#activate/activate.c:1067       vg/locktest is active
#locking/cluster_locking.c:508       Locking LV 
oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN EX 
(LV|NONBLOCK|CLUSTER|LOCAL|CONVERT) (0x40dd)
#locking/cluster_locking.c:395   Error locking on node 7400a8c0: Invalid 
argument


Regards
Andreas

Re: [linux-lvm] LVM snapshot with Clustered VG

[Vladislav Bogdanov]

[-- Attachment #1: Type: text/plain, Size: 4135 bytes --]

06.03.2013 10:40, Andreas Pflug wrote:
> Am 01.03.13 16:41, schrieb Vladislav Bogdanov:
>> Sorry, forgot to reply to list from the first attempt.
>>
>> Andreas Pflug <pgadmin@pse-consulting.de> wrote:
>>
>>> Hi Vladislav,
>>>
>>> I tried the patch you mentioned in
>>> https://www.redhat.com/archives/linux-lvm/2013-January/msg00006.html ,
>>> but apparently it doesn't work for me. On a locally active LV, I still
>>> get
>>>
>>> ./lvm lvchange -aey -vvvv vg/locktest
>>>
>>> ...
>>> Activating logical volume "locktest" exclusively
>>> #activate/dev_manager.c:284         Getting device info for vg-locktest
>>>
>>> [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
>>> #ioctl/libdm-iface.c:1724         dm info
>>> LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN
>>> NF   [16384] (*1)
>>> #locking/cluster_locking.c:568         Lock held for
>>> oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN, node
>>> 7400a8c0 : CR
>>> #activate/activate.c:1067       vg/locktest is active
>>> #locking/cluster_locking.c:508       Locking LV
>>> oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN EX
>>> (LV|NONBLOCK|CLUSTER|LOCAL) (0xdd)
>>> #locking/cluster_locking.c:395   Error locking on node 7400a8c0: Device
>>>
>>> or resource busy
>>>
>>> Can you give me a hint what to try further?
>>>
>>> Regards
>>> Andreas
>> Hi Andreas,
>> Lock convertion is only enabled if you pass --force flag.
>> Also, to upgrade local lock to exclusive one, you need to ensure IIRC
>> that no more node holds local lock.
> 
> Hm, tried that as well:
> 
> tools/lvm lvchange --force -aey  -vvvv vg/locktest
> 
> --force changes the error from "resource busy" to "invalid argument":

Is volume active on other nodes at that time?
And do you run clvmd from that build tree as well?

Also, can you please try attached patch (on top of that one you have)? I
polished conversion a bit more, denying -an if volume is ex-locked
somewhere and other fixes to logic.
This patch also allows locking (activation) to be performed on remote
nodes. I only tested this with corosync2 (which is set up in a way
latest pacemaker - post-1.1.8 git master - needs, nodes has additional
'name' value in nodelist, please see
http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-name.html).

I did not have a time to split this all into git commits suitable for
sending upstream, so it all is a little bit messy yet.

> 
> #lvchange.c:243     Activating logical volume "locktest" exclusively
> (forced)
> #activate/dev_manager.c:284         Getting device info for vg-locktest
> [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
> #ioctl/libdm-iface.c:1724         dm info
> LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN
> NF   [16384] (*1)
> #locking/cluster_locking.c:568         Lock held for
> oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN, node
> 7400a8c0 : CR
> #activate/activate.c:1067       vg/locktest is active
> #activate/dev_manager.c:284         Getting device info for vg-locktest
> [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
> #ioctl/libdm-iface.c:1724         dm info
> LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN
> NF   [16384] (*1)
> #activate/activate.c:1067       vg/locktest is active
> #activate/dev_manager.c:284         Getting device info for vg-locktest
> [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
> #ioctl/libdm-iface.c:1724         dm info
> LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN
> NF   [16384] (*1)
> #locking/cluster_locking.c:568         Lock held for
> oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN, node
> 7400a8c0 : CR
> #activate/activate.c:1067       vg/locktest is active
> #locking/cluster_locking.c:508       Locking LV
> oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN EX
> (LV|NONBLOCK|CLUSTER|LOCAL|CONVERT) (0x40dd)
> #locking/cluster_locking.c:395   Error locking on node 7400a8c0: Invalid
> argument
> 
> 
> Regards
> Andreas


[-- Attachment #2: LVM2.2.02.98-remote-activation.patch --]
[-- Type: text/x-patch, Size: 26826 bytes --]

diff -urNp LVM2.2.02.98.orig/daemons/clvmd/clvmd.c LVM2.2.02.98/daemons/clvmd/clvmd.c
--- LVM2.2.02.98.orig/daemons/clvmd/clvmd.c	2012-12-28 13:32:10.000000000 +0000
+++ LVM2.2.02.98/daemons/clvmd/clvmd.c	2013-03-05 10:59:47.874192528 +0000
@@ -1293,7 +1293,7 @@ static int read_from_local_sock(struct l
 		}
 
 		/* Check the node name for validity */
-		if (inheader->node[0] && clops->csid_from_name(csid, inheader->node)) {
+		if (inheader->node[0] && clops->csid_from_name(csid, inheader->node) <= 0) {
 			/* Error, node is not in the cluster */
 			struct clvm_header reply = {
 				.cmd = CLVMD_CMD_REPLY,
@@ -1436,7 +1436,7 @@ static int distribute_command(struct loc
                         /* Do it on a single node */
 			char csid[MAX_CSID_LEN];
 
-			if (clops->csid_from_name(csid, inheader->node)) {
+			if (clops->csid_from_name(csid, inheader->node) <= 0) {
 				/* This has already been checked so should not happen */
 				return 0;
 			} else {
@@ -1448,6 +1448,8 @@ static int distribute_command(struct loc
 				/* Are we the requested node ?? */
 				if (memcmp(csid, our_csid, max_csid_len) == 0) {
 					DEBUGLOG("Doing command on local node only\n");
+					inheader->flags &= ~CLVMD_FLAG_REMOTE;
+					inheader->flags |= CLVMD_FLAG_LOCAL;
 					add_to_lvmqueue(thisfd, inheader, len, NULL);
 				} else {
 					DEBUGLOG("Sending message to single node: %s\n",
@@ -1497,7 +1499,7 @@ static void process_remote_command(struc
 	   clvmd-command.c */
 	if (msg->cmd == CLVMD_CMD_VERSION) {
 		int version_nums[3];
-		char node[256];
+		char node[max_cluster_member_name_len];
 
 		memcpy(version_nums, msg->args, sizeof(version_nums));
 
diff -urNp LVM2.2.02.98.orig/daemons/clvmd/clvmd-corosync.c LVM2.2.02.98/daemons/clvmd/clvmd-corosync.c
--- LVM2.2.02.98.orig/daemons/clvmd/clvmd-corosync.c	2012-12-28 13:32:10.000000000 +0000
+++ LVM2.2.02.98/daemons/clvmd/clvmd-corosync.c	2013-03-05 10:59:47.874192528 +0000
@@ -72,6 +72,9 @@ static struct local_client *cluster_clie
 /* Corosync handles */
 static cpg_handle_t cpg_handle;
 static quorum_handle_t quorum_handle;
+#if defined HAVE_COROSYNC_CMAP_H
+static cmap_handle_t cmap_handle = 0;
+#endif
 
 /* DLM Handle */
 static dlm_lshandle_t *lockspace;
@@ -298,6 +301,17 @@ static int _init_cluster(void)
 		return cs_to_errno(err);
 	}
 
+#if defined HAVE_COROSYNC_CMAP_H
+	err = cmap_initialize(&cmap_handle);
+	if (err != CS_OK) {
+		syslog(LOG_ERR, "Cannot initialise Corosync CMAP service: %d",
+		       err);
+		DEBUGLOG("Cannot initialise Corosync CMAP service: %d", err);
+		cpg_finalize(cpg_handle);
+		return cs_to_errno(err);
+	}
+#endif
+
 #ifdef QUORUM_SET
 	err = quorum_initialize(&quorum_handle,
 				&quorum_callbacks,
@@ -306,6 +320,10 @@ static int _init_cluster(void)
 	if (quorum_type != QUORUM_SET) {
 		syslog(LOG_ERR, "Corosync quorum service is not configured");
 		DEBUGLOG("Corosync quorum service is not configured");
+#if defined HAVE_COROSYNC_CMAP_H
+		cmap_finalize(cmap_handle);
+#endif
+		cpg_finalize(cpg_handle);
 		return EINVAL;
 	}
 #else
@@ -316,6 +334,10 @@ static int _init_cluster(void)
 	if (err != CS_OK) {
 		syslog(LOG_ERR, "Cannot initialise Corosync quorum service: %d",
 		       err);
+#if defined HAVE_COROSYNC_CMAP_H
+		cmap_finalize(cmap_handle);
+#endif
+		cpg_finalize(cpg_handle);
 		DEBUGLOG("Cannot initialise Corosync quorum service: %d", err);
 		return cs_to_errno(err);
 	}
@@ -326,6 +348,10 @@ static int _init_cluster(void)
 		lockspace = dlm_create_lockspace(LOCKSPACE_NAME, 0600);
 		if (!lockspace) {
 			syslog(LOG_ERR, "Unable to create DLM lockspace for CLVM: %m");
+#if defined HAVE_COROSYNC_CMAP_H
+			cmap_finalize(cmap_handle);
+#endif
+			cpg_finalize(cpg_handle);
 			return -1;
 		}
 		DEBUGLOG("Created DLM lockspace for CLVMD.\n");
@@ -340,6 +366,9 @@ static int _init_cluster(void)
 	cpg_group_name.length = strlen((char *)cpg_group_name.value);
 	err = cpg_join(cpg_handle, &cpg_group_name);
 	if (err != CS_OK) {
+#if defined HAVE_COROSYNC_CMAP_H
+		cmap_finalize(cmap_handle);
+#endif
 		cpg_finalize(cpg_handle);
 		quorum_finalize(quorum_handle);
 		dlm_release_lockspace(LOCKSPACE_NAME, lockspace, 1);
@@ -351,6 +380,9 @@ static int _init_cluster(void)
 	err = cpg_local_get(cpg_handle,
 			    &our_nodeid);
 	if (err != CS_OK) {
+#if defined HAVE_COROSYNC_CMAP_H
+		cmap_finalize(cmap_handle);
+#endif
 		cpg_finalize(cpg_handle);
 		quorum_finalize(quorum_handle);
 		dlm_release_lockspace(LOCKSPACE_NAME, lockspace, 1);
@@ -372,6 +404,9 @@ static void _cluster_closedown(void)
 	dlm_release_lockspace(LOCKSPACE_NAME, lockspace, 1);
 	cpg_finalize(cpg_handle);
 	quorum_finalize(quorum_handle);
+#if defined HAVE_COROSYNC_CMAP_H
+	cmap_finalize(cmap_handle);
+#endif
 }
 
 static void _get_our_csid(char *csid)
@@ -380,17 +415,56 @@ static void _get_our_csid(char *csid)
 }
 
 /* Corosync doesn't really have nmode names so we
-   just use the node ID in hex instead */
+   just use the node ID in hex instead.
+   The only case when we have name is when nodelist is configured for
+   pacemaker the way described in pacemaker docs (has nodelist.node.X.name) */
 static int _csid_from_name(char *csid, const char *name)
 {
-	int nodeid;
+	int nodeid = 0;
 	struct node_info *ninfo;
 
-	if (sscanf(name, "%x", &nodeid) == 1) {
-		ninfo = dm_hash_lookup_binary(node_hash, csid, COROSYNC_CSID_LEN);
-		if (ninfo)
+#if defined HAVE_COROSYNC_CMAP_H
+	int idx = 0;
+	char key[128];
+
+	while (1) {
+		cs_error_t rc;
+		char *n_name = NULL;
+
+		snprintf(key, 128, "nodelist.node.%d.name", idx);
+		rc = cmap_get_string(cmap_handle, key, &n_name);
+
+		if (rc != CS_OK) {
+			snprintf(key, 128, "nodelist.node.%d.ring0_ipaddr", idx);
+			rc = cmap_get_string(cmap_handle, key, &n_name);
+
+			if (rc != CS_OK) {
+				break;
+			}
+		}
+		if (strcmp(name, n_name) == 0) {
+			free(n_name);
+			snprintf(key, 128, "nodelist.node.%d.nodeid", idx);
+			rc = cmap_get_uint32(cmap_handle, key, (uint32_t *)&nodeid);
+			if (rc != CS_OK) {
+				break;
+			}
+			
+		} else {
+			free(n_name);
+		}
+		idx++;
+	}
+#endif
+
+	if (nodeid || sscanf(name, "%d", &nodeid) == 1) {
+		ninfo = dm_hash_lookup_binary(node_hash, (char *)&nodeid, COROSYNC_CSID_LEN);
+		if (ninfo) {
+			memcpy(csid, &nodeid, COROSYNC_CSID_LEN);
 			return nodeid;
+		}
 	}
+	*csid = 0;
 	return -1;
 }
 
@@ -398,14 +472,64 @@ static int _name_from_csid(const char *c
 {
 	struct node_info *ninfo;
 
+#if defined HAVE_COROSYNC_CMAP_H
+	int nodeid = 0;
+	int idx = 0;
+	char key[128];
+
+	memcpy(&nodeid, csid, COROSYNC_CSID_LEN);
+
+	*name = '\0';
+
+	while (1) {
+		cs_error_t rc;
+		char *n_name = NULL;
+		uint32_t id;
+
+		snprintf(key, 128, "nodelist.node.%d.nodeid", idx);
+		rc = cmap_get_uint32(cmap_handle, key, &id);
+		if (rc != CS_OK) {
+			break;
+		}
+		if (id == nodeid) {
+			snprintf(key, 128, "nodelist.node.%d.name", idx);
+			rc = cmap_get_string(cmap_handle, key, &n_name);
+			if (rc != CS_OK || !n_name) {
+				int octet;
+				snprintf(key, 128, "nodelist.node.%d.ring0_ipaddr", idx);
+				rc = cmap_get_string(cmap_handle, key, &n_name);
+
+				if (rc != CS_OK || !n_name) {
+					break;
+				}
+				if (sscanf(n_name, "%d.%d.%d.%d", &octet, &octet, &octet, &octet) == 4 || strstr(n_name, ":") != NULL) {
+					/* ring0_ipaddr contains IP address */
+					free(n_name);
+					n_name = NULL;
+					break;
+				}
+			}
+			snprintf(name, COROSYNC_MAX_CLUSTER_MEMBER_NAME_LEN, "%s", n_name);
+			free(n_name);
+			break;
+		}
+
+		idx++;
+	}
+#endif
+
 	ninfo = dm_hash_lookup_binary(node_hash, csid, COROSYNC_CSID_LEN);
 	if (!ninfo)
 	{
-		sprintf(name, "UNKNOWN %s", print_corosync_csid(csid));
+		if (! *name)
+			snprintf(name, COROSYNC_MAX_CLUSTER_MEMBER_NAME_LEN, "UNKNOWN %s", print_corosync_csid(csid));
+
 		return -1;
 	}
 
-	sprintf(name, "%x", ninfo->nodeid);
+	if (! *name)
+		snprintf(name, COROSYNC_MAX_CLUSTER_MEMBER_NAME_LEN, "%d", ninfo->nodeid);
+
 	return 0;
 }
 
@@ -629,18 +753,12 @@ out:
 
 static int _get_cluster_name(char *buf, int buflen)
 {
-	cmap_handle_t cmap_handle = 0;
 	int result;
 	char *name = NULL;
 
 	/* This is a default in case everything else fails */
 	strncpy(buf, "Corosync", buflen);
 
-	/* Look for a cluster name in cmap */
-	result = cmap_initialize(&cmap_handle);
-	if (result != CS_OK)
-		return 0;
-
 	result = cmap_get_string(cmap_handle, "totem.cluster_name", &name);
 	if (result != CS_OK)
 		goto out;
@@ -651,7 +769,6 @@ static int _get_cluster_name(char *buf, 
 out:
 	if (name)
 		free(name);
-	cmap_finalize(cmap_handle);
 	return 0;
 }
 
diff -urNp LVM2.2.02.98.orig/lib/activate/activate.c LVM2.2.02.98/lib/activate/activate.c
--- LVM2.2.02.98.orig/lib/activate/activate.c	2012-10-15 14:24:58.000000000 +0000
+++ LVM2.2.02.98/lib/activate/activate.c	2013-03-05 10:59:47.876252746 +0000
@@ -270,7 +270,7 @@ int lv_is_active_exclusive_locally(const
 {
 	return 0;
 }
-int lv_is_active_exclusive_remotely(const struct logical_volume *lv)
+int lv_is_active_exclusive_remotely(const struct logical_volume *lv, const char *node)
 {
 	return 0;
 }
@@ -1014,7 +1014,7 @@ int lvs_in_vg_opened(const struct volume
  * Returns: 0 or 1
  */
 static int _lv_is_active(const struct logical_volume *lv,
-			 int *locally, int *exclusive)
+			 int *locally, int *exclusive, const char *node)
 {
 	int r, l, e; /* remote, local, and exclusive */
 
@@ -1033,7 +1033,7 @@ static int _lv_is_active(const struct lo
 	if (l && !exclusive)
 		goto out;
 
-	if ((r = remote_lock_held(lv->lvid.s, &e)) >= 0)
+	if ((r = remote_lock_held(lv->lvid.s, &e, node)) >= 0)
 		goto out;
 
 	/*
@@ -1071,34 +1071,40 @@ out:
 
 int lv_is_active(const struct logical_volume *lv)
 {
-	return _lv_is_active(lv, NULL, NULL);
+	return _lv_is_active(lv, NULL, NULL, NULL);
+}
+
+int lv_is_active_remotely(const struct logical_volume *lv, const char *node)
+{
+	int l;
+	return _lv_is_active(lv, &l, NULL, node) && !l;
 }
 
 int lv_is_active_but_not_locally(const struct logical_volume *lv)
 {
 	int l;
-	return _lv_is_active(lv, &l, NULL) && !l;
+	return _lv_is_active(lv, &l, NULL, NULL) && !l;
 }
 
 int lv_is_active_exclusive(const struct logical_volume *lv)
 {
 	int e;
 
-	return _lv_is_active(lv, NULL, &e) && e;
+	return _lv_is_active(lv, NULL, &e, NULL) && e;
 }
 
 int lv_is_active_exclusive_locally(const struct logical_volume *lv)
 {
 	int l, e;
 
-	return _lv_is_active(lv, &l, &e) && l && e;
+	return _lv_is_active(lv, &l, &e, NULL) && l && e;
 }
 
-int lv_is_active_exclusive_remotely(const struct logical_volume *lv)
+int lv_is_active_exclusive_remotely(const struct logical_volume *lv, const char *node)
 {
 	int l, e;
 
-	return _lv_is_active(lv, &l, &e) && !l && e;
+	return _lv_is_active(lv, &l, &e, node) && !l && e;
 }
 
 #ifdef DMEVENTD
diff -urNp LVM2.2.02.98.orig/lib/activate/activate.h LVM2.2.02.98/lib/activate/activate.h
--- LVM2.2.02.98.orig/lib/activate/activate.h	2012-10-15 14:24:58.000000000 +0000
+++ LVM2.2.02.98/lib/activate/activate.h	2013-03-05 10:59:47.878201236 +0000
@@ -130,10 +130,11 @@ int lvs_in_vg_activated(const struct vol
 int lvs_in_vg_opened(const struct volume_group *vg);
 
 int lv_is_active(const struct logical_volume *lv);
+int lv_is_active_remotely(const struct logical_volume *lv, const char *node);
 int lv_is_active_but_not_locally(const struct logical_volume *lv);
 int lv_is_active_exclusive(const struct logical_volume *lv);
 int lv_is_active_exclusive_locally(const struct logical_volume *lv);
-int lv_is_active_exclusive_remotely(const struct logical_volume *lv);
+int lv_is_active_exclusive_remotely(const struct logical_volume *lv, const char *node);
 
 int lv_has_target_type(struct dm_pool *mem, struct logical_volume *lv,
 		       const char *layer, const char *target_type);
diff -urNp LVM2.2.02.98.orig/lib/commands/toolcontext.h LVM2.2.02.98/lib/commands/toolcontext.h
--- LVM2.2.02.98.orig/lib/commands/toolcontext.h	2012-10-15 14:24:58.000000000 +0000
+++ LVM2.2.02.98/lib/commands/toolcontext.h	2013-03-05 10:59:47.879245750 +0000
@@ -112,6 +112,7 @@ struct cmd_context {
 	char dev_dir[PATH_MAX];
 	char proc_dir[PATH_MAX];
 	char sysfs_dir[PATH_MAX]; /* FIXME Use global value instead. */
+	const char *node;
 };
 
 /*
diff -urNp LVM2.2.02.98.orig/lib/locking/cluster_locking.c LVM2.2.02.98/lib/locking/cluster_locking.c
--- LVM2.2.02.98.orig/lib/locking/cluster_locking.c	2012-12-28 13:32:10.000000000 +0000
+++ LVM2.2.02.98/lib/locking/cluster_locking.c	2013-03-05 10:59:47.881190506 +0000
@@ -190,8 +190,10 @@ static void _build_header(struct clvm_he
 	} else if (!strcmp(node, NODE_REMOTE)) {
 		head->node[0] = '\0';
 		head->flags = CLVMD_FLAG_REMOTE;
-	} else
-		strcpy(head->node, node);
+	} else {
+		memcpy(head->node, node, strlen(node) + 1);
+		head->flags = CLVMD_FLAG_REMOTE;
+	}
 }
 
 /*
@@ -315,6 +317,9 @@ static int _lock_for_cluster(struct cmd_
 
 	assert(name);
 
+	if (node == NULL) {
+		node = "";
+	}
 	len = strlen(name) + 3;
 	args = alloca(len);
 	strcpy(args + 2, name);
@@ -374,7 +379,7 @@ static int _lock_for_cluster(struct cmd_
 		     !(flags & LCK_CLUSTER_VG)))
 			node = NODE_LOCAL;
 		else if (flags & LCK_REMOTE)
-			node = NODE_REMOTE;
+			node = cmd->node ? cmd->node : NODE_REMOTE;
 	}
 
 	status = _cluster_request(clvmd_cmd, node, args, len,
@@ -527,16 +532,18 @@ static int decode_lock_type(const char *
 }
 
 #ifdef CLUSTER_LOCKING_INTERNAL
-static int _query_resource(const char *resource, int *mode)
+static int _query_resource(const char *resource, int *mode, const char *node)
 #else
-int query_resource(const char *resource, int *mode)
+int query_resource(const char *resource, int *mode, const char *node)
 #endif
 {
 	int i, status, len, num_responses, saved_errno;
-	const char *node = "";
 	char *args;
 	lvm_response_t *response = NULL;
 
+	if (node == NULL) {
+		node = "";
+	}
 	saved_errno = errno;
 	len = strlen(resource) + 3;
 	args = alloca(len);
diff -urNp LVM2.2.02.98.orig/lib/locking/external_locking.c LVM2.2.02.98/lib/locking/external_locking.c
--- LVM2.2.02.98.orig/lib/locking/external_locking.c	2012-10-15 14:24:58.000000000 +0000
+++ LVM2.2.02.98/lib/locking/external_locking.c	2013-03-05 10:59:47.882313739 +0000
@@ -28,7 +28,7 @@ static int (*_lock_fn) (struct cmd_conte
 			uint32_t flags) = NULL;
 static int (*_init_fn) (int type, struct dm_config_tree * cft,
 			uint32_t *flags) = NULL;
-static int (*_lock_query_fn) (const char *resource, int *mode) = NULL;
+static int (*_lock_query_fn) (const char *resource, int *mode, const char *node) = NULL;
 
 static int _lock_resource(struct cmd_context *cmd, const char *resource,
 			  uint32_t flags)
diff -urNp LVM2.2.02.98.orig/lib/locking/locking.c LVM2.2.02.98/lib/locking/locking.c
--- LVM2.2.02.98.orig/lib/locking/locking.c	2012-12-28 13:32:10.000000000 +0000
+++ LVM2.2.02.98/lib/locking/locking.c	2013-03-06 06:42:59.043441030 +0000
@@ -544,10 +544,12 @@ int deactivate_lv(struct cmd_context *cm
 {
 	if (vg_is_clustered(lv->vg)) {
 		if (lv_is_active_exclusive(lv) && ! lv_is_active_exclusive_locally(lv)) {
+			log_error("Failed to deactivate %s: exclusively activated on a remote node. Use -aln --node Node.", lv->name);
 			return_0;
 		}
 	}
 	lock_lv_vol(cmd, lv, LCK_LV_DEACTIVATE);
+	return 1;
 }
 
 /*
@@ -561,20 +563,29 @@ int activate_lv_excl(struct cmd_context 
 	if (!vg_is_clustered(lv->vg))
 		return activate_lv_excl_local(cmd, lv);
 
-	if (lv_is_active_exclusive_locally(lv))
-		return 1;
+	if (!cmd->node) {
+		if (lv_is_active_exclusive_locally(lv))
+			return 1;
 
-	if (!activate_lv_excl_local(cmd, lv))
-		return_0;
+		if (!activate_lv_excl_local(cmd, lv))
+			return_0;
 
-	if (lv_is_active_exclusive(lv))
-		return 1;
 
-	/* FIXME Deal with error return codes. */
-	if (activate_lv_excl_remote(cmd, lv))
-		stack;
+		if (lv_is_active_exclusive(lv))
+			return 1;
+	} else {
+		if (lv_is_active_exclusive_remotely(lv, cmd->node))
+			return 1;
+
+		if (!activate_lv_excl_remote(cmd, lv))
+			return_0;
+
+
+		if (lv_is_active_exclusive(lv))
+			return 1;
+	}
 
-	return lv_is_active_exclusive(lv);
+	return_0;
 }
 
 int activate_lv_excl_force(struct cmd_context *cmd, struct logical_volume *lv) 
@@ -583,20 +594,29 @@ int activate_lv_excl_force(struct cmd_co
 	if (!vg_is_clustered(lv->vg))
 		return activate_lv_excl_local(cmd, lv);
 
-	if (lv_is_active_exclusive_locally(lv))
-		return 1;
+	if (!cmd->node) {
+		if (lv_is_active_exclusive_locally(lv))
+			return 1;
 
-	if (!activate_lv_excl_local_force(cmd, lv))
-		return_0;
+		if (!activate_lv_excl_local_force(cmd, lv))
+			return_0;
 
-	if (lv_is_active_exclusive(lv))
-		return 1;
 
-	/* FIXME Deal with error return codes. */
-	if (activate_lv_excl_remote_force(cmd, lv))
-		stack;
+		if (lv_is_active_exclusive(lv))
+			return 1;
+	} else {
+		if (lv_is_active_exclusive_remotely(lv, cmd->node))
+			return 1;
+
+		if (!activate_lv_excl_remote_force(cmd, lv))
+			return_0;
+
+
+		if (lv_is_active_exclusive(lv))
+			return 1;
+	}
 
-	return lv_is_active_exclusive(lv);
+	return_0;
 }
 
 /* Lock a list of LVs */
@@ -635,7 +655,7 @@ int locking_is_clustered(void)
 	return (_locking.flags & LCK_CLUSTERED) ? 1 : 0;
 }
 
-int remote_lock_held(const char *vol, int *exclusive)
+int remote_lock_held(const char *vol, int *exclusive, const char *node)
 {
 	int mode = LCK_NULL;
 
@@ -648,7 +668,7 @@ int remote_lock_held(const char *vol, in
 	/*
 	 * If an error occured, expect that volume is active
 	 */
-	if (!_locking.query_resource(vol, &mode)) {
+	if (!_locking.query_resource(vol, &mode, node)) {
 		stack;
 		return 1;
 	}
diff -urNp LVM2.2.02.98.orig/lib/locking/locking.h LVM2.2.02.98/lib/locking/locking.h
--- LVM2.2.02.98.orig/lib/locking/locking.h	2012-12-28 13:32:10.000000000 +0000
+++ LVM2.2.02.98/lib/locking/locking.h	2013-03-05 10:59:47.885191648 +0000
@@ -25,7 +25,7 @@ void reset_locking(void);
 int vg_write_lock_held(void);
 int locking_is_clustered(void);
 
-int remote_lock_held(const char *vol, int *exclusive);
+int remote_lock_held(const char *vol, int *exclusive, const char *node);
 
 /*
  * LCK_VG:
@@ -186,7 +186,7 @@ int check_lvm1_vg_inactive(struct cmd_co
 #define activate_lv_excl_local_force(cmd, lv)	\
 				lock_lv_vol(cmd, lv, LCK_LV_EXCLUSIVE | LCK_HOLD | LCK_LOCAL | (lv_is_active(lv) && ! lv_is_active_exclusive(lv) ? LCK_TRY_CONVERT : 0))
 #define activate_lv_excl_remote_force(cmd, lv)	\
-				lock_lv_vol(cmd, lv, LCK_LV_EXCLUSIVE | LCK_HOLD | LCK_REMOTE | (lv_is_active(lv) && ! lv_is_active_exclusive(lv) ? LCK_TRY_CONVERT : 0))
+				lock_lv_vol(cmd, lv, LCK_LV_EXCLUSIVE | LCK_HOLD | LCK_REMOTE | (lv_is_active_remotely(lv, cmd->node) && ! lv_is_active_exclusive(lv) ? LCK_TRY_CONVERT : 0))
 
 struct logical_volume;
 int activate_lv_excl(struct cmd_context *cmd, struct logical_volume *lv);
@@ -197,8 +197,14 @@ int deactivate_lv(struct cmd_context *cm
 	lock_lv_vol(cmd, lv, LCK_LV_ACTIVATE | LCK_HOLD | LCK_LOCAL)
 #define activate_lv_local_force(cmd, lv)	\
 	lock_lv_vol(cmd, lv, LCK_LV_ACTIVATE | LCK_HOLD | LCK_LOCAL | (lv_is_active_exclusive_locally(lv) ? LCK_TRY_CONVERT : 0))
+#define activate_lv_remote(cmd, lv)	\
+	lock_lv_vol(cmd, lv, LCK_LV_ACTIVATE | LCK_HOLD | LCK_REMOTE)
+#define activate_lv_remote_force(cmd, lv)	\
+	lock_lv_vol(cmd, lv, LCK_LV_ACTIVATE | LCK_HOLD | LCK_REMOTE | (lv_is_active_exclusive_remotely(lv, cmd->node) ? LCK_TRY_CONVERT : 0))
 #define deactivate_lv_local(cmd, lv)	\
 	lock_lv_vol(cmd, lv, LCK_LV_DEACTIVATE | LCK_LOCAL)
+#define deactivate_lv_remote(cmd, lv)	\
+	lock_lv_vol(cmd, lv, LCK_LV_DEACTIVATE | LCK_REMOTE)
 #define drop_cached_metadata(vg)	\
 	lock_vol((vg)->cmd, (vg)->name, LCK_VG_DROP_CACHE)
 #define remote_commit_cached_metadata(vg)	\
diff -urNp LVM2.2.02.98.orig/lib/locking/locking_types.h LVM2.2.02.98/lib/locking/locking_types.h
--- LVM2.2.02.98.orig/lib/locking/locking_types.h	2012-10-15 14:24:58.000000000 +0000
+++ LVM2.2.02.98/lib/locking/locking_types.h	2013-03-05 10:59:47.886265186 +0000
@@ -18,7 +18,7 @@
 
 typedef int (*lock_resource_fn) (struct cmd_context * cmd, const char *resource,
 				 uint32_t flags);
-typedef int (*query_resource_fn) (const char *resource, int *mode);
+typedef int (*query_resource_fn) (const char *resource, int *mode, const char *node);
 
 typedef void (*fin_lock_fn) (void);
 typedef void (*reset_lock_fn) (void);
diff -urNp LVM2.2.02.98.orig/tools/args.h LVM2.2.02.98/tools/args.h
--- LVM2.2.02.98.orig/tools/args.h	2012-10-15 14:24:59.000000000 +0000
+++ LVM2.2.02.98/tools/args.h	2013-03-05 10:59:47.888190332 +0000
@@ -76,6 +76,7 @@ arg(discards_ARG, '\0', "discards", disc
 arg(stripes_long_ARG, '\0', "stripes", int_arg, 0)
 arg(sysinit_ARG, '\0', "sysinit", NULL, 0)
 arg(thinpool_ARG, '\0', "thinpool", string_arg, 0)
+arg(node_ARG, '\0', "node", string_arg, 0)
 
 /* Allow some variations */
 arg(resizable_ARG, '\0', "resizable", yes_no_arg, 0)
diff -urNp LVM2.2.02.98.orig/tools/commands.h LVM2.2.02.98/tools/commands.h
--- LVM2.2.02.98.orig/tools/commands.h	2012-10-15 14:24:59.000000000 +0000
+++ LVM2.2.02.98/tools/commands.h	2013-03-05 10:59:47.889258249 +0000
@@ -87,6 +87,7 @@ xx(lvchange,
    "\t[-y|--yes]\n"
    "\t[--version]\n"
    "\t[-Z|--zero {y|n}]\n"
+   "\t[--node Node ]\n"
    "\tLogicalVolume[Path] [LogicalVolume[Path]...]\n",
 
    alloc_ARG, autobackup_ARG, activate_ARG, available_ARG, contiguous_ARG,
@@ -94,7 +95,7 @@ xx(lvchange,
    major_ARG, minor_ARG, monitor_ARG, noudevsync_ARG, partial_ARG,
    permission_ARG, persistent_ARG, poll_ARG, readahead_ARG, resync_ARG,
    refresh_ARG, addtag_ARG, deltag_ARG, sysinit_ARG, test_ARG, yes_ARG,
-   zero_ARG)
+   zero_ARG, node_ARG)
 
 xx(lvconvert,
    "Change logical volume layout",
diff -urNp LVM2.2.02.98.orig/tools/lvchange.c LVM2.2.02.98/tools/lvchange.c
--- LVM2.2.02.98.orig/tools/lvchange.c	2012-12-28 13:32:10.000000000 +0000
+++ LVM2.2.02.98/tools/lvchange.c	2013-03-05 10:59:47.891443499 +0000
@@ -226,11 +226,22 @@ static int _lvchange_activate(struct cmd
 	}
 
 	if (activate == CHANGE_ALN) {
-		log_verbose("Deactivating logical volume \"%s\" locally",
-			    lv->name);
-		if (!deactivate_lv_local(cmd, lv))
-			return_0;
+		if (cmd->node) {
+			log_verbose("Deactivating logical volume \"%s\" locally on node %s",
+				    lv->name, cmd->node);
+			if (!deactivate_lv_remote(cmd, lv))
+				return_0;
+		} else {
+			log_verbose("Deactivating logical volume \"%s\" locally",
+				    lv->name);
+			if (!deactivate_lv_local(cmd, lv))
+				return_0;
+		}
 	} else if (activate == CHANGE_AN) {
+		if (cmd->node) {
+			log_error("Use -aln to deactivate volume on a remote node");
+			return_0;
+		}
 		log_verbose("Deactivating logical volume \"%s\"", lv->name);
 		if (!deactivate_lv(cmd, lv))
 			return_0;
@@ -239,29 +250,55 @@ static int _lvchange_activate(struct cmd
 		    lv_is_origin(lv) ||
 		    lv_is_thin_type(lv)) {
 			if (arg_count(cmd, force_ARG)) {
-				log_verbose("Activating logical volume \"%s\" "
-					    "exclusively (forced)", lv->name);
+				if (cmd->node)
+					log_verbose("Activating logical volume \"%s\" "
+						    "exclusively on node %s (forced)", lv->name, cmd->node);
+				else
+					log_verbose("Activating logical volume \"%s\" "
+						    "exclusively (forced)", lv->name);
 				if (!activate_lv_excl_force(cmd, lv))
 					return_0;
 			} else {
-				log_verbose("Activating logical volume \"%s\" "
-					    "exclusively", lv->name);
+				if (cmd->node)
+					log_verbose("Activating logical volume \"%s\" "
+						    "exclusively on node %s", lv->name, cmd->node);
+				else
+					log_verbose("Activating logical volume \"%s\" "
+						    "exclusively", lv->name);
 				if (!activate_lv_excl(cmd, lv))
 					return_0;
 			}
 		} else if (activate == CHANGE_ALY) {
-			if (arg_count(cmd, force_ARG)) {
-				log_verbose("Activating logical volume \"%s\" locally (forced)",
-					    lv->name);
-				if (!activate_lv_local_force(cmd, lv))
-					return_0;
+			if (cmd->node) {
+				if (arg_count(cmd, force_ARG)) {
+					log_verbose("Activating logical volume \"%s\" locally on node %s (forced)",
+						    lv->name, cmd->node);
+					if (!activate_lv_remote_force(cmd, lv))
+						return_0;
+				} else {
+					log_verbose("Activating logical volume \"%s\" locally on node %s",
+						    lv->name, cmd->node);
+					if (!activate_lv_remote(cmd, lv))
+						return_0;
+				}
 			} else {
-				log_verbose("Activating logical volume \"%s\" locally",
-					    lv->name);
-				if (!activate_lv_local(cmd, lv))
-					return_0;
+				if (arg_count(cmd, force_ARG)) {
+					log_verbose("Activating logical volume \"%s\" locally (forced)",
+						    lv->name);
+					if (!activate_lv_local_force(cmd, lv))
+						return_0;
+				} else {
+					log_verbose("Activating logical volume \"%s\" locally",
+						    lv->name);
+					if (!activate_lv_local(cmd, lv))
+						return_0;
+				}
 			}
 		} else {
+			if (cmd->node) {
+				log_error("Use -aly or -aey to activate volume on a remote node");
+				return_0;
+			}
 			log_verbose("Activating logical volume \"%s\"",
 				    lv->name);
 			if (!activate_lv(cmd, lv))
@@ -808,6 +845,25 @@ static int lvchange_single(struct cmd_co
 		return ECMD_FAILED;
 	}
 
+	switch (arg_count(cmd, node_ARG)) {
+		case 1:
+			if (!arg_count(cmd, activate_ARG)) {
+				log_error("--node argument may be used only with -a");
+				return ECMD_FAILED;
+			}
+			if (!vg_is_clustered(lv->vg)) {
+				log_error("--node argument may be used only with clustered volume groups");
+				return ECMD_FAILED;
+			}
+			cmd->node = arg_str_value(cmd, node_ARG, NULL);
+			break;
+		case 0:
+			break;
+		default:
+			log_error("Only one --node argument is supported");
+			return ECMD_FAILED;
+			break;
+	}
 	/*
 	 * FIXME: DEFAULT_BACKGROUND_POLLING should be "unspecified".
 	 * If --poll is explicitly provided use it; otherwise polling
diff -urNp LVM2.2.02.98.orig/tools/pvmove.c LVM2.2.02.98/tools/pvmove.c
--- LVM2.2.02.98.orig/tools/pvmove.c	2012-10-15 14:24:59.000000000 +0000
+++ LVM2.2.02.98/tools/pvmove.c	2013-03-05 10:59:47.893194162 +0000
@@ -240,7 +240,7 @@ static struct logical_volume *_set_up_pv
 		}
 
 		if (vg_is_clustered(vg) &&
-		    lv_is_active_exclusive_remotely(lv)) {
+		    lv_is_active_exclusive_remotely(lv, NULL)) {
 			lv_skipped = 1;
 			log_print_unless_silent("Skipping LV %s which is activated "
 						"exclusively on remote node.", lv->name);
diff -urNp LVM2.2.02.98.orig/tools/vgchange.c LVM2.2.02.98/tools/vgchange.c
--- LVM2.2.02.98.orig/tools/vgchange.c	2012-10-15 14:24:59.000000000 +0000
+++ LVM2.2.02.98/tools/vgchange.c	2013-03-05 10:59:47.894198836 +0000
@@ -125,7 +125,7 @@ static int _activate_lvs_in_vg(struct cm
 		 * If the LV is active exclusive remotely,
 		 * then ignore it here
 		 */
-		if (lv_is_active_exclusive_remotely(lv)) {
+		if (lv_is_active_exclusive_remotely(lv, NULL)) {
 			log_verbose("%s/%s is exclusively active on"
 				    " a remote node", vg->name, lv->name);
 			continue;

Re: [linux-lvm] LVM snapshot with Clustered VG

[Andreas Pflug]

Am 06.03.13 08:58, schrieb Vladislav Bogdanov:
> 06.03.2013 10:40, Andreas Pflug wrote:
>> Am 01.03.13 16:41, schrieb Vladislav Bogdanovr?
>>
>>> Hi Andreas,
>>> Lock convertion is only enabled if you pass --force flag.
>>> Also, to upgrade local lock to exclusive one, you need to ensure IIRC
>>> that no more node holds local lock.
>> Hm, tried that as well:
>>
>> tools/lvm lvchange --force -aey  -vvvv vg/locktest
>>
>> --force changes the error from "resource busy" to "invalid argument":
> Is volume active on other nodes at that time?

I made sure it's not active on other nodes: lvchange -an vg/locktest ; 
lvchange -aly vg/locktest
> And do you run clvmd from that build tree as well?
>
> Also, can you please try attached patch (on top of that one you have)? I
> polished conversion a bit more, denying -an if volume is ex-locked
> somewhere and other fixes to logic.
I tried that additional patch. I'm running this test versions on my test 
node only (including clvmd), the other nodes are still running clvmd 
2.2.95 (I guess this shouldn't matter since all are inactive). Same result:

#lvchange.c:258     Activating logical volume "locktest" exclusively 
(forced)
#activate/dev_manager.c:284         Getting device info for vg-locktest 
[LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
#ioctl/libdm-iface.c:1724         dm info 
LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN 
NF   [16384] (*1)
#activate/activate.c:1067       vg/locktest is active
#activate/dev_manager.c:284         Getting device info for vg-locktest 
[LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
#ioctl/libdm-iface.c:1724         dm info 
LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN 
NF   [16384] (*1)
#activate/activate.c:1067       vg/locktest is active
#activate/dev_manager.c:284         Getting device info for vg-locktest 
[LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
#ioctl/libdm-iface.c:1724         dm info 
LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN 
NF   [16384] (*1)
#activate/activate.c:1067       vg/locktest is active
#locking/cluster_locking.c:513       Locking LV 
oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN EX 
(LV|NONBLOCK|CLUSTER|LOCAL|CONVERT) (0x40dd)
#locking/cluster_locking.c:400   Error locking on node 7400a8c0: Invalid 
argument


> This patch also allows locking (activation) to be performed on remote
> nodes. I only tested this with corosync2 (which is set up in a way
> latest pacemaker - post-1.1.8 git master - needs, nodes has additional
> 'name' value in nodelist, please see
> http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-name.html).

I'm running corosync 1.4.2 (debian wheezy).

Regards,
Andreas

Re: [linux-lvm] LVM snapshot with Clustered VG

[Vladislav Bogdanov]

06.03.2013 12:15, Andreas Pflug wrote:
> Am 06.03.13 08:58, schrieb Vladislav Bogdanov:
>> 06.03.2013 10:40, Andreas Pflug wrote:
>>> Am 01.03.13 16:41, schrieb Vladislav Bogdanovr?
>>>
>>>> Hi Andreas,
>>>> Lock convertion is only enabled if you pass --force flag.
>>>> Also, to upgrade local lock to exclusive one, you need to ensure IIRC
>>>> that no more node holds local lock.
>>> Hm, tried that as well:
>>>
>>> tools/lvm lvchange --force -aey  -vvvv vg/locktest
>>>
>>> --force changes the error from "resource busy" to "invalid argument":
>> Is volume active on other nodes at that time?
> 
> I made sure it's not active on other nodes: lvchange -an vg/locktest ;
> lvchange -aly vg/locktest
>> And do you run clvmd from that build tree as well?
>>
>> Also, can you please try attached patch (on top of that one you have)? I
>> polished conversion a bit more, denying -an if volume is ex-locked
>> somewhere and other fixes to logic.
> I tried that additional patch. I'm running this test versions on my test
> node only (including clvmd), the other nodes are still running clvmd
> 2.2.95 (I guess this shouldn't matter since all are inactive). Same result:

I believe this matters, because error you see is received from a remote
node. Is node with ID 7400a8c0 local?

> 
> #lvchange.c:258     Activating logical volume "locktest" exclusively
> (forced)
> #activate/dev_manager.c:284         Getting device info for vg-locktest
> [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
> #ioctl/libdm-iface.c:1724         dm info
> LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN
> NF   [16384] (*1)
> #activate/activate.c:1067       vg/locktest is active
> #activate/dev_manager.c:284         Getting device info for vg-locktest
> [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
> #ioctl/libdm-iface.c:1724         dm info
> LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN
> NF   [16384] (*1)
> #activate/activate.c:1067       vg/locktest is active
> #activate/dev_manager.c:284         Getting device info for vg-locktest
> [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
> #ioctl/libdm-iface.c:1724         dm info
> LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN
> NF   [16384] (*1)
> #activate/activate.c:1067       vg/locktest is active
> #locking/cluster_locking.c:513       Locking LV
> oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN EX
> (LV|NONBLOCK|CLUSTER|LOCAL|CONVERT) (0x40dd)
> #locking/cluster_locking.c:400   Error locking on node 7400a8c0: Invalid
> argument
> 
> 
>> This patch also allows locking (activation) to be performed on remote
>> nodes. I only tested this with corosync2 (which is set up in a way
>> latest pacemaker - post-1.1.8 git master - needs, nodes has additional
>> 'name' value in nodelist, please see
>> http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-name.html).
>>
> 
> I'm running corosync 1.4.2 (debian wheezy).

Which cluster manager interface does clvmd detect? corosync or openais?
You should use former, openais one is(was) using LCK service which is
very unstable.

> 
> Regards,
> Andreas

Re: [linux-lvm] LVM snapshot with Clustered VG

[Andreas Pflug]

Am 06.03.13 10:35, schrieb Vladislav Bogdanov:
> 06.03.2013 12:15, Andreas Pflug wrote:
>> I made sure it's not active on other nodes: lvchange -an vg/locktest ;
>> lvchange -aly vg/locktest
>>> And do you run clvmd from that build tree as well?
>>>
>>> Also, can you please try attached patch (on top of that one you have)? I
>>> polished conversion a bit more, denying -an if volume is ex-locked
>>> somewhere and other fixes to logic.
>> I tried that additional patch. I'm running this test versions on my test
>> node only (including clvmd), the other nodes are still running clvmd
>> 2.2.95 (I guess this shouldn't matter since all are inactive). Same result:
> I believe this matters, because error you see is received from a remote
> node. Is node with ID 7400a8c0 local?
Yes, that's the test node.
Hm, not funny if I have to upgrade all nodes on the production system... 
I'm a little surprised that remote inactive nodes need to be aware of 
that force-exclusive stuff.
>
> I'm running corosync 1.4.2 (debian wheezy).
> Which cluster manager interface does clvmd detect? corosync or openais?
> You should use former, openais one is(was) using LCK service which is
> very unstable.
It's using openais. I'm not too happy about the stability, so maybe I'd 
switch to corosync now.
Could this be a reason for the x-lock failure as well?


Regards,
Andreas

Re: [linux-lvm] LVM snapshot with Clustered VG

[Vladislav Bogdanov]

06.03.2013 12:59, Andreas Pflug wrote:
> Am 06.03.13 10:35, schrieb Vladislav Bogdanov:
>> 06.03.2013 12:15, Andreas Pflug wrote:
>>> I made sure it's not active on other nodes: lvchange -an vg/locktest ;
>>> lvchange -aly vg/locktest
>>>> And do you run clvmd from that build tree as well?
>>>>
>>>> Also, can you please try attached patch (on top of that one you
>>>> have)? I
>>>> polished conversion a bit more, denying -an if volume is ex-locked
>>>> somewhere and other fixes to logic.
>>> I tried that additional patch. I'm running this test versions on my test
>>> node only (including clvmd), the other nodes are still running clvmd
>>> 2.2.95 (I guess this shouldn't matter since all are inactive). Same
>>> result:
>> I believe this matters, because error you see is received from a remote
>> node. Is node with ID 7400a8c0 local?
> Yes, that's the test node.

But you receive "Invalid argument" error from the cluster layer, so is
should go from a local clvmd if you are correct about node IDs. You may
run clvmd in a debugging mode as well to catch what goes wrong.

> Hm, not funny if I have to upgrade all nodes on the production system...
> I'm a little surprised that remote inactive nodes need to be aware of
> that force-exclusive stuff.

You probably should try to test that on a test cluster first (as I do)
although I'm almost sure it works correctly in setup I described.

>>
>> I'm running corosync 1.4.2 (debian wheezy).
>> Which cluster manager interface does clvmd detect? corosync or openais?
>> You should use former, openais one is(was) using LCK service which is
>> very unstable.
> It's using openais. I'm not too happy about the stability, so maybe I'd
> switch to corosync now.

That could be the problem btw. I did neither test nor look at openais
module implementation in clvmd, because I had plenty problems with it
(actually with LCK under it) in the past, so I even forced to use
corosync (CPG) + dlm instead of detected openais (CPG+LCK) for older
systems (look at -I switch of clvmd).

And, openais is deprecated upstream, so I do not see any reason to use
it. Even gfs_controld (which is probably the only well-known user of
openais services) actually does not strictly require it, at least I was
able to port it to pure CPG+DLM with dlm4 on top of corosync2, which is
not compatible with openais plugins.

Also you may need quorum patch found in this list, it does its job well.

> Could this be a reason for the x-lock failure as well?

Can you please reword?

Vladislav

Re: [linux-lvm] LVM snapshot with Clustered VG

[Andreas Pflug]

Am 06.03.13 12:20, schrieb Vladislav Bogdanov:
>>> I'm running corosync 1.4.2 (debian wheezy).
>>> Which cluster manager interface does clvmd detect? corosync or openais?
>>> You should use former, openais one is(was) using LCK service which is
>>> very unstable.
>> It's using openais. I'm not too happy about the stability, so maybe I'd
>> switch to corosync now.
> That could be the problem btw. I did neither test nor look at openais
> module implementation in clvmd, because I had plenty problems with it
> (actually with LCK under it) in the past, so I even forced to use
> corosync (CPG) + dlm instead of detected openais (CPG+LCK) for older
> systems (look at -I switch of clvmd).
>
> And, openais is deprecated upstream, so I do not see any reason to use
> it. Even gfs_controld (which is probably the only well-known user of
> openais services) actually does not strictly require it, at least I was
> able to port it to pure CPG+DLM with dlm4 on top of corosync2, which is
> not compatible with openais plugins.
>
> Also you may need quorum patch found in this list, it does its job well.
>
>> Could this be a reason for the x-lock failure as well?
You just answered the quirky question :-)

Unfortunately, corosync/dlm don't work for me as expected. When starting 
clvmd -I corosync (with dlm kernel module loaded), creating the dlm 
lockspace "clvmd" fails, with

dlm: no local IP address has been set
dlm: cannot start dlm lowcomms -107

I haven't found any hint what might go wrong on the machine (checked 
already hostname resolves correctly to its ip address via /etc/hosts; 
corosync uses that network too).

Regards
Andreas

Re: [linux-lvm] LVM snapshot with Clustered VG

[Vladislav Bogdanov]

06.03.2013 15:17, Andreas Pflug wrote:
> Am 06.03.13 12:20, schrieb Vladislav Bogdanov:
>>>> I'm running corosync 1.4.2 (debian wheezy).
>>>> Which cluster manager interface does clvmd detect? corosync or openais?
>>>> You should use former, openais one is(was) using LCK service which is
>>>> very unstable.
>>> It's using openais. I'm not too happy about the stability, so maybe I'd
>>> switch to corosync now.
>> That could be the problem btw. I did neither test nor look at openais
>> module implementation in clvmd, because I had plenty problems with it
>> (actually with LCK under it) in the past, so I even forced to use
>> corosync (CPG) + dlm instead of detected openais (CPG+LCK) for older
>> systems (look at -I switch of clvmd).
>>
>> And, openais is deprecated upstream, so I do not see any reason to use
>> it. Even gfs_controld (which is probably the only well-known user of
>> openais services) actually does not strictly require it, at least I was
>> able to port it to pure CPG+DLM with dlm4 on top of corosync2, which is
>> not compatible with openais plugins.
>>
>> Also you may need quorum patch found in this list, it does its job well.
>>
>>> Could this be a reason for the x-lock failure as well?
> You just answered the quirky question :-)
> 
> Unfortunately, corosync/dlm don't work for me as expected. When starting
> clvmd -I corosync (with dlm kernel module loaded), creating the dlm
> lockspace "clvmd" fails, with
> 
> dlm: no local IP address has been set
> dlm: cannot start dlm lowcomms -107
> 

You need to have dlm_controld running on all nodes.
And that is not trivial with corosync1. You need to either use cman or
use deprecated dlm_controld.pcmk which was removed from a cluster (cman)
package after 3.0.17. Latter does not work well without heavy patching
(http://www.mail-archive.com/pacemaker@oss.clusterlabs.org/msg09959.html
,
http://www.mail-archive.com/pacemaker@oss.clusterlabs.org/msg11123.html
and
http://oss.clusterlabs.org/pipermail/pacemaker/2012-February/013073.html).
Even after that there are some problems with it.

Latest versions (4.x) of dlm (it was split from cman) work fine with
corosync2 and pacemaker 1.1.6+. But I doubt it can ever be compiled with
corosync1. Yes, it cannot, because it requires quorum which appeared in
corosync2.

You may look at thread
http://oss.clusterlabs.org/pipermail/pacemaker/2012-October/015826.html
for some info.

That's all why I moved to corosync2. You can search through pacemaker
list archives for dlm-related messages from me to get full picture.

You may also look at slides I presented at LVEE winter 2013:
http://lvee.org/uploads/image_upload/file/267/Linux_Clusters_at_LVEE_2013.pdf


> I haven't found any hint what might go wrong on the machine (checked
> already hostname resolves correctly to its ip address via /etc/hosts;
> corosync uses that network too).
> 
> Regards
> Andreas

Re: [linux-lvm] LVM snapshot with Clustered VG

[Andreas Pflug]

Am 06.03.13 14:28, schrieb Vladislav Bogdanov:
> 06.03.2013 15:17, Andreas Pflug wrote:
>> Am 06.03.13 12:20, schrieb Vladislav Bogdanov:
>>>>> I'm running corosync 1.4.2 (debian wheezy).
>>>>> Which cluster manager interface does clvmd detect? corosync or openais?
>>>>> You should use former, openais one is(was) using LCK service which is
>>>>> very unstable.
>>>> It's using openais. I'm not too happy about the stability, so maybe I'd
>>>> switch to corosync now.
>>> That could be the problem btw. I did neither test nor look at openais
>>> module implementation in clvmd, because I had plenty problems with it
>>> (actually with LCK under it) in the past, so I even forced to use
>>> corosync (CPG) + dlm instead of detected openais (CPG+LCK) for older
>>> systems (look at -I switch of clvmd).
>>>
>>> And, openais is deprecated upstream, so I do not see any reason to use
>>> it. Even gfs_controld (which is probably the only well-known user of
>>> openais services) actually does not strictly require it, at least I was
>>> able to port it to pure CPG+DLM with dlm4 on top of corosync2, which is
>>> not compatible with openais plugins.
>>>
>>> Also you may need quorum patch found in this list, it does its job well.
>>>
>>>> Could this be a reason for the x-lock failure as well?
>> You just answered the quirky question :-)
>>
>> Unfortunately, corosync/dlm don't work for me as expected. When starting
>> clvmd -I corosync (with dlm kernel module loaded), creating the dlm
>> lockspace "clvmd" fails, with
>>
>> dlm: no local IP address has been set
>> dlm: cannot start dlm lowcomms -107
>>
> You need to have dlm_controld running on all nodes.
> And that is not trivial with corosync1. You need to either use cman or
> use deprecated dlm_controld.pcmk which was removed from a cluster (cman)
> package after 3.0.17. Latter does not work well without heavy patching
> (http://www.mail-archive.com/pacemaker@oss.clusterlabs.org/msg09959.html
> ,
> http://www.mail-archive.com/pacemaker@oss.clusterlabs.org/msg11123.html
> and
> http://oss.clusterlabs.org/pipermail/pacemaker/2012-February/013073.html).
> Even after that there are some problems with it.
>
> Latest versions (4.x) of dlm (it was split from cman) work fine with
> corosync2 and pacemaker 1.1.6+. But I doubt it can ever be compiled with
> corosync1. Yes, it cannot, because it requires quorum which appeared in
> corosync2.
>
> You may look at thread
> http://oss.clusterlabs.org/pipermail/pacemaker/2012-October/015826.html
> for some info.
>
> That's all why I moved to corosync2. You can search through pacemaker
> list archives for dlm-related messages from me to get full picture.
>
> You may also look at slides I presented at LVEE winter 2013:
> http://lvee.org/uploads/image_upload/file/267/Linux_Clusters_at_LVEE_2013.pdf

Thanks for the pointers, seems there's no way around reorganizing all 
clustering stuff.

Regards,
Andreas

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Andreas Pflug]

Hi Vladislav,

I finally succeeded making all working as expected!

What I did:
- compile corosync 2.3.0 
(http://corosync.org/download/corosync-2.3.0.tar.gz)
- compile libqb 0.14.4 
(https://fedorahosted.org/releases/q/u/quarterback/libqb-0.14.4.tar.gz)
- compile dlm_4.0.1 
(http://git.fedorahosted.org/cgit/dlm.git/snapshot/dlm-4.0.1.tar.gz)
- Compiled LVM 2.2.98 
(http://git.fedorahosted.org/cgit/lvm2.git/snapshot/lvm2-2_02_98.tar.gz)
    with your PREVIOUS patch ( 
https://www.redhat.com/archives/linux-lvm/2013-January/msg00006.html )

I made a 2-node test cluster, both accessing a test iscsi target, all 
daemons load fine.
Everything works as expected! Snapshotting is possible after -aey 
--force explicit exclusive locking, lvresize works as well.

Previously, I tried your latest patch. I had a plethiora of errors, 
starting with lvcreate failing:

#metadata/lv_manip.c:4137     Clearing start of logical volume "testvol2"
#device/dev-cache.c:599       /dev/san2/testvol2: stat failed: No such 
file or directory
#metadata/lv_manip.c:4140   /dev/san2/testvol2: not found: device not 
cleared
#metadata/lv_manip.c:4620   Aborting. Failed to wipe start of new LV.

I need to add -Z n to have it created. After that, most activations 
failed, with error code 5. I  went back to vanilla 2.2.98, all ok except 
exclusive locking, applied the initial locking patch and voila! All done.

Thanks for your support!
Andreas

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Vladislav Bogdanov]

13.03.2013 18:14, Andreas Pflug wrote:
> Hi Vladislav,
> 
> I finally succeeded making all working as expected!
> 
> What I did:
> - compile corosync 2.3.0
> (http://corosync.org/download/corosync-2.3.0.tar.gz)
> - compile libqb 0.14.4
> (https://fedorahosted.org/releases/q/u/quarterback/libqb-0.14.4.tar.gz)
> - compile dlm_4.0.1
> (http://git.fedorahosted.org/cgit/dlm.git/snapshot/dlm-4.0.1.tar.gz)
> - Compiled LVM 2.2.98
> (http://git.fedorahosted.org/cgit/lvm2.git/snapshot/lvm2-2_02_98.tar.gz)
>    with your PREVIOUS patch (
> https://www.redhat.com/archives/linux-lvm/2013-January/msg00006.html )
> 
> I made a 2-node test cluster, both accessing a test iscsi target, all
> daemons load fine.
> Everything works as expected! Snapshotting is possible after -aey
> --force explicit exclusive locking, lvresize works as well.
> 
> Previously, I tried your latest patch. I had a plethiora of errors,
> starting with lvcreate failing:
> 
> #metadata/lv_manip.c:4137     Clearing start of logical volume "testvol2"
> #device/dev-cache.c:599       /dev/san2/testvol2: stat failed: No such
> file or directory
> #metadata/lv_manip.c:4140   /dev/san2/testvol2: not found: device not
> cleared
> #metadata/lv_manip.c:4620   Aborting. Failed to wipe start of new LV.

Yep, I know about this, I reworked that patch a lot, and now it does
every thing I wanted without regressions.
I will try to post updated version in the next few days when I manage to
apply it to a git tree.

> 
> I need to add -Z n to have it created. After that, most activations
> failed, with error code 5. I  went back to vanilla 2.2.98, all ok except
> exclusive locking, applied the initial locking patch and voila! All done.
> 
> Thanks for your support!
> Andreas
> 
> 

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Andreas Pflug]

Am 13.03.13 17:53, schrieb Vladislav Bogdanov:
>
> #metadata/lv_manip.c:4137     Clearing start of logical volume "testvol2"
> #device/dev-cache.c:599       /dev/san2/testvol2: stat failed: No such
> file or directory
> #metadata/lv_manip.c:4140   /dev/san2/testvol2: not found: device not
> cleared
> #metadata/lv_manip.c:4620   Aborting. Failed to wipe start of new LV.
> Yep, I know about this, I reworked that patch a lot, and now it does
> every thing I wanted without regressions.
> I will try to post updated version in the next few days when I manage to
> apply it to a git tree.

Is there a way to find out if a LV is locked exclusively? lvs displaying 
-e-- instead of -a-- would be nice. Seems not even lvdisplay knows about 
exclusive locking.

Regards,
Andreas

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Vladislav Bogdanov]

13.03.2013 20:37, Andreas Pflug wrote:
> Am 13.03.13 17:53, schrieb Vladislav Bogdanov:
>>
>> #metadata/lv_manip.c:4137     Clearing start of logical volume "testvol2"
>> #device/dev-cache.c:599       /dev/san2/testvol2: stat failed: No such
>> file or directory
>> #metadata/lv_manip.c:4140   /dev/san2/testvol2: not found: device not
>> cleared
>> #metadata/lv_manip.c:4620   Aborting. Failed to wipe start of new LV.
>> Yep, I know about this, I reworked that patch a lot, and now it does
>> every thing I wanted without regressions.
>> I will try to post updated version in the next few days when I manage to
>> apply it to a git tree.
> 
> Is there a way to find out if a LV is locked exclusively? lvs displaying
> -e-- instead of -a-- would be nice. Seems not even lvdisplay knows about
> exclusive locking.

That would break other tools which rely on their output. F.e. cluster
resource agents of libvirt (yes, it runs lvm tools rather then using
API, which is not yet complete btw). As I also need to obtain this
information, I think about writing simple tool (f.e. clvm_tool) which
would display needed info.

As a workaround you can run lvchange -aly without force parameter. If it
succeeds, the volume is locked in a shared mode, otherwise it is locked
exclusively.

> 
> Regards,
> Andreas

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Andreas Pflug]

On 03/13/13 19:30, Vladislav Bogdanov wrote:
>
>> Is there a way to find out if a LV is locked exclusively? lvs displaying
>> -e-- instead of -a-- would be nice. Seems not even lvdisplay knows about
>> exclusive locking.
> That would break other tools which rely on their output. F.e. cluster
> resource agents of libvirt (yes, it runs lvm tools rather then using
> API, which is not yet complete btw). As I also need to obtain this
> information, I think about writing simple tool (f.e. clvm_tool) which
> would display needed info.
>
> As a workaround you can run lvchange -aly without force parameter. If it
> succeeds, the volume is locked in a shared mode, otherwise it is locked
> exclusively.

Hm, thats one ugly workaround...
How about a clvmd option, something like -l to list all locks and exit.

Regards,
Andreas

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Zdenek Kabelac]

Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
> On 03/13/13 19:30, Vladislav Bogdanov wrote:
>>
>>> Is there a way to find out if a LV is locked exclusively? lvs displaying
>>> -e-- instead of -a-- would be nice. Seems not even lvdisplay knows about
>>> exclusive locking.
>> That would break other tools which rely on their output. F.e. cluster
>> resource agents of libvirt (yes, it runs lvm tools rather then using
>> API, which is not yet complete btw). As I also need to obtain this
>> information, I think about writing simple tool (f.e. clvm_tool) which
>> would display needed info.
>>
>> As a workaround you can run lvchange -aly without force parameter. If it
>> succeeds, the volume is locked in a shared mode, otherwise it is locked
>> exclusively.
>
> Hm, thats one ugly workaround...
> How about a clvmd option, something like -l to list all locks and exit.
>


I think - the extension to  'lvs' command could be relatively simple
(adding a new column)

You may query  for  exclusive/local activation on the node.
(So you cannot just tell on which other node is the device active,
but you could print about these states:

active exclusive local
active exclusive
active local
active

Zdenek

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Vladislav Bogdanov]

15.03.2013 12:00, Zdenek Kabelac wrote:
> Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
>> On 03/13/13 19:30, Vladislav Bogdanov wrote:
>>>
>>>> Is there a way to find out if a LV is locked exclusively? lvs
>>>> displaying
>>>> -e-- instead of -a-- would be nice. Seems not even lvdisplay knows
>>>> about
>>>> exclusive locking.
>>> That would break other tools which rely on their output. F.e. cluster
>>> resource agents of libvirt (yes, it runs lvm tools rather then using
>>> API, which is not yet complete btw). As I also need to obtain this
>>> information, I think about writing simple tool (f.e. clvm_tool) which
>>> would display needed info.
>>>
>>> As a workaround you can run lvchange -aly without force parameter. If it
>>> succeeds, the volume is locked in a shared mode, otherwise it is locked
>>> exclusively.
>>
>> Hm, thats one ugly workaround...
>> How about a clvmd option, something like -l to list all locks and exit.
>>
> 
> 
> I think - the extension to  'lvs' command could be relatively simple
> (adding a new column)

Yes, that's correct.

> 
> You may query  for  exclusive/local activation on the node.
> (So you cannot just tell on which other node is the device active,
> but you could print about these states:
> 
> active exclusive local
> active exclusive
> active local
> active

You also may poll all know nodes, but that is a hack too.

That's why I prefer to have this as a separate tool (with dlm_tool-like
params and output) which lists node IDs and lock mode. Unfortunately do
not have power to write it now.

Are core LVM devels interested in these two features: lock conversion
and managing remote node locks? If yes, then I can (hopefully) prepare
git patches next week.

Vladislav

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Zdenek Kabelac]

Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
> 15.03.2013 12:00, Zdenek Kabelac wrote:
>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
>>> On 03/13/13 19:30, Vladislav Bogdanov wrote:
>>>>
>>>>> Is there a way to find out if a LV is locked exclusively? lvs
>>>>> displaying
>>>>> -e-- instead of -a-- would be nice. Seems not even lvdisplay knows
>>>>> about
>>>>> exclusive locking.
>>>> That would break other tools which rely on their output. F.e. cluster
>>>> resource agents of libvirt (yes, it runs lvm tools rather then using
>>>> API, which is not yet complete btw). As I also need to obtain this
>>>> information, I think about writing simple tool (f.e. clvm_tool) which
>>>> would display needed info.
>>>>
>>>> As a workaround you can run lvchange -aly without force parameter. If it
>>>> succeeds, the volume is locked in a shared mode, otherwise it is locked
>>>> exclusively.
>>>
>>> Hm, thats one ugly workaround...
>>> How about a clvmd option, something like -l to list all locks and exit.
>>>
>>
>>
>> I think - the extension to  'lvs' command could be relatively simple
>> (adding a new column)
>
> Yes, that's correct.
>
>>
>> You may query  for  exclusive/local activation on the node.
>> (So you cannot just tell on which other node is the device active,
>> but you could print about these states:
>>
>> active exclusive local
>> active exclusive
>> active local
>> active
>
> You also may poll all know nodes, but that is a hack too.
>
> That's why I prefer to have this as a separate tool (with dlm_tool-like
> params and output) which lists node IDs and lock mode. Unfortunately do
> not have power to write it now.
>
> Are core LVM devels interested in these two features: lock conversion
> and managing remote node locks? If yes, then I can (hopefully) prepare
> git patches next week.


I'm not quite sure what do you mean by  'managing remote node locks' ?

Current login behind lvm command is  -

You could activate LVs with the above syntax [ael]
(there is a tag support - so you could exclusively activate LV on remote
node in via some configuration tags)

And you want to 'upgrade' remote locks to something else ?

What would be the use-case you could not resolve with current command line args?

Is that supported by dlm (since lvm locks are mapped to dlm)?
How would you resolve error path fallbacks ?
Also I believe the clvmd protocol is out of free bits for extension,
so how the protocol would look like ?

Zdenek

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Vladislav Bogdanov]

15.03.2013 12:37, Zdenek Kabelac wrote:
> Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
>> 15.03.2013 12:00, Zdenek Kabelac wrote:
>>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
>>>> On 03/13/13 19:30, Vladislav Bogdanov wrote:
>>>>>
>>>>>> Is there a way to find out if a LV is locked exclusively? lvs
>>>>>> displaying
>>>>>> -e-- instead of -a-- would be nice. Seems not even lvdisplay knows
>>>>>> about
>>>>>> exclusive locking.
>>>>> That would break other tools which rely on their output. F.e. cluster
>>>>> resource agents of libvirt (yes, it runs lvm tools rather then using
>>>>> API, which is not yet complete btw). As I also need to obtain this
>>>>> information, I think about writing simple tool (f.e. clvm_tool) which
>>>>> would display needed info.
>>>>>
>>>>> As a workaround you can run lvchange -aly without force parameter.
>>>>> If it
>>>>> succeeds, the volume is locked in a shared mode, otherwise it is
>>>>> locked
>>>>> exclusively.
>>>>
>>>> Hm, thats one ugly workaround...
>>>> How about a clvmd option, something like -l to list all locks and exit.
>>>>
>>>
>>>
>>> I think - the extension to  'lvs' command could be relatively simple
>>> (adding a new column)
>>
>> Yes, that's correct.
>>
>>>
>>> You may query  for  exclusive/local activation on the node.
>>> (So you cannot just tell on which other node is the device active,
>>> but you could print about these states:
>>>
>>> active exclusive local
>>> active exclusive
>>> active local
>>> active
>>
>> You also may poll all know nodes, but that is a hack too.
>>
>> That's why I prefer to have this as a separate tool (with dlm_tool-like
>> params and output) which lists node IDs and lock mode. Unfortunately do
>> not have power to write it now.
>>
>> Are core LVM devels interested in these two features: lock conversion
>> and managing remote node locks? If yes, then I can (hopefully) prepare
>> git patches next week.
> 
> 
> I'm not quite sure what do you mean by  'managing remote node locks' ?

Activation/deactivation of LVs on a different corosync cluster node,
specified by its node name (with pacemaker-like method to determine that
name).
Also conversion of locks on that node.

> 
> Current login behind lvm command is  -
> 
> You could activate LVs with the above syntax [ael]
> (there is a tag support - so you could exclusively activate LV on remote
> node in via some configuration tags)

Could you please explain this - I do not see anything relevant in man pages.

> 
> And you want to 'upgrade' remote locks to something else ?

Yes, shared-to-exclusive end vice verse.

> 
> What would be the use-case you could not resolve with current command
> line args?

I need to convert lock on a remote node during last stage of ver3
migration in libvirt/qemu. That is a "confirm" stage, which runs on a
"source" node, during which "old" vm is killed and disk is released.
So, I first ("begin" stage) convert lock from exclusive to shared on a
source node, then obtain shared lock on a target node (during "prepare"
stage, when receiving qemu instance is started), then migrate vm between
two processes which have LV opened, and then release shared lock on a
source node ("confirm" stage, after source qemu is killed).

There is no other events on a destination node in ver3 migration
protocol, so I'm unable to convert lock to exclusive there after
migration is finished. So I do that from a source node, after it
released lock.

> 
> Is that supported by dlm (since lvm locks are mapped to dlm)?
Command just sent to a specific clvm instance and performed there.

> How would you resolve error path fallbacks ?

Could you please tell  what exactly do you mean?
If dlm on a remote node is unable to perform requested operation, then
error is returned to a initiator.

> Also I believe the clvmd protocol is out of free bits for extension,
> so how the protocol would look like ?
It contains 'node' field (I assume it was never actually used before)
and with some fixes that works.

Vladislav

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Vladislav Bogdanov]

15.03.2013 15:53, Vladislav Bogdanov wrote:
> 15.03.2013 12:37, Zdenek Kabelac wrote:
>> Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
>>> 15.03.2013 12:00, Zdenek Kabelac wrote:
>>>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
>>>>> On 03/13/13 19:30, Vladislav Bogdanov wrote:
>>>>>>
>>>>>>> Is there a way to find out if a LV is locked exclusively? lvs
>>>>>>> displaying
>>>>>>> -e-- instead of -a-- would be nice. Seems not even lvdisplay knows
>>>>>>> about
>>>>>>> exclusive locking.
>>>>>> That would break other tools which rely on their output. F.e. cluster
>>>>>> resource agents of libvirt (yes, it runs lvm tools rather then using
>>>>>> API, which is not yet complete btw). As I also need to obtain this
>>>>>> information, I think about writing simple tool (f.e. clvm_tool) which
>>>>>> would display needed info.
>>>>>>
>>>>>> As a workaround you can run lvchange -aly without force parameter.
>>>>>> If it
>>>>>> succeeds, the volume is locked in a shared mode, otherwise it is
>>>>>> locked
>>>>>> exclusively.
>>>>>
>>>>> Hm, thats one ugly workaround...
>>>>> How about a clvmd option, something like -l to list all locks and exit.
>>>>>
>>>>
>>>>
>>>> I think - the extension to  'lvs' command could be relatively simple
>>>> (adding a new column)
>>>
>>> Yes, that's correct.
>>>
>>>>
>>>> You may query  for  exclusive/local activation on the node.
>>>> (So you cannot just tell on which other node is the device active,
>>>> but you could print about these states:
>>>>
>>>> active exclusive local
>>>> active exclusive
>>>> active local
>>>> active
>>>
>>> You also may poll all know nodes, but that is a hack too.
>>>
>>> That's why I prefer to have this as a separate tool (with dlm_tool-like
>>> params and output) which lists node IDs and lock mode. Unfortunately do
>>> not have power to write it now.
>>>
>>> Are core LVM devels interested in these two features: lock conversion
>>> and managing remote node locks? If yes, then I can (hopefully) prepare
>>> git patches next week.
>>
>>
>> I'm not quite sure what do you mean by  'managing remote node locks' ?
> 
> Activation/deactivation of LVs on a different corosync cluster node,
> specified by its node name (with pacemaker-like method to determine that
> name).
> Also conversion of locks on that node.
> 
>>
>> Current login behind lvm command is  -
>>
>> You could activate LVs with the above syntax [ael]
>> (there is a tag support - so you could exclusively activate LV on remote
>> node in via some configuration tags)
> 
> Could you please explain this - I do not see anything relevant in man pages.
> 
>>
>> And you want to 'upgrade' remote locks to something else ?
> 
> Yes, shared-to-exclusive end vice verse.
> 
>>
>> What would be the use-case you could not resolve with current command
>> line args?
> 
> I need to convert lock on a remote node during last stage of ver3
> migration in libvirt/qemu. That is a "confirm" stage, which runs on a
> "source" node, during which "old" vm is killed and disk is released.
> So, I first ("begin" stage) convert lock from exclusive to shared on a
> source node, then obtain shared lock on a target node (during "prepare"
> stage, when receiving qemu instance is started), then migrate vm between
> two processes which have LV opened, and then release shared lock on a
> source node ("confirm" stage, after source qemu is killed).
> 
> There is no other events on a destination node in ver3 migration
> protocol, so I'm unable to convert lock to exclusive there after
> migration is finished. So I do that from a source node, after it
> released lock.
> 
>>
>> Is that supported by dlm (since lvm locks are mapped to dlm)?
> Command just sent to a specific clvm instance and performed there.
> 
>> How would you resolve error path fallbacks ?
> 
> Could you please tell  what exactly do you mean?
> If dlm on a remote node is unable to perform requested operation, then
> error is returned to a initiator.
> 
>> Also I believe the clvmd protocol is out of free bits for extension,
>> so how the protocol would look like ?
> It contains 'node' field (I assume it was never actually used before)
> and with some fixes that works.

Fix to myself. clvm client API has that field. And there is no need for
changes in on-wire protocol - corosync module already allows to send a
message to a specific csid (node).

> 
> Vladislav
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Zdenek Kabelac]

Dne 15.3.2013 13:53, Vladislav Bogdanov napsal(a):
> 15.03.2013 12:37, Zdenek Kabelac wrote:
>> Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
>>> 15.03.2013 12:00, Zdenek Kabelac wrote:
>>>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
>>>>> On 03/13/13 19:30, Vladislav Bogdanov wrote:
>>>>>>
>>> You could activate LVs with the above syntax [ael]
>> (there is a tag support - so you could exclusively activate LV on remote
>> node in via some configuration tags)
>
> Could you please explain this - I do not see anything relevant in man pages.

Let's say - you have 3 nodes  A, B, C - each have a TAG_A, TAG_B, TAG_C,
then on node A you may exclusively activate LV which has TAG_B - this
will try to exclusively active LV on the node which has it configured
in lvm.conf  (see the  volume_list= [])

>
>>
>> And you want to 'upgrade' remote locks to something else ?
>
> Yes, shared-to-exclusive end vice verse.

So how do you convert the lock from   shared to exclusive without unlock
(if I get it right - you keep the ConcurrentRead lock - and you want to take 
Exlusive -  to  make change state from  'active' to 'active exlusive')
https://en.wikipedia.org/wiki/Distributed_lock_manager

Clvmd 'communicates' via these locks.
So the proper algorithm needs to be there for ending with some proper state 
after locks changes (and sorry I'm not a dlm expert here)


>>
>> What would be the use-case you could not resolve with current command
>> line args?
>
> I need to convert lock on a remote node during last stage of ver3
> migration in libvirt/qemu. That is a "confirm" stage, which runs on a
> "source" node, during which "old" vm is killed and disk is released.
> So, I first ("begin" stage) convert lock from exclusive to shared on a
> source node, then obtain shared lock on a target node (during "prepare"

Which most probably works only your environment - since you do not try to
'break' the logic - but it's probably not a generic concept - i.e.
in this controlled environment you may live probably happily even with
local activation of LVs - since you always know what the tool is doing.

> There is no other events on a destination node in ver3 migration
> protocol, so I'm unable to convert lock to exclusive there after
> migration is finished. So I do that from a source node, after it
> released lock.
>
>>
>> Is that supported by dlm (since lvm locks are mapped to dlm)?
> Command just sent to a specific clvm instance and performed there.

As said - the 'lock' is the thing which controls the activation state.
So faking it on the software level may possible lead to inconsistency between 
the dlm and clvmd view of the lock state.

Zdenek

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Vladislav Bogdanov]

15.03.2013 16:32, Zdenek Kabelac wrote:
> Dne 15.3.2013 13:53, Vladislav Bogdanov napsal(a):
>> 15.03.2013 12:37, Zdenek Kabelac wrote:
>>> Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
>>>> 15.03.2013 12:00, Zdenek Kabelac wrote:
>>>>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
>>>>>> On 03/13/13 19:30, Vladislav Bogdanov wrote:
>>>>>>>
>>>> You could activate LVs with the above syntax [ael]
>>> (there is a tag support - so you could exclusively activate LV on remote
>>> node in via some configuration tags)
>>
>> Could you please explain this - I do not see anything relevant in man
>> pages.
> 
> Let's say - you have 3 nodes  A, B, C - each have a TAG_A, TAG_B, TAG_C,
> then on node A you may exclusively activate LV which has TAG_B - this
> will try to exclusively active LV on the node which has it configured
> in lvm.conf  (see the  volume_list= [])

Aha, if I understand correctly this is absolutely not what I need.
I want all this to be fully dynamic without any "config-editing voodoo".

> 
>>
>>>
>>> And you want to 'upgrade' remote locks to something else ?
>>
>> Yes, shared-to-exclusive end vice verse.
> 
> So how do you convert the lock from   shared to exclusive without unlock
> (if I get it right - you keep the ConcurrentRead lock - and you want to
> take Exlusive -  to  make change state from  'active' to 'active exlusive')
> https://en.wikipedia.org/wiki/Distributed_lock_manager

I just pass LCKF_CONVERT to dlm_controld if requested and needed. And
that is dlm's task to either satisfy conversion or to refuse it.

> 
> Clvmd 'communicates' via these locks.

Not exactly true.

clvmd does cluster communications with corosync, which implements
virtual synchrony, so all cluster nodes receive messages in the same order.
At the bottom, clvmd uses libdlm to communicate with dlm_controld and
request it to lock/unlock.
dlm_controld instances use corosync for membership and locally manages
in-kernel dlm counter-part, which uses tcp/sctp mesh-like connections to
communicate.
So request from one clvmd instance goes to another and goes to kernel
from there, and then it is distributed to other nodes. Actually that
does not matter where does it hits kernel space if it supports
delegation of locks to remote nodes, but I suspect it doesn't. But if it
doesn't support such thing, then the only option to manage lock on a
remote node is to request that's node dlm instance to do the locking job.

> So the proper algorithm needs to be there for ending with some proper
> state after locks changes (and sorry I'm not a dlm expert here)

That is what actually happens.
There is just no difference between running (to upgrade local lock to
exclusive on node <node>.

ssh <node> lvchange -aey --force VG/LV

or

lvchange -aey --node <node> --force VG/LV

The same command, it is just sent via different channels.

Again, I just send locking request to a remote clvmd instance through
corosync.
It asks its local dlm to convert (acquire, release) lock and returns its
answer back. After dlm answers, operation is either performed, and then
OK is send back to a initiator, or refused, and the error is sent back.

> 
> 
>>>
>>> What would be the use-case you could not resolve with current command
>>> line args?
>>
>> I need to convert lock on a remote node during last stage of ver3
>> migration in libvirt/qemu. That is a "confirm" stage, which runs on a
>> "source" node, during which "old" vm is killed and disk is released.
>> So, I first ("begin" stage) convert lock from exclusive to shared on a
>> source node, then obtain shared lock on a target node (during "prepare"
> 
> Which most probably works only your environment - since you do not try to
> 'break' the logic - but it's probably not a generic concept - i.e.
> in this controlled environment you may live probably happily even with
> local activation of LVs - since you always know what the tool is doing.

I prefer to do not allow anybody else to open an LV at all (and to be
able to take snapshots of it when needed on a node where it is
exclusively locked). That's why I implemented a lock-manager driver for
libvirt, which controls inter-node locks with clvm exclusive activation,
and then controls inner open attempts with local locking mechanism.

While libvirt already has two lok drivers, neither of fully satisfy me
because or shared storage requirement. And, I want to take snapshots,
that's why I need exclusive activation.

I plan to send libvirt patches to the libvirt list if lvm part is
accepted (otherwise it makes no sense to do that and I will need to
maintain everything out-of-the-tree).

> 
>> There is no other events on a destination node in ver3 migration
>> protocol, so I'm unable to convert lock to exclusive there after
>> migration is finished. So I do that from a source node, after it
>> released lock.
>>
>>>
>>> Is that supported by dlm (since lvm locks are mapped to dlm)?
>> Command just sent to a specific clvm instance and performed there.
> 
> As said - the 'lock' is the thing which controls the activation state.
> So faking it on the software level may possible lead to inconsistency
> between the dlm and clvmd view of the lock state.

No faking. Just a remote management of the same lock.

Vladislav

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Zdenek Kabelac]

Dne 15.3.2013 15:51, Vladislav Bogdanov napsal(a):
> 15.03.2013 16:32, Zdenek Kabelac wrote:
>> Dne 15.3.2013 13:53, Vladislav Bogdanov napsal(a):
>>> 15.03.2013 12:37, Zdenek Kabelac wrote:
>>>> Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
>>>>> 15.03.2013 12:00, Zdenek Kabelac wrote:
>>>>>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
>>>>>>> On 03/13/13 19:30, Vladislav Bogdanov wrote:
>>>>>>>>
>>>>> You could activate LVs with the above syntax [ael]
>>>> (there is a tag support - so you could exclusively activate LV on remote
>>>> node in via some configuration tags)
>>>
>>> Could you please explain this - I do not see anything relevant in man
>>> pages.
>>
>> Let's say - you have 3 nodes  A, B, C - each have a TAG_A, TAG_B, TAG_C,
>> then on node A you may exclusively activate LV which has TAG_B - this
>> will try to exclusively active LV on the node which has it configured
>> in lvm.conf  (see the  volume_list= [])
>
> Aha, if I understand correctly this is absolutely not what I need.
> I want all this to be fully dynamic without any "config-editing voodoo".
>
>>
>>>
>>>>
>>>> And you want to 'upgrade' remote locks to something else ?
>>>
>>> Yes, shared-to-exclusive end vice verse.
>>
>> So how do you convert the lock from   shared to exclusive without unlock
>> (if I get it right - you keep the ConcurrentRead lock - and you want to
>> take Exlusive -  to  make change state from  'active' to 'active exlusive')
>> https://en.wikipedia.org/wiki/Distributed_lock_manager
>
> I just pass LCKF_CONVERT to dlm_controld if requested and needed. And
> that is dlm's task to either satisfy conversion or to refuse it.
>

So to understand myself better this thing -

the dlm sends 'unlock' requests to all other nodes except the one which
should be converted to exclusive mode and send exclusive lock to the prefered 
node?

>>
>> Clvmd 'communicates' via these locks.
>
> Not exactly true.
>
> clvmd does cluster communications with corosync, which implements
> virtual synchrony, so all cluster nodes receive messages in the same order.
> At the bottom, clvmd uses libdlm to communicate with dlm_controld and
> request it to lock/unlock.
> dlm_controld instances use corosync for membership and locally manages
> in-kernel dlm counter-part, which uses tcp/sctp mesh-like connections to
> communicate.
> So request from one clvmd instance goes to another and goes to kernel
> from there, and then it is distributed to other nodes. Actually that
> does not matter where does it hits kernel space if it supports
> delegation of locks to remote nodes, but I suspect it doesn't. But if it
> doesn't support such thing, then the only option to manage lock on a
> remote node is to request that's node dlm instance to do the locking job.
>
>> So the proper algorithm needs to be there for ending with some proper
>> state after locks changes (and sorry I'm not a dlm expert here)
>
> That is what actually happens.
> There is just no difference between running (to upgrade local lock to
> exclusive on node <node>.
>
> ssh <node> lvchange -aey --force VG/LV
>
> or
>
> lvchange -aey --node <node> --force VG/LV


--node is exactly what the tag is for - each node may have it's tag.
lvm doesn't work with cluster nodes.

The question is - could be the code transformed to use this logic ?
I guess you need to know  dlm node name here right ?


> The same command, it is just sent via different channels.
>
> Again, I just send locking request to a remote clvmd instance through
> corosync.
> It asks its local dlm to convert (acquire, release) lock and returns its
> answer back. After dlm answers, operation is either performed, and then
> OK is send back to a initiator, or refused, and the error is sent back.


>>> There is no other events on a destination node in ver3 migration
>>> protocol, so I'm unable to convert lock to exclusive there after
>>> migration is finished. So I do that from a source node, after it
>>> released lock.
>>>
>>>>
>>>> Is that supported by dlm (since lvm locks are mapped to dlm)?
>>> Command just sent to a specific clvm instance and performed there.
>>
>> As said - the 'lock' is the thing which controls the activation state.
>> So faking it on the software level may possible lead to inconsistency
>> between the dlm and clvmd view of the lock state.
>
> No faking. Just a remote management of the same lock.

Could you repost patches against git ?
With some usage examples ?

Zdenek

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Vladislav Bogdanov]

15.03.2013 18:02, Zdenek Kabelac wrote:
> Dne 15.3.2013 15:51, Vladislav Bogdanov napsal(a):
>> 15.03.2013 16:32, Zdenek Kabelac wrote:
>>> Dne 15.3.2013 13:53, Vladislav Bogdanov napsal(a):
>>>> 15.03.2013 12:37, Zdenek Kabelac wrote:
>>>>> Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
>>>>>> 15.03.2013 12:00, Zdenek Kabelac wrote:
>>>>>>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
>>>>>>>> On 03/13/13 19:30, Vladislav Bogdanov wrote:
>>>>>>>>>
>>>>>> You could activate LVs with the above syntax [ael]
>>>>> (there is a tag support - so you could exclusively activate LV on
>>>>> remote
>>>>> node in via some configuration tags)
>>>>
>>>> Could you please explain this - I do not see anything relevant in man
>>>> pages.
>>>
>>> Let's say - you have 3 nodes  A, B, C - each have a TAG_A, TAG_B, TAG_C,
>>> then on node A you may exclusively activate LV which has TAG_B - this
>>> will try to exclusively active LV on the node which has it configured
>>> in lvm.conf  (see the  volume_list= [])
>>
>> Aha, if I understand correctly this is absolutely not what I need.
>> I want all this to be fully dynamic without any "config-editing voodoo".
>>
>>>
>>>>
>>>>>
>>>>> And you want to 'upgrade' remote locks to something else ?
>>>>
>>>> Yes, shared-to-exclusive end vice verse.
>>>
>>> So how do you convert the lock from   shared to exclusive without unlock
>>> (if I get it right - you keep the ConcurrentRead lock - and you want to
>>> take Exlusive -  to  make change state from  'active' to 'active
>>> exlusive')
>>> https://en.wikipedia.org/wiki/Distributed_lock_manager
>>
>> I just pass LCKF_CONVERT to dlm_controld if requested and needed. And
>> that is dlm's task to either satisfy conversion or to refuse it.
>>
> 
> So to understand myself better this thing -
> 
> the dlm sends 'unlock' requests to all other nodes except the one which
> should be converted to exclusive mode and send exclusive lock to the
> prefered node?

No.
clvmd sends request to a remote clvmd to upgrade or acquire or release
the lock.
That remote instance asks local dlm to do the job. dlm either says OK or
says ERROR.
It does not do anything except that.
It LV is locked on a remote node, be it shared or exclusive lock, dlm
says ERROR if exclusive lock (or conversion to it) is requested.

My patches also allow "-an --force" to release shared locks on other
nodes. Exclusive lock may be released or downgraded only on node which
holds it (or with --node <node>).

> 
>>>
>>> Clvmd 'communicates' via these locks.
>>
>> Not exactly true.
>>
>> clvmd does cluster communications with corosync, which implements
>> virtual synchrony, so all cluster nodes receive messages in the same
>> order.
>> At the bottom, clvmd uses libdlm to communicate with dlm_controld and
>> request it to lock/unlock.
>> dlm_controld instances use corosync for membership and locally manages
>> in-kernel dlm counter-part, which uses tcp/sctp mesh-like connections to
>> communicate.
>> So request from one clvmd instance goes to another and goes to kernel
>> from there, and then it is distributed to other nodes. Actually that
>> does not matter where does it hits kernel space if it supports
>> delegation of locks to remote nodes, but I suspect it doesn't. But if it
>> doesn't support such thing, then the only option to manage lock on a
>> remote node is to request that's node dlm instance to do the locking job.
>>
>>> So the proper algorithm needs to be there for ending with some proper
>>> state after locks changes (and sorry I'm not a dlm expert here)
>>
>> That is what actually happens.
>> There is just no difference between running (to upgrade local lock to
>> exclusive on node <node>.
>>
>> ssh <node> lvchange -aey --force VG/LV
>>
>> or
>>
>> lvchange -aey --node <node> --force VG/LV
> 
> 
> --node is exactly what the tag is for - each node may have it's tag.
> lvm doesn't work with cluster nodes.

But corosync and dlm operate node IDs, and pacemaker operates node names
and IDs. None of them use tags.

> 
> The question is - could be the code transformed to use this logic ?
> I guess you need to know  dlm node name here right ?

Node IDs are obtained from corosync membership list, and may be used for
that. If corosync is configured with nodelist in a way pacemaker wants
it
(http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-name.html),
then node names may be used too.

> 
> 
>> The same command, it is just sent via different channels.
>>
>> Again, I just send locking request to a remote clvmd instance through
>> corosync.
>> It asks its local dlm to convert (acquire, release) lock and returns its
>> answer back. After dlm answers, operation is either performed, and then
>> OK is send back to a initiator, or refused, and the error is sent back.
> 
> 
>>>> There is no other events on a destination node in ver3 migration
>>>> protocol, so I'm unable to convert lock to exclusive there after
>>>> migration is finished. So I do that from a source node, after it
>>>> released lock.
>>>>
>>>>>
>>>>> Is that supported by dlm (since lvm locks are mapped to dlm)?
>>>> Command just sent to a specific clvm instance and performed there.
>>>
>>> As said - the 'lock' is the thing which controls the activation state.
>>> So faking it on the software level may possible lead to inconsistency
>>> between the dlm and clvmd view of the lock state.
>>
>> No faking. Just a remote management of the same lock.
> 
> Could you repost patches against git ?

I plan that for the next week.

> With some usage examples ?

Yes, if you give me an "example of example" ;)

Vladislav

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Zdenek Kabelac]

Dne 15.3.2013 16:36, Vladislav Bogdanov napsal(a):
> 15.03.2013 18:02, Zdenek Kabelac wrote:
>> Dne 15.3.2013 15:51, Vladislav Bogdanov napsal(a):
>>> 15.03.2013 16:32, Zdenek Kabelac wrote:
>>>> Dne 15.3.2013 13:53, Vladislav Bogdanov napsal(a):
>>>>> 15.03.2013 12:37, Zdenek Kabelac wrote:
>>>>>> Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
>>>>>>> 15.03.2013 12:00, Zdenek Kabelac wrote:
>>>>>>>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
>>>>>>>>> On 03/13/13 19:30, Vladislav Bogdanov wrote:
>>>>>>>>>>
>>>>>>> You could activate LVs with the above syntax [ael]
>>>>>> (there is a tag support - so you could exclusively activate LV on
>>>>>> remote
>>>>>> node in via some configuration tags)
>>>>>
>>>>> Could you please explain this - I do not see anything relevant in man
>>>>> pages.
>>>>
>>>> Let's say - you have 3 nodes  A, B, C - each have a TAG_A, TAG_B, TAG_C,
>>>> then on node A you may exclusively activate LV which has TAG_B - this
>>>> will try to exclusively active LV on the node which has it configured
>>>> in lvm.conf  (see the  volume_list= [])
>>>
>>> Aha, if I understand correctly this is absolutely not what I need.
>>> I want all this to be fully dynamic without any "config-editing voodoo".
>>>
>>>>
>>>>>
>>>>>>
>>>>>> And you want to 'upgrade' remote locks to something else ?
>>>>>
>>>>> Yes, shared-to-exclusive end vice verse.
>>>>
>>>> So how do you convert the lock from   shared to exclusive without unlock
>>>> (if I get it right - you keep the ConcurrentRead lock - and you want to
>>>> take Exlusive -  to  make change state from  'active' to 'active
>>>> exlusive')
>>>> https://en.wikipedia.org/wiki/Distributed_lock_manager
>>>
>>> I just pass LCKF_CONVERT to dlm_controld if requested and needed. And
>>> that is dlm's task to either satisfy conversion or to refuse it.
>>>
>>
>> So to understand myself better this thing -
>>
>> the dlm sends 'unlock' requests to all other nodes except the one which
>> should be converted to exclusive mode and send exclusive lock to the
>> prefered node?
>
> No.
> clvmd sends request to a remote clvmd to upgrade or acquire or release
> the lock.
> That remote instance asks local dlm to do the job. dlm either says OK or
> says ERROR.
> It does not do anything except that.
> It LV is locked on a remote node, be it shared or exclusive lock, dlm
> says ERROR if exclusive lock (or conversion to it) is requested.
>
> My patches also allow "-an --force" to release shared locks on other
> nodes. Exclusive lock may be released or downgraded only on node which
> holds it (or with --node <node>).
>
>>
>>>>
>>>> Clvmd 'communicates' via these locks.
>>>
>>> Not exactly true.
>>>
>>> clvmd does cluster communications with corosync, which implements
>>> virtual synchrony, so all cluster nodes receive messages in the same
>>> order.
>>> At the bottom, clvmd uses libdlm to communicate with dlm_controld and
>>> request it to lock/unlock.
>>> dlm_controld instances use corosync for membership and locally manages
>>> in-kernel dlm counter-part, which uses tcp/sctp mesh-like connections to
>>> communicate.
>>> So request from one clvmd instance goes to another and goes to kernel
>>> from there, and then it is distributed to other nodes. Actually that
>>> does not matter where does it hits kernel space if it supports
>>> delegation of locks to remote nodes, but I suspect it doesn't. But if it
>>> doesn't support such thing, then the only option to manage lock on a
>>> remote node is to request that's node dlm instance to do the locking job.
>>>
>>>> So the proper algorithm needs to be there for ending with some proper
>>>> state after locks changes (and sorry I'm not a dlm expert here)
>>>
>>> That is what actually happens.
>>> There is just no difference between running (to upgrade local lock to
>>> exclusive on node <node>.
>>>
>>> ssh <node> lvchange -aey --force VG/LV
>>>
>>> or
>>>
>>> lvchange -aey --node <node> --force VG/LV
>>
>>
>> --node is exactly what the tag is for - each node may have it's tag.
>> lvm doesn't work with cluster nodes.
>
> But corosync and dlm operate node IDs, and pacemaker operates node names
> and IDs. None of them use tags.
>
>>
>> The question is - could be the code transformed to use this logic ?
>> I guess you need to know  dlm node name here right ?
>
> Node IDs are obtained from corosync membership list, and may be used for
> that. If corosync is configured with nodelist in a way pacemaker wants
> it
> (http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-name.html),
> then node names may be used too.

clvmd knows the dlm node name - but lvm command should reference things via 
tags. There will be probably no way to add '--node' option into lvm command.

Can you think about using 'tags'?

So your machines will have configured tags (be it machine name) and instead of 
the node - you would use 'tag' ?

Zdenek

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Vladislav Bogdanov]

15.03.2013 18:55, Zdenek Kabelac пишет:
> Dne 15.3.2013 16:36, Vladislav Bogdanov napsal(a):
>> 15.03.2013 18:02, Zdenek Kabelac wrote:
>>> Dne 15.3.2013 15:51, Vladislav Bogdanov napsal(a):
>>>> 15.03.2013 16:32, Zdenek Kabelac wrote:
>>>>> Dne 15.3.2013 13:53, Vladislav Bogdanov napsal(a):
>>>>>> 15.03.2013 12:37, Zdenek Kabelac wrote:
>>>>>>> Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
>>>>>>>> 15.03.2013 12:00, Zdenek Kabelac wrote:
>>>>>>>>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
>>>>>>>>>> On 03/13/13 19:30, Vladislav Bogdanov wrote:
>>>>>>>>>>>
>>>>>>>> You could activate LVs with the above syntax [ael]
>>>>>>> (there is a tag support - so you could exclusively activate LV on
>>>>>>> remote
>>>>>>> node in via some configuration tags)
>>>>>>
>>>>>> Could you please explain this - I do not see anything relevant in man
>>>>>> pages.
>>>>>
>>>>> Let's say - you have 3 nodes  A, B, C - each have a TAG_A, TAG_B,
>>>>> TAG_C,
>>>>> then on node A you may exclusively activate LV which has TAG_B - this
>>>>> will try to exclusively active LV on the node which has it configured
>>>>> in lvm.conf  (see the  volume_list= [])
>>>>
>>>> Aha, if I understand correctly this is absolutely not what I need.
>>>> I want all this to be fully dynamic without any "config-editing
>>>> voodoo".
>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>> And you want to 'upgrade' remote locks to something else ?
>>>>>>
>>>>>> Yes, shared-to-exclusive end vice verse.
>>>>>
>>>>> So how do you convert the lock from   shared to exclusive without
>>>>> unlock
>>>>> (if I get it right - you keep the ConcurrentRead lock - and you
>>>>> want to
>>>>> take Exlusive -  to  make change state from  'active' to 'active
>>>>> exlusive')
>>>>> https://en.wikipedia.org/wiki/Distributed_lock_manager
>>>>
>>>> I just pass LCKF_CONVERT to dlm_controld if requested and needed. And
>>>> that is dlm's task to either satisfy conversion or to refuse it.
>>>>
>>>
>>> So to understand myself better this thing -
>>>
>>> the dlm sends 'unlock' requests to all other nodes except the one which
>>> should be converted to exclusive mode and send exclusive lock to the
>>> prefered node?
>>
>> No.
>> clvmd sends request to a remote clvmd to upgrade or acquire or release
>> the lock.
>> That remote instance asks local dlm to do the job. dlm either says OK or
>> says ERROR.
>> It does not do anything except that.
>> It LV is locked on a remote node, be it shared or exclusive lock, dlm
>> says ERROR if exclusive lock (or conversion to it) is requested.
>>
>> My patches also allow "-an --force" to release shared locks on other
>> nodes. Exclusive lock may be released or downgraded only on node which
>> holds it (or with --node <node>).
>>
>>>
>>>>>
>>>>> Clvmd 'communicates' via these locks.
>>>>
>>>> Not exactly true.
>>>>
>>>> clvmd does cluster communications with corosync, which implements
>>>> virtual synchrony, so all cluster nodes receive messages in the same
>>>> order.
>>>> At the bottom, clvmd uses libdlm to communicate with dlm_controld and
>>>> request it to lock/unlock.
>>>> dlm_controld instances use corosync for membership and locally manages
>>>> in-kernel dlm counter-part, which uses tcp/sctp mesh-like
>>>> connections to
>>>> communicate.
>>>> So request from one clvmd instance goes to another and goes to kernel
>>>> from there, and then it is distributed to other nodes. Actually that
>>>> does not matter where does it hits kernel space if it supports
>>>> delegation of locks to remote nodes, but I suspect it doesn't. But
>>>> if it
>>>> doesn't support such thing, then the only option to manage lock on a
>>>> remote node is to request that's node dlm instance to do the locking
>>>> job.
>>>>
>>>>> So the proper algorithm needs to be there for ending with some proper
>>>>> state after locks changes (and sorry I'm not a dlm expert here)
>>>>
>>>> That is what actually happens.
>>>> There is just no difference between running (to upgrade local lock to
>>>> exclusive on node <node>.
>>>>
>>>> ssh <node> lvchange -aey --force VG/LV
>>>>
>>>> or
>>>>
>>>> lvchange -aey --node <node> --force VG/LV
>>>
>>>
>>> --node is exactly what the tag is for - each node may have it's tag.
>>> lvm doesn't work with cluster nodes.
>>
>> But corosync and dlm operate node IDs, and pacemaker operates node names
>> and IDs. None of them use tags.
>>
>>>
>>> The question is - could be the code transformed to use this logic ?
>>> I guess you need to know  dlm node name here right ?
>>
>> Node IDs are obtained from corosync membership list, and may be used for
>> that. If corosync is configured with nodelist in a way pacemaker wants
>> it
>> (http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-name.html),
>>
>> then node names may be used too.
> 
> clvmd knows the dlm node name - but lvm command should reference things
> via tags. There will be probably no way to add '--node' option into lvm
> command.
> 
> Can you think about using 'tags'?
> 
> So your machines will have configured tags (be it machine name) and
> instead of the node - you would use 'tag' ?

So, --tag should be used instead on --node?
I'd better make --node a alias for --tag for exactly this case if that
is possible...
But... For those with cluster background --node is a native term while
--tag is... alien one... So, I'd better leave it as is.

> 
> Zdenek
> 

[linux-lvm] LVM snapshot with Clustered VG

[Rob]

Is there a way to convert an active LV on a clustered VG to an exclusive 
lock without deactivating it first? Or some way of taking a snapshot of 
a clustered LVM? I know I have read snapshots are not possible on CLVM, 
but the information seems outdated.

ha node1: /dev/vg_clus/logvol_clus
ha node2: /dev/vg_clus/logvol_clus

1) I can deactivate the the entire vg on node1 and node2 with 'vgchange 
-an vg_clus' and activate with -ay. Just stating cluster vg is working.

2) I can deactivate locally on node2 with 'vgchange -aln vg_clus' , it 
shows inactive on node2 and active still on node1
     - After deactivating, I cannot get exclusive lock on node1 with 
'vgchange -aey vg_clus'
       I get an error locking on node1

3) I can get exclusive an lock if I deactivate the VG on both nodes and 
then run 'vgchange -aey vg_clus' on either node. The lvcreate snapshop 
works, but again, it requires deactivating the vg, which isn't an option 
in production.


Setup:
  Centos 6.3 (current)
  cman+pacemaker dual primary drbd for HA KVM w/ live migration
  disk: lvm -> drbd -> clvm -> kvm virt

Currently I am taking a lvm snapshot of the drbd lvm backing device, but 
it's not ideal ( as there are other issues )

Thanks for the time!

-Rob

Re: [linux-lvm] LVM snapshot with Clustered VG

[Vladislav Bogdanov]

[-- Attachment #1: Type: text/plain, Size: 1601 bytes --]

04.01.2013 05:56, Rob wrote:
> Is there a way to convert an active LV on a clustered VG to an exclusive
> lock without deactivating it first? Or some way of taking a snapshot of
> a clustered LVM? I know I have read snapshots are not possible on CLVM,
> but the information seems outdated.
> 
> ha node1: /dev/vg_clus/logvol_clus
> ha node2: /dev/vg_clus/logvol_clus
> 
> 1) I can deactivate the the entire vg on node1 and node2 with 'vgchange
> -an vg_clus' and activate with -ay. Just stating cluster vg is working.
> 
> 2) I can deactivate locally on node2 with 'vgchange -aln vg_clus' , it
> shows inactive on node2 and active still on node1
>     - After deactivating, I cannot get exclusive lock on node1 with
> 'vgchange -aey vg_clus'
>       I get an error locking on node1
> 
> 3) I can get exclusive an lock if I deactivate the VG on both nodes and
> then run 'vgchange -aey vg_clus' on either node. The lvcreate snapshop
> works, but again, it requires deactivating the vg, which isn't an option
> in production.

You may try (experimental, not-complete) patch attached.

> 
> 
> Setup:
>  Centos 6.3 (current)
>  cman+pacemaker dual primary drbd for HA KVM w/ live migration
>  disk: lvm -> drbd -> clvm -> kvm virt
> 
> Currently I am taking a lvm snapshot of the drbd lvm backing device, but
> it's not ideal ( as there are other issues )
> 
> Thanks for the time!
> 
> -Rob
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/


[-- Attachment #2: LVM2.2.02.98-lock-convert.patch --]
[-- Type: text/x-patch, Size: 6498 bytes --]

diff -urNp LVM2.2.02.98.orig/lib/locking/cluster_locking.c LVM2.2.02.98/lib/locking/cluster_locking.c
--- LVM2.2.02.98.orig/lib/locking/cluster_locking.c	2012-12-28 13:20:07.218355620 +0000
+++ LVM2.2.02.98/lib/locking/cluster_locking.c	2012-12-15 12:40:02.934170948 +0000
@@ -330,6 +330,9 @@ static int _lock_for_cluster(struct cmd_
 	if (flags & LCK_REVERT)
 		args[1] |= LCK_REVERT_MODE;
 
+	if (flags & LCK_TRY_CONVERT)
+		args[1] |= LCK_CONVERT;
+
 	if (mirror_in_sync())
 		args[1] |= LCK_MIRROR_NOSYNC_MODE;
 
@@ -491,7 +494,7 @@ int lock_resource(struct cmd_context *cm
 		return 0;
 	}
 
-	log_very_verbose("Locking %s %s %s (%s%s%s%s%s%s%s%s%s) (0x%x)", lock_scope, lockname,
+	log_very_verbose("Locking %s %s %s (%s%s%s%s%s%s%s%s%s%s) (0x%x)", lock_scope, lockname,
 			 lock_type, lock_scope,
 			 flags & LCK_NONBLOCK ? "|NONBLOCK" : "",
 			 flags & LCK_HOLD ? "|HOLD" : "",
@@ -501,6 +504,7 @@ int lock_resource(struct cmd_context *cm
 			 flags & LCK_CACHE ? "|CACHE" : "",
 			 flags & LCK_ORIGIN_ONLY ? "|ORIGIN_ONLY" : "",
 			 flags & LCK_REVERT ? "|REVERT" : "",
+			 flags & LCK_TRY_CONVERT ? "|CONVERT" : "",
 			 flags);
 
 	/* Send a message to the cluster manager */
diff -urNp LVM2.2.02.98.orig/lib/locking/locking.c LVM2.2.02.98/lib/locking/locking.c
--- LVM2.2.02.98.orig/lib/locking/locking.c	2012-12-28 13:20:07.219355605 +0000
+++ LVM2.2.02.98/lib/locking/locking.c	2012-12-28 13:25:10.218097986 +0000
@@ -540,6 +540,16 @@ int suspend_lvs(struct cmd_context *cmd,
 	return 1;
 }
 
+int deactivate_lv(struct cmd_context *cmd, struct logical_volume *lv)
+{
+	if (vg_is_clustered(lv->vg)) {
+		if (lv_is_active_exclusive(lv) && ! lv_is_active_exclusive_locally(lv)) {
+			return_0;
+		}
+	}
+	lock_lv_vol(cmd, lv, LCK_LV_DEACTIVATE);
+}
+
 /*
  * First try to activate exclusively locally.
  * Then if the VG is clustered and the LV is not yet active (e.g. due to 
@@ -567,6 +577,28 @@ int activate_lv_excl(struct cmd_context 
 	return lv_is_active_exclusive(lv);
 }
 
+int activate_lv_excl_force(struct cmd_context *cmd, struct logical_volume *lv) 
+{
+	/* Non-clustered VGs are only activated locally. */
+	if (!vg_is_clustered(lv->vg))
+		return activate_lv_excl_local(cmd, lv);
+
+	if (lv_is_active_exclusive_locally(lv))
+		return 1;
+
+	if (!activate_lv_excl_local_force(cmd, lv))
+		return_0;
+
+	if (lv_is_active_exclusive(lv))
+		return 1;
+
+	/* FIXME Deal with error return codes. */
+	if (activate_lv_excl_remote_force(cmd, lv))
+		stack;
+
+	return lv_is_active_exclusive(lv);
+}
+
 /* Lock a list of LVs */
 int activate_lvs(struct cmd_context *cmd, struct dm_list *lvs, unsigned exclusive)
 {
diff -urNp LVM2.2.02.98.orig/lib/locking/locking.h LVM2.2.02.98/lib/locking/locking.h
--- LVM2.2.02.98.orig/lib/locking/locking.h	2012-12-28 13:20:07.225307455 +0000
+++ LVM2.2.02.98/lib/locking/locking.h	2012-12-28 13:25:41.166100204 +0000
@@ -101,6 +101,7 @@ int check_lvm1_vg_inactive(struct cmd_co
 #define LCK_CACHE	0x00000100U	/* Operation on cache only using P_ lock */
 #define LCK_ORIGIN_ONLY	0x00000200U	/* Operation should bypass any snapshots */
 #define LCK_REVERT	0x00000400U	/* Revert any incomplete change */
+#define LCK_TRY_CONVERT			0x00004000U	/* Convert existing lock */
 
 /*
  * Additional lock bits for cluster communication via args[1]
@@ -176,19 +177,26 @@ int check_lvm1_vg_inactive(struct cmd_co
 #define revert_lv(cmd, lv)	lock_lv_vol(cmd, lv, LCK_LV_RESUME | LCK_REVERT)
 #define suspend_lv(cmd, lv)	lock_lv_vol(cmd, lv, LCK_LV_SUSPEND | LCK_HOLD)
 #define suspend_lv_origin(cmd, lv)	lock_lv_vol(cmd, lv, LCK_LV_SUSPEND | LCK_HOLD | LCK_ORIGIN_ONLY)
-#define deactivate_lv(cmd, lv)	lock_lv_vol(cmd, lv, LCK_LV_DEACTIVATE)
 
 #define activate_lv(cmd, lv)	lock_lv_vol(cmd, lv, LCK_LV_ACTIVATE | LCK_HOLD)
 #define activate_lv_excl_local(cmd, lv)	\
 				lock_lv_vol(cmd, lv, LCK_LV_EXCLUSIVE | LCK_HOLD | LCK_LOCAL)
 #define activate_lv_excl_remote(cmd, lv)	\
 				lock_lv_vol(cmd, lv, LCK_LV_EXCLUSIVE | LCK_HOLD | LCK_REMOTE)
+#define activate_lv_excl_local_force(cmd, lv)	\
+				lock_lv_vol(cmd, lv, LCK_LV_EXCLUSIVE | LCK_HOLD | LCK_LOCAL | (lv_is_active(lv) && ! lv_is_active_exclusive(lv) ? LCK_TRY_CONVERT : 0))
+#define activate_lv_excl_remote_force(cmd, lv)	\
+				lock_lv_vol(cmd, lv, LCK_LV_EXCLUSIVE | LCK_HOLD | LCK_REMOTE | (lv_is_active(lv) && ! lv_is_active_exclusive(lv) ? LCK_TRY_CONVERT : 0))
 
 struct logical_volume;
 int activate_lv_excl(struct cmd_context *cmd, struct logical_volume *lv);
+int activate_lv_excl_force(struct cmd_context *cmd, struct logical_volume *lv);
+int deactivate_lv(struct cmd_context *cmd, struct logical_volume *lv);
 
 #define activate_lv_local(cmd, lv)	\
 	lock_lv_vol(cmd, lv, LCK_LV_ACTIVATE | LCK_HOLD | LCK_LOCAL)
+#define activate_lv_local_force(cmd, lv)	\
+	lock_lv_vol(cmd, lv, LCK_LV_ACTIVATE | LCK_HOLD | LCK_LOCAL | (lv_is_active_exclusive_locally(lv) ? LCK_TRY_CONVERT : 0))
 #define deactivate_lv_local(cmd, lv)	\
 	lock_lv_vol(cmd, lv, LCK_LV_DEACTIVATE | LCK_LOCAL)
 #define drop_cached_metadata(vg)	\
diff -urNp LVM2.2.02.98.orig/tools/lvchange.c LVM2.2.02.98/tools/lvchange.c
--- LVM2.2.02.98.orig/tools/lvchange.c	2012-12-28 13:20:07.226156912 +0000
+++ LVM2.2.02.98/tools/lvchange.c	2012-12-15 12:40:02.937164567 +0000
@@ -238,15 +238,29 @@ static int _lvchange_activate(struct cmd
 		if ((activate == CHANGE_AE) ||
 		    lv_is_origin(lv) ||
 		    lv_is_thin_type(lv)) {
-			log_verbose("Activating logical volume \"%s\" "
-				    "exclusively", lv->name);
-			if (!activate_lv_excl(cmd, lv))
-				return_0;
+			if (arg_count(cmd, force_ARG)) {
+				log_verbose("Activating logical volume \"%s\" "
+					    "exclusively (forced)", lv->name);
+				if (!activate_lv_excl_force(cmd, lv))
+					return_0;
+			} else {
+				log_verbose("Activating logical volume \"%s\" "
+					    "exclusively", lv->name);
+				if (!activate_lv_excl(cmd, lv))
+					return_0;
+			}
 		} else if (activate == CHANGE_ALY) {
-			log_verbose("Activating logical volume \"%s\" locally",
-				    lv->name);
-			if (!activate_lv_local(cmd, lv))
-				return_0;
+			if (arg_count(cmd, force_ARG)) {
+				log_verbose("Activating logical volume \"%s\" locally (forced)",
+					    lv->name);
+				if (!activate_lv_local_force(cmd, lv))
+					return_0;
+			} else {
+				log_verbose("Activating logical volume \"%s\" locally",
+					    lv->name);
+				if (!activate_lv_local(cmd, lv))
+					return_0;
+			}
 		} else {
 			log_verbose("Activating logical volume \"%s\"",
 				    lv->name);