Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Andreas Pflug]

Hi Vladislav,

I finally succeeded making all working as expected!

What I did:
- compile corosync 2.3.0 
(http://corosync.org/download/corosync-2.3.0.tar.gz)
- compile libqb 0.14.4 
(https://fedorahosted.org/releases/q/u/quarterback/libqb-0.14.4.tar.gz)
- compile dlm_4.0.1 
(http://git.fedorahosted.org/cgit/dlm.git/snapshot/dlm-4.0.1.tar.gz)
- Compiled LVM 2.2.98 
(http://git.fedorahosted.org/cgit/lvm2.git/snapshot/lvm2-2_02_98.tar.gz)
    with your PREVIOUS patch ( 
https://www.redhat.com/archives/linux-lvm/2013-January/msg00006.html )

I made a 2-node test cluster, both accessing a test iscsi target, all 
daemons load fine.
Everything works as expected! Snapshotting is possible after -aey 
--force explicit exclusive locking, lvresize works as well.

Previously, I tried your latest patch. I had a plethiora of errors, 
starting with lvcreate failing:

#metadata/lv_manip.c:4137     Clearing start of logical volume "testvol2"
#device/dev-cache.c:599       /dev/san2/testvol2: stat failed: No such 
file or directory
#metadata/lv_manip.c:4140   /dev/san2/testvol2: not found: device not 
cleared
#metadata/lv_manip.c:4620   Aborting. Failed to wipe start of new LV.

I need to add -Z n to have it created. After that, most activations 
failed, with error code 5. I  went back to vanilla 2.2.98, all ok except 
exclusive locking, applied the initial locking patch and voila! All done.

Thanks for your support!
Andreas

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Vladislav Bogdanov]

13.03.2013 18:14, Andreas Pflug wrote:
> Hi Vladislav,
> 
> I finally succeeded making all working as expected!
> 
> What I did:
> - compile corosync 2.3.0
> (http://corosync.org/download/corosync-2.3.0.tar.gz)
> - compile libqb 0.14.4
> (https://fedorahosted.org/releases/q/u/quarterback/libqb-0.14.4.tar.gz)
> - compile dlm_4.0.1
> (http://git.fedorahosted.org/cgit/dlm.git/snapshot/dlm-4.0.1.tar.gz)
> - Compiled LVM 2.2.98
> (http://git.fedorahosted.org/cgit/lvm2.git/snapshot/lvm2-2_02_98.tar.gz)
>    with your PREVIOUS patch (
> https://www.redhat.com/archives/linux-lvm/2013-January/msg00006.html )
> 
> I made a 2-node test cluster, both accessing a test iscsi target, all
> daemons load fine.
> Everything works as expected! Snapshotting is possible after -aey
> --force explicit exclusive locking, lvresize works as well.
> 
> Previously, I tried your latest patch. I had a plethiora of errors,
> starting with lvcreate failing:
> 
> #metadata/lv_manip.c:4137     Clearing start of logical volume "testvol2"
> #device/dev-cache.c:599       /dev/san2/testvol2: stat failed: No such
> file or directory
> #metadata/lv_manip.c:4140   /dev/san2/testvol2: not found: device not
> cleared
> #metadata/lv_manip.c:4620   Aborting. Failed to wipe start of new LV.

Yep, I know about this, I reworked that patch a lot, and now it does
every thing I wanted without regressions.
I will try to post updated version in the next few days when I manage to
apply it to a git tree.

> 
> I need to add -Z n to have it created. After that, most activations
> failed, with error code 5. I  went back to vanilla 2.2.98, all ok except
> exclusive locking, applied the initial locking patch and voila! All done.
> 
> Thanks for your support!
> Andreas
> 
> 

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Andreas Pflug]

Am 13.03.13 17:53, schrieb Vladislav Bogdanov:
>
> #metadata/lv_manip.c:4137     Clearing start of logical volume "testvol2"
> #device/dev-cache.c:599       /dev/san2/testvol2: stat failed: No such
> file or directory
> #metadata/lv_manip.c:4140   /dev/san2/testvol2: not found: device not
> cleared
> #metadata/lv_manip.c:4620   Aborting. Failed to wipe start of new LV.
> Yep, I know about this, I reworked that patch a lot, and now it does
> every thing I wanted without regressions.
> I will try to post updated version in the next few days when I manage to
> apply it to a git tree.

Is there a way to find out if a LV is locked exclusively? lvs displaying 
-e-- instead of -a-- would be nice. Seems not even lvdisplay knows about 
exclusive locking.

Regards,
Andreas

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Vladislav Bogdanov]

13.03.2013 20:37, Andreas Pflug wrote:
> Am 13.03.13 17:53, schrieb Vladislav Bogdanov:
>>
>> #metadata/lv_manip.c:4137     Clearing start of logical volume "testvol2"
>> #device/dev-cache.c:599       /dev/san2/testvol2: stat failed: No such
>> file or directory
>> #metadata/lv_manip.c:4140   /dev/san2/testvol2: not found: device not
>> cleared
>> #metadata/lv_manip.c:4620   Aborting. Failed to wipe start of new LV.
>> Yep, I know about this, I reworked that patch a lot, and now it does
>> every thing I wanted without regressions.
>> I will try to post updated version in the next few days when I manage to
>> apply it to a git tree.
> 
> Is there a way to find out if a LV is locked exclusively? lvs displaying
> -e-- instead of -a-- would be nice. Seems not even lvdisplay knows about
> exclusive locking.

That would break other tools which rely on their output. F.e. cluster
resource agents of libvirt (yes, it runs lvm tools rather then using
API, which is not yet complete btw). As I also need to obtain this
information, I think about writing simple tool (f.e. clvm_tool) which
would display needed info.

As a workaround you can run lvchange -aly without force parameter. If it
succeeds, the volume is locked in a shared mode, otherwise it is locked
exclusively.

> 
> Regards,
> Andreas

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Andreas Pflug]

On 03/13/13 19:30, Vladislav Bogdanov wrote:
>
>> Is there a way to find out if a LV is locked exclusively? lvs displaying
>> -e-- instead of -a-- would be nice. Seems not even lvdisplay knows about
>> exclusive locking.
> That would break other tools which rely on their output. F.e. cluster
> resource agents of libvirt (yes, it runs lvm tools rather then using
> API, which is not yet complete btw). As I also need to obtain this
> information, I think about writing simple tool (f.e. clvm_tool) which
> would display needed info.
>
> As a workaround you can run lvchange -aly without force parameter. If it
> succeeds, the volume is locked in a shared mode, otherwise it is locked
> exclusively.

Hm, thats one ugly workaround...
How about a clvmd option, something like -l to list all locks and exit.

Regards,
Andreas

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Zdenek Kabelac]

Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
> On 03/13/13 19:30, Vladislav Bogdanov wrote:
>>
>>> Is there a way to find out if a LV is locked exclusively? lvs displaying
>>> -e-- instead of -a-- would be nice. Seems not even lvdisplay knows about
>>> exclusive locking.
>> That would break other tools which rely on their output. F.e. cluster
>> resource agents of libvirt (yes, it runs lvm tools rather then using
>> API, which is not yet complete btw). As I also need to obtain this
>> information, I think about writing simple tool (f.e. clvm_tool) which
>> would display needed info.
>>
>> As a workaround you can run lvchange -aly without force parameter. If it
>> succeeds, the volume is locked in a shared mode, otherwise it is locked
>> exclusively.
>
> Hm, thats one ugly workaround...
> How about a clvmd option, something like -l to list all locks and exit.
>


I think - the extension to  'lvs' command could be relatively simple
(adding a new column)

You may query  for  exclusive/local activation on the node.
(So you cannot just tell on which other node is the device active,
but you could print about these states:

active exclusive local
active exclusive
active local
active

Zdenek

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Vladislav Bogdanov]

15.03.2013 12:00, Zdenek Kabelac wrote:
> Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
>> On 03/13/13 19:30, Vladislav Bogdanov wrote:
>>>
>>>> Is there a way to find out if a LV is locked exclusively? lvs
>>>> displaying
>>>> -e-- instead of -a-- would be nice. Seems not even lvdisplay knows
>>>> about
>>>> exclusive locking.
>>> That would break other tools which rely on their output. F.e. cluster
>>> resource agents of libvirt (yes, it runs lvm tools rather then using
>>> API, which is not yet complete btw). As I also need to obtain this
>>> information, I think about writing simple tool (f.e. clvm_tool) which
>>> would display needed info.
>>>
>>> As a workaround you can run lvchange -aly without force parameter. If it
>>> succeeds, the volume is locked in a shared mode, otherwise it is locked
>>> exclusively.
>>
>> Hm, thats one ugly workaround...
>> How about a clvmd option, something like -l to list all locks and exit.
>>
> 
> 
> I think - the extension to  'lvs' command could be relatively simple
> (adding a new column)

Yes, that's correct.

> 
> You may query  for  exclusive/local activation on the node.
> (So you cannot just tell on which other node is the device active,
> but you could print about these states:
> 
> active exclusive local
> active exclusive
> active local
> active

You also may poll all know nodes, but that is a hack too.

That's why I prefer to have this as a separate tool (with dlm_tool-like
params and output) which lists node IDs and lock mode. Unfortunately do
not have power to write it now.

Are core LVM devels interested in these two features: lock conversion
and managing remote node locks? If yes, then I can (hopefully) prepare
git patches next week.

Vladislav

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Zdenek Kabelac]

Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
> 15.03.2013 12:00, Zdenek Kabelac wrote:
>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
>>> On 03/13/13 19:30, Vladislav Bogdanov wrote:
>>>>
>>>>> Is there a way to find out if a LV is locked exclusively? lvs
>>>>> displaying
>>>>> -e-- instead of -a-- would be nice. Seems not even lvdisplay knows
>>>>> about
>>>>> exclusive locking.
>>>> That would break other tools which rely on their output. F.e. cluster
>>>> resource agents of libvirt (yes, it runs lvm tools rather then using
>>>> API, which is not yet complete btw). As I also need to obtain this
>>>> information, I think about writing simple tool (f.e. clvm_tool) which
>>>> would display needed info.
>>>>
>>>> As a workaround you can run lvchange -aly without force parameter. If it
>>>> succeeds, the volume is locked in a shared mode, otherwise it is locked
>>>> exclusively.
>>>
>>> Hm, thats one ugly workaround...
>>> How about a clvmd option, something like -l to list all locks and exit.
>>>
>>
>>
>> I think - the extension to  'lvs' command could be relatively simple
>> (adding a new column)
>
> Yes, that's correct.
>
>>
>> You may query  for  exclusive/local activation on the node.
>> (So you cannot just tell on which other node is the device active,
>> but you could print about these states:
>>
>> active exclusive local
>> active exclusive
>> active local
>> active
>
> You also may poll all know nodes, but that is a hack too.
>
> That's why I prefer to have this as a separate tool (with dlm_tool-like
> params and output) which lists node IDs and lock mode. Unfortunately do
> not have power to write it now.
>
> Are core LVM devels interested in these two features: lock conversion
> and managing remote node locks? If yes, then I can (hopefully) prepare
> git patches next week.


I'm not quite sure what do you mean by  'managing remote node locks' ?

Current login behind lvm command is  -

You could activate LVs with the above syntax [ael]
(there is a tag support - so you could exclusively activate LV on remote
node in via some configuration tags)

And you want to 'upgrade' remote locks to something else ?

What would be the use-case you could not resolve with current command line args?

Is that supported by dlm (since lvm locks are mapped to dlm)?
How would you resolve error path fallbacks ?
Also I believe the clvmd protocol is out of free bits for extension,
so how the protocol would look like ?

Zdenek

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Vladislav Bogdanov]

15.03.2013 12:37, Zdenek Kabelac wrote:
> Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
>> 15.03.2013 12:00, Zdenek Kabelac wrote:
>>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
>>>> On 03/13/13 19:30, Vladislav Bogdanov wrote:
>>>>>
>>>>>> Is there a way to find out if a LV is locked exclusively? lvs
>>>>>> displaying
>>>>>> -e-- instead of -a-- would be nice. Seems not even lvdisplay knows
>>>>>> about
>>>>>> exclusive locking.
>>>>> That would break other tools which rely on their output. F.e. cluster
>>>>> resource agents of libvirt (yes, it runs lvm tools rather then using
>>>>> API, which is not yet complete btw). As I also need to obtain this
>>>>> information, I think about writing simple tool (f.e. clvm_tool) which
>>>>> would display needed info.
>>>>>
>>>>> As a workaround you can run lvchange -aly without force parameter.
>>>>> If it
>>>>> succeeds, the volume is locked in a shared mode, otherwise it is
>>>>> locked
>>>>> exclusively.
>>>>
>>>> Hm, thats one ugly workaround...
>>>> How about a clvmd option, something like -l to list all locks and exit.
>>>>
>>>
>>>
>>> I think - the extension to  'lvs' command could be relatively simple
>>> (adding a new column)
>>
>> Yes, that's correct.
>>
>>>
>>> You may query  for  exclusive/local activation on the node.
>>> (So you cannot just tell on which other node is the device active,
>>> but you could print about these states:
>>>
>>> active exclusive local
>>> active exclusive
>>> active local
>>> active
>>
>> You also may poll all know nodes, but that is a hack too.
>>
>> That's why I prefer to have this as a separate tool (with dlm_tool-like
>> params and output) which lists node IDs and lock mode. Unfortunately do
>> not have power to write it now.
>>
>> Are core LVM devels interested in these two features: lock conversion
>> and managing remote node locks? If yes, then I can (hopefully) prepare
>> git patches next week.
> 
> 
> I'm not quite sure what do you mean by  'managing remote node locks' ?

Activation/deactivation of LVs on a different corosync cluster node,
specified by its node name (with pacemaker-like method to determine that
name).
Also conversion of locks on that node.

> 
> Current login behind lvm command is  -
> 
> You could activate LVs with the above syntax [ael]
> (there is a tag support - so you could exclusively activate LV on remote
> node in via some configuration tags)

Could you please explain this - I do not see anything relevant in man pages.

> 
> And you want to 'upgrade' remote locks to something else ?

Yes, shared-to-exclusive end vice verse.

> 
> What would be the use-case you could not resolve with current command
> line args?

I need to convert lock on a remote node during last stage of ver3
migration in libvirt/qemu. That is a "confirm" stage, which runs on a
"source" node, during which "old" vm is killed and disk is released.
So, I first ("begin" stage) convert lock from exclusive to shared on a
source node, then obtain shared lock on a target node (during "prepare"
stage, when receiving qemu instance is started), then migrate vm between
two processes which have LV opened, and then release shared lock on a
source node ("confirm" stage, after source qemu is killed).

There is no other events on a destination node in ver3 migration
protocol, so I'm unable to convert lock to exclusive there after
migration is finished. So I do that from a source node, after it
released lock.

> 
> Is that supported by dlm (since lvm locks are mapped to dlm)?
Command just sent to a specific clvm instance and performed there.

> How would you resolve error path fallbacks ?

Could you please tell  what exactly do you mean?
If dlm on a remote node is unable to perform requested operation, then
error is returned to a initiator.

> Also I believe the clvmd protocol is out of free bits for extension,
> so how the protocol would look like ?
It contains 'node' field (I assume it was never actually used before)
and with some fixes that works.

Vladislav

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Vladislav Bogdanov]

15.03.2013 15:53, Vladislav Bogdanov wrote:
> 15.03.2013 12:37, Zdenek Kabelac wrote:
>> Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
>>> 15.03.2013 12:00, Zdenek Kabelac wrote:
>>>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
>>>>> On 03/13/13 19:30, Vladislav Bogdanov wrote:
>>>>>>
>>>>>>> Is there a way to find out if a LV is locked exclusively? lvs
>>>>>>> displaying
>>>>>>> -e-- instead of -a-- would be nice. Seems not even lvdisplay knows
>>>>>>> about
>>>>>>> exclusive locking.
>>>>>> That would break other tools which rely on their output. F.e. cluster
>>>>>> resource agents of libvirt (yes, it runs lvm tools rather then using
>>>>>> API, which is not yet complete btw). As I also need to obtain this
>>>>>> information, I think about writing simple tool (f.e. clvm_tool) which
>>>>>> would display needed info.
>>>>>>
>>>>>> As a workaround you can run lvchange -aly without force parameter.
>>>>>> If it
>>>>>> succeeds, the volume is locked in a shared mode, otherwise it is
>>>>>> locked
>>>>>> exclusively.
>>>>>
>>>>> Hm, thats one ugly workaround...
>>>>> How about a clvmd option, something like -l to list all locks and exit.
>>>>>
>>>>
>>>>
>>>> I think - the extension to  'lvs' command could be relatively simple
>>>> (adding a new column)
>>>
>>> Yes, that's correct.
>>>
>>>>
>>>> You may query  for  exclusive/local activation on the node.
>>>> (So you cannot just tell on which other node is the device active,
>>>> but you could print about these states:
>>>>
>>>> active exclusive local
>>>> active exclusive
>>>> active local
>>>> active
>>>
>>> You also may poll all know nodes, but that is a hack too.
>>>
>>> That's why I prefer to have this as a separate tool (with dlm_tool-like
>>> params and output) which lists node IDs and lock mode. Unfortunately do
>>> not have power to write it now.
>>>
>>> Are core LVM devels interested in these two features: lock conversion
>>> and managing remote node locks? If yes, then I can (hopefully) prepare
>>> git patches next week.
>>
>>
>> I'm not quite sure what do you mean by  'managing remote node locks' ?
> 
> Activation/deactivation of LVs on a different corosync cluster node,
> specified by its node name (with pacemaker-like method to determine that
> name).
> Also conversion of locks on that node.
> 
>>
>> Current login behind lvm command is  -
>>
>> You could activate LVs with the above syntax [ael]
>> (there is a tag support - so you could exclusively activate LV on remote
>> node in via some configuration tags)
> 
> Could you please explain this - I do not see anything relevant in man pages.
> 
>>
>> And you want to 'upgrade' remote locks to something else ?
> 
> Yes, shared-to-exclusive end vice verse.
> 
>>
>> What would be the use-case you could not resolve with current command
>> line args?
> 
> I need to convert lock on a remote node during last stage of ver3
> migration in libvirt/qemu. That is a "confirm" stage, which runs on a
> "source" node, during which "old" vm is killed and disk is released.
> So, I first ("begin" stage) convert lock from exclusive to shared on a
> source node, then obtain shared lock on a target node (during "prepare"
> stage, when receiving qemu instance is started), then migrate vm between
> two processes which have LV opened, and then release shared lock on a
> source node ("confirm" stage, after source qemu is killed).
> 
> There is no other events on a destination node in ver3 migration
> protocol, so I'm unable to convert lock to exclusive there after
> migration is finished. So I do that from a source node, after it
> released lock.
> 
>>
>> Is that supported by dlm (since lvm locks are mapped to dlm)?
> Command just sent to a specific clvm instance and performed there.
> 
>> How would you resolve error path fallbacks ?
> 
> Could you please tell  what exactly do you mean?
> If dlm on a remote node is unable to perform requested operation, then
> error is returned to a initiator.
> 
>> Also I believe the clvmd protocol is out of free bits for extension,
>> so how the protocol would look like ?
> It contains 'node' field (I assume it was never actually used before)
> and with some fixes that works.

Fix to myself. clvm client API has that field. And there is no need for
changes in on-wire protocol - corosync module already allows to send a
message to a specific csid (node).

> 
> Vladislav
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Zdenek Kabelac]

Dne 15.3.2013 13:53, Vladislav Bogdanov napsal(a):
> 15.03.2013 12:37, Zdenek Kabelac wrote:
>> Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
>>> 15.03.2013 12:00, Zdenek Kabelac wrote:
>>>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
>>>>> On 03/13/13 19:30, Vladislav Bogdanov wrote:
>>>>>>
>>> You could activate LVs with the above syntax [ael]
>> (there is a tag support - so you could exclusively activate LV on remote
>> node in via some configuration tags)
>
> Could you please explain this - I do not see anything relevant in man pages.

Let's say - you have 3 nodes  A, B, C - each have a TAG_A, TAG_B, TAG_C,
then on node A you may exclusively activate LV which has TAG_B - this
will try to exclusively active LV on the node which has it configured
in lvm.conf  (see the  volume_list= [])

>
>>
>> And you want to 'upgrade' remote locks to something else ?
>
> Yes, shared-to-exclusive end vice verse.

So how do you convert the lock from   shared to exclusive without unlock
(if I get it right - you keep the ConcurrentRead lock - and you want to take 
Exlusive -  to  make change state from  'active' to 'active exlusive')
https://en.wikipedia.org/wiki/Distributed_lock_manager

Clvmd 'communicates' via these locks.
So the proper algorithm needs to be there for ending with some proper state 
after locks changes (and sorry I'm not a dlm expert here)


>>
>> What would be the use-case you could not resolve with current command
>> line args?
>
> I need to convert lock on a remote node during last stage of ver3
> migration in libvirt/qemu. That is a "confirm" stage, which runs on a
> "source" node, during which "old" vm is killed and disk is released.
> So, I first ("begin" stage) convert lock from exclusive to shared on a
> source node, then obtain shared lock on a target node (during "prepare"

Which most probably works only your environment - since you do not try to
'break' the logic - but it's probably not a generic concept - i.e.
in this controlled environment you may live probably happily even with
local activation of LVs - since you always know what the tool is doing.

> There is no other events on a destination node in ver3 migration
> protocol, so I'm unable to convert lock to exclusive there after
> migration is finished. So I do that from a source node, after it
> released lock.
>
>>
>> Is that supported by dlm (since lvm locks are mapped to dlm)?
> Command just sent to a specific clvm instance and performed there.

As said - the 'lock' is the thing which controls the activation state.
So faking it on the software level may possible lead to inconsistency between 
the dlm and clvmd view of the lock state.

Zdenek

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Vladislav Bogdanov]

15.03.2013 16:32, Zdenek Kabelac wrote:
> Dne 15.3.2013 13:53, Vladislav Bogdanov napsal(a):
>> 15.03.2013 12:37, Zdenek Kabelac wrote:
>>> Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
>>>> 15.03.2013 12:00, Zdenek Kabelac wrote:
>>>>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
>>>>>> On 03/13/13 19:30, Vladislav Bogdanov wrote:
>>>>>>>
>>>> You could activate LVs with the above syntax [ael]
>>> (there is a tag support - so you could exclusively activate LV on remote
>>> node in via some configuration tags)
>>
>> Could you please explain this - I do not see anything relevant in man
>> pages.
> 
> Let's say - you have 3 nodes  A, B, C - each have a TAG_A, TAG_B, TAG_C,
> then on node A you may exclusively activate LV which has TAG_B - this
> will try to exclusively active LV on the node which has it configured
> in lvm.conf  (see the  volume_list= [])

Aha, if I understand correctly this is absolutely not what I need.
I want all this to be fully dynamic without any "config-editing voodoo".

> 
>>
>>>
>>> And you want to 'upgrade' remote locks to something else ?
>>
>> Yes, shared-to-exclusive end vice verse.
> 
> So how do you convert the lock from   shared to exclusive without unlock
> (if I get it right - you keep the ConcurrentRead lock - and you want to
> take Exlusive -  to  make change state from  'active' to 'active exlusive')
> https://en.wikipedia.org/wiki/Distributed_lock_manager

I just pass LCKF_CONVERT to dlm_controld if requested and needed. And
that is dlm's task to either satisfy conversion or to refuse it.

> 
> Clvmd 'communicates' via these locks.

Not exactly true.

clvmd does cluster communications with corosync, which implements
virtual synchrony, so all cluster nodes receive messages in the same order.
At the bottom, clvmd uses libdlm to communicate with dlm_controld and
request it to lock/unlock.
dlm_controld instances use corosync for membership and locally manages
in-kernel dlm counter-part, which uses tcp/sctp mesh-like connections to
communicate.
So request from one clvmd instance goes to another and goes to kernel
from there, and then it is distributed to other nodes. Actually that
does not matter where does it hits kernel space if it supports
delegation of locks to remote nodes, but I suspect it doesn't. But if it
doesn't support such thing, then the only option to manage lock on a
remote node is to request that's node dlm instance to do the locking job.

> So the proper algorithm needs to be there for ending with some proper
> state after locks changes (and sorry I'm not a dlm expert here)

That is what actually happens.
There is just no difference between running (to upgrade local lock to
exclusive on node <node>.

ssh <node> lvchange -aey --force VG/LV

or

lvchange -aey --node <node> --force VG/LV

The same command, it is just sent via different channels.

Again, I just send locking request to a remote clvmd instance through
corosync.
It asks its local dlm to convert (acquire, release) lock and returns its
answer back. After dlm answers, operation is either performed, and then
OK is send back to a initiator, or refused, and the error is sent back.

> 
> 
>>>
>>> What would be the use-case you could not resolve with current command
>>> line args?
>>
>> I need to convert lock on a remote node during last stage of ver3
>> migration in libvirt/qemu. That is a "confirm" stage, which runs on a
>> "source" node, during which "old" vm is killed and disk is released.
>> So, I first ("begin" stage) convert lock from exclusive to shared on a
>> source node, then obtain shared lock on a target node (during "prepare"
> 
> Which most probably works only your environment - since you do not try to
> 'break' the logic - but it's probably not a generic concept - i.e.
> in this controlled environment you may live probably happily even with
> local activation of LVs - since you always know what the tool is doing.

I prefer to do not allow anybody else to open an LV at all (and to be
able to take snapshots of it when needed on a node where it is
exclusively locked). That's why I implemented a lock-manager driver for
libvirt, which controls inter-node locks with clvm exclusive activation,
and then controls inner open attempts with local locking mechanism.

While libvirt already has two lok drivers, neither of fully satisfy me
because or shared storage requirement. And, I want to take snapshots,
that's why I need exclusive activation.

I plan to send libvirt patches to the libvirt list if lvm part is
accepted (otherwise it makes no sense to do that and I will need to
maintain everything out-of-the-tree).

> 
>> There is no other events on a destination node in ver3 migration
>> protocol, so I'm unable to convert lock to exclusive there after
>> migration is finished. So I do that from a source node, after it
>> released lock.
>>
>>>
>>> Is that supported by dlm (since lvm locks are mapped to dlm)?
>> Command just sent to a specific clvm instance and performed there.
> 
> As said - the 'lock' is the thing which controls the activation state.
> So faking it on the software level may possible lead to inconsistency
> between the dlm and clvmd view of the lock state.

No faking. Just a remote management of the same lock.

Vladislav

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Zdenek Kabelac]

Dne 15.3.2013 15:51, Vladislav Bogdanov napsal(a):
> 15.03.2013 16:32, Zdenek Kabelac wrote:
>> Dne 15.3.2013 13:53, Vladislav Bogdanov napsal(a):
>>> 15.03.2013 12:37, Zdenek Kabelac wrote:
>>>> Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
>>>>> 15.03.2013 12:00, Zdenek Kabelac wrote:
>>>>>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
>>>>>>> On 03/13/13 19:30, Vladislav Bogdanov wrote:
>>>>>>>>
>>>>> You could activate LVs with the above syntax [ael]
>>>> (there is a tag support - so you could exclusively activate LV on remote
>>>> node in via some configuration tags)
>>>
>>> Could you please explain this - I do not see anything relevant in man
>>> pages.
>>
>> Let's say - you have 3 nodes  A, B, C - each have a TAG_A, TAG_B, TAG_C,
>> then on node A you may exclusively activate LV which has TAG_B - this
>> will try to exclusively active LV on the node which has it configured
>> in lvm.conf  (see the  volume_list= [])
>
> Aha, if I understand correctly this is absolutely not what I need.
> I want all this to be fully dynamic without any "config-editing voodoo".
>
>>
>>>
>>>>
>>>> And you want to 'upgrade' remote locks to something else ?
>>>
>>> Yes, shared-to-exclusive end vice verse.
>>
>> So how do you convert the lock from   shared to exclusive without unlock
>> (if I get it right - you keep the ConcurrentRead lock - and you want to
>> take Exlusive -  to  make change state from  'active' to 'active exlusive')
>> https://en.wikipedia.org/wiki/Distributed_lock_manager
>
> I just pass LCKF_CONVERT to dlm_controld if requested and needed. And
> that is dlm's task to either satisfy conversion or to refuse it.
>

So to understand myself better this thing -

the dlm sends 'unlock' requests to all other nodes except the one which
should be converted to exclusive mode and send exclusive lock to the prefered 
node?

>>
>> Clvmd 'communicates' via these locks.
>
> Not exactly true.
>
> clvmd does cluster communications with corosync, which implements
> virtual synchrony, so all cluster nodes receive messages in the same order.
> At the bottom, clvmd uses libdlm to communicate with dlm_controld and
> request it to lock/unlock.
> dlm_controld instances use corosync for membership and locally manages
> in-kernel dlm counter-part, which uses tcp/sctp mesh-like connections to
> communicate.
> So request from one clvmd instance goes to another and goes to kernel
> from there, and then it is distributed to other nodes. Actually that
> does not matter where does it hits kernel space if it supports
> delegation of locks to remote nodes, but I suspect it doesn't. But if it
> doesn't support such thing, then the only option to manage lock on a
> remote node is to request that's node dlm instance to do the locking job.
>
>> So the proper algorithm needs to be there for ending with some proper
>> state after locks changes (and sorry I'm not a dlm expert here)
>
> That is what actually happens.
> There is just no difference between running (to upgrade local lock to
> exclusive on node <node>.
>
> ssh <node> lvchange -aey --force VG/LV
>
> or
>
> lvchange -aey --node <node> --force VG/LV


--node is exactly what the tag is for - each node may have it's tag.
lvm doesn't work with cluster nodes.

The question is - could be the code transformed to use this logic ?
I guess you need to know  dlm node name here right ?


> The same command, it is just sent via different channels.
>
> Again, I just send locking request to a remote clvmd instance through
> corosync.
> It asks its local dlm to convert (acquire, release) lock and returns its
> answer back. After dlm answers, operation is either performed, and then
> OK is send back to a initiator, or refused, and the error is sent back.


>>> There is no other events on a destination node in ver3 migration
>>> protocol, so I'm unable to convert lock to exclusive there after
>>> migration is finished. So I do that from a source node, after it
>>> released lock.
>>>
>>>>
>>>> Is that supported by dlm (since lvm locks are mapped to dlm)?
>>> Command just sent to a specific clvm instance and performed there.
>>
>> As said - the 'lock' is the thing which controls the activation state.
>> So faking it on the software level may possible lead to inconsistency
>> between the dlm and clvmd view of the lock state.
>
> No faking. Just a remote management of the same lock.

Could you repost patches against git ?
With some usage examples ?

Zdenek

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Vladislav Bogdanov]

15.03.2013 18:02, Zdenek Kabelac wrote:
> Dne 15.3.2013 15:51, Vladislav Bogdanov napsal(a):
>> 15.03.2013 16:32, Zdenek Kabelac wrote:
>>> Dne 15.3.2013 13:53, Vladislav Bogdanov napsal(a):
>>>> 15.03.2013 12:37, Zdenek Kabelac wrote:
>>>>> Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
>>>>>> 15.03.2013 12:00, Zdenek Kabelac wrote:
>>>>>>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
>>>>>>>> On 03/13/13 19:30, Vladislav Bogdanov wrote:
>>>>>>>>>
>>>>>> You could activate LVs with the above syntax [ael]
>>>>> (there is a tag support - so you could exclusively activate LV on
>>>>> remote
>>>>> node in via some configuration tags)
>>>>
>>>> Could you please explain this - I do not see anything relevant in man
>>>> pages.
>>>
>>> Let's say - you have 3 nodes  A, B, C - each have a TAG_A, TAG_B, TAG_C,
>>> then on node A you may exclusively activate LV which has TAG_B - this
>>> will try to exclusively active LV on the node which has it configured
>>> in lvm.conf  (see the  volume_list= [])
>>
>> Aha, if I understand correctly this is absolutely not what I need.
>> I want all this to be fully dynamic without any "config-editing voodoo".
>>
>>>
>>>>
>>>>>
>>>>> And you want to 'upgrade' remote locks to something else ?
>>>>
>>>> Yes, shared-to-exclusive end vice verse.
>>>
>>> So how do you convert the lock from   shared to exclusive without unlock
>>> (if I get it right - you keep the ConcurrentRead lock - and you want to
>>> take Exlusive -  to  make change state from  'active' to 'active
>>> exlusive')
>>> https://en.wikipedia.org/wiki/Distributed_lock_manager
>>
>> I just pass LCKF_CONVERT to dlm_controld if requested and needed. And
>> that is dlm's task to either satisfy conversion or to refuse it.
>>
> 
> So to understand myself better this thing -
> 
> the dlm sends 'unlock' requests to all other nodes except the one which
> should be converted to exclusive mode and send exclusive lock to the
> prefered node?

No.
clvmd sends request to a remote clvmd to upgrade or acquire or release
the lock.
That remote instance asks local dlm to do the job. dlm either says OK or
says ERROR.
It does not do anything except that.
It LV is locked on a remote node, be it shared or exclusive lock, dlm
says ERROR if exclusive lock (or conversion to it) is requested.

My patches also allow "-an --force" to release shared locks on other
nodes. Exclusive lock may be released or downgraded only on node which
holds it (or with --node <node>).

> 
>>>
>>> Clvmd 'communicates' via these locks.
>>
>> Not exactly true.
>>
>> clvmd does cluster communications with corosync, which implements
>> virtual synchrony, so all cluster nodes receive messages in the same
>> order.
>> At the bottom, clvmd uses libdlm to communicate with dlm_controld and
>> request it to lock/unlock.
>> dlm_controld instances use corosync for membership and locally manages
>> in-kernel dlm counter-part, which uses tcp/sctp mesh-like connections to
>> communicate.
>> So request from one clvmd instance goes to another and goes to kernel
>> from there, and then it is distributed to other nodes. Actually that
>> does not matter where does it hits kernel space if it supports
>> delegation of locks to remote nodes, but I suspect it doesn't. But if it
>> doesn't support such thing, then the only option to manage lock on a
>> remote node is to request that's node dlm instance to do the locking job.
>>
>>> So the proper algorithm needs to be there for ending with some proper
>>> state after locks changes (and sorry I'm not a dlm expert here)
>>
>> That is what actually happens.
>> There is just no difference between running (to upgrade local lock to
>> exclusive on node <node>.
>>
>> ssh <node> lvchange -aey --force VG/LV
>>
>> or
>>
>> lvchange -aey --node <node> --force VG/LV
> 
> 
> --node is exactly what the tag is for - each node may have it's tag.
> lvm doesn't work with cluster nodes.

But corosync and dlm operate node IDs, and pacemaker operates node names
and IDs. None of them use tags.

> 
> The question is - could be the code transformed to use this logic ?
> I guess you need to know  dlm node name here right ?

Node IDs are obtained from corosync membership list, and may be used for
that. If corosync is configured with nodelist in a way pacemaker wants
it
(http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-name.html),
then node names may be used too.

> 
> 
>> The same command, it is just sent via different channels.
>>
>> Again, I just send locking request to a remote clvmd instance through
>> corosync.
>> It asks its local dlm to convert (acquire, release) lock and returns its
>> answer back. After dlm answers, operation is either performed, and then
>> OK is send back to a initiator, or refused, and the error is sent back.
> 
> 
>>>> There is no other events on a destination node in ver3 migration
>>>> protocol, so I'm unable to convert lock to exclusive there after
>>>> migration is finished. So I do that from a source node, after it
>>>> released lock.
>>>>
>>>>>
>>>>> Is that supported by dlm (since lvm locks are mapped to dlm)?
>>>> Command just sent to a specific clvm instance and performed there.
>>>
>>> As said - the 'lock' is the thing which controls the activation state.
>>> So faking it on the software level may possible lead to inconsistency
>>> between the dlm and clvmd view of the lock state.
>>
>> No faking. Just a remote management of the same lock.
> 
> Could you repost patches against git ?

I plan that for the next week.

> With some usage examples ?

Yes, if you give me an "example of example" ;)

Vladislav

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Zdenek Kabelac]

Dne 15.3.2013 16:36, Vladislav Bogdanov napsal(a):
> 15.03.2013 18:02, Zdenek Kabelac wrote:
>> Dne 15.3.2013 15:51, Vladislav Bogdanov napsal(a):
>>> 15.03.2013 16:32, Zdenek Kabelac wrote:
>>>> Dne 15.3.2013 13:53, Vladislav Bogdanov napsal(a):
>>>>> 15.03.2013 12:37, Zdenek Kabelac wrote:
>>>>>> Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
>>>>>>> 15.03.2013 12:00, Zdenek Kabelac wrote:
>>>>>>>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
>>>>>>>>> On 03/13/13 19:30, Vladislav Bogdanov wrote:
>>>>>>>>>>
>>>>>>> You could activate LVs with the above syntax [ael]
>>>>>> (there is a tag support - so you could exclusively activate LV on
>>>>>> remote
>>>>>> node in via some configuration tags)
>>>>>
>>>>> Could you please explain this - I do not see anything relevant in man
>>>>> pages.
>>>>
>>>> Let's say - you have 3 nodes  A, B, C - each have a TAG_A, TAG_B, TAG_C,
>>>> then on node A you may exclusively activate LV which has TAG_B - this
>>>> will try to exclusively active LV on the node which has it configured
>>>> in lvm.conf  (see the  volume_list= [])
>>>
>>> Aha, if I understand correctly this is absolutely not what I need.
>>> I want all this to be fully dynamic without any "config-editing voodoo".
>>>
>>>>
>>>>>
>>>>>>
>>>>>> And you want to 'upgrade' remote locks to something else ?
>>>>>
>>>>> Yes, shared-to-exclusive end vice verse.
>>>>
>>>> So how do you convert the lock from   shared to exclusive without unlock
>>>> (if I get it right - you keep the ConcurrentRead lock - and you want to
>>>> take Exlusive -  to  make change state from  'active' to 'active
>>>> exlusive')
>>>> https://en.wikipedia.org/wiki/Distributed_lock_manager
>>>
>>> I just pass LCKF_CONVERT to dlm_controld if requested and needed. And
>>> that is dlm's task to either satisfy conversion or to refuse it.
>>>
>>
>> So to understand myself better this thing -
>>
>> the dlm sends 'unlock' requests to all other nodes except the one which
>> should be converted to exclusive mode and send exclusive lock to the
>> prefered node?
>
> No.
> clvmd sends request to a remote clvmd to upgrade or acquire or release
> the lock.
> That remote instance asks local dlm to do the job. dlm either says OK or
> says ERROR.
> It does not do anything except that.
> It LV is locked on a remote node, be it shared or exclusive lock, dlm
> says ERROR if exclusive lock (or conversion to it) is requested.
>
> My patches also allow "-an --force" to release shared locks on other
> nodes. Exclusive lock may be released or downgraded only on node which
> holds it (or with --node <node>).
>
>>
>>>>
>>>> Clvmd 'communicates' via these locks.
>>>
>>> Not exactly true.
>>>
>>> clvmd does cluster communications with corosync, which implements
>>> virtual synchrony, so all cluster nodes receive messages in the same
>>> order.
>>> At the bottom, clvmd uses libdlm to communicate with dlm_controld and
>>> request it to lock/unlock.
>>> dlm_controld instances use corosync for membership and locally manages
>>> in-kernel dlm counter-part, which uses tcp/sctp mesh-like connections to
>>> communicate.
>>> So request from one clvmd instance goes to another and goes to kernel
>>> from there, and then it is distributed to other nodes. Actually that
>>> does not matter where does it hits kernel space if it supports
>>> delegation of locks to remote nodes, but I suspect it doesn't. But if it
>>> doesn't support such thing, then the only option to manage lock on a
>>> remote node is to request that's node dlm instance to do the locking job.
>>>
>>>> So the proper algorithm needs to be there for ending with some proper
>>>> state after locks changes (and sorry I'm not a dlm expert here)
>>>
>>> That is what actually happens.
>>> There is just no difference between running (to upgrade local lock to
>>> exclusive on node <node>.
>>>
>>> ssh <node> lvchange -aey --force VG/LV
>>>
>>> or
>>>
>>> lvchange -aey --node <node> --force VG/LV
>>
>>
>> --node is exactly what the tag is for - each node may have it's tag.
>> lvm doesn't work with cluster nodes.
>
> But corosync and dlm operate node IDs, and pacemaker operates node names
> and IDs. None of them use tags.
>
>>
>> The question is - could be the code transformed to use this logic ?
>> I guess you need to know  dlm node name here right ?
>
> Node IDs are obtained from corosync membership list, and may be used for
> that. If corosync is configured with nodelist in a way pacemaker wants
> it
> (http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-name.html),
> then node names may be used too.

clvmd knows the dlm node name - but lvm command should reference things via 
tags. There will be probably no way to add '--node' option into lvm command.

Can you think about using 'tags'?

So your machines will have configured tags (be it machine name) and instead of 
the node - you would use 'tag' ?

Zdenek

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[David Teigland]

> I need to convert lock on a remote node during last stage of ver3
> migration in libvirt/qemu

Hi, I wrote and maintain the dlm and have more recently written a new
disk-based lock manager called sanlock, https://fedorahosted.org/sanlock/
which operates with only shared storage among nodes (no networking or
other cluster manager.)

sanlock was written to allow RHEV to manage leases for vm images on shared
storage, including the ability to migrate leases among hosts (which is the
most complicated part, as you've mentioned above.)  sanlock plugs into the
libvirt locking api, which also supports file locks (usable on local or
nfs file systems.)  (Search for "virtlockd".)

Trying to use/control vm locks via lvm commands is not a good way to solve
the vm management/migration problem at the present time (but see below).
Instead, I'd suggest doing the locking via the libvirt locking api which
was designed for this purpose.  As I mentioned, libvirt supports both
sanlock and file locks, but another option is to write a new libvirt
locking plug-in for dlm/corosync.  This would be the best way to use dlm
locks to protect vm images on shared storage; I've been hoping someone
would do this for some time.

Incidentally, I've recently started a new project which is to replace
clvmd with a new "lvmlockd".  I'm designing lvmlockd to support both dlm
and sanlock on the back side (transparent to lvm itself).  With sanlock,
you will not need a dlm/corosync cluster to have the benefits of locking
vgs and lvs on shared storage.  This project is requiring a lot of
preliminary work in the lvm code, because the clvmd approach reaches
deeply into lvm itself.  Relating back to virt environments, lvmlockd will
give you direct control of the lock types, modes and objects in each
command.  This will hopefully make it much easier to use lvm locks in a
controlled and programatic way to solve problems like vm management.

So, in preparation for lvmlockd, you should not assume that lvm commands
will operate in a dlm/corosync environment.  Any new options or
capabilities should be considered more generally.  Also, the concept of
lvm commands executing remote operations in a cluster was a poor design
choice for clvm.  lvmlockd will probably not support this notion.
Executing remote commands should be done at a higher layer.

Dave

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Vladislav Bogdanov]

15.03.2013 18:55, Zdenek Kabelac пишет:
> Dne 15.3.2013 16:36, Vladislav Bogdanov napsal(a):
>> 15.03.2013 18:02, Zdenek Kabelac wrote:
>>> Dne 15.3.2013 15:51, Vladislav Bogdanov napsal(a):
>>>> 15.03.2013 16:32, Zdenek Kabelac wrote:
>>>>> Dne 15.3.2013 13:53, Vladislav Bogdanov napsal(a):
>>>>>> 15.03.2013 12:37, Zdenek Kabelac wrote:
>>>>>>> Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
>>>>>>>> 15.03.2013 12:00, Zdenek Kabelac wrote:
>>>>>>>>> Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
>>>>>>>>>> On 03/13/13 19:30, Vladislav Bogdanov wrote:
>>>>>>>>>>>
>>>>>>>> You could activate LVs with the above syntax [ael]
>>>>>>> (there is a tag support - so you could exclusively activate LV on
>>>>>>> remote
>>>>>>> node in via some configuration tags)
>>>>>>
>>>>>> Could you please explain this - I do not see anything relevant in man
>>>>>> pages.
>>>>>
>>>>> Let's say - you have 3 nodes  A, B, C - each have a TAG_A, TAG_B,
>>>>> TAG_C,
>>>>> then on node A you may exclusively activate LV which has TAG_B - this
>>>>> will try to exclusively active LV on the node which has it configured
>>>>> in lvm.conf  (see the  volume_list= [])
>>>>
>>>> Aha, if I understand correctly this is absolutely not what I need.
>>>> I want all this to be fully dynamic without any "config-editing
>>>> voodoo".
>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>> And you want to 'upgrade' remote locks to something else ?
>>>>>>
>>>>>> Yes, shared-to-exclusive end vice verse.
>>>>>
>>>>> So how do you convert the lock from   shared to exclusive without
>>>>> unlock
>>>>> (if I get it right - you keep the ConcurrentRead lock - and you
>>>>> want to
>>>>> take Exlusive -  to  make change state from  'active' to 'active
>>>>> exlusive')
>>>>> https://en.wikipedia.org/wiki/Distributed_lock_manager
>>>>
>>>> I just pass LCKF_CONVERT to dlm_controld if requested and needed. And
>>>> that is dlm's task to either satisfy conversion or to refuse it.
>>>>
>>>
>>> So to understand myself better this thing -
>>>
>>> the dlm sends 'unlock' requests to all other nodes except the one which
>>> should be converted to exclusive mode and send exclusive lock to the
>>> prefered node?
>>
>> No.
>> clvmd sends request to a remote clvmd to upgrade or acquire or release
>> the lock.
>> That remote instance asks local dlm to do the job. dlm either says OK or
>> says ERROR.
>> It does not do anything except that.
>> It LV is locked on a remote node, be it shared or exclusive lock, dlm
>> says ERROR if exclusive lock (or conversion to it) is requested.
>>
>> My patches also allow "-an --force" to release shared locks on other
>> nodes. Exclusive lock may be released or downgraded only on node which
>> holds it (or with --node <node>).
>>
>>>
>>>>>
>>>>> Clvmd 'communicates' via these locks.
>>>>
>>>> Not exactly true.
>>>>
>>>> clvmd does cluster communications with corosync, which implements
>>>> virtual synchrony, so all cluster nodes receive messages in the same
>>>> order.
>>>> At the bottom, clvmd uses libdlm to communicate with dlm_controld and
>>>> request it to lock/unlock.
>>>> dlm_controld instances use corosync for membership and locally manages
>>>> in-kernel dlm counter-part, which uses tcp/sctp mesh-like
>>>> connections to
>>>> communicate.
>>>> So request from one clvmd instance goes to another and goes to kernel
>>>> from there, and then it is distributed to other nodes. Actually that
>>>> does not matter where does it hits kernel space if it supports
>>>> delegation of locks to remote nodes, but I suspect it doesn't. But
>>>> if it
>>>> doesn't support such thing, then the only option to manage lock on a
>>>> remote node is to request that's node dlm instance to do the locking
>>>> job.
>>>>
>>>>> So the proper algorithm needs to be there for ending with some proper
>>>>> state after locks changes (and sorry I'm not a dlm expert here)
>>>>
>>>> That is what actually happens.
>>>> There is just no difference between running (to upgrade local lock to
>>>> exclusive on node <node>.
>>>>
>>>> ssh <node> lvchange -aey --force VG/LV
>>>>
>>>> or
>>>>
>>>> lvchange -aey --node <node> --force VG/LV
>>>
>>>
>>> --node is exactly what the tag is for - each node may have it's tag.
>>> lvm doesn't work with cluster nodes.
>>
>> But corosync and dlm operate node IDs, and pacemaker operates node names
>> and IDs. None of them use tags.
>>
>>>
>>> The question is - could be the code transformed to use this logic ?
>>> I guess you need to know  dlm node name here right ?
>>
>> Node IDs are obtained from corosync membership list, and may be used for
>> that. If corosync is configured with nodelist in a way pacemaker wants
>> it
>> (http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-name.html),
>>
>> then node names may be used too.
> 
> clvmd knows the dlm node name - but lvm command should reference things
> via tags. There will be probably no way to add '--node' option into lvm
> command.
> 
> Can you think about using 'tags'?
> 
> So your machines will have configured tags (be it machine name) and
> instead of the node - you would use 'tag' ?

So, --tag should be used instead on --node?
I'd better make --node a alias for --tag for exactly this case if that
is possible...
But... For those with cluster background --node is a native term while
--tag is... alien one... So, I'd better leave it as is.

> 
> Zdenek
> 

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Vladislav Bogdanov]

15.03.2013 19:31, David Teigland wrote:
>> I need to convert lock on a remote node during last stage of ver3
>> migration in libvirt/qemu
> 
> Hi, I wrote and maintain the dlm and have more recently written a new
> disk-based lock manager called sanlock, https://fedorahosted.org/sanlock/
> which operates with only shared storage among nodes (no networking or
> other cluster manager.)
> 
> sanlock was written to allow RHEV to manage leases for vm images on shared
> storage, including the ability to migrate leases among hosts (which is the
> most complicated part, as you've mentioned above.)  sanlock plugs into the
> libvirt locking api, which also supports file locks (usable on local or
> nfs file systems.)  (Search for "virtlockd".)

Yes, I know, please see below.

> 
> Trying to use/control vm locks via lvm commands is not a good way to solve
> the vm management/migration problem at the present time (but see below).
> Instead, I'd suggest doing the locking via the libvirt locking api which
> was designed for this purpose.  As I mentioned, libvirt supports both
> sanlock and file locks, but another option is to write a new libvirt
> locking plug-in for dlm/corosync.  This would be the best way to use dlm
> locks to protect vm images on shared storage; I've been hoping someone
> would do this for some time.

I almost agree. That's why I developed one more locking mech for libvirt
which works and solves what I need.
I also thought about dlm one (may be the next step :) ), but I need lvm
to activate volumes, because I do not like current libvirt's "-aln
everywhere" approach with VLM, so I decided to use clvm-based locking
driver (with my additions to LVM). So, I made new "clvm2" pool type
(subtype of a "logical" together with "lvm2") which does not activate
volumes at all by itself, and "clvm" locking driver. Of course, that was
not so easy, because all current locking drivers assume that disk device
used by VM is always accessible (file on NFS f.e.), and that is not true
for clvm with exclusive activation. But, I really made it all work, so I
expect this to be a very strong alternative to all other locking mechs
in libvirt (but only for LVM volumes). Local (prevent LV from being
opened on the same node) part is not yet fully baked, only a PoC, but
that can be easily replaced with some flock-based implementation (not
released yet, but I saw some words about it is being written by Daniel)
or even allow chaining of another driver (sanlock on a local storage?)
for local (per-node) locking.
Please keep an eye on libvirt list, I want to send my work there if my
ideas are accepted here on lvm list, there is strong dependency on them.

It is very hard to inject ideas simultaneously to related projects, but
I'll try.

> 
> Incidentally, I've recently started a new project which is to replace
> clvmd with a new "lvmlockd".  I'm designing lvmlockd to support both dlm
> and sanlock on the back side (transparent to lvm itself).  With sanlock,
> you will not need a dlm/corosync cluster to have the benefits of locking
> vgs and lvs on shared storage.  This project is requiring a lot of
> preliminary work in the lvm code, because the clvmd approach reaches
> deeply into lvm itself.  Relating back to virt environments, lvmlockd will
> give you direct control of the lock types, modes and objects in each
> command.  This will hopefully make it much easier to use lvm locks in a
> controlled and programatic way to solve problems like vm management.

Anyways, I decided to go with pacemaker, so what I have now fully fits
my needs (and I have it now, not with EL7).

Of course, I fully agree that clvm is just a big hack from the today's
point of view and something need to change here.

Unfortunately I do not have a power and fundings to fully redesign
clustered LVM (I do what I do for projects which did not give me a one
dollar yet), so I decided to just solve my task, being not intrusive.

And I like a state I have it all now in.

Your idea is clean, and it reminds me idea of moving quorum to corosync,
so please keep going. ;)

> 
> So, in preparation for lvmlockd, you should not assume that lvm commands
> will operate in a dlm/corosync environment.  Any new options or
> capabilities should be considered more generally.  Also, the concept of
> lvm commands executing remote operations in a cluster was a poor design
> choice for clvm.  lvmlockd will probably not support this notion.
> Executing remote commands should be done at a higher layer.

It all takes soooo long, but I need it all to be done yesterday ;)

Anyways, please feel free to add me to CC of lvmlockd discussions.

Vladislav

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[David Teigland]

On Fri, Mar 15, 2013 at 08:46:53PM +0300, Vladislav Bogdanov wrote:
> I also thought about dlm one (may be the next step :) ), 

Let me know if you decide to start work on this and I can probably lend
some help.

> Of course, I fully agree that clvm is just a big hack from the today's
> point of view and something need to change here.

Since you're open to hacking components together, here's another possible
hack:  you could create a gfs2 file system on the shared storage, and use
file locks on gfs2 files.  The gfs2 files could be the actual vm images,
but they do not have to be.  You could still use lv's for vms directly,
and create empty files on gfs2 representing each lv.  The virtlockd file
locks would be acquired on the empty gfs2 files, representing the lvs.
This is another indirect way of using dlm locks, since the gfs2 file locks
are passed to the dlm.

Dave

Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]

[Vladislav Bogdanov]

15.03.2013 21:38, David Teigland wrote:
> On Fri, Mar 15, 2013 at 08:46:53PM +0300, Vladislav Bogdanov wrote:
>> I also thought about dlm one (may be the next step :) ), 
> 
> Let me know if you decide to start work on this and I can probably lend
> some help.
> 
>> Of course, I fully agree that clvm is just a big hack from the today's
>> point of view and something need to change here.
> 
> Since you're open to hacking components together, here's another possible
> hack:  you could create a gfs2 file system on the shared storage, and use
> file locks on gfs2 files.  The gfs2 files could be the actual vm images,
> but they do not have to be.  You could still use lv's for vms directly,
> and create empty files on gfs2 representing each lv.  The virtlockd file
> locks would be acquired on the empty gfs2 files, representing the lvs.
> This is another indirect way of using dlm locks, since the gfs2 file locks
> are passed to the dlm.

That would not solve issue with exclusive LV activation which is
required to take snapshots (and just for more safety), but prevents
migration.

Vladislav