lvmlockd: adopt locks always failed

lvmlockd: adopt locks always failed

[Eric Ren]

Hi David!

I'm testing "dlm + lvmlockd". Everything goes well so far, until I try 
the steps:
1. do everything to have a LV active on a shared VGs successfully;
2. kill lvmlockd on purpose;
3. restart lvmlockd with "--adopt 1".

Every time I get:
"""
Aug 07 13:09:14 tw1 lvmlockd[27299]: 1502082554 lvmlockd started
Aug 07 13:09:14 tw1 lvmlockd[27299]: 1502082554 adopt_locks failed, 
reset host
"""

After some debugging, I narrow down it here:

1.
adopt_locks()
{
...

        if (lm_support_dlm()) {
                 rv = lm_get_lockspaces_dlm(&ls_found); ====> OK.
                 if ((rv < 0) && (rv != -ECONNREFUSED))
                         goto fail;
         }

         if (lm_support_sanlock()) {
                 rv = lm_get_lockspaces_sanlock(&ls_found);       ====> 
rv gets -2
                 if ((rv < 0) && (rv != -ECONNREFUSED))
                         goto fail;
         }
...
}

2.
sanlock_get_lockspaces()
{
...
         rv = connect_socket(&fd);
         if (rv < 0)
                 return rv;                 ===> it return -2, because 
sanlock plugin is not running
...
}

The problem here seems we should not ask sanlock for left lockspaces 
when it's not running?

Thanks in advance!
Eric

lvmlockd: adopt locks always failed

[David Teigland]

> The problem here seems we should not ask sanlock for left lockspaces when
> it's not running?

Yes, thanks for debugging that, I've just pushed a fix.
Dave

lvmlockd: adopt locks always failed

[Eric Ren]

Hi David,

On 08/07/2017 11:47 PM, David Teigland wrote:
>> The problem here seems we should not ask sanlock for left lockspaces when
>> it's not running?
> Yes, thanks for debugging that, I've just pushed a fix.
Thanks!

I add a resource agent for lvmlockd. It would be great if you can give some
review and comments there ;-P  The PR is here:

https://github.com/ClusterLabs/resource-agents/pull/1013

Regards,
Eric

lvmlockd: adopt locks always failed

[David Teigland]

On Tue, Aug 08, 2017 at 10:03:58AM +0800, Eric Ren wrote:
> I add a resource agent for lvmlockd. It would be great if you can give some
> review and comments there ;-P  The PR is here:
> 
> https://github.com/ClusterLabs/resource-agents/pull/1013

Hi, thanks for working on that, resource agents have always seemed mostly
redundant to me, so I'm not one to give the best feedback.

You could simplify things by just starting lvmlockd from systemd during
system startup using
https://sourceware.org/git/?p=lvm2.git;a=blob_plain;f=scripts/lvm2_lvmlockd_systemd_red_hat.service.in;hb=HEAD

since there's no clustering dependency on doing that, i.e. no problem
starting the daemon before clustering components are started.  It doesn't
actually do anything until you start shared VGs.

I think we could make lock-adoption the default during startup.  I may
change the default in lvmlockd itself.  You always want to adopt locks if
they exist; things aren't going to work if you restart lvmlockd without
adopting locks when previous orphaned locks exist.  However, I'm not sure
that resource agents will be sophisticated enough to restart lvmlockd and
adopt locks after a previous instance failed; they've always seemed eager
to just reset the node.

Once the clustering system is running (in particular the dlm), you can
then start and use shared VGs.  You could use or copy this unit file for
that:

https://sourceware.org/git/?p=lvm2.git;a=blob_plain;f=scripts/lvm2_lvmlocking_systemd_red_hat.service.in;hb=HEAD

For activation, you need to use vgchange -aay to specify auto-activation.
You're also including 's' to activate LVs with a shared lock, but it's
only safe to do that if you have a cluster file system on the LV.  If you
don't have a way to know that, I've thought in the past that we could add
a new lvm.conf list, similar to auto_activation_volume_list, that
specifies which LVs should be activated with shared locks.  For
deactivation, use -an, (the 'l' was a clvm-ism, and is meaningless with
lvmlockd.)

Dave

lvmlockd: adopt locks always failed

[Eric Ren]

Hi,


On 08/08/2017 10:28 PM, David Teigland wrote:
> On Tue, Aug 08, 2017 at 10:03:58AM +0800, Eric Ren wrote:
>> I add a resource agent for lvmlockd. It would be great if you can give some
>> review and comments there ;-P  The PR is here:
>>
>> https://github.com/ClusterLabs/resource-agents/pull/1013
> Hi, thanks for working on that, resource agents have always seemed mostly
> redundant to me, so I'm not one to give the best feedback.

Yeah, agree. But, many resource agents are still created for daemons
that already have systemd service though, like dlm, clvm, etc. It seems
HA software wants to manage these daemons itself. Anyway, just provide
a another option for HA users who used to stack resource agents ;-P

> You could simplify things by just starting lvmlockd from systemd during
> system startup using
> https://sourceware.org/git/?p=lvm2.git;a=blob_plain;f=scripts/lvm2_lvmlockd_systemd_red_hat.service.in;hb=HEAD
>
> since there's no clustering dependency on doing that, i.e. no problem
> starting the daemon before clustering components are started.  It doesn't
> actually do anything until you start shared VGs.

Thanks, I didn't notice this point before.

>
> I think we could make lock-adoption the default during startup.  I may
> change the default in lvmlockd itself.  You always want to adopt locks if
> they exist; things aren't going to work if you restart lvmlockd without
> adopting locks when previous orphaned locks exist.  However, I'm not sure
> that resource agents will be sophisticated enough to restart lvmlockd and
> adopt locks after a previous instance failed; they've always seemed eager
> to just reset the node.

Yes, I also realized that. In the resource agent, I've added "-A 1" when 
starting the daemon.

>
> Once the clustering system is running (in particular the dlm), you can
> then start and use shared VGs.  You could use or copy this unit file for
> that:
>
> https://sourceware.org/git/?p=lvm2.git;a=blob_plain;f=scripts/lvm2_lvmlocking_systemd_red_hat.service.in;hb=HEAD
>
> For activation, you need to use vgchange -aay to specify auto-activation.

OK, thanks.  For the resource agent, we can select "-aay" way as 
default, while providing the option to
activate all shared VGs when users have many shared VGs and they don't 
want to add them into
lvm.conf manually.

> You're also including 's' to activate LVs with a shared lock, but it's
> only safe to do that if you have a cluster file system on the LV.  If you
> don't have a way to know that, I've thought in the past that we could add
> a new lvm.conf list, similar to auto_activation_volume_list, that
> specifies which LVs should be activated with shared locks.  For
> deactivation, use -an, (the 'l' was a clvm-ism, and is meaningless with
> lvmlockd.)
Got it, thanks. I will change that ;-)

Regards,
Eric