Question: the failure handling for sanlock

Question: the failure handling for sanlock

[Leo Yan]

Hi there,

I have couple questions for the drive failure handling for sanlock; let me
elaborate for the questions:

If any drive or fabric failures happen and it's impossible to access the
drives, thus the sanlock lock manager fails to renew the lease for locks.
In this case, as described in the sanlock wrapper of lvmlockd [1], it firstly
invokes command 'lvmlockctl --kill vg_name' to prevent lvmlockd to handle any
further requests for the VG which is killed; and it "attempts to quit using
the VG."

For "quit using the VG", the comment suggests to use blkdeactivate() or a more
forceful equivalent method to disable block devices, and needs to release
lease for VG/LV locks.  But as we can see, in the code of lvmlockctl, it
comments out the failure handling operations [2]:

 538         /*
 539          * FIXME: here is where we should implement a strong form of
 540          * blkdeactivate, and if it completes successfully, automatically call
 541          * do_drop() afterward.  (The drop step may not always be necessary
 542          * if the lvm commands run while shutting things down release all the
 543          * leases.)
 544          *
 545          * run_strong_blkdeactivate();
 546          * do_drop();
 547          */

My first question is why lvmlockctl comments out the failure handling code so
cannot automatically deactivate drives and cleanup VG/LV locks by default?
I think one possible reason is lvmlockd gives the priority for DLM locking scheme,
the DLM lock manager is based on networking protocol, and DLM has its path for
failure handling, like using pace maker and CPG [3].  But on the other hand, this
means the current implementation for lvmlockd/lvmlockctl is absent to support
driver failure handling by default and need to manually deactivate block devices.

The second question is what's a suggested flow for failure handling for sanlock?
If I understand correctly, we can rely on the command "blkdeactivate" to
deactivates block devices; but if we deactivate the whole block device, it
might also hurt other VGs/LVs resident on the same drive.  So should we firstly
deactivate VGs or LVs with "vgchange" or "lvchange" commands rather than
deactivate the whole block device?  We might consider there have no chance to
access drive for the drive or fabric malfunction, so cannot make succuss for
commands "vgchange" and "lvchange".

Hope I have described my questions clearly, if no, please let me know.
And thanks in advance for suggestions!

Leo

[1] https://sourceware.org/git/?p=lvm2.git;a=blob;f=daemons/lvmlockd/lvmlockd-sanlock.c;h=4bc6402cf85a374a49695c6bca5bc10a7e5f042b;hb=refs/heads/master#l119
[2] https://sourceware.org/git/?p=lvm2.git;a=blob;f=daemons/lvmlockd/lvmlockctl.c;h=c2a998c8c4a19bdf835d1020c64af2e8918c3915;hb=refs/heads/master#l539
[3] https://pagure.io/dlm/blob/master/f/dlm_controld/cpg.c#_46

Question: the failure handling for sanlock

[David Teigland]

On Sun, Feb 14, 2021 at 06:13:03PM +0800, Leo Yan wrote:
>  538         /*
>  539          * FIXME: here is where we should implement a strong form of
>  540          * blkdeactivate, and if it completes successfully, automatically call
>  541          * do_drop() afterward.  (The drop step may not always be necessary
>  542          * if the lvm commands run while shutting things down release all the
>  543          * leases.)
>  544          *
>  545          * run_strong_blkdeactivate();
>  546          * do_drop();
>  547          */
> 
> My first question is why lvmlockctl comments out the failure handling code so
> cannot automatically deactivate drives and cleanup VG/LV locks by default?

Only because that forceful deactivate has not been implemented.

> The second question is what's a suggested flow for failure handling for sanlock?
> If I understand correctly, we can rely on the command "blkdeactivate" to
> deactivates block devices; but if we deactivate the whole block device, it
> might also hurt other VGs/LVs resident on the same drive.  So should we firstly
> deactivate VGs or LVs with "vgchange" or "lvchange" commands rather than
> deactivate the whole block device?  We might consider there have no chance to
> access drive for the drive or fabric malfunction, so cannot make succuss for
> commands "vgchange" and "lvchange".

We only need to forcefully deactivate LVs in the failed VG.  The lvmlockd
man page section "sanlock lease storage failure" describes how this
process can work manually, and mentions doing it automatically in the
future.  In the manual process you'd try to stop applications using the
fs, unmount file systems on the LVs, then try to deactivate the LVs.
Since there are storage issues, the unmount and deactivate might hang, in
which case you could try dmsetup wipe_table on the LVs.  You might just
skip immediately to that last step since there's not a lot of time.

There was a script written in 2017 to automate that final dmsetup step.
It was finished or very close, but I don't remember what kept us from
including it.  An idea to make dmsetup do more of the work perhaps left us
unsure what to do.  Here is the script and some discussion:
https://listman.redhat.com/archives/lvm-devel/2017-September/msg00011.html

There was also a small lvmlockctl patch to replace the
run_strong_blkdeactivate() comment with running the script.

If you can get this to work for your case I'd like to get it included in
lvm.

Dave

Question: the failure handling for sanlock

[Leo Yan]

Hi David,

On Mon, Feb 15, 2021 at 10:31:27AM -0600, David Teigland wrote:
> On Sun, Feb 14, 2021 at 06:13:03PM +0800, Leo Yan wrote:
> >  538         /*
> >  539          * FIXME: here is where we should implement a strong form of
> >  540          * blkdeactivate, and if it completes successfully, automatically call
> >  541          * do_drop() afterward.  (The drop step may not always be necessary
> >  542          * if the lvm commands run while shutting things down release all the
> >  543          * leases.)
> >  544          *
> >  545          * run_strong_blkdeactivate();
> >  546          * do_drop();
> >  547          */
> > 
> > My first question is why lvmlockctl comments out the failure handling code so
> > cannot automatically deactivate drives and cleanup VG/LV locks by default?
> 
> Only because that forceful deactivate has not been implemented.
> 
> > The second question is what's a suggested flow for failure handling for sanlock?
> > If I understand correctly, we can rely on the command "blkdeactivate" to
> > deactivates block devices; but if we deactivate the whole block device, it
> > might also hurt other VGs/LVs resident on the same drive.  So should we firstly
> > deactivate VGs or LVs with "vgchange" or "lvchange" commands rather than
> > deactivate the whole block device?  We might consider there have no chance to
> > access drive for the drive or fabric malfunction, so cannot make succuss for
> > commands "vgchange" and "lvchange".
> 
> We only need to forcefully deactivate LVs in the failed VG.  The lvmlockd
> man page section "sanlock lease storage failure" describes how this
> process can work manually, and mentions doing it automatically in the
> future.  In the manual process you'd try to stop applications using the
> fs, unmount file systems on the LVs, then try to deactivate the LVs.
> Since there are storage issues, the unmount and deactivate might hang, in
> which case you could try dmsetup wipe_table on the LVs.  You might just
> skip immediately to that last step since there's not a lot of time.
> 
> There was a script written in 2017 to automate that final dmsetup step.
> It was finished or very close, but I don't remember what kept us from
> including it.  An idea to make dmsetup do more of the work perhaps left us
> unsure what to do.  Here is the script and some discussion:
> https://listman.redhat.com/archives/lvm-devel/2017-September/msg00011.html
> 
> There was also a small lvmlockctl patch to replace the
> run_strong_blkdeactivate() comment with running the script.

Yes, I went through the two patches, one is for the script
blkdeactivate.sh and another is for lvmlockctl.

And I saw you suggested to use the dmsetup commands:

  dmsetup info -c -S "uuid=~LVM && vgname=$VG && lv_layer=\"\"" \
        -o name --noheadings | xargs dmsetup wipe_table

> If you can get this to work for your case I'd like to get it included in
> lvm.

I will verify the changes and share back the result.

Thank you for suggestions!

Leo

Question: the failure handling for sanlock

[Leo Yan]

Hi David,

On Mon, Feb 15, 2021 at 10:31:27AM -0600, David Teigland wrote:

[...]

> We only need to forcefully deactivate LVs in the failed VG.  The lvmlockd
> man page section "sanlock lease storage failure" describes how this
> process can work manually, and mentions doing it automatically in the
> future.  In the manual process you'd try to stop applications using the
> fs, unmount file systems on the LVs, then try to deactivate the LVs.
> Since there are storage issues, the unmount and deactivate might hang, in
> which case you could try dmsetup wipe_table on the LVs.  You might just
> skip immediately to that last step since there's not a lot of time.
> 
> There was a script written in 2017 to automate that final dmsetup step.
> It was finished or very close, but I don't remember what kept us from
> including it.  An idea to make dmsetup do more of the work perhaps left us
> unsure what to do.  Here is the script and some discussion:
> https://listman.redhat.com/archives/lvm-devel/2017-September/msg00011.html
> 
> There was also a small lvmlockctl patch to replace the
> run_strong_blkdeactivate() comment with running the script.
> 
> If you can get this to work for your case I'd like to get it included in
> lvm.

I verified the two patches you mentioned, and applied your suggestion
to use the command:

  "$DMSETUP" info -c -S "uuid=~LVM && vgname=$VG_NAME && lv_layer=\"\"" \
  	-o name --noheadings | xargs "$DMSETUP" wipe_table

At the end, I can see the device is changed to use DM "error" backend
driver:

  Broadcast message from systemd-journald at localhost.localdomain (Tue 2021-02-23 21:55:16 EST):
  lvmlockctl[2032]: Deactivated LVs in VG TESTVG1 successfully.

  Broadcast message from systemd-journald at localhost.localdomain (Tue 2021-02-23 21:55:16 EST):
  lvmlockctl[2032]: Runing lvmlockctl --drop TESTVG1.

  # dmsetup table
  TESTPV1: 0 163840 linear 8:146 0
  TESTVG1-TESTVG1LV1: 0 155648 error

Please be aware, to avoid misleading, I tested this with another
in-house locking scheme rather than sanlock, but the working flow for
failure handling is exactly same.

I have updated a bit for the two patches [1]; but want to check with you
how to proceed to merge these patches?  Would you directly merge patches
at your side or you want me to resend these two patches so you could
take a review before merging?  For the later case, I am glad to send the
patches to lvm-devel mailing list and follow up comments/suggestions as
needed.

Thanks,
Leo

[1] https://people.linaro.org/~leo.yan/lvm/

Question: the failure handling for sanlock

[David Teigland]

On Wed, Feb 24, 2021 at 11:16:30AM +0800, Leo Yan wrote:
> I verified the two patches you mentioned, and applied your suggestion
> to use the command:
> 
>   "$DMSETUP" info -c -S "uuid=~LVM && vgname=$VG_NAME && lv_layer=\"\"" \
>   	-o name --noheadings | xargs "$DMSETUP" wipe_table
> 
> At the end, I can see the device is changed to use DM "error" backend
> driver:
> 
>   Broadcast message from systemd-journald at localhost.localdomain (Tue 2021-02-23 21:55:16 EST):
>   lvmlockctl[2032]: Deactivated LVs in VG TESTVG1 successfully.
> 
>   Broadcast message from systemd-journald at localhost.localdomain (Tue 2021-02-23 21:55:16 EST):
>   lvmlockctl[2032]: Runing lvmlockctl --drop TESTVG1.
> 
>   # dmsetup table
>   TESTPV1: 0 163840 linear 8:146 0
>   TESTVG1-TESTVG1LV1: 0 155648 error
> 
> Please be aware, to avoid misleading, I tested this with another
> in-house locking scheme rather than sanlock, but the working flow for
> failure handling is exactly same.
> 
> I have updated a bit for the two patches [1]; but want to check with you
> how to proceed to merge these patches?  Would you directly merge patches
> at your side or you want me to resend these two patches so you could
> take a review before merging?  For the later case, I am glad to send the
> patches to lvm-devel mailing list and follow up comments/suggestions as
> needed.
> 
> Thanks,
> Leo
> 
> [1] https://people.linaro.org/~leo.yan/lvm/

Thanks for verifying that, please send the patches to the list and I'll
take them from there.  Another part I now remember we need to address is
making this configurable (blkdeactivate isn't an established command.)
I'll need to come up with a solution to that before pushing them out.  The
possible solutions I'm seeing are somewhat ugly, and this is one of the
reasons we didn't finish the integration last time.

Dave

Question: the failure handling for sanlock

[Leo Yan]

On Wed, Feb 24, 2021 at 12:21:01PM -0600, David Teigland wrote:

[...]

> Thanks for verifying that, please send the patches to the list and I'll
> take them from there.

Have sent out the patches for reviewing:
https://listman.redhat.com/archives/lvm-devel/2021-February/msg00143.html

Just remind, I saw Zhuang Huan mentioned in the original patch for
"disabling queueing on multipath device", this part I have no
knowledge for how to handle it, so I left it out in the latest patch.
Please note this when review the patch.

> Another part I now remember we need to address is
> making this configurable (blkdeactivate isn't an established command.)
> I'll need to come up with a solution to that before pushing them out.  The
> possible solutions I'm seeing are somewhat ugly, and this is one of the
> reasons we didn't finish the integration last time.

Seems to me, we need to add some configurations in lvm.conf and can
teach lvmlockctl to do automatic handling for failure or bypass the
relevant operations.   If need me to follow up anything for this, let
me know.

Thanks,
Leo