Question: LVM2 testing with sanlock lock manager

Question: LVM2 testing with sanlock lock manager

[Leo Yan]

Hi there,

I have one question for LVM test suites with sanlock locking manager.
After tried both the master branch and with the LVM v2 (with the tag
'v2_02_187'), the testing seems to me is fragile.

I observed some below issues:

- The sanlock daemon preparation script
  test/shell/aa-lvmlockd-sanlock-prepare.sh has several issues, e.g.
  the dmsetup command cannot work well for global lock; the
  gl_file.img is stored in the temporary folder and it will be removed
  after the script aa-lvmlockd-sanlock-prepare.sh is finished, so
  gl_file.img cannot be used cross all sequential test cases.  Thus
  the global lock will malfunction for this case.

  Have sent related patches to lvm-devel mailing list for this.

- I also compared the testing result between DLM and sanlock, if I
  using DLM as the locking scheme, the testing result is quite
  reliable.  But for sanlock related testing, the testing cases have
  many failures, even after fixed the observed issues in
  aa-lvmlockd-sanlock-prepare.sh, still can see many failures.

  I used the command as below:
  make check_lvmlockd_sanlock LVM_TEST_BACKING_DEVICE=/dev/sdj3 S=component-cache.sh

  From the testing I found, component-cache.sh is easily cause stuck
  at the last command to remove VG and after the testing case
  lvconvert-cache-chunks.sh the almost later cases fail.  So the final
  testing result is:
  ### 330 tests: 34 passed, 98 skipped, 0 timed out, 1 warned, 197 failed

Could you kindly confirm if I miss something (configurations, or
should co-work with DLM) for sanlock related testing?

Thank you,
Leo

Question: LVM2 testing with sanlock lock manager

[David Teigland]

On Thu, Jun 25, 2020 at 05:34:12PM +0800, Leo Yan wrote:
> Hi there,
> 
> I have one question for LVM test suites with sanlock locking manager.
> After tried both the master branch and with the LVM v2 (with the tag
> 'v2_02_187'), the testing seems to me is fragile.

Hi, we switched to using check_lvmlockd_test (running lvmlockd in test
mode without a lock manager backend) because it accomplished most of what
we needed in a much simpler way.  Also, a forcable cleanup of the lock
managers after a test failure had problems I never fully resolved.  And,
the value of check_lvmlockd_sanlock and check_lvmlockd_dlm is limited
since these are local tests.  As you've seen, I left the old methods
around in case someone found a use for them.

> I observed some below issues:
> 
> - The sanlock daemon preparation script
>   test/shell/aa-lvmlockd-sanlock-prepare.sh has several issues, e.g.
>   the dmsetup command cannot work well for global lock; the
>   gl_file.img is stored in the temporary folder and it will be removed
>   after the script aa-lvmlockd-sanlock-prepare.sh is finished, so
>   gl_file.img cannot be used cross all sequential test cases.  Thus
>   the global lock will malfunction for this case.
> 
>   Have sent related patches to lvm-devel mailing list for this.
> 
> - I also compared the testing result between DLM and sanlock, if I
>   using DLM as the locking scheme, the testing result is quite
>   reliable.  But for sanlock related testing, the testing cases have
>   many failures, even after fixed the observed issues in
>   aa-lvmlockd-sanlock-prepare.sh, still can see many failures.
> 
>   I used the command as below:
>   make check_lvmlockd_sanlock LVM_TEST_BACKING_DEVICE=/dev/sdj3 S=component-cache.sh
> 
>   From the testing I found, component-cache.sh is easily cause stuck
>   at the last command to remove VG and after the testing case
>   lvconvert-cache-chunks.sh the almost later cases fail.  So the final
>   testing result is:
>   ### 330 tests: 34 passed, 98 skipped, 0 timed out, 1 warned, 197 failed
> 
> Could you kindly confirm if I miss something (configurations, or
> should co-work with DLM) for sanlock related testing?

Thanks for the patches.  There are some issues that prevent a number of
tests from being used with check_sanlock, the main one being the internal
lvmlock LV that holds the sanlock locks and keeps one of the PVs in use.
We sometimes change the tests so they'll work, or will flag them to skip
with sanlock.  I will try component-cache.sh to see if that's the issue,
or something else.

Dave

Question: LVM2 testing with sanlock lock manager

[David Teigland]

On Thu, Jun 25, 2020 at 09:55:23AM -0500, David Teigland wrote:
> with sanlock.  I will try component-cache.sh to see if that's the issue,
> or something else.

I suspect that test is hitting an issue where a stray lock is being left
behind and blocking the test cleanup.  The test is using some unusual
commands, activating internal LVs that are usually disallowed.

I couldn't get the GL_DEV/GL_FILE stuff to work, with or without your
patches.  I removed all that and specified a literal sd device.  I'd
suggest we do that instead of the complicated attempt to automate it; i.e.
require a path to an unused device that the test will use for glvg.

Dave

Question: LVM2 testing with sanlock lock manager

[Leo Yan]

Hi David,

Thank you for kindly replying.

On Thu, Jun 25, 2020 at 02:11:00PM -0500, David Teigland wrote:
> On Thu, Jun 25, 2020 at 09:55:23AM -0500, David Teigland wrote:
> > with sanlock.  I will try component-cache.sh to see if that's the issue,
> > or something else.
> 
> I suspect that test is hitting an issue where a stray lock is being left
> behind and blocking the test cleanup.  The test is using some unusual
> commands, activating internal LVs that are usually disallowed.
> 
> I couldn't get the GL_DEV/GL_FILE stuff to work, with or without your
> patches.  I removed all that and specified a literal sd device.  I'd
> suggest we do that instead of the complicated attempt to automate it; i.e.
> require a path to an unused device that the test will use for glvg.

I followed your suggestion and tried to use a dedicated /dev/sdX device
for global lock with sanlock, after run several test cases, it looks
to me is much stable than before for global lock.

For component-cache.sh, I still can see it's stuck at the last line of
the test case:

[ 1:01] lvs -a $vg
[ 1:01] #component-cache.sh:89+ lvs -a @PREFIX at vg
[ 1:01]   LV                  VG             Attr       LSize   Pool    Origin              Data%  Meta%  Move Log Cpy%Sync Convert
[ 1:01]   [cpool]             @PREFIX at vg Cwi---C---   2.00m
[ 1:01]   [cpool_cdata]       @PREFIX at vg Cwi-------   2.00m
[ 1:01]   [cpool_cmeta]       @PREFIX at vg eRi-a-----   4.50m
[ 1:01]   [lvmlock]           @PREFIX at vg -wi-ao---- 256.00m
[ 1:01]   [lvol0_pmspare]     @PREFIX at vg ewi-------   4.50m
[ 1:01]   tpool               @PREFIX at vg twi---tz--   4.00m
[ 1:01]   [tpool_tdata]       @PREFIX at vg Cwi---C---   4.00m [cpool] [tpool_tdata_corig]
[ 1:01]   [tpool_tdata_corig] @PREFIX at vg owi---C---   4.00m
[ 1:01]   [tpool_tmeta]       @PREFIX at vg ewi-------   2.00m
[ 1:01]
[ 1:01] # And final removal works                                        
[ 1:01] vgremove -f $vg                                                  
[ 1:01] #component-cache.sh:92+ vgremove -f @PREFIX at vg                   
[ 1:01]   WARNING: This metadata update is NOT backed up.                
[ 1:01]   Logical volume "tpool" successfully removed                    

So the LV "tpool" has been removed but seems it stuck on other LV's
deleting?  Should I note other component, e.g dmeventd for this?

Thanks,
Leo

Question: LVM2 testing with sanlock lock manager

[Leo Yan]

On Thu, Jun 25, 2020 at 09:55:23AM -0500, David Teigland wrote:
> On Thu, Jun 25, 2020 at 05:34:12PM +0800, Leo Yan wrote:
> > Hi there,
> > 
> > I have one question for LVM test suites with sanlock locking manager.
> > After tried both the master branch and with the LVM v2 (with the tag
> > 'v2_02_187'), the testing seems to me is fragile.
> 
> Hi, we switched to using check_lvmlockd_test (running lvmlockd in test
> mode without a lock manager backend) because it accomplished most of what
> we needed in a much simpler way.  Also, a forcable cleanup of the lock
> managers after a test failure had problems I never fully resolved.  And,
> the value of check_lvmlockd_sanlock and check_lvmlockd_dlm is limited
> since these are local tests.  As you've seen, I left the old methods
> around in case someone found a use for them.

Thanks a lot for the info.

I noted check_lvmlockd_test, this feature is more related for testing
lvmlockd + LVM and in the middle to hijack so disallow LVM tools to talk
to lock managers.

> > I observed some below issues:
> > 
> > - The sanlock daemon preparation script
> >   test/shell/aa-lvmlockd-sanlock-prepare.sh has several issues, e.g.
> >   the dmsetup command cannot work well for global lock; the
> >   gl_file.img is stored in the temporary folder and it will be removed
> >   after the script aa-lvmlockd-sanlock-prepare.sh is finished, so
> >   gl_file.img cannot be used cross all sequential test cases.  Thus
> >   the global lock will malfunction for this case.
> > 
> >   Have sent related patches to lvm-devel mailing list for this.
> > 
> > - I also compared the testing result between DLM and sanlock, if I
> >   using DLM as the locking scheme, the testing result is quite
> >   reliable.  But for sanlock related testing, the testing cases have
> >   many failures, even after fixed the observed issues in
> >   aa-lvmlockd-sanlock-prepare.sh, still can see many failures.
> > 
> >   I used the command as below:
> >   make check_lvmlockd_sanlock LVM_TEST_BACKING_DEVICE=/dev/sdj3 S=component-cache.sh
> > 
> >   From the testing I found, component-cache.sh is easily cause stuck
> >   at the last command to remove VG and after the testing case
> >   lvconvert-cache-chunks.sh the almost later cases fail.  So the final
> >   testing result is:
> >   ### 330 tests: 34 passed, 98 skipped, 0 timed out, 1 warned, 197 failed
> > 
> > Could you kindly confirm if I miss something (configurations, or
> > should co-work with DLM) for sanlock related testing?
> 
> Thanks for the patches.  There are some issues that prevent a number of
> tests from being used with check_sanlock, the main one being the internal
> lvmlock LV that holds the sanlock locks and keeps one of the PVs in use.

Understood, I do see some logs like below:

udev-lvmlockd-sanlock:shell_zero-usage.sh.txt:[ 0:14] device-mapper: remove ioctl on LVMTEST7803vg-lvmlock  failed: Device or resource busy  
udev-lvmlockd-sanlock:shell_zero-usage.sh.txt:[ 0:14] device-mapper: remove ioctl on LVMTEST27669vg_1-lvmlock  failed: Device or resource bus
udev-lvmlockd-sanlock:shell_zero-usage.sh.txt:[ 0:14] device-mapper: remove ioctl on LVMTEST22552vg-lvmlock  failed: Device or resource busy 
udev-lvmlockd-sanlock:shell_zero-usage.sh.txt:[ 0:14] device-mapper: remove ioctl on LVMTEST5984vg-lvmlock  failed: Device or resource busy  
udev-lvmlockd-sanlock:shell_zero-usage.sh.txt:[ 0:14] device-mapper: remove ioctl on LVMTEST9398vg-lvmlock  failed: Device or resource busy  
udev-lvmlockd-sanlock:shell_zero-usage.sh.txt:[ 0:14] device-mapper: remove ioctl on LVMTEST4978vg-lvmlock  failed: Device or resource busy  

Thanks,
Leo