Is: Contention in block script when doing guest saving. Was:Re: an issue with 'xm save'

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Fri, Sep 28, 2012 at 06:34:51PM +0800, Zhenzhong Duan wrote:
> On 2012-09-27 19:59, Konrad Rzeszutek Wilk wrote:
> >On Thu, Sep 27, 2012 at 01:58:19PM +0800, Zhenzhong Duan wrote:
> >>
> >>On 2012-09-26 20:35, Konrad Rzeszutek Wilk wrote:
> >>>On Wed, Sep 26, 2012 at 04:48:42PM +0800, Zhenzhong Duan wrote:
> >>>>Konrad Rzeszutek Wilk wrote:
> >>>>>On Fri, Sep 21, 2012 at 05:41:27PM +0800, Zhenzhong Duan wrote:
> >>>>>>Hi maintainers,
> >>>>>>
> >>>>>>I found there is an issue when 'xm save' a pvm guest. See below:
> >>>>>>
> >>>>>>When I do save then restore once, CPU(%) in xentop showed around 99%.
> >>>>>>When I do that second time, CPU(%) showed 199%
> >>>>>>
> >>>>>>top in dom0 showed:
> >>>>>>     PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> >>>>>>    20946 root      18  -2 10984 1284  964 S 19.8  0.3   0:48.93 block
> >>>>>>    4939 root      18  -2 10984 1288  964 S 19.5  0.3   1:34.68 block
> >>>>>>
> >>>>>>I could kill the block process, then all look normal again.
> >>>>>What is the 'block' process? If you attach 'perf' to it do you get an idea
> >>>>>of what it is spinning at?
> >>>>It's /etc/xen/scripts/block
> >>>>I add 'set -x' to /etc/xen/scripts/block, found it blocked at claim_lock.
> >>>>When domU was created first time, claim_lock/release_lock finished quickly,
> >>>>when 'xm save' was called, claim_lock spin in its own while loop.
> >>>>I can ensure no other domU create/save/etc happen when I test.
> >>>OK, so how come you have two block processes? Is it b/c you have two
> >>>disks attached to the guest? The are multiple claim_lock in the shell
> >>>script - do you know where each of two threads are spinning? Are they
> >>>spinning on the same function?
> >>In above test, I run save/restore twice, so two block processes.
> >>In other test, run save/restore once, there is only one block process.
> >>After do 'xm save', I see block process spin at line 328:
> >>321   remove)
> >>322     case $t in
> >>323       phy)
> >>324         exit 0
> >>325         ;;
> >>326
> >>327       file)
> >>328         claim_lock "block"
> >>329         node=$(xenstore_read "$XENBUS_PATH/node")
> >>330         losetup -d "$node"
> >>331         release_lock "block"
> >>332         exit 0
> >>333         ;;
> >So with the patches in OVM - do they have this fixed? Can they be upstreamed
> >or are the dependent on some magic OVM sauce?
> After replace locking.sh with OVM's, it worked.
> But xen-tools evolved to use flock based locking currently. We can't
> revert back.
> It seems changeset 25595:497e2fe49455 bring the issue.
> Finally, I came with a small patch that workaround the issue.

Ian, any thoughts on how this can be fixed properly? The locking
is quite heavy and causes two scripts that do deal with blocks
to spin for a long time.

> 
> diff -r d364becfb083 tools/hotplug/Linux/locking.sh
> --- a/tools/hotplug/Linux/locking.sh    Thu Sep 20 13:31:19 2012 +0200
> +++ b/tools/hotplug/Linux/locking.sh    Fri Sep 28 18:27:31 2012 +0800
> @@ -66,6 +66,7 @@
>  release_lock()
>  {
>      _setlockfd $1
> +    flock -u $_lockfd
>      rm "$_lockfile"
>  }