Re: [BUG] Xen live migration dom0 memory

[Ian Campbell <ian.campbell@citrix.com>]

On Fri, 2015-09-04 at 17:04 +0100, Ian Campbell wrote:
> On Tue, 2015-07-07 at 17:26 +0200, Epiontis IT wrote:
> > Hello,
> > 
> > I already posted on the xen-users list and Ian told me to post a bug 
> > report here. We have the following problem:
> > 
> > When I start a migration with "xl migrate <vm> <destination>" the 
> > destination machine sets up a vm "<vm>--incoming" and seems to sync 
> > memory because "xl list" shows "Mem" for the incoming vm going up 
> > stepwise until 2048. After that though the vm doesn't get launched. The 
> > 
> > vm is frozen for about a minute, the dom0 begins swapping out the 2GB 
> > to 
> > disk (because it only has 512M available for itself) and a process
> 
> I spoke with Ian Jackson about this this afternoon and we spent some
> staring at the pmap, and we are now both very perplexed...
> 
> So I've just tried to repro this.
> 
> I installed latest 4.5-testing and a 512M dom0 (which as it happens is 
> also
> what our automated test system uses). I then migrated a 3GB domU with "xl
> migrate <dom> localhost" and it completed successfully. I had to do
> localhost migrate due to only having one box, but I don't think this 
> should
> matter.
>  
> I observed the save helper with pmap and never saw any large allocations.
> 
> This matches the behaviour that both Ian and I expected.
> 
> Your logs show you are running 4.5.1-rc1. I've looked over
> git log 4.5.1-rc1..origin/staging-4.5 -- tools/libx[cl]/
> and I don't see anything like a fix for a leak etc.
> 
> Next step is to revert my test box from staging-4.5 to 4.5.1-rc1 and see
> what changes.

Nothing changed.

I then tried building this tree with 

EXTRA_CFLAGS_XEN_TOOLS="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security" APPEND_CPPFLAGS="-D_FORTIFY_SOURCE=2" APPEND_LDFLAGS="-Wl,-z,relro" 

which is how the Debian package is built, still no change.

My next step was to build patched with the Debian patches, still no change.

So I'm at even more of a loss than I was before :-/

Your report said you were running Debian 8.0 (Jessie) but with Xen 4.5.1
-rc1, while Jessie only has 4.4.1.

Was 4.5.1-rc1 from experimental or built yourself? Did you install anything
else from Sid/Experimental (or Stretch)?

If not from the packages in experimental then how did you install Xen?

If you can still reproduce then you could try installing the latest
valgrind, move libxl-save-helper aside and replace it with a script which
runs the tool under valgrind.

Modern valgrind understands many Xen hypercalls etc and so is quite useful
even on the Xen toolstack binaries.

Ian.

> 
> Ian.
> 
> > 
> > /usr/lib/xen-4.5/bin/libxl-save-helper --restore-domain
> > 
> > takes about 90% of the dom0 memory. After that the vm ist launched, 
> > memory consumption goes back to normal, swap space is freed.
> > 
> > I attached several log and output files in xen_bug_report.txt. In 
> > particular Ian wanted to see the pmap of the save helper process which 
> > you can find at the end of the file taken during several phases of the 
> > migration process.
> > 
> > 
> > Thank you,
> > Alex
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel