Re: can not use all available memory

[Dan Magenheimer <dan.magenheimer@oracle.com>]

> From: Pasi Kärkkäinen [mailto:pasik@iki.fi]
> Subject: Re: [Xen-devel] can not use all available memory
> 
> On Mon, Nov 26, 2012 at 12:20:40PM -0800, Dan Magenheimer wrote:
> > > From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> > > Subject: Re: [Xen-devel] can not use all available memory
> > >
> > > Monday, November 26, 2012, 5:58:28 PM, you wrote:
> > >
> > > >> From: Ian Campbell [mailto:Ian.Campbell@citrix.com]
> > > >> Sent: Friday, November 23, 2012 6:34 AM
> > > >> To: Pasi Kärkkäinen
> > > >> Cc: xen-devel@lists.xensource.com; Alexander Bienzeisler
> > > >> Subject: Re: [Xen-devel] can not use all available memory
> > > >>
> > > >> On Fri, 2012-11-23 at 13:29 +0000, Pasi Kärkkäinen wrote:
> > > >> > On Fri, Nov 23, 2012 at 01:15:53PM +0000, Ian Campbell wrote:
> > > >> > > On Fri, 2012-11-23 at 12:55 +0000, Pasi Kärkkäinen wrote:
> > > >> > >
> > > >> > > > > This has been discussed at length on the list before, please check the
> > > >> > > > > archives.
> > > >> > > > >
> > > >> > > >
> > > >> > > > I tried googling quickly but I didn't find anything relevant..
> > > >> > >
> > > >> > > I found
> > > >> > > http://web.archiveorange.com/archive/v/zCKz5T3PLvtyZDSPQc9i
> > > >> > > in a matter of seconds, then:
> > > >> > > http://lists.xen.org/archives/html/xen-devel/2011-11/msg01415.html
> > > >> > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00139.html
> > > >> > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00146.html
> > > >> > >
> > > >> > > > To me this behaviour still seems wrong. What's the point of autoballoon=1 trying to
> > > >> > > > balloon down dom0 if the hypervisor already has enough free memory for the VM ?
> > > >> > > >
> > > >> > > > In this case:
> > > >> > > >         - dom0_mem=2G
> > > >> > > >         - new VM to launch with size 16 GB.
> > > >> > > >         - Xen has 28 GB of free memory.
> > > >> > > >
> > > >> > > > So clearly there's no need to try to balloon down dom0..
> > > >> > >
> > > >> > > Right, so don't set autoballoon then.
> > > >> > >
> > > >> >
> > > >> > It's enabled as a default.. so many people hit this problem.
> > > >> >
> > > >> >
> > > >> > > > not-yet-implemented check/feature in xl, or a bug?
> > > >> > >
> > > >> > > Neither, it is the intended behaviour of xl autoballoon, this option
> > > >> > > means exactly "take the required memory from dom0".
> > > >> > >
> > > >> >
> > > >> > http://xenbits.xen.org/docs/4.2-testing/man/xl.conf.5.html
> > > >> >
> > > >> > "autoballoon=BOOLEAN
> > > >> >
> > > >> >     If disabled then xl will not attempt to reduce the amount of
> > > >> > memory assigned to domain 0 in order to create free memory when
> > > >> > starting a new domain. You should set this if you use the dom0_mem
> > > >> > hypervisor command line to reduce the amount of memory given to domain
> > > >> > 0 by default.
> > > >> >
> > > >> >     Default: 1"
> > > >> >
> > > >> >
> > > >> > I think we should modify that to say "You should set autoballoon=0 if you use the dom0_mem
> > > >> hypervisor command line .."
> > > >> > At least I understood that text in the opposite way..
> > > >>
> > > >> Yes, there should be a s/set/unset/ in there I think.
> > > >>
> > > >> > Also: What happens if you have autoballoon=1 and you start some VMs, then stop the VMs,
> > > >> > so most of the memory is now free in Xen..
> > > >>
> > > >> xl balloons dom0 back up when it destroy domains with autoballoon=1.
> > > >>
> > > >> > and then you try to start a big VM ?
> > > >> > Aren't you going to hit the same problem as in this thread?
> > >
> > > > Hmmm... this behavior and default may make sense for the Citrix
> > > > memory model (single machine, dom0 is "the user" so you want it
> > > > to always hold most of physical RAM not used by guests).  But
> > > > probably not so for a more cloud-like memory model.
> > >
> > > > Is there any (easy) way to force autoballoon=0 if the hypervisor
> > > > dom0_mem boot option is specified?  Or is there some reasonably
> > > > sane case I am missing where a user would want both dom0_mem
> > > > and autoballoon=1?
> > >
> > > > Oracle VM always boots servers with dom0_mem= set so (if/when
> > > > OVM switches to use xl), OVM will always set autoballoon off.
> > > > So it's the large number of non-Citrix-non-Oracle Xen-as-a-service
> > > > providers I am trying to help here.
> > >
> > >
> > >
> > > Hmm i was bitten by this 2 weeks ago, i found it a bit not intuitive that:
> > > - While the guest i was trying to start required less memory than was freely available (according
> to
> > > xentop) outside of dom0
> > > - It would fail, because xl started to try to balloon dom0 down which failed.
> >
> > Hi Sander --
> >
> > I could be wrong (and I am confident someone will correct me if I am) but
> > I think this is because the Citrix memory model assumes there is an
> > inference-driven policy engine for load-balancing memory across competing
> > virtual machines ("squeezed").  I suspect squeezed returns unallocated
> > xen "free" memory to dom0.

I forgot... it is called Dynamic Memory Control (DMC), not squeezed
in the XenServer product.

> > IMHO, such policy engines are good for demos and so salespeople can
> > say "yes, this product supports memory overcommit" but Transcendent
> > Memory goes quite a bit further (albeit not for guests with proprietary
> > OS's).
> 
> Afaik XenServer/XCP and XenClient both use dom0_mem= option for Xen.

Hmmm... looks like you are right.

Dan