Re: domain creation vs querying free memory (xend and xl)

[Olaf Hering <olaf@aepfle.de>]

On Mon, Oct 01, Dan Magenheimer wrote:

> > From: Ian Jackson [mailto:Ian.Jackson@eu.citrix.com]
> > Sent: Friday, September 28, 2012 11:12 AM
> > To: Dan Magenheimer
> > Cc: xen-devel@lists.xen.org; Kurt Hackel; Konrad Wilk
> > Subject: Re: [Xen-devel] domain creation vs querying free memory (xend and xl)
> > 
> > Dan Magenheimer writes ("[Xen-devel] domain creation vs querying free memory (xend and xl)"):
> > > But the second domain launch fails, possibly after
> > > several minutes because, actually, there isn't enough
> > > physical RAM for both.
> > 
> > This is a real problem.  The solution is not easy, and may not make it
> > for 4.3.  It would involve a rework of the memory handling code in
> > libxl.
> 
> [broadening cc to "Xen memory technology people", please forward/add
> if I missed someone]

Dan,

I'm sure there has been already alot of thought and discussion about
this issue, So here are my thoughts:

In my opinion the code which is about to start a domain has to take all
currently created/starting/running/dying domains, and their individual
"allocation behaviour", into account before it can finally launch the
domain. All of this needs math, not locking.

A domain (domU or dom0) has a couple of constraints:

- current nr_pages vs. target_nr_pages vs. max_pages
- current PoD allocation vs. max_PoD
- current paged_pages vs. target_nr_pages vs. max_paged_pages
- some shared_pages
- some tmem
- maybe grant_pages
- ...

Depending on the state (starting and working towards a target number,
running, dying) the "current" numbers above will increase or shrink. So
the algorithm which turns the parameters above for each domain into a
total number of allocated (or soon to be allocated) host memory has to
work with "target numbers" instead of what is currently allocated.

Some examples that come to mind:
- a PoD domain will most likely use all of the pages configured with
  memory=, so that number should be used
- the number shared pages is eventually not predictable. If so, this
  number could be handled as "allocated to the guest". Maybe a knob to
  say "running domains will have amount N shared" can exist? Dont know
  much about how sharing looks in practice.
- ballooning may not reach the configured target, and the guest admin
  can just balloon up to the limit without notifying the toolstack
- a new paging target will take some time until its reached, there is
  always some jitter during page-in/page-out, mapping guest pages will
  cause nomination failures.
- tmem does something, I dont know.
- no idea if grant pages are needed in the math


Since the central management of xend is gone each libxl process is
likely on its own, so two "xl create" can race when doing the math.
Maybe a libxl process dies and leaves a mess behind. So that could make
it difficult to get a good snapshot of the memory situation on the host.
Maybe each domain could get some metadata to record the individual
current/target/max numbers. Or if xenstore is good enough, something can
cleanup zombie numbers.

As IanJ said, the memory handling code in libxl needs such a feature to
do the math right. The proposed handling of
sharing/paging/ballooning/PoD/tmem/... in libxl is just a small part of
it.


Olaf