Re: Proposed XENMEM_claim_pages hypercall: Analysis of problem and alternate solutions

[Dan Magenheimer <dan.magenheimer@oracle.com>]

> From: Ian Campbell [mailto:Ian.Campbell@citrix.com]
> 
> On Thu, 2013-01-03 at 18:49 +0000, Dan Magenheimer wrote:
> >
> > Well, perhaps my statement is a bit heavy-handed, but I don't see
> > how it ends the discussion... you simply need to prove my statement
> > incorrect! ;-)  To me, that would mean pointing out any existing
> > implementation or even university research that successfully
> > predicts or externally infers future memory demand for guests.
> > (That's a good approximation of my definition of an omniscient
> > toolstack.)
> 
> I don't think a solution involving massaging of tot_pages need involve
> either frequent changes to tot_pages nor omniscience from the tool
> stack.
> 
> Start by separating the lifetime_maxmem from current_maxmem. The
> lifetime_maxmem is internal to the toolstack (it is effectively your
> tot_pages from today) and current_maxmem becomes whatever the toolstack
> has actually pushed down into tot_pages at any given time.
> 
> In the normal steady state lifetime_maxmem == current_maxmem.
> 
> When you want to claim some memory in order to start a new domain of
> size M you *temporarily* reduce current_maxmem for some set of domains
> on the chosen host and arrange that the total of all the current_maxmems
> on the host is such that "HOST_MEM - SUM(current_maxmems) > M".
> 
> Once the toolstack has built (or failed to build) the domain it can set
> all the current_maxmems back to their lifetime_maxmem values.
> 
> If you want to build multiple domains in parallel then M just becomes
> the sum over all the domains currently being built.

Hi Ian --

Happy New Year!

Perhaps you are missing an important point that is leading
you to oversimplify and draw conclusions based on that
oversimplification...

We are _primarily_ discussing the case where physical RAM is
overcommitted, or to use your terminology IIUC:

   SUM(lifetime_maxmem) > HOST_MEM

Thus:

> In the normal steady state lifetime_maxmem == current_maxmem.

is a flawed assumption, except perhaps as an initial condition
or in systems where RAM is almost never a bottleneck.

Without that assumption, in your model, the toolstack must
make intelligent policy decisions about how to vary
current_maxmem relative to lifetime_maxmem, across all the
domains on the system.  Since the memory demands of any domain
often vary frequently, dramatically and unpredictably (i.e.
"spike") and since the performance consequences of inadequate
memory can be dire (i.e. "swap storm"), that is why I say the
toolstack (in your model) must both make frequent changes
to tot_pages and "be omniscient".

FWIW, I fully acknowledge that your model works fine when
there are no memory overcommitment technologies active.
I also acknowledge that your model is the best that can
be expected with legacy proprietary domains.  The Oracle
model however assumes both that RAM is frequently a bottleneck,
and that open-source guest kernels can intelligently participate
in optimizing their own memory usage; such guest kernels are
now shipping.

So, Ian, would you please acknowledge that the Oracle model
is valid and, in such cases where your maxmem assumption
is incorrect, that hypervisor-controlled capacity allocation
(i.e. XENMEM_claim_pages) is an acceptable solution?

Thanks,
Dan