Re: Proposed XENMEM_claim_pages hypercall: Analysis of problem and alternate solutions

[Tim Deegan <tim@xen.org>]

Hi,

At 17:17 -0500 on 18 Dec (1355851071), Konrad Rzeszutek Wilk wrote:
> In essence, the max_pages does work - _if_ one does these operations
> in serial. We are trying to make this work in parallel and without
> any failures - for that we - one way that is quite simplistic
> is the claim hypercall. It sets up a 'stake' of the amount of
> memory that the hypervisor should reserve. This way other
> guests creations/ ballonning do not infringe on the 'claimed' amount.
> 
> I believe with this hypercall the Xapi can be made to do its operations
> in parallel as well.

The question of starting VMs in parallel seems like a red herring to me:
- TTBOMK Xapi already can start VMs in parallel.  Since it knows what
  constraints it's placed on existing VMs and what VMs it's currently
  building, there is nothing stopping it.  Indeed, AFAICS any toolstack
  that can guarantee enough RAM to build one VM at a time could do the
  same for multiple parallel builds with a bit of bookkeeping.
- Dan's stated problem (failure during VM build in the presence of
  unconstrained guest-controlled allocations) happens even if there is
  only one VM being created.

> > > Andres Lagar-Cavilla says "... this is because of shortcomings in the
> > > [Xen] mm layer and its interaction with wait queues, documented
> > > elsewhere."  In other words, this batching proposal requires
> > > significant changes to the hypervisor, which I think we
> > > all agreed we were trying to avoid.
> >
> > Let me nip this at the bud. I use page sharing and other techniques in an environment that doesn't use Citrix's DMC, nor is focused only on proprietary kernels...
> 
> I believe Dan is saying is that it is not enabled by default.
> Meaning it does not get executed in by /etc/init.d/xencommons and
> as such it never gets run (or does it now?) - unless one knows
> about it - or it is enabled by default in a product. But perhaps
> we are both mistaken? Is it enabled by default now on xen-unstable?

I think the point Dan was trying to make is that if you use page-sharing
to do overcommit, you can end up with the same problem that self-balloon
has: guest activity might consume all your RAM while you're trying to
build a new VM.

That could be fixed by a 'further hypervisor change' (constraining the
total amount of free memory that CoW unsharing can consume).  I suspect
that it can also be resolved by using d->max_pages on each shared-memory
VM to put a limit on how much memory they can (severally) consume.

> Just as a summary as this is getting to be a long thread - my
> understanding has been that the hypervisor is suppose to toolstack
> independent.

Let's keep calm.  If people were arguing "xl (or xapi) doesn't need this
so we shouldn't do it" that would certainly be wrong, but I don't think
that's the case.  At least I certainly hope not!

The discussion ought to be around the actual problem, which is (as far
as I can see) that in a system where guests are ballooning without
limits, VM creation failure can happen after a long delay.  In
particular it is the delay that is the problem, rather than the failure.
Some solutions that have been proposed so far:
 - don't do that, it's silly (possibly true but not helpful);
 - this reservation hypercall, to pull the failure forward;
 - make allocation faster to avoid the delay (a good idea anyway,
   but can it be made fast enough?);
 - use max_pages or similar to stop other VMs using all of RAM.

My own position remains that I can live with the reservation hypercall,
as long as it's properly done - including handling PV 32-bit and PV
superpage guests.

Cheers,

Tim.