Please ack XENMEM_claim_pages hypercall?

Please ack XENMEM_claim_pages hypercall?

[Dan Magenheimer]

George, Ian, Ian, Tim --

After the very extended debate about Oracle's proposed
"memory capacity claim" hypercall, it seemed prudent to
first implement and demonstrate a working model.

A hypervisor patch (and a two-line libxc patch) has
now been completed and reviewed and Jan Beulich has found
it acceptable, so it is now time to return to the
discussion of whether the feature is needed.  The
patch description provides the detailed technical argument:
http://lists.xen.org/archives/html/xen-devel/2012-11/msg01427.html 

I believe, in the earlier debate, you were the most
vociferous opponents of this proposed hypercall,
primarily -- if I may attempt to summarize -- because
you believe that your paradigm of a how a toolstack
should manage memory ("omnisciently") renders the proposed
feature unnecessary.  I believe I adequately described
the differences between your paradigm and Oracle's, and
also demonstrated Oracle's customer need for the feature,
but the debate fell silent without your acknowledgement.

While it has always been my understanding that the
hypervisor is intended to be toolstack-independent,
Jan would like your Ack before committing the hypervisor
changes.

So if you still object, please state your objections.
Otherwise, please ack, so Jan can commit it and Oracle
can move forward with toolstack development built
on the hypercall.

Thanks,
Dan

Re: Please ack XENMEM_claim_pages hypercall?

[Jan Beulich]

>>> On 26.11.12 at 20:07, Dan Magenheimer <dan.magenheimer@oracle.com> wrote:
> While it has always been my understanding that the
> hypervisor is intended to be toolstack-independent,
> Jan would like your Ack before committing the hypervisor
> changes.
> 
> So if you still object, please state your objections.
> Otherwise, please ack, so Jan can commit it and Oracle
> can move forward with toolstack development built
> on the hypercall.

Sorry, there must have been some misunderstanding here: First
of all, without a maintainer's ack (Keir's in this case) I can't commit
anything to code that I'm not explicitly listed for as maintainer.

Second, while I said the code itself looks acceptable, I also pointed
out that in the shape it is right now it is dead code, as there's no
user for it. So all we would get would be the risk of new bugs (and
the one I just pointed out worries me in so far as how much testing
this code really has seen).

Third, deferral (or denial) of the patch going in is certainly not a
blocking factor for tools side development at Oracle. In the worst
case, you'd have to maintain the patch in your own tree(s); I do
realize that you want to avoid that (as I would, but there are
examples of patches that we carry in our trees that didn't get
accepted into the community one - luckily they're of smaller size).

Jan

Re: Please ack XENMEM_claim_pages hypercall?

[Keir Fraser]

On 27/11/2012 08:48, "Jan Beulich" <JBeulich@suse.com> wrote:

> Sorry, there must have been some misunderstanding here: First
> of all, without a maintainer's ack (Keir's in this case) I can't commit
> anything to code that I'm not explicitly listed for as maintainer.
> 
> Second, while I said the code itself looks acceptable, I also pointed
> out that in the shape it is right now it is dead code, as there's no
> user for it. So all we would get would be the risk of new bugs (and
> the one I just pointed out worries me in so far as how much testing
> this code really has seen).
> 
> Third, deferral (or denial) of the patch going in is certainly not a
> blocking factor for tools side development at Oracle. In the worst
> case, you'd have to maintain the patch in your own tree(s); I do
> realize that you want to avoid that (as I would, but there are
> examples of patches that we carry in our trees that didn't get
> accepted into the community one - luckily they're of smaller size).

It's actually not that large a patch, and perhaps we could take a
domain_adjust_tot_pages() hook which would reduce the size of any private
patch, while not inflicting a maintenance or bug burden on mainline.

 -- Keir

Re: Please ack XENMEM_claim_pages hypercall?

[Dan Magenheimer]

> From: Keir Fraser [mailto:keir@xen.org]
> Subject: Re: [Xen-devel] Please ack XENMEM_claim_pages hypercall?
> 
> On 27/11/2012 08:48, "Jan Beulich" <JBeulich@suse.com> wrote:
> 
> > Sorry, there must have been some misunderstanding here: First
> > of all, without a maintainer's ack (Keir's in this case) I can't commit
> > anything to code that I'm not explicitly listed for as maintainer.
> >
> > Second, while I said the code itself looks acceptable, I also pointed
> > out that in the shape it is right now it is dead code, as there's no
> > user for it. So all we would get would be the risk of new bugs (and
> > the one I just pointed out worries me in so far as how much testing
> > this code really has seen).
> >
> > Third, deferral (or denial) of the patch going in is certainly not a
> > blocking factor for tools side development at Oracle. In the worst
> > case, you'd have to maintain the patch in your own tree(s); I do
> > realize that you want to avoid that (as I would, but there are
> > examples of patches that we carry in our trees that didn't get
> > accepted into the community one - luckily they're of smaller size).
> 
> It's actually not that large a patch, and perhaps we could take a
> domain_adjust_tot_pages() hook which would reduce the size of any private
> patch, while not inflicting a maintenance or bug burden on mainline.

Thanks for your kind words.  That would be very helpful.
Also reserving the subops would be much more valuable.

Thanks,
Dan

Re: Please ack XENMEM_claim_pages hypercall?

[Keir Fraser]

On 27/11/2012 14:08, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:

>> It's actually not that large a patch, and perhaps we could take a
>> domain_adjust_tot_pages() hook which would reduce the size of any private
>> patch, while not inflicting a maintenance or bug burden on mainline.
> 
> Thanks for your kind words.  That would be very helpful.
> Also reserving the subops would be much more valuable.

Yes yes, reserving subop(s) is certainly doable. We've done it for other
vendors in the past, and happy to again, within reason.

 -- Keir

Re: Please ack XENMEM_claim_pages hypercall?

[Ian Campbell]

On Tue, 2012-11-27 at 08:48 +0000, Jan Beulich wrote:
> Second, while I said the code itself looks acceptable, I also pointed
> out that in the shape it is right now it is dead code, as there's no
> user for it. So all we would get would be the risk of new bugs (and
> the one I just pointed out worries me in so far as how much testing
> this code really has seen).

I agree with this.

> Third, deferral (or denial) of the patch going in is certainly not a
> blocking factor for tools side development at Oracle. In the worst
> case, you'd have to maintain the patch in your own tree(s); I do
> realize that you want to avoid that (as I would, but there are
> examples of patches that we carry in our trees that didn't get
> accepted into the community one - luckily they're of smaller size).

I think Oracle carrying this patch in their tree is probably the best
approach for now.

I'm also a little surprised that this patch is being so aggressively
pushed upstream when the toolstack work which would use it is seemingly
not fully formed yet.

Anyway, it seems to me that this argument seems to me to be starting
from the wrong end, it starts from the hypercall and tries to justify it
based on requirements imposed by an toolstack which is presented as
something of a fixed black box from the xen-devel point of view, which
is not something I find particularly convincing.

So a possible alternative to Oracle carrying this patch long term is
that someone who understands Oracle's toolstack's requirements and
constraints takes over from Dan (who I think has said several times that
he is not familiar with all the details of the Oracle toolstack) as
advocate for finding a solution to the underlying issue here and can
engage xen-devel in a discussion about the design decisions involved
from the toolstack downwards. Either this leads to the proposed solution
in the hypervisor (or something similar) or it results in a completely
different solution which everyone is happy with (or I suppose it might
still end up with Oracle carrying this patch long term).

I also feel I should also point out that contrary to the claims in
http://lists.xen.org/archives/html/xen-devel/2012-11/msg01427.html and
elsewhere the acceptance or otherwise of this patch has nothing to do
with Citrix. Although some of the folks involved in the discussion are
employed by Citrix they are all members of the "platform team" which
operates independently, is concerned with the state of Xen.org provided
Xen bits and is not tied to any product team (Citrix or otherwise). So
this has nothing whatsoever to do with Citrix's plans to use this
mechanism (and such conspiracy theories IMHO add nothing to the
discussion). AFAIK no one who is involved with any of Citrix's products
has said anything at all in any thread on the matter one way or the
other.

Ian

Re: Please ack XENMEM_claim_pages hypercall?

[Tim Deegan]

At 10:31 +0000 on 27 Nov (1354012308), Ian Campbell wrote:
> On Tue, 2012-11-27 at 08:48 +0000, Jan Beulich wrote:
> > Second, while I said the code itself looks acceptable, I also pointed
> > out that in the shape it is right now it is dead code, as there's no
> > user for it. So all we would get would be the risk of new bugs (and
> > the one I just pointed out worries me in so far as how much testing
> > this code really has seen).
> 
> I agree with this.

Me too (and with Jan and Ian's other points).

> > Third, deferral (or denial) of the patch going in is certainly not a
> > blocking factor for tools side development at Oracle. In the worst
> > case, you'd have to maintain the patch in your own tree(s); I do
> > realize that you want to avoid that (as I would, but there are
> > examples of patches that we carry in our trees that didn't get
> > accepted into the community one - luckily they're of smaller size).
> 
> I think Oracle carrying this patch in their tree is probably the best
> approach for now.

Agreed.  We can always take a patch reserving hypercall numbers for now,
so there's no risk that an Oracle private patch will clash with an
upstream allocation later on.

Tim.

Re: Please ack XENMEM_claim_pages hypercall?

[Dan Magenheimer]

> From: Ian Campbell [mailto:Ian.Campbell@citrix.com]
> Sent: Tuesday, November 27, 2012 3:32 AM
> To: Jan Beulich
> Cc: Dan Magenheimer; George Dunlap; Ian Jackson; xen-devel@lists.xen.org; Konrad Wilk; Zhigang Wang;
> Keir (Xen.org); Tim (Xen.org)
> Subject: Re: Please ack XENMEM_claim_pages hypercall?

Hi Ian --
 
> I'm also a little surprised that this patch is being so aggressively
> pushed upstream when the toolstack work which would use it is seemingly
> not fully formed yet.

The deficiency in the Oracle toolstack has been known for years
and this solution has been discussed and approved at the VP level.
 
> Anyway, it seems to me that this argument seems to me to be starting
> from the wrong end, it starts from the hypercall and tries to justify it
> based on requirements imposed by an toolstack which is presented as
> something of a fixed black box from the xen-devel point of view, which
> is not something I find particularly convincing.

I fully recognize that the existence of the black box is annoying.
But the argument began not with a hypercall but with a real
customer problem.  The early discussion, in which others suggested
"change your toolstack", uncovered a rather dramatic paradigm
difference that re-emphasized the need for the hypercall.
See below.

> So a possible alternative to Oracle carrying this patch long term is
> that someone who understands Oracle's toolstack's requirements and
> constraints takes over from Dan (who I think has said several times that
> he is not familiar with all the details of the Oracle toolstack) as
> advocate for finding a solution to the underlying issue here and can
> engage xen-devel in a discussion about the design decisions involved
> from the toolstack downwards. Either this leads to the proposed solution
> in the hypervisor (or something similar) or it results in a completely
> different solution which everyone is happy with (or I suppose it might
> still end up with Oracle carrying this patch long term).

>From a single-system-xl-toolstack-centric perspective ("paradigm"),
I can see your point.  This is not an xl-toolstack-centric problem.

> I also feel I should also point out that contrary to the claims in
> http://lists.xen.org/archives/html/xen-devel/2012-11/msg01427.html and
> elsewhere the acceptance or otherwise of this patch has nothing to do
> with Citrix. Although some of the folks involved in the discussion are
> employed by Citrix they are all members of the "platform team" which
> operates independently, is concerned with the state of Xen.org provided
> Xen bits and is not tied to any product team (Citrix or otherwise). So
> this has nothing whatsoever to do with Citrix's plans to use this
> mechanism (and such conspiracy theories IMHO add nothing to the
> discussion). AFAIK no one who is involved with any of Citrix's products
> has said anything at all in any thread on the matter one way or the
> other.

Conspiracy theory?  Some old guy from your neck of the woods said
something like: "Methinks, the maintainer doth protest too much." ;-)

I apologize if my words have implied that maintainers employed
by Citrix are intentionally making decisions that favor Citrix.
I _am_ implying, however, that the single-system-inference/policy-driven
memory-load-balancer paradigm _has_ influenced the open source
hypervisor _and_ heavily influenced the objections to this proposal.
AFAIK, Citrix's Dynamic Memory Controller (DMC) in XenServer is
the only shipping example (in the Xen universe) of that, so I call
it the "Citrix paradigm".  Xen decisions made with this paradigm
in mind heavily favor a single-system model, which lead to
solutions to problems which are not as applicable to a data-center
paradigm such as Oracle's.  I _do_ consider this a serious issue
and suggest that the maintainers consider it deeply because
the hypervisor must serve both paradigms.

Now, with that in mind, I still haven't heard any objections
other than insufficient testing (fair, but also very true of
most patches accepted into xen-unstable months before a release);
and "your toolstack should use the 'Citrix paradigm'" (and
I believe I have adequately explained why we cannot).  Are
there other valid objections?

Thanks,
Dan

Re: Please ack XENMEM_claim_pages hypercall?

[George Dunlap]

On 27/11/12 14:38, Dan Magenheimer wrote:
> Now, with that in mind, I still haven't heard any objections
> other than insufficient testing (fair, but also very true of
> most patches accepted into xen-unstable months before a release);
> and "your toolstack should use the 'Citrix paradigm'" (and
> I believe I have adequately explained why we cannot).  Are
> there other valid objections?

Yes -- the main one being, you have not convinced anyone it's necessary 
yet.  Maybe you should ask someone else to take up this cause -- we seem 
to have major problems communicating with you.

  -George

Re: Please ack XENMEM_claim_pages hypercall?

[Ian Jackson]

Dan Magenheimer writes ("RE: Please ack XENMEM_claim_pages hypercall?"):
> From a single-system-xl-toolstack-centric perspective ("paradigm"),
> I can see your point.

I don't think this is the case.  What you are doing is putting this
node-specific claim functionality in the hypervisor.  I still think it
should be done outside the hypervisor.  This does not mean that there
has to be a single omniscient piece of software for an entire
cluster.

It just means that there has to be a single omniscient piece of
software _for a particular host_.  In your proposal that piece of
software is the hypervisor - specifically, the part of the hypervisor
that does the bookkeeping of these claims.  However I am still
unconvinced that this can't be implemented outside the hypervisor.


In general it is a key design principle of a system like Xen that the
hypervisor should provide only those facilities which are strictly
necessary.  Any functionality which can be reasonably provided outside
the hypervisor should be excluded from it.  It is this principle which
you are running up against.


It may be (and it seems likely from what you've said in private email)
that some of your existing guests make some assumptions about the
semantics of the Xen ballooning memory interface, which might be
violated by such a design.  Specifically it appears that your design
has guests deliberately balloon down further than requested by the
toolstack, in a kind of attempt to negotiate with other users of
memory on the system by actually releasing and claiming Xen memory.

But I don't think it's reasonable to demand that the shared codebase
reflect such undocumented prior assumptions.  Particularly when this
design seems poor.  I find it poor because (a) using actual memory
allocation and release provides only a very impoverished signalling
mechanism (b) it imposes on every part of the system the possibility
of unexpected memory allocation failure.  (Your claim hypercall is an
attempt to mitigate (b) for some but not all cases.)

It seems to me that the correct approach is to design and implement a
new interface which allows a guest (whether that be its kernel or an
agent) to conduct a richer negotiation with out-of-hypervisor
toolstack software.  But that toolstack software (which might take any
particular shape - certainly we don't want to make any assumptions
about its policies and nature) needs to be sufficiently aware of the
claims and arbitrate between them, in a way that arranges that guests
which obey the rules never see an "out of memory" from the hypervisor.

Of course if it really is desired to have each guest make its own
decisions and simply for them to somehow agree to divvy up the
available resources, then even so a new hypervisor mechanism is not
needed.  All that is needed is a way for those guests to synchronise
their accesses and updates to shared records of the available and
in-use memory.


> AFAIK, Citrix's Dynamic Memory Controller (DMC) in XenServer is
> the only shipping example (in the Xen universe) of that,

I have never worked on the XenServer codebase and I have no clear idea
what this "Dynamic Memory Controller" is.  No-one here has explained
it to me and I have no particular desire to know about it.  Its
design, and its requirements or lack of them, have not influenced my
opinion on your proposals.

As an example to demonstrate that the reaction you are seeing is
nothing to do with whether the originators of the proposal are inside
our outside Citrix, please refer to our cool reaction to the v4v
proposals which also involved new hypervisor functionality.  There
too, we require the case to be made: we need to be able to see that an
out-of-hypervisor approach is not sufficient.


>    Xen decisions made with this paradigm
> in mind heavily favor a single-system model,

Nothing in my end of this conversation is predicated on any particular
deployment paradigm.  It is clear that in both your proposal and my
counter-proposal[1] there is a single place in each host where memory
allocation decisions are made and in particular where the memory needs
of competing guests etc. are arbitrated.

In your proposal this place is in the hypervisor and the negotiation
between the competing resource users is "grab the memory if you want
to".  Naturally an in-hypervisor arbitration facility has to be very
simple and a sophisticated policy is difficult to apply.

In my counter-proposal this negotiation occurs between the guest and
an out-of-hypervisor per-host arbitrator of some kind.

I think you are going to say that in your system the guests decide for
themselves how much memory to claim based on their views of how much
is free, and whether their allocations fail.  However, there is no
particular reason why the information about how much memory is free,
and how much has been committed for each purpose, could not be
collected somewhere outside the hypervisor.

[1] I don't have a detailed counter-proposal design of course, but
that's mostly because the information and reasoning you have provided
about your objectives and constraints is rather vague.


I agree with George that you should consider allowing someone else to
have a go at explaining things to us.  If this new hypercall is indeed
needed then all that is required is a clear and logical explanation of
why this is so.  I'm sorry to say that your efforts in this direction
so far have not been sufficient, and I feel that our attempts to
elicit explanations from you have not been as successful as needed.

I would love to help Oracle out by solving this problem which is
evidently causing a lot of trouble.  But it's difficult, and in
particular we do seem to be having serious trouble communicating with
you.


Sorry,
Ian.

Re: Please ack XENMEM_claim_pages hypercall?

[Dan Magenheimer]

> From: Ian Jackson [mailto:Ian.Jackson@eu.citrix.com]
> Subject: RE: Please ack XENMEM_claim_pages hypercall?
> 
> Dan Magenheimer writes ("RE: Please ack XENMEM_claim_pages hypercall?"):
> > From a single-system-xl-toolstack-centric perspective ("paradigm"),
> > I can see your point.
> 
> I don't think this is the case.  What you are doing is putting this
> node-specific claim functionality in the hypervisor.  I still think it
> should be done outside the hypervisor.  This does not mean that there
> has to be a single omniscient piece of software for an entire
> cluster.
>
>  <remainder deleted>

Hi Ian --

Thanks for taking the time to write a detailed response.
I realized that I had promised you a complete summary
of the problem and alternate solutions; but then I got
involved in demonstrating and refining a prototype
and failed to deliver.  I had thought the patch
summary would serve that purpose, but now see that it
is not sufficiently comprehensive.

So in an attempt to fulfill my promise and provide
all of the necessary information you may require, I
am in the process of writing up a complete summary
and will send it on a new thread, hopefully later
today (US time).

Dan

Re: Please ack XENMEM_claim_pages hypercall?

[Dan Magenheimer]

> From: Jan Beulich [mailto:JBeulich@suse.com]
> Subject: Re: Please ack XENMEM_claim_pages hypercall?

Hi Jan --

> >>> On 26.11.12 at 20:07, Dan Magenheimer <dan.magenheimer@oracle.com> wrote:
> > While it has always been my understanding that the
> > hypervisor is intended to be toolstack-independent,
> > Jan would like your Ack before committing the hypervisor
> > changes.
> >
> > So if you still object, please state your objections.
> > Otherwise, please ack, so Jan can commit it and Oracle
> > can move forward with toolstack development built
> > on the hypercall.
> 
> Sorry, there must have been some misunderstanding here: First
> of all, without a maintainer's ack (Keir's in this case) I can't commit
> anything to code that I'm not explicitly listed for as maintainer.

Oops, sorry.  Since you contribute so widely to the hypervisor,
the maintainership division between you and Keir is not particularly 
clear.
 
> Second, while I said the code itself looks acceptable, I also pointed
> out that in the shape it is right now it is dead code, as there's no
> user for it. So all we would get would be the risk of new bugs (and
> the one I just pointed out worries me in so far as how much testing
> this code really has seen).

The proposed hypercall is dead code to the _xl_ toolstack.
As I said, I personally can write an xm/xend patch that uses it
(which will undoubtedly launch another firestorm, so I was
trying to avoid that).

But most importantly, many of the code changes _will_ be
tested without any toolstack changes at all, as the existing
toolstack (with no claim hypercalls) exercises the changes.
Perhaps this is the most important testing of all and,
as I understand it, is a primary purpose of xen-unstable.

I do agree though that it is difficult to get adequate testing
without a toolstack user.  I, too, would like to see that fixed. :-(

> Third, deferral (or denial) of the patch going in is certainly not a
> blocking factor for tools side development at Oracle. In the worst
> case, you'd have to maintain the patch in your own tree(s); I do
> realize that you want to avoid that (as I would, but there are
> examples of patches that we carry in our trees that didn't get
> accepted into the community one - luckily they're of smaller size).

I guess I've failed to make a very important point because
I thought it was obvious... Because of the addition of the
hypercall subops, this is an ABI change.  I think you would
agree that maintaining an ABI change out-of-tree is much
more difficult than maintaining non-ABI changes out-of-tree.

If the hypercall subops are reserved, whether the remainder
of the patch is accepted now or not, that might be a
reasonable compromise to Oracle.

Thanks,
Dan