Re: [PATCH] x86: expose XENMEM_get_pod_target to subject domain

[George Dunlap <george.dunlap@eu.citrix.com>]

On 04/02/2014 09:52 AM, Jan Beulich wrote:
>>>> On 01.04.14 at 19:05, <George.Dunlap@eu.citrix.com> wrote:
>> On Tue, Apr 1, 2014 at 5:55 PM, George Dunlap
>> <George.Dunlap@eu.citrix.com> wrote:
>>> So the basic issue, regardless of PoD, is that the guest is told to
>>> balloon down to X, by handing back (T - X) pages to Xen.  But it
>>> doesn't actually know what T is, for a number of reasons; mainly due
>>> to having holes in lowmem for vga ram and things like that.  When PoD
>>> is enabled, this may cause a guest to end up killed as a result of
>>> accidentally not ballooning down enough; but even on non-PoD systems,
>>> this would lead to a small mismatch between the toolstack's idea of
>>> how much a VM was using (or should be using) and how much the guest is
>>> using.
>>>
>>> It looks like libxl anyway already writes "static-max" into xenstore,
>>> right next to the balloon target.  Is there any reason we can't use
>>> the toolstack-written static-max to calculate how big a balloon we
>>> need to inflate?
>>>
>>> If it's not accruate or consistent at the moment, we should be able to
>>> make it consistent going forward.
>> One potential problem I see at the moment is that during domain
>> creation, "static-max" is set to maxmem, but "target" is set to
>> "memory-video_memkb"; but the actual number of pages will be memory -
>> VGA_HOLE, which probably won't match either one.  Then in
>> libxl_set_memory_target(), it's just straight-up set to "target".
>> This may lead to a rather unexpected situation where after starting
>> with memory=2048, setting the target to 1024, and then setting it back
>> to 2048, the guest has a different amount of memory than it started
>> with.  (Or perhaps the balloon driver will be running up against the
>> memory ceiling.)
>>
>> It would be ideal if the balloon driver could just release (static-max
>> - target) memory and be assured that everything would be OK.
> Another problem is that "target" (or in fact anything in xenstore) isn't
> tightly coupled with what the hypervisor knows for the domain. Yet
> the main goal (preventing to be killed due to ballooning down to much)
> depends on only the hypervisor's knowledge, and hence I think that
> only the hypervisor should be consulted to find out.

I understand the sentiment; but as I said, the real problem is a lack of 
clarity about what exactly the toolstack is asking the VM to do.  This 
is obviously a particular problem in the case of PoD, but it's still a 
problem even for non-PoD guests; it's just that the outcomes are less 
severe.  If we solve the general problem, we'll solve the PoD problem.

The other thing is that the whole point of PoD is to be transparent to 
the guest.  Xen is already careful in how it handles post-creation 
adjustments to the PoD size -- always increasing the PoD cache, never 
decreasing it -- specifically so that the guest doesn't need to know.

What should really happen at some point is for PoD to just become a 
special case of swapping.  In a sense, that's almost the same issue: you 
could have a situation where the toolstack asks a guest to balloon down, 
and the guest does so; but not as low as the toolstack expected, so the 
toolstack labels the guest as "misbehaving" and tells Xen to swap out 
pages until it reaches what the toolstack thinks is the correct value.  
The guest won't crash, but performance will be impacted.

The target in xenstore could be made tightly coupled: if the toolstack 
always wrote into xenstore exactly what it reported to Xen, then it 
would be the same.

Alternately, since now Xen is involved with ballooning targets -- 
whether you're doing PoD or swapping -- maybe we should consider moving 
the "target" into Xen entirely.  Then there would be no chance for 
"drift", as Xen and the balloon driver would be working from the same 
data.  This would be basically repurposing the get/set pod_target 
hypercall to something specifically for ballooning.

  -George