Re: Linux Xen Balloon Driver Improvement (Draft 2)

[David Vrabel <david.vrabel@citrix.com>]

On 27/10/14 16:29, Wei Liu wrote:
> On Mon, Oct 27, 2014 at 02:23:22PM +0000, David Vrabel wrote:
>> On 27/10/14 12:33, Wei Liu wrote:
>>>
>>> Changes in this version:
>>>
>>> 1. Style, grammar and typo fixes.
>>> 2. Make this document Linux centric.
>>> 3. Add a new section for NUMA-aware ballooning.
>>
>> You've not included the required changes to the toolstack and
>> autoballoon driver to always use 2M multiples when creating VMs and
>> setting targets.
>>
> 
> When creating VM, toolstack already tries to use as many huge pages as
> possible.
> 
> Setting target doesn't use 2M multiples.  But I don't think this is
> necessary. To balloon in / out X MB memory
> 
>   nr_2m = X % 2M
>   nr_4k = (X / 2M) / 4k
> 
> The remainder just goes to 4K queue.

I understand that it will work with 4k multiples but it is not /optimal/
to do so since it will result in more fragmentation.

> And what do you mean by "autoballoon" driver? Do you mean functionality
> of xl? In the end the request is still fulfilled by Xen balloon driver
> in kernel. So if dom0 is using the new balloon driver proposed here, it
> should balloon down in 2M multiples automatically.

Both xl and the auto-balloon driver in the kernel should only set the
target in multiples of 2M.

>>> ## Goal of improvement
>>>
>>> The balloon driver makes use of as many huge pages as possible,
>>> defragmenting guest address space. Contiguous guest address space
>>> permits huge page ballooning which helps prevent host address space
>>> fragmentation.
>>>
>>> This should be achieved without any particular hypervisor side
>>> feature.
>>
>> I really think you need to be taking whole-system view and not focusing
>> on just the guest balloon driver.
>>
> 
> I don't think there's terribly tight linkage between hypervisor side
> change and guest side change.

I don't see how you can think this unless you also have a design for the
hypervisor side.

I do not want a situation were effective and efficient host
defragmentation requires balloon driver changes to avoid a regression.

> To have guest automatically defragmenting it's address space while at
> the same time helps prevent hypervisor memory from fragmenting (at least
> this is what the design aims for, as for how it works in practice, it
> needs to be prototyped and benchmarked).
> 
> The above reasoning is good enough to justify this change, isn't it?

Having a whole system design does not mean that it must be all
implemented.  If one part has benefits independently from the rest then
it can be implemented and merged.

>>> ### Make use of balloon page compaction
>>>
>>> The core of migration callback is XENMEM\_exchange hypercall. This
>>> makes sure that inflation of old page and deflation of new page is
>>> done atomically, so even if a domain is beyond its memory target and
>>> the target is being enforced, it can still compact memory.
>>
>> Having looked at what XENMEM_exchange actually does, I can't see how
>> you're using it to give this behaviour.

Never mind.  I misread the docs.

>>> ### Periodically exchange normal size pages with huge pages
>>>
>>> Worker thread wakes up periodically to check if there are enough pages
>>> in normal size page queue to coalesce into a huge page. If so, it will
>>> try to exchange that huge page into a number of normal size pages with
>>> XENMEM\_exchange hypercall.
>>
>> I don't see what this is supposed to achieve.  This is going to take a
>> (potentially) non-fragmented superpage and fragment it.
>>
> 
> Let's look at this from start of day.
> 
> Guest always tries to balloon in / out as many 2M pages as possible. So
> if we have a long list of 4K pages, it means the underlying host super
> frames are fragmented already.
> 
> So if 1) there are enough 4K pages in ballooned out list, 2) there is a
> spare 2M page, it means that the 2M page comes from the result of
> balloon page compaction, which means the underlying host super frame is
> fragmented.

This assumption is only true because your page migration isn't trying
hard enough to defragment super frames, and it is assuming that Xen does
nothing to address host super frame fragmentation.  This highlights the
importance of looking at a system-level for designs, IMO.

David