Re: [PATCH RFC 00/10] Xen balloon page compaction support

[Ian Campbell <ian.campbell@citrix.com>]

On Wed, 2014-10-15 at 18:14 +0100, Andrew Cooper wrote:
> On 15/10/14 18:00, Wei Liu wrote:
> > On Wed, Oct 15, 2014 at 05:54:24PM +0100, Andrew Cooper wrote:
> >> On 15/10/14 16:54, Wei Liu wrote:
> >>> Hi all
> >>>
> >>> This is a prototype to make Xen balloon driver work with balloon page
> >>> compaction. The goal is to reduce guest physical address space fragmentation
> >>> introduced by balloon pages. Having guest physical address space as contiguous
> >>> as possible is generally useful (because guest can have higher order pages), and
> >>> it should also help improve HVM performance (provided that guest kernel knows
> >>> how to use huge pages -- Linux has hugetlbfs and transparent huge page).
> >> After you have defragmented guest physical memory, does Linux use
> >> 2MB/1GB superpages more readily?
> >>
> > That's completely up to the guest. Having contiguous guest physical
> > address space is prerequisite.
> >
> >> On what basis do you think this will improve HVM performance?  The HAP
> >> tables still remain fragmented after ballooning.
> >>
> > That's the other side of this problem and orthogonal to this patch
> > series. It should be fixed separately on the hypervisor side, presumably
> > with similar mechanism to coalesce HAP table in hypervisor.
> 
> You can't rearrange the memory of any domain with passthrough, or any
> subregion which is mapped by another domain.  Even if the underlying
> pages are in order,

That's fine. This work still improves things for plenty of other
domains.

I suspect with a suitable amount of cunning it might be possible to do
this for domains with passthrough using smmu. Whether it is worth the
time and effort (since I certain concede it won't be easy) I don't know.

> you can't coalesce 4K pages to 2M pages for any
> region containing a grant.

An *active* grant (i.e. something which is granted and actually used by
another domain. At least on ARM it is perfectly fine to grant a page in
the midst of a 2M mapping and no need to shatter on the granting domain.

> By far the best solution for both guest and hypervisor is for Xen to do
> its utmost to allocate superpages to start with, and then for the guest
> not to shoot holes in its physical address space in the first place.

Which having the guest try and ballooning in 2M increments help to
achieve, especially since e.g. grant mappings are placed into ballooned
out holes, meaning they aren't poking holes in other bits of the address
space.

> This would involve intelligently choosing which pages to balloon/grant
> first, rather than blindly choosing the first free page.

By trying to do 2M balloons first and also trying to coalesce any 4k
ballooned pages into 2M using THP we start to achieve this.

I don't think what you are suggesting is going to be practical within
the constraints of the kernel infrastructure.

Ian.