Re: Analysis of using balloon page compaction in Xen balloon driver

[David Vrabel <david.vrabel@citrix.com>]

On 16/10/14 18:12, Wei Liu wrote:
> This document analyses the impact of using balloon compaction
> infrastructure in Xen balloon driver.

Thanks for writing this.   This is a excellent starting point for a
productive design discussion.

> ## Benefit for auto-translated guest
> 
> HVM/PVH/ARM guest can have contiguous guest physical address space
> after balloon pages are compacted, which potentially improves memory
> performance provided guest makes use of huge pages, either via
> Hugetlbfs or Transparent Huge Page (THP).
> 
> Consider memory access pattern of these guests, one access to guest
> physical address involves several accesses to machine memory. The
> total number of memory accesses can be represented as:
> 
>> X = H1 * G1 + H2 * G2 + ... + Hn * Gn + 1
> 
> Hx denotes second stage page table walk levels and Gx denotes guest
> page table walk levels.
> 
> By having contiguous guest physical address, guest can make use of
> huge pages. This can reduce the number of G's in formula.
> 
> Reducing number of H's is another project for hypervisor side
> improvement and should be decoupled from Linux side changes.

Whilst this analysis is fine, I don't think this is the real benefit of
using superpages which is reducing TLB usage and reducing the number of
TLB misses.

With fragmented stage 2 tables I don't think you will see see much
improvement in TLB usage.

> ## HAP table fragmentation is not made worse

This reasoning looks reasonable to me.  But it suggests that the balloon
compaction isn't doing it's job properly.  It seems like it should be
much more proactive in resolving fragmentation.

> ## Beyond Linux balloon compaction infrastructure
> 
> Currently there's no mechanism in Xen to coalesce HAP table
> entries. To coalesce HAP entries we would need to make sure all
> discrete entries belong to one huge page, are in correct order and
> correct state.

I would like to see a more detailed description of the Xen-side
solution.  So we can be sure the Linux half is compatible with it.

Before accepting any series I would also need to see real world
performance improvements, not just theoretical ones.

David