Re: HV KVM fails on 970 due to HTAB allocation

[Paul Mackerras <paulus@samba.org>]

On Thu, Jan 12, 2012 at 07:16:51PM +0100, Alexander Graf wrote:

> While trying to run HV KVM for something useful on 970, we stumbled
> over the following code path:
> 
>         /* Allocate guest's hashed page table */
>         hpt =
> __get_free_pages(GFP_KERNEL|__GFP_ZERO|__GFP_REPEAT|__GFP_NOWARN,
>                                HPT_ORDER - PAGE_SHIFT);
>         if (!hpt) {
>                 pr_err("kvm_alloc_hpt: Couldn't alloc HPT\n");
>                 return -ENOMEM;
>         }
>         kvm->arch.hpt_virt = hpt;
> 
> We're most of the time running into the !hpt case, because we simply
> don't have 16MB of contiguous memory lying around.
> 
> I was trying to check if we could maybe allocate a huge_tlb page
> from within kernel space, since that usually matches the 16MB pretty
> well. However that seems to be very tricky. Maybe something similar
> to the RMA thing would be a good idea?

In discussing this with David Gibson in the past, one idea we have had
is to have userspace allocate the HPT using hugetlbfs and supply it to
KVM via an ioctl.  If userspace doesn't call that ioctl then we try to
do a high-order allocation, as at present, when they do the first
VCPU_RUN ioctl.

The other thing the code could do is to fall back to lower-order
allocations.  The HPT doesn't have to be 16MB in size; any power of 2
that is at least 256kB will do (there is an upper limit, but it is
enormous).  Smaller sizes will potentially reduce performance, of
course (and the size of the VRMA on POWER7, but on 970 we have to use
an RMO region, which isn't affected by the HPT size).

Paul.