Re: [PATCH 0/3] KVM-userspace: add NUMA support for guests

[Andi Kleen <andi@firstfloor.org>]

On Sun, Nov 30, 2008 at 05:38:34PM +0200, Avi Kivity wrote:
> Andi Kleen wrote:
> >>I don't think the first one works without the second.  Calling getcpu() 
> >>on startup is meaningless since the initial placement doesn't take the 
> >>    
> >
> >Who said anything about startup? The idea behind getcpu() is to call
> >it every time you allocate someting.
> >
> >  
> 
> Qemu only allocates on startup (though of course the kernel actually 
> allocates the memory lazily).

> Please explain.  When would you call getcpu() and what would you do at 
> that time?

When the guest allocates on the node of its current CPU get memory on
the node pool getcpu() tells you it is running on. More tricky 
is handling guest explicitely accessing other node for NUMA policy
purposes, but in this case you can access the cache of the getcpu
information of other vcpus. 

This is roughly equivalent of getting a fresh new demand fault page,
but doesn't require to unmap/free/remap.

The tricky bit is probably figuring out what is a fresh new page for
the guest. That might need some paravirtual help.

It's an approximate scheme, I don't know how well it would really
work. 

> >I think I would prefer to fix that in the kernel. user space will never
> >have the full picture.
> >  
> 
> On the other hand, getting everyone happy so this can get into the 
> kernel will be very difficult.  Many workloads will lose from this; 
> we're trying to balance both memory affinity and cpu balancing, and each 
> workload has a different tradeoff.

Yes it's unlikely to be a win in general, at least on small systems
with moderate NUMA factor. I expect KVM would need to turn it on with 
some explicit hints.

-Andi

-- 
ak@linux.intel.com