Re: [PATCH 0/3] KVM-userspace: add NUMA support for guests

[Avi Kivity <avi@redhat.com>]

Andi Kleen wrote:
> On Sun, Nov 30, 2008 at 06:38:14PM +0200, Avi Kivity wrote:
>   
>> The guest allocates when it touches the page for the first time.  This 
>> means very little since all of memory may be touched during guest bootup 
>> or shortly afterwards.  Even if not, it is still a one-time operation, 
>> and any choices we make based on it will last the lifetime of the guest.
>>     
>
> I was more thinking about some heuristics that checks when a page
> is first mapped into user space. The only problem is that it is zeroed
> through the direct mapping before, but perhaps there is a way around it. 
> That's one of the rare cases when 32bit highmem actually makes things easier.
> It might be also easier on some other OS than Linux who don't use
> direct mapping that aggressively.
>   

In the context of kvm, the mmap() calls happen before the guest ever 
executes.  First access happens somewhat later, but still we cannot 
count on the majority of accesses to come from the same cpu as the first 
access.

>>> This is roughly equivalent of getting a fresh new demand fault page,
>>> but doesn't require to unmap/free/remap.
>>>  
>>>       
>> Lost again, sorry.
>>     
>
> free/unmap/remap gives you normally local memory. I tend to call
> it poor man's NUMA policy API.
>
> The alternative is to keep your own pools and allocate from the
> correct pool, but then you either need pinning or getcpu()
>   

This is meaningless in kvm context.  Other than small bits of memory 
needed for I/O and shadow page tables, the bulk of memory is allocated 
once.  Guest processes may repeatedly allocate and free memory, but kvm 
will never see this.

>> We need to mimic real hardware.
>>     
>
> The underlying allocation is in pages, so the NUMA affinity can 
> be as well handled by this. 
>
> Basic algorithm:
> - If guest touches virtual node that is the same as the local node
> of the current vcpu assume it's a local allocation.
>   

The guest is not making the same assumption; lying to the guest is 
counterproductive.  The big problem is that a local decision takes 
effect indefinitely.

> - On allocation get the underlying page from the correct underlying
> node based on a dynamic getcpu relationship.
> - Find some way to get rid of unused pages. e.g. keep track of 
> the number of mappings to a page and age or use pv help.
>
>   

(1) with npt/ept we have no clue as to guest mappings
(2) even without npt/ept, we have no idea how often mappings are used 
and by which cpu.  finding out is expensive.
(3) for many workloads, there are no unused pages.  the guest 
application allocates all memory and manages memory by itself.
>> The static case is simple.  We allocate memory from a few nodes (for 
>> small guests, only one) and establish a guest_node -> host_node 
>> mapping.  vcpus on guest node X are constrained to host node according 
>> to this mapping.
>>
>> The dynamic case is really complicated.  We can allow vcpus to wander to 
>> other cpus on cpu overcommit, but need to pull them back soonish, or 
>> alternatively migrate the entire node, taking into account the cost of 
>> the migration, cpu availability on the target node, and memory 
>> availability on the target node.  Since the cost is so huge, this needs 
>> to be done on a very coarse scale.
>>     
>
> I wrote a scheduler that did that on 2.4 (it was called homenode scheduling),
> but it never worked well on small systems. It was moderately successfull on
> some big NUMA boxes though. The fundamental problem is that not using
> a CPU is always worse than using remote memory on the small systems.
>
>   

Right.  The situation I'm trying to avoid is process A with memory on 
node X running on node Y, and process B with memory on node Y running on 
node X.  The scheduler arrives at a local optimum, caused by some 
spurious load, and won't move to the global optimum because migrating 
processes across cpus is considered expensive.

I don't know, perhaps the current scheduler is clever enough to do this 
already.

> Always migrating memory on CPU migration is also too costly in the general
> case, but it might be possible to make it work in the special case 
> of vCPU guests with some tweaks.
>   
Yes, virtual machines are easier since there are a smaller number of 
mm_structs and tasks compared to more general workloads.

-- 
error compiling committee.c: too many arguments to function