Re: [PATCH 0/3] KVM-userspace: add NUMA support for guests

[Avi Kivity <avi@redhat.com>]

Andi Kleen wrote:
>>> I was more thinking about some heuristics that checks when a page
>>> is first mapped into user space. The only problem is that it is zeroed
>>> through the direct mapping before, but perhaps there is a way around it. 
>>> That's one of the rare cases when 32bit highmem actually makes things 
>>> easier.
>>> It might be also easier on some other OS than Linux who don't use
>>> direct mapping that aggressively.
>>>  
>>>       
>> In the context of kvm, the mmap() calls happen before the guest ever 
>>     
>
> The mmap call doesn't matter at all, what matters is when the
> page is allocated.
>
>   

The page is allocated at an uninteresting point in time.  For example, 
the boot loaded allocates a bunch of pages.

>> executes.  First access happens somewhat later, but still we cannot 
>> count on the majority of accesses to come from the same cpu as the first 
>> access.
>>     
>
> It is a reasonable heuristic. It's just like the rather
> successfull default local allocation heuristic the native kernel uses.
>   

It's very different.  The kernel expects an application that touched 
page X on node Y to continue using page X on node Y.  Because 
applications know this, they are written to this assumption.  However, 
in a virtualization context, the guest kernel expects that page X 
belongs to whatever node the SRAT table points at, without regard to the 
first access.

Guest kernels behave differently from applications, because real 
hardware doesn't allocate pages dynamically like the kernel can for 
applications.

(btw, what do you do with cpu-less nodes? I think some sgi hardware has 
them)

>>> The alternative is to keep your own pools and allocate from the
>>> correct pool, but then you either need pinning or getcpu()
>>>  
>>>       
>> This is meaningless in kvm context.  Other than small bits of memory 
>> needed for I/O and shadow page tables, the bulk of memory is allocated 
>> once. 
>>     
>
> Mapped once. Anyways that could be changed too if there was need.
>
>   

Mapped once and allocated once (not at the same time, but fairly close).

We can't change it without changing the guest.

>>> Basic algorithm:
>>> - If guest touches virtual node that is the same as the local node
>>> of the current vcpu assume it's a local allocation.
>>>  
>>>       
>> The guest is not making the same assumption; lying to the guest is 
>>     
>
> Huh? Pretty much all NUMA aware OS should. Linux will definitely.
>
>   

No.  Linux will assume a page belongs to the node the SRAT table says it 
belongs to.  Whether first access will be from the local node depends on 
the workload.  If the first application running accesses all memory from 
a single cpu, we will allocate all memory from one node, but this is wrong.

>> (2) even without npt/ept, we have no idea how often mappings are used 
>> and by which cpu.  finding out is expensive.
>>     
>
> You see a fault on the first mapping. That fault is on the CPU that
> did the access.  Therefore you know which one it was.
>   

It's meaningless information.  First access means nothing.  And again, 
the guest doesn't expect the page to move to the node where it touched it.

(we also see first access with ept)

>> (3) for many workloads, there are no unused pages.  the guest 
>> application allocates all memory and manages memory by itself.
>>     
>
> First a common case of guest using all memory is file cache,
> but for NUMA purposes file cache locality typically doesn't
> matter because it's not accessed frequently enough that
> non locality is a problem. It really only matters for mapping
> that are used often by the CPU.
>
> When a single application allocates everything and keeps it that is fine
> too because you'll give it approximately local memory on the initial
> set up (assuming the application has reasonable NUMA behaviour by itself
> on a first touch local allocation policy)
>   

Sure, for the simple cases it works.  But consider your first example 
followed by the second (you can even reboot the guest in the middle, but 
the bad assignment sticks).

And if the vcpu moves for some reason, things get screwed up permanently.

We should try to be predictable, not depend on behavior the guest has no 
real reason to follow, if it follows hardware specs.


-- 
error compiling committee.c: too many arguments to function