Re: [PATCH RESEND 05/12] xen: numa-sched: make space for per-vcpu node-affinity

[George Dunlap <george.dunlap@eu.citrix.com>]

On 06/11/13 15:14, Jan Beulich wrote:
>>>> On 06.11.13 at 15:56, George Dunlap <george.dunlap@eu.citrix.com> wrote:
>> On 06/11/13 14:26, Dario Faggioli wrote:
>>> On mer, 2013-11-06 at 11:44 +0000, George Dunlap wrote:
>>>> On 06/11/13 10:00, Dario Faggioli wrote:
>>>>> I see, and that sounds sensible to me... It's mostly a matter a matter
>>>>> of deciding whether o not we want something like that, and, if yes,
>>>>> whether we want it based on hard of soft.
>>>>>
>>>>> Personally, I think I agree with you on having it based on hard
>>>>> affinities by default.
>>>>>
>>>>> Let's see if George get to say something before I get to that part of
>>>>> the (re)implementation. :-)
>>>> I would probably have it based on soft affinities, since that's where we
>>>> expect to have the domain's vcpus actually running most if the time;
>>>>
>>> True. However, doing that would rule out cpupool and vcpu-pinning.
>> I guess I was assuming that a vcpu's soft affinity would always be
>> considered a subset of its hard affinity and the cpus in its cpupool.
> For CPU pools I agree, but didn't we mean hard affinity to control
> memory allocation, and soft affinity scheduling decisions (in which
> case there's no strict ordering between the two)? If not, I guess I'd
> need a brief but clear definition what "soft" and "hard" are supposed
> to represent...

We have two "affinities" at the moment:
* per-vcpu cpu affinity.  This is this is a scheduling construct, and is 
used to restrict vcpus to run on specific pcpus.  This is what we would 
call hard affinity -- "hard" because vcpus are *not allowed* to run on 
cpus not in this mask.
* Domain NUMA affinity.  As of 4.3 this has two functions: both a memory 
allocation function, and a scheduling function.  For the scheduling 
function, it acts as a "soft affinity", but it's domain-wide, rather 
than being per-vcpu.  It's "soft" because the scheduler will *try* to 
run it on pcpus in that mask, but if it cannot, it *will allow* them to 
run on pcpus not in that mask.

At the moment you can set this manually, or if it's not set, then Xen 
will set it automatically based on the vcpu hard affinities (and I think 
the cpupool that it's in).

What we're proposing is to remove the domain-wide soft affinity and 
replace it with a per-vcpu soft affinity.  That way each vcpu has some 
pcpus that it prefers (in the soft affinity *and* the hard affinity), 
some that it doesn't prefer but is OK with (hard affinity but not soft 
affinity), and some that it cannot run on at all (not in the hard affinity).

The question Dario has is this: given that we now have per-vcpu hard and 
soft scheduling affinity, how should we automatically construct the 
per-domain memory allocation affinity, if at all?  Should we construct 
it from the "hard" scheduling affinities, or from the "soft" scheduling 
affinities?

I said that I thought we should use the soft affinity; but I really 
meant the "effective soft affinity" -- i.e., the union of soft, hard, 
and cpupools.

  -George