Re: [PATCH v2 08/16] xen: derive NUMA node affinity from hard and soft CPU affinity

[George Dunlap <george.dunlap@eu.citrix.com>]

On 14/11/13 16:30, Dario Faggioli wrote:
> On gio, 2013-11-14 at 15:21 +0000, George Dunlap wrote:
>>> Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
>>> diff --git a/xen/common/domain.c b/xen/common/domain.c
>>> @@ -373,8 +379,12 @@ void domain_update_node_affinity(struct domain *d)
>>>    
>>>        for_each_vcpu ( d, v )
>>>        {
>>> +        /* Build up the mask of online pcpus we have hard affinity with */
>>>            cpumask_and(online_affinity, v->cpu_hard_affinity, online);
>>>            cpumask_or(cpumask, cpumask, online_affinity);
>>> +
>>> +        /* As well as the mask of all pcpus we have soft affinity with */
>>> +        cpumask_or(cpumask_soft, cpumask_soft, v->cpu_soft_affinity);
>> Is this really the most efficient way to do this?  I would have thought
>> or'ing cpumask and cpumask_soft, and then if it's not empty, then use
>> it; maybe use a pointer so you don't have to copy one into the other one?
>>
> I'm not sure I fully get what you mean... I cannot afford neglecting
> online_affinity, independently from how cpumask and cpumask_soft look
> like, because that's what's needed to account for cpupools. Anyway, I'll
> think more about it and see if I can make it better.

So what you have here is (in pseudocode):

  online_affinity = hard_affinity & online;
  cpumask |= online_affinity;
  cpumask_soft |= soft_affinity;

if ( intersects(cpumask, cpumask_soft) )
   cpumask &= cpumask_soft;

So at least four full bitwise operations, plus "intersects" which will 
be the equivalent of a full bitwise operation if it turns out to be false.

How about something like the following:

cpumask_hard = hard_affinity & online;
cpumask_soft = cpumask_hard & soft_affinity;

cpumask_p = is_empty(cpumask_soft) ? &cpumask_hard : &cpumask_soft;

So only two full bitwise operations, plus "is_empty", which I think 
should be faster than "intersects" even if it turns out to be true, 
since we can (I think) check full words at a time.

Then use cpumask_p in the subsequent node_affinity update.  (Or just do 
a full copy, since doing the nodewise check for intersection is going to 
be fairly slow anyway.)

  -George