From mboxrd@z Thu Jan  1 00:00:00 1970
From: George Dunlap <george.dunlap@eu.citrix.com>
Subject: Re: [PATCH v2 08/16] xen: derive NUMA node affinity
 from hard and soft CPU affinity
Date: Fri, 15 Nov 2013 10:52:35 +0000
Message-ID: <5285FCF3.5030207@eu.citrix.com>
References: <20131113190852.18086.5437.stgit@Solace>	
	<20131113191214.18086.51921.stgit@Solace>
	<5284EA84.30506@eu.citrix.com> <1384446603.29902.170.camel@Abyss>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; Format="flowed"
Content-Transfer-Encoding: 7bit
Return-path: <xen-devel-bounces@lists.xen.org>
In-Reply-To: <1384446603.29902.170.camel@Abyss>
List-Unsubscribe: <http://lists.xen.org/cgi-bin/mailman/options/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=unsubscribe>
List-Post: <mailto:xen-devel@lists.xen.org>
List-Help: <mailto:xen-devel-request@lists.xen.org?subject=help>
List-Subscribe: <http://lists.xen.org/cgi-bin/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=subscribe>
Sender: xen-devel-bounces@lists.xen.org
Errors-To: xen-devel-bounces@lists.xen.org
To: Dario Faggioli <dario.faggioli@citrix.com>
Cc: Marcus Granado <Marcus.Granado@eu.citrix.com>, Keir Fraser <keir@xen.org>, Ian Campbell <Ian.Campbell@citrix.com>, Li Yechen <lccycc123@gmail.com>, Andrew Cooper <Andrew.Cooper3@citrix.com>, Juergen Gross <juergen.gross@ts.fujitsu.com>, Ian Jackson <Ian.Jackson@eu.citrix.com>, xen-devel@lists.xen.org, Jan Beulich <JBeulich@suse.com>, Justin Weaver <jtweaver@hawaii.edu>, Matt Wilson <msw@amazon.com>, Elena Ufimtseva <ufimtseva@gmail.com>
List-Id: xen-devel@lists.xenproject.org

On 14/11/13 16:30, Dario Faggioli wrote:
> On gio, 2013-11-14 at 15:21 +0000, George Dunlap wrote:
>>> Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
>>> diff --git a/xen/common/domain.c b/xen/common/domain.c
>>> @@ -373,8 +379,12 @@ void domain_update_node_affinity(struct domain *d)
>>>    
>>>        for_each_vcpu ( d, v )
>>>        {
>>> +        /* Build up the mask of online pcpus we have hard affinity with */
>>>            cpumask_and(online_affinity, v->cpu_hard_affinity, online);
>>>            cpumask_or(cpumask, cpumask, online_affinity);
>>> +
>>> +        /* As well as the mask of all pcpus we have soft affinity with */
>>> +        cpumask_or(cpumask_soft, cpumask_soft, v->cpu_soft_affinity);
>> Is this really the most efficient way to do this?  I would have thought
>> or'ing cpumask and cpumask_soft, and then if it's not empty, then use
>> it; maybe use a pointer so you don't have to copy one into the other one?
>>
> I'm not sure I fully get what you mean... I cannot afford neglecting
> online_affinity, independently from how cpumask and cpumask_soft look
> like, because that's what's needed to account for cpupools. Anyway, I'll
> think more about it and see if I can make it better.

So what you have here is (in pseudocode):

  online_affinity = hard_affinity & online;
  cpumask |= online_affinity;
  cpumask_soft |= soft_affinity;

if ( intersects(cpumask, cpumask_soft) )
   cpumask &= cpumask_soft;

So at least four full bitwise operations, plus "intersects" which will 
be the equivalent of a full bitwise operation if it turns out to be false.

How about something like the following:

cpumask_hard = hard_affinity & online;
cpumask_soft = cpumask_hard & soft_affinity;

cpumask_p = is_empty(cpumask_soft) ? &cpumask_hard : &cpumask_soft;

So only two full bitwise operations, plus "is_empty", which I think 
should be faster than "intersects" even if it turns out to be true, 
since we can (I think) check full words at a time.

Then use cpumask_p in the subsequent node_affinity update.  (Or just do 
a full copy, since doing the nodewise check for intersection is going to 
be fairly slow anyway.)

  -George