From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from e32.co.us.ibm.com (e32.co.us.ibm.com [32.97.110.150]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "e32.co.us.ibm.com", Issuer "Equifax" (verified OK)) by ozlabs.org (Postfix) with ESMTPS id 9C690B72DE for ; Tue, 10 May 2011 07:26:37 +1000 (EST) Received: from d03relay02.boulder.ibm.com (d03relay02.boulder.ibm.com [9.17.195.227]) by e32.co.us.ibm.com (8.14.4/8.13.1) with ESMTP id p49LF9Hj011653 for ; Mon, 9 May 2011 15:15:09 -0600 Received: from d03av04.boulder.ibm.com (d03av04.boulder.ibm.com [9.17.195.170]) by d03relay02.boulder.ibm.com (8.13.8/8.13.8/NCO v9.1) with ESMTP id p49LQQ6P150166 for ; Mon, 9 May 2011 15:26:26 -0600 Received: from d03av04.boulder.ibm.com (loopback [127.0.0.1]) by d03av04.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id p49FQO6G019071 for ; Mon, 9 May 2011 09:26:25 -0600 Message-ID: <4DC85BFE.7060900@linux.vnet.ibm.com> Date: Mon, 09 May 2011 16:26:22 -0500 From: Jesse Larrew MIME-Version: 1.0 To: Benjamin Herrenschmidt Subject: Re: [BUG] rebuild_sched_domains considered dangerous References: <1299639487.22236.256.camel@pasglop> <1299665998.2308.2753.camel@twins> <1299675674.2308.2924.camel@twins> <1299766211.2308.4468.camel@twins> <1303294056.8345.122.camel@twins> <1303336869.2513.26.camel@pasglop> In-Reply-To: <1303336869.2513.26.camel@pasglop> Content-Type: multipart/alternative; boundary="------------050809090000080802010600" Cc: Peter Zijlstra , Martin Schwidefsky , linuxppc-dev , "linux-kernel@vger.kernel.org" List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , This is a multi-part message in MIME format. --------------050809090000080802010600 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit On 04/20/2011 05:01 PM, Benjamin Herrenschmidt wrote: > On Wed, 2011-04-20 at 12:07 +0200, Peter Zijlstra wrote: >> On Thu, 2011-03-10 at 15:10 +0100, Peter Zijlstra wrote: >>> >>> Also, still waiting to hear from the Power7 folks on how often >>> they think to rebuild the topology and how they think that makes >>> sense, afaict Power7 does have actual NUMA nodes unlike s390, so >>> I'm still not seeing how that's going to work properly at all. >> >> Jesse care to answer? I hear from Ben you're responsible for that >> mess. > > "responsible for this mess" is a big word :-) > > But he's the one to last play with that code ... Jesse ? > Hi Peter! According the the Power firmware folks, updating the home node of a virtual cpu happens rather infrequently. The VPHN code currently checks for topology updates every 60 seconds, but we can poll less frequently if it helps. I chose 60 second intervals simply because that's how often they check the topology on s390. ;-) As for updating the memory topology, there are cases where changing the home node of a virtual cpu doesn't affect the memory topology. If it does, there is a separate notification system for memory topology updates that is independent from the cpu updates. I plan to start working on a patch set to enable memory topology updates in the kernel in the coming weeks, but I wanted to get the cpu patches out on the list so we could start having these debates. :) Sincerely, Jesse Larrew Software Engineer, Linux on Power Kernel Team IBM Linux Technology Center Phone: (512) 973-2052 (T/L: 363-2052) jlarrew@linux.vnet.ibm.com --------------050809090000080802010600 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 7bit On 04/20/2011 05:01 PM, Benjamin Herrenschmidt wrote:
> On Wed, 2011-04-20 at 12:07 +0200, Peter Zijlstra wrote:
>> On Thu, 2011-03-10 at 15:10 +0100, Peter Zijlstra wrote:
>>>
>>> Also, still waiting to hear from the Power7 folks on how often
>>> they think to rebuild the topology and how they think that makes
>>> sense, afaict Power7 does have actual NUMA nodes unlike s390, so
>>> I'm still not seeing how that's going to work properly at all.
>>
>> Jesse care to answer? I hear from Ben you're responsible for that
>> mess.
>
> "responsible for this mess" is a big word :-)
>
> But he's the one to last play with that code ... Jesse ?
>

Hi Peter!

According the the Power firmware folks, updating the home node of a virtual cpu happens rather infrequently. The VPHN code currently checks for topology updates every 60 seconds, but we can poll less frequently if it helps. I chose 60 second intervals simply because that's how often they check the topology on s390. ;-)

As for updating the memory topology, there are cases where changing the home node of a virtual cpu doesn't affect the memory topology. If it does, there is a separate notification system for memory topology updates that is independent from the cpu updates. I plan to start working on a patch set to enable memory topology updates in the kernel in the coming weeks, but I wanted to get the cpu patches out on the list so we could start having these debates. :)

Sincerely,

Jesse Larrew
Software Engineer, Linux on Power Kernel Team
IBM Linux Technology Center
Phone: (512) 973-2052 (T/L: 363-2052)
jlarrew@linux.vnet.ibm.com

--------------050809090000080802010600--