From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Fri, 15 Jul 2011 10:45:47 +1000 From: Anton Blanchard To: Peter Zijlstra Subject: Re: [regression] 3.0-rc boot failure -- bisected to cd4ea6ae3982 Message-ID: <20110715104547.29c3c509@kryten> In-Reply-To: <1310649379.2586.273.camel@twins> References: <20110707102107.GA16666@in.ibm.com> <1310036375.3282.509.camel@twins> <20110714103418.7ef25b68@kryten> <20110714143521.5fe4fab6@kryten> <1310649379.2586.273.camel@twins> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Cc: mahesh@linux.vnet.ibm.com, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, mingo@elte.hu, torvalds@linux-foundation.org List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Hi, > Urgh.. so those spans are generated by sched_domain_node_span(), and > it looks like that simply picks the 15 nearest nodes to the one we've > got without consideration for overlap with previously generated spans. I do wonder if we need this extra level at all on ppc64. From memory SGI added it for their massive setups, but our largest setup is 32 nodes and breaking that down into 16 node chunks seems overkill. I just realised we were setting NEWIDLE on our node definition and that was causing large amounts of rebalance work even with SD_NODES_PER_DOMAIN=16. After removing it and bumping SD_NODES_PER_DOMAIN to 32, things look pretty good. Perhaps we should allow an arch to override SD_NODES_PER_DOMAIN so this extra level is only used by SGI boxes. Anton From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932680Ab1GOAp6 (ORCPT ); Thu, 14 Jul 2011 20:45:58 -0400 Received: from ozlabs.org ([203.10.76.45]:46908 "EHLO ozlabs.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932409Ab1GOAp5 (ORCPT ); Thu, 14 Jul 2011 20:45:57 -0400 Date: Fri, 15 Jul 2011 10:45:47 +1000 From: Anton Blanchard To: Peter Zijlstra Cc: mahesh@linux.vnet.ibm.com, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, mingo@elte.hu, benh@kernel.crashing.org, torvalds@linux-foundation.org Subject: Re: [regression] 3.0-rc boot failure -- bisected to cd4ea6ae3982 Message-ID: <20110715104547.29c3c509@kryten> In-Reply-To: <1310649379.2586.273.camel@twins> References: <20110707102107.GA16666@in.ibm.com> <1310036375.3282.509.camel@twins> <20110714103418.7ef25b68@kryten> <20110714143521.5fe4fab6@kryten> <1310649379.2586.273.camel@twins> X-Mailer: Claws Mail 3.7.8 (GTK+ 2.24.4; i686-pc-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi, > Urgh.. so those spans are generated by sched_domain_node_span(), and > it looks like that simply picks the 15 nearest nodes to the one we've > got without consideration for overlap with previously generated spans. I do wonder if we need this extra level at all on ppc64. From memory SGI added it for their massive setups, but our largest setup is 32 nodes and breaking that down into 16 node chunks seems overkill. I just realised we were setting NEWIDLE on our node definition and that was causing large amounts of rebalance work even with SD_NODES_PER_DOMAIN=16. After removing it and bumping SD_NODES_PER_DOMAIN to 32, things look pretty good. Perhaps we should allow an arch to override SD_NODES_PER_DOMAIN so this extra level is only used by SGI boxes. Anton