From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752484AbbJJTYn (ORCPT ); Sat, 10 Oct 2015 15:24:43 -0400 Received: from e39.co.us.ibm.com ([32.97.110.160]:43862 "EHLO e39.co.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751483AbbJJTYl (ORCPT ); Sat, 10 Oct 2015 15:24:41 -0400 X-IBM-Helo: d03dlp02.boulder.ibm.com X-IBM-MailFrom: paulmck@linux.vnet.ibm.com X-IBM-RcptTo: linux-kernel@vger.kernel.org Date: Sat, 10 Oct 2015 12:24:39 -0700 From: "Paul E. McKenney" To: Meelis Roos Cc: Linux Kernel list , Thomas Gleixner , Frederic Weisbecker Subject: Re: 4.2: CONFIG_NO_HZ_FULL_ALL effectively disabling non-boot CPUs Message-ID: <20151010192439.GY3910@linux.vnet.ibm.com> Reply-To: paulmck@linux.vnet.ibm.com References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 15101019-0033-0000-0000-00000653B399 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sat, Oct 10, 2015 at 10:14:25PM +0300, Meelis Roos wrote: > Short summary: turning on CONFIG_NO_HZ_FULL_ALL seems to disable all > non-boot CPUs for scheduler. > > A couple of days ago I noticed that make -j8 on a 4-core i5 is very slow > (with 4.3.0-rc4+git). Looking at top ('1' for per-cpu states), only > first CPU is loaded and 3 other CPUs are 100% idle. This seems to be a > problem on 3 of my desktop machines (different generation Intel: i5-660, > i5-2400, i3-3220). All the computers run custom kernels. > > Further investigation showed that CPU affinity was set to 1 (CPU0 only) > for init and all the children. Kernel threads had affinities 1,2,4,8 > and f (seems normal). > > Even more interesting was the behaviour after setting affinity to f for > all userland processes and then running make -j4. The other cores were > still idle! > > Switching back to 4.2.0 with my config, the problem persisted. 4.2.3 as > packaged by Debian worked fine. 4.0.0 and 4.1.0 with my config worked > also fine. systemd and sysvinit behaved the same and no affinity was > configured for systemd. > > So did a kernel config bisection between my kernel config and Debian > config and came to CONFIG_NO_HZ_FULL_ALL. Debian has it off, I had it > on. Turning that off fixed the scheduling and the system spread the > tasks to all the cores. > > I do not remember changing this value for a long time, I set them after > the settings were introduced and used it. So it seems it broken in 4.2.0 > but was working in 4.1 but I do not have 4.1 config saved anywhere > (many make oldconfigs since). > > Bisection between 4.1 and 4.2 is possible but not easy since the > machines are usually actively used when I am near them. This is expected and intended behavior. The whole point of CONFIG_NO_HZ_FULL_ALL is to keep everything off of the non-boot CPUs that is not explicitly placed there. Without CONFIG_NO_HZ_FULL_ALL, you can use the nohz_full boot parameter to select exactly which CPUs are to behave this way. Thanx, Paul