From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752780AbbJLAvY (ORCPT ); Sun, 11 Oct 2015 20:51:24 -0400 Received: from mail-wi0-f179.google.com ([209.85.212.179]:33212 "EHLO mail-wi0-f179.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752730AbbJLAvW (ORCPT ); Sun, 11 Oct 2015 20:51:22 -0400 Date: Mon, 12 Oct 2015 02:51:19 +0200 From: Frederic Weisbecker To: "Paul E. McKenney" Cc: Meelis Roos , Linux Kernel list , Thomas Gleixner Subject: Re: 4.2: CONFIG_NO_HZ_FULL_ALL effectively disabling non-boot CPUs Message-ID: <20151012005117.GA15270@lerouge> References: <20151010192439.GY3910@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151010192439.GY3910@linux.vnet.ibm.com> User-Agent: Mutt/1.5.24 (2015-08-30) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sat, Oct 10, 2015 at 12:24:39PM -0700, Paul E. McKenney wrote: > On Sat, Oct 10, 2015 at 10:14:25PM +0300, Meelis Roos wrote: > > Short summary: turning on CONFIG_NO_HZ_FULL_ALL seems to disable all > > non-boot CPUs for scheduler. > > > > A couple of days ago I noticed that make -j8 on a 4-core i5 is very slow > > (with 4.3.0-rc4+git). Looking at top ('1' for per-cpu states), only > > first CPU is loaded and 3 other CPUs are 100% idle. This seems to be a > > problem on 3 of my desktop machines (different generation Intel: i5-660, > > i5-2400, i3-3220). All the computers run custom kernels. > > > > Further investigation showed that CPU affinity was set to 1 (CPU0 only) > > for init and all the children. Kernel threads had affinities 1,2,4,8 > > and f (seems normal). > > > > Even more interesting was the behaviour after setting affinity to f for > > all userland processes and then running make -j4. The other cores were > > still idle! > > > > Switching back to 4.2.0 with my config, the problem persisted. 4.2.3 as > > packaged by Debian worked fine. 4.0.0 and 4.1.0 with my config worked > > also fine. systemd and sysvinit behaved the same and no affinity was > > configured for systemd. > > > > So did a kernel config bisection between my kernel config and Debian > > config and came to CONFIG_NO_HZ_FULL_ALL. Debian has it off, I had it > > on. Turning that off fixed the scheduling and the system spread the > > tasks to all the cores. > > > > I do not remember changing this value for a long time, I set them after > > the settings were introduced and used it. So it seems it broken in 4.2.0 > > but was working in 4.1 but I do not have 4.1 config saved anywhere > > (many make oldconfigs since). > > > > Bisection between 4.1 and 4.2 is possible but not easy since the > > machines are usually actively used when I am near them. > > This is expected and intended behavior. The whole point of > CONFIG_NO_HZ_FULL_ALL is to keep everything off of the non-boot CPUs > that is not explicitly placed there. Without CONFIG_NO_HZ_FULL_ALL, > you can use the nohz_full boot parameter to select exactly which > CPUs are to behave this way. I'm preparing a revert of this. Many people are complaining about this. Most of the time it's about accidentally enbling NO_HZ_FULL_ALL and I could fix this with a warning to avoid time spent by users to chase a non-bug. But Mike says that CONFIG_NO_HZ_FULL_ALL makes the machine unusable for anything else than isolation workloads whereas some "normal" workload may be needed as well by the machine before or after an isolation task.