From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753482AbbDBPT4 (ORCPT ); Thu, 2 Apr 2015 11:19:56 -0400 Received: from mail-wg0-f47.google.com ([74.125.82.47]:33276 "EHLO mail-wg0-f47.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752715AbbDBPTy (ORCPT ); Thu, 2 Apr 2015 11:19:54 -0400 Date: Thu, 2 Apr 2015 17:19:51 +0200 From: Frederic Weisbecker To: Don Zickus Cc: Chris Metcalf , Andrew Morton , Andrew Jones , chai wen , Ingo Molnar , Ulrich Obergfell , Fabian Frederick , Aaron Tomlin , Ben Zhang , Christoph Lameter , Gilad Ben-Yossef , Steven Rostedt , open list Subject: Re: [PATCH v2] watchdog: nohz: don't run watchdog on nohz_full cores Message-ID: <20150402151950.GB10357@lerouge> References: <1427741465-15747-1-git-send-email-cmetcalf@ezchip.com> <20150330191245.GO162412@redhat.com> <5519A4E7.8040708@ezchip.com> <20150330200206.GQ162412@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150330200206.GQ162412@redhat.com> User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Mar 30, 2015 at 04:02:06PM -0400, Don Zickus wrote: > On Mon, Mar 30, 2015 at 03:32:55PM -0400, Chris Metcalf wrote: > > On 03/30/2015 03:12 PM, Don Zickus wrote: > > >On Mon, Mar 30, 2015 at 02:51:05PM -0400, cmetcalf@ezchip.com wrote: > > >>From: Chris Metcalf > > >> > > >>Running watchdog can be a helpful debugging feature on regular > > >>cores, but it's incompatible with nohz_full, since it forces > > >>regular scheduling events. Accordingly, just exit out immediately > > >>from any nohz_full core. > > >> > > >>An alternate approach would be to add a flags field or function to > > >>smp_hotplug_thread to control on which cores the percpu threads > > >>are created, but it wasn't clear that much mechanism was useful. > > >Hi Chris, > > > > > >It seems like the correct solution would be to hook into the idle_loop > > >somehow. If the cpu is idle, then it seems unlikely that a lockup could > > >occur. > > > > With nohz_full, though, the cpu might be running userspace code > > with the intention of keeping kernel ticks disabled. Even returning > > to kernel mode to try to figure out if we "should" be running the > > watchdog on a given core will induce exactly the kind of interrupts > > that nohz_full is designed to prevent. > > > > My assumption is generally that nohz_full cores don't spend a lot of > > time in the kernel anyway, as they are optimized for user space. > > > > I guess you could imagine doing something per-cpu on the nohz_full > > cores where we effectively call watchdog_disable() whenever a > > nohz_full core enters userspace, and watchdog_enable() whenever it > > enters the kernel. We could add some per-cpu state in the watchdog > > code to track whether that core was currently enabled or disabled > > to avoid double-enabling or double-disabling. I would think > > context_tracking_user_exit()/_enter() would be the place to do this. > > > > This feels like a lot of overhead, potentially. Thoughts? > > A few months ago I might have thought that a reasonable approach. But > recently we have added code to make the watchdog an all or nothing approach > across the system. This might make it difficult to do what you are > suggesting. > > I do not know enough about the nohz code to know what the right approach is > here. Perhaps Federic can enlighten me? Well, cancelling/rearming a timer on every userspace round trip sounds way too much overhead to me :-) But Ingo's suggestion to disable it properly (only on nohz full core) looks good. And we should be able to re-enable it everywhere with "sysctl -w kernel.watchdog=1" and you need to warn about this on boot.