Re: [PATCH] watchdog: nohz: don't run watchdog on nohz_full cores

[Chris Metcalf <cmetcalf@ezchip.com>]

On 4/2/2015 9:35 AM, Don Zickus wrote:
> On Tue, Mar 31, 2015 at 02:30:44PM -0400, Chris Metcalf wrote:
>> On 03/31/2015 03:25 AM, Ingo Molnar wrote:
>>> * cmetcalf@ezchip.com <cmetcalf@ezchip.com> wrote:
>>>
>>>> From: Chris Metcalf <cmetcalf@ezchip.com>
>>>>
>>>> Running watchdog can be a helpful debugging feature on regular
>>>> cores, but it's incompatible with nohz_full, since it forces
>>>> regular scheduling events.  Accordingly, just exit out immediately
>>> >from any nohz_full core.
>>>> An alternate approach would be to add a flags field or function to
>>>> smp_hotplug_thread to control on which cores the percpu threads
>>>> are created, but it wasn't clear that much mechanism was useful.
>>>>
>>>> [...]
>>> So what happens if someone wants to enable the lockup detector, with a
>>> long timeout, even on nohz-full CPUs? This patch makes that
>>> impossible.
>>>
>>> A better solution would be to tweak the defaults:
>>>
>>>   - to default the watchdog(s) to disabled when nohz-full is
>>>     enabled, even if HARDLOCKUP_DETECTOR=y or DETECT_HUNG_TASK=y, and
>>>     allow it to be re-enabled via its sysctl.
>> That's certainly a reasonable thing to do; it looks like just an #ifdef
>> at the top of watchdog.c would suffice.  Does this look right?
>>
>> diff --git a/kernel/watchdog.c b/kernel/watchdog.c
>> index 8a46d9d8a66f..c8555c211e65 100644
>> --- a/kernel/watchdog.c
>> +++ b/kernel/watchdog.c
>> @@ -25,7 +25,11 @@
>>   #include <linux/kvm_para.h>
>>   #include <linux/perf_event.h>
>> +#ifdef CONFIG_NO_HZ_FULL
>> +int watchdog_user_enabled = 0;
>> +#else
>>   int watchdog_user_enabled = 1;
>> +#endif
>>   int __read_mostly watchdog_thresh = 10;
>>   #ifdef CONFIG_SMP
>>   int __read_mostly sysctl_softlockup_all_cpu_backtrace;
>>
>> It doesn't look like I need to do anything else special to disable
>> HARDLOCKUP_DETECTOR, and khungtaskd can happily run on
>> a non-nohz core, so that should be OK.
>>
>> What I was trying to achieve with my proposed patch was kind
>> of orthogonal: to allow the watchdog to run on standard cores,
>> but not run on nohz cores, so we could benefit from it on the
>> cores where it was safe for it to run.  Do you see value in this,
>> or better to just enable/disable all watchdog threads collectively?
>
> Hmm, I am not sure I am a big fan of this approach.  I know RHEL keeps the
> watchdogs enabled for customers and it would be a regression if we disabled
> it.  And at the same time, I could see RHEL leaning towards enabling
> CONFIG_NO_HZ_FULL, which would just delay this problem a number of years
> until RHEL-8 gets around to ramping up.
>
> So I guess I would prefer to figure out a better co-existing solution now.
>
> Can I ask how the NO_HZ_FULL technology works from userspace?  Is there a
> system command that has to be sent?  How does the kernel know to turn off
> ticks and trust userspace to do the right thing?

The NO_HZ_FULL option, when configured into the kernel, lets
you boot with "nohz_full=1-15" (or whatever cpumask you like),
typically in conjunction with "isolcpus=1-15".  At this point no tasks
will run on those cores until explicitly placed there by affinity, and
once there and running in userspace, the kernel will automatically
get out of their way and not interrupt at all.  This lets those tasks
run with 100.000% of the cpu, which is a requirement for many
user-space device drivers running high throughput devices.
(This is typically the use case for the tile architecture customers.)

So, other than a boot flag, there are no system commands or
other APIs to deal with.

Part of the requirement, though, is that there can be only one task
bound and runnable on that cpu, otherwise the kernel has to be
involved to do the context-switching off of the scheduler tick.
This is why having the standard watchdog kernel thread doesn't
work in this context.

I continue to suspect that the right model here is to disable the
watchdog specifically on the cores that the user has tagged with
the nohz_full boot argument.  I agree that there might be a case
to be made for leaving the watchdog conditionally (as suggested
by Ingo) but it should be possible to have the watchdogs on
the nohz_full cores be turned off completely if desired.

-- 
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com