Re: [PATCH] x86/nmi: lower initial watchdog frequency to avoid boot hangs

From: Andrew Cooper <andrew.cooper3@citrix.com>
To: Jan Beulich <JBeulich@suse.com>
Cc: Igor Druzhinin <igor.druzhinin@citrix.com>, xen-devel@lists.xen.org
Subject: Re: [PATCH] x86/nmi: lower initial watchdog frequency to avoid boot hangs
Date: Thu, 8 Feb 2018 12:18:30 +0000	[thread overview]
Message-ID: <9042c7e8-6ac0-525b-4b3e-fb376dbcc604@citrix.com> (raw)
In-Reply-To: <5A7C228D02000078001A649E@prv-mh.provo.novell.com>

On 08/02/18 09:12, Jan Beulich wrote:
>>>> On 07.02.18 at 18:08, <andrew.cooper3@citrix.com> wrote:
>> On 07/02/18 15:06, Jan Beulich wrote:
>>> Also you completely ignore my argument against the seemingly
>>> random division by 10, including the resulting question of what you
>>> mean to do once 10Hz also turns out too high a frequency.
>> We've got to pick a frequency.  The current 100Hz is just as arbitrary
>> as the proposed new 10Hz.
> Not exactly - the 100Hz is simply the frequency we run the main tick
> at, so while random it is not as random as any further derived value
> which has no proper reason behind it.
>
> There's one more point wrt your argument of overhead: If servicing
> an NMI takes that long on those boxes, you're basically saying you
> are happy to waste at least 1% of a core's bandwidth on a
> debugging feature. Is that reasonable for a production setup? And
> considering that I'd expect the patch to have chosen e.g. HZ / 5,
> HZ / 4, or even HZ / 2 if that worked reliably, I could even conclude
> you're happy to spend somewhere between 5 and 10% of one
> core's bandwidth. (FAOD all this is based on the 1Hz frequency we
> - iirc - run the NMI at later on.) To me this is another clear argument
> to turn off the watchdog on those systems, rather than trying to
> "fix" its probing.

It is not a debugging feature; it's a reliability feature.  With
clustered storage in particular, it is absolutely paramount to guarantee
that a struggling host fences itself cleanly, or you lose the entire
cluster.

This particular issue is a failure to boot, but by far the most common
issue we see in the field is a fence when all-but-one CPU is waiting in
the calibration rendezvous, by which point the host has effectively been
dead for 5 seconds already.  Turning the watchdog off isn't a viable or
reasonable solution to the problem.

We switch the NMI frequency to ~2Hz after the calibration, but that is
after having run the BSP at 100Hz for a long period of time, and the APs
at this rate for a short while.  Irrespective of the exact fix here, it
is simply not a good idea to be running with this NMI frequency, other
than possibly during the immediate calibration logic.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel