Re: PPro arch_cpu_idle: NMI watchdog: Watchdog detected hard LOCKUP on cpu 1

Frederic Weisbecker
2017-03-05T23:20:00Z

On Sun, Mar 05, 2017 at 10:48:50PM +0200, Meelis Roos wrote:
> Added some CC-s because of bisect find. Whole context should be still
> here.
>
> > > > > > > This is on my trusty IBM PC365, dual Pentium Pro. 4.10 worked fine,
> > > > > > > 4.10.0-09686-g9e314890292c and 4.10.0-10770-g2d6be4abf514 exhibit a
> > > > > > > problem. Ocassionally NMI watchdog kicks in and discovers one of the
> > > > > > > CPUs in LOCKUP. The system keeps running fine. The first lockup was
> > > > > > > different, all the others were from arch_cpu_idle. Sometime ecey couple
> > > > > > > of seconds (after some activity), sometimes nothing for a long time
> > > > > > > (idle, no SSH logins).
> > > > > >
> > > > > > The only watchdog related patch which hit after 4.10 is:
> > > > > >
> > > > > > 8dcde9def5a1 kernel/watchdog.c: do not hardcode CPU 0 as the initial thread
> > > > > >
> > > > > > Can you try to revert that for a start? I'm not seeing why it should be the
> > > > > > culprit from a quick glance, but ...
> > > > >
> > > > > Reverting this patch does not help.
> > > >
> > > > I did not expect that, but excluding it was a valid shot in the
> > > > dark. Thanmks for trying.
> > > >
> > > > To be honest, I have no idea what causes that at the moment, but I will
> > > > come back to you tomorrow after thinking it through (with brain awake) how
> > > > to debug this.
> > >
> > > Went through the related changes which came in during the merge window. One
> > > which affects the per cpu timers is: 914122c389d0
> > >
> > > Can you try to revert that one please?
> >
> > Running out of obvious culprits. Any chance that you can do a bisect or
> > this too painful on that box?
>
> Done on a P4 where the problem also appeared. The bisecting resulted in
> this commit. Does it seem realistic? I will also try if this help son
> the old PPro.
>
> 93825f2ec736f30e034ab7c9d56b42849c5b00da is the first bad commit
> commit 93825f2ec736f30e034ab7c9d56b42849c5b00da
> Author: Frederic Weisbecker <fweisbec@gmail.com>
> Date: Tue Jan 31 04:09:16 2017 +0100
>
> jiffies: Reuse TICK_NSEC instead of NSEC_PER_JIFFY
>
> NSEC_PER_JIFFY is an ad-hoc redefinition of TICK_NSEC. Let's rather
> use a unique and well maintained version.
>
> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> Cc: Paul Mackerras <paulus@samba.org>
> Cc: Michael Ellerman <mpe@ellerman.id.au>
> Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
> Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
> Cc: Tony Luck <tony.luck@intel.com>
> Cc: Fenghua Yu <fenghua.yu@intel.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Rik van Riel <riel@redhat.com>
> Cc: Stanislaw Gruszka <sgruszka@redhat.com>
> Cc: Wanpeng Li <wanpeng.li@hotmail.com>
> Link: http://lkml.kernel.org/r/1485832191-26889-1-git-send-email-fweisbec@gmail.com
> Signed-off-by: Ingo Molnar <mingo@kernel.org>

Ouch, looking at that patch again, I probably had a delusional moment when I wrote this:

diff --git a/kernel/time/jiffies.c b/kernel/time/jiffies.c
index a4a0e47..7906b3f 100644
--- a/kernel/time/jiffies.c
+++ b/kernel/time/jiffies.c
@@ -125,7 +125,7 @@ int register_refined_jiffies(long cycles_per_second)
shift_hz += cycles_per_tick/2;
do_div(shift_hz, cycles_per_tick);
/* Calculate nsec_per_tick using shift_hz */
- nsec_per_tick = (u64)NSEC_PER_SEC << 8;
+ nsec_per_tick = (u64)TICK_NSEC << 8;
nsec_per_tick += (u32)shift_hz/2;
do_div(nsec_per_tick, (u32)shift_hz);

Could you please retry after reverting this specific chunk? (that would be the very fix
to apply).

Thanks!