Re: [sparc] BUG: NMI Watchdog detected LOCKUP on CPU0

Re: [sparc] BUG: NMI Watchdog detected LOCKUP on CPU0

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 1504 bytes --]

Summary for the SPARC maintainers:

The NMI watchdog is firing on Sunfire 280R and Sun Blade2500 systems
with one or both processors in cheetah_xcall_deliver().  This has been
seen under 3.0, 3.2 and 3.3 and seems to be associated with disk I/O.

Full bug log is at: http://bugs.debian.org/648766

On Tue, 2012-04-03 at 20:56 -0400, Kieron Gillespie wrote:
> I have also noticed, that if I am reading the trace correctly that in 
> both of my cases, and the original bug submitter's, and a bug posted on 
> old.nabble.com's case the crash always seems to happen when one CPU is 
> doing cheetah_xcall_deliver, and the other CPU is in the same 
> instruction in tl0_irq15. Here is a link to the post.
[...]

tl0_irq15 seems to be part of the NMI watchdog (for detecting that the
kernel has hung), so you should always see that in a backtrace when the
NMI watchdog fires.  It's not part of the problem.

cheetah_xcall_deliver() does appear to be relevant to the problem and it
looks like it could loop indefinitely - though presumably only if a
processor is behaving strangely?  It appears to periodically enable and
disable interrupts, but then I'm not sure how the PSTATE.IE and PIL
interrupt control fields interact and I don't think this will reset the
NMI watchdog.  In any case, it seems like there's a serious problem if
it's looping for a long time, whether or not interrupts remain disabled.

Ben.

-- 
Ben Hutchings
Larkinson's Law: All laws are basically false.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [sparc] BUG: NMI Watchdog detected LOCKUP on CPU0

[David Miller]

From: Ben Hutchings <ben@decadent.org.uk>
Date: Sat, 07 Apr 2012 18:21:38 +0100

> cheetah_xcall_deliver() does appear to be relevant to the problem and it
> looks like it could loop indefinitely - though presumably only if a
> processor is behaving strangely?

I can only loop indefinitely if one of the cpus is hung and
does not respond to the cross-call interrupt.

> It appears to periodically enable and disable interrupts, but then
> I'm not sure how the PSTATE.IE and PIL interrupt control fields
> interact and I don't think this will reset the NMI watchdog.

PSTATE.IE controls delivery of all interrupts, both PIL
based and vectored interrupts.

PIL only controls delivery of PIL interrupts.

The NMI watchdog interrupt is a special PIL interrupt, and
most of the standard local_irq_disable() et al. routines on
sparc will adjust the %pil such that NMI watchdog interrupts
are still delivered.

See include/asm/pil.h for details.

Re: [sparc] BUG: NMI Watchdog detected LOCKUP on CPU0

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 1962 bytes --]

On Sat, 2012-04-07 at 13:40 -0400, David Miller wrote:
> From: Ben Hutchings <ben@decadent.org.uk>
> Date: Sat, 07 Apr 2012 18:21:38 +0100
> 
> > cheetah_xcall_deliver() does appear to be relevant to the problem and it
> > looks like it could loop indefinitely - though presumably only if a
> > processor is behaving strangely?
> 
> I can only loop indefinitely if one of the cpus is hung and
> does not respond to the cross-call interrupt.

Well, it has to keep responding with a NACK, right?

Will the recipient NACK if the cross-call interrupt is disabled, or do
the processors have a buffer/FIFO for such IRQs?

If there's no buffer then what stops two processors live-locking here?
(The 'random time' where some interrupts are enabled isn't really very
random, so it seems to be possible for two processors to go round the
loop in lock-step.)

> > It appears to periodically enable and disable interrupts, but then
> > I'm not sure how the PSTATE.IE and PIL interrupt control fields
> > interact and I don't think this will reset the NMI watchdog.
> 
> PSTATE.IE controls delivery of all interrupts, both PIL
> based and vectored interrupts.
>
> PIL only controls delivery of PIL interrupts.
> 
> The NMI watchdog interrupt is a special PIL interrupt, and
> most of the standard local_irq_disable() et al. routines on
> sparc will adjust the %pil such that NMI watchdog interrupts
> are still delivered.
> 
> See include/asm/pil.h for details.

Obviously the NMI watchdog is not being disabled, but I was wondering
how its timer gets reset.

Having RTFS, it appears that it is never really reset but is held off by
either an hrtimer interrupt or an explicit call to watchdog_nmi_touch()
during each interval.  If I'm not mistaken, the hrtimer interrupt is
being disabled by xcall_deliver() and remains disabled.

Ben.

-- 
Ben Hutchings
If more than one person is responsible for a bug, no one is at fault.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [sparc] BUG: NMI Watchdog detected LOCKUP on CPU0

[David Miller]

From: Ben Hutchings <ben@decadent.org.uk>
Date: Sun, 08 Apr 2012 22:12:06 +0100

> Will the recipient NACK if the cross-call interrupt is disabled, or do
> the processors have a buffer/FIFO for such IRQs?

Recipient's NACK when their incoming cross-call queue is
full.  A cpu hung with PSTATE_IE clear will not take
vectorred interrupts and therefore not empty the queue.

> Obviously the NMI watchdog is not being disabled, but I was wondering
> how its timer gets reset.

The NMI watchdog is controlled by the performance counter
infrastructure.

When the performance counters are not being used for 'perf',
they are used to implement the NMI watchdog.

See arch/sparc/kernel/{pcr,nmi}.c