Re: [PATCH] x86, UV: Fix NMI handler for UV platforms

[Jack Steiner <steiner@sgi.com>]

On Tue, Mar 22, 2011 at 02:44:50PM -0400, Don Zickus wrote:
> On Tue, Mar 22, 2011 at 12:11:18PM -0500, Jack Steiner wrote:
> > How certain are you that multiple NMIs triggered at about the same time will
> > deliver discrete NMI events? I updated the patch so that I'm running with:
> 
> I think as long as there isn't more than two (1 active, 1 latched), you
> would be ok.  A third one looks like it would get dropped.
> 
> > 
> > 	- no special code in traps.c (I removed the traps.c code that was
> > 	  in the patch I posted)
> > 	- used die_notifier for calling the UV nmi handler
> > 	- UV priority is higher than the hw_perf priority
> > 
> > Both hw_perf (perf top) & UV NMIs work correctly under light loads. However, if I
> > run for 10 - 15 minutes injecting UV NMIs at a rate of about 30/min, "perf top"
> > stops generating output. Strace shows that it continues to poll() but no data
> > is received.
> 
> That's a low frequency and it still gets stuck?
> 
> > 
> > While "perf top" is hung, if I inject an NMI into the system in a way that will NOT
> > be consumed by the UV nmi handler, "perf top" resumes output but will stop again after
> > a few minutes.
> 
> So that means the PMU set its interrupt bit but the cpu failed to get the
> NMI.
> 
> > 
> > 
> > AFAICT, the UV nmi handler is not consuming extra NMI interrupts. I can't
> > rule out that I'm missing something but I don't see it.
> 
> What happens if you put the UV nmi handler below the hw_perf handler in
> priority?  I assume the DIE_NMIUNKNOWN snippet in the hw_perf handler will
> swallow some of the UV NMIs, but more importantly does it still generate
> the hang you see?

I verified that the failures ("perf top" stops) are the same on both RHEL6.1 & the
latest x86 2.6.38+ tree.

I switched priorities & as expected, "perf top" no longer hangs. I see an occassional
missed UV NMI - about 1 every minute. I also see a few "dazed" messages as
well - 3 in a 5 minute period. This testing was done on a 2.6.38+ kernel.

I'm running on a 48p system.

Ideas?

> 
> > 
> > 
> > Do you have any ideas or clues???
> 
> Part of the problem is most of the NMI testing is done with perf and maybe
> kgdb.  So high frequency NMI sharing is probably exposing more bugs.
> 
> Also is it a problem to move your testing on to the latest upstream code
> instead of RHEL-6?  Not all the latest NMI work is there.  I want to make
> sure we are all starting at the same code. :-)
> 
> Cheers,
> Don
> 
> > 
> > 
> > > 
> > > > 
> > > > The root cause of the problem is that architecturally, x86 does not
> > > > have a way to identifies the source(s) that cause an NMI. If multiple
> > > > events occur at about the same time, there is no way that I can see that the
> > > > OS can detect it.
> > > 
> > > There are registers we can check to see who owns trigger the NMI (at least
> > > for the perf code, the SGI code maybe not, which is why I set it to a
> > > lower priority to be a catch-all).
> > > 
> > > I'm not aware of the x86 architecture dropping NMIs, so they should all
> > > get processed.  It is just a matter of which subsystems get determine if
> > > they are the source of the NMI or not.
> > > 
> > > > 
> > > > > 
> > > > > My first impression is the skip nmi logic in the perf handler is probably
> > > > > accidentally thinking the SGI external nmi is the perf's 'extra' nmi it is
> > > > > supposed to skip and thus swallows it.  At least that is the impression I
> > > > 
> > > > Agree
> > > > 
> > > > 
> > > > > get from the RedHat bugzilla which says SGI is running 'perf top', getting
> > > > > a hang, then pressing their nmi button to see the stack traces.
> > > > > 
> > > > > Jack,
> > > > > 
> > > > > I worked through a number of these issues upstream and I already talked to
> > > > > George and Russ over here at RedHat about working through the issue over
> > > > > here with them.  They can help me get access to your box to help debug.
> > > > 
> > > > Russ is right down the hall.
> > > 
> > > Great!
> > > 
> > > Cheers,
> > > Don