x86's nmi_hz wrt. oprofile's nmi_timer_int.c

x86's nmi_hz wrt. oprofile's nmi_timer_int.c

[David Miller]

While working on an NMI watchdog implementation on sparc64
I noticed what seems to be a peculiar behavior of the NMI
timer int oprofile support on x86.

When the NMI watchdog tests itself at boot timer we start
with nmi_hz equal to HZ.

After the NMI watchdog self-test passes, nmi_hz is reduced
down to '1'.

The NMI timer int oprofile support simply uses DIE_NMI notifiers for
it's implementation.  But I don't see anything in the code of
arch/x86/oprofile/nmi_timer_int.c nor the NMI watchdog infrastructure
which will re-adjust nmi_hz back to HZ or something similar.

Am I missing something?

Re: x86's nmi_hz wrt. oprofile's nmi_timer_int.c

[Ingo Molnar]

* David Miller <davem@davemloft.net> wrote:

> While working on an NMI watchdog implementation on sparc64 I noticed 
> what seems to be a peculiar behavior of the NMI timer int oprofile 
> support on x86.
> 
> When the NMI watchdog tests itself at boot timer we start with nmi_hz 
> equal to HZ.
> 
> After the NMI watchdog self-test passes, nmi_hz is reduced down to '1'.
> 
> The NMI timer int oprofile support simply uses DIE_NMI notifiers for 
> it's implementation.  But I don't see anything in the code of 
> arch/x86/oprofile/nmi_timer_int.c nor the NMI watchdog infrastructure 
> which will re-adjust nmi_hz back to HZ or something similar.
> 
> Am I missing something?

Reducing it to 1 HZ was kind of a performance hack: running NMIs at HZ 
needlessly interrupts the CPU HZ times a second. It's more than enough to 
have 1 nmi-watchdog tick per second to notice deadlocks that take longer 
than 5 seconds.

Can you see a problem with that approach, or was this just a question 
about why it's reduced to 1 Hz?

	Ingo

Re: x86's nmi_hz wrt. oprofile's nmi_timer_int.c

[David Miller]

From: Ingo Molnar <mingo@elte.hu>
Date: Fri, 30 Jan 2009 16:01:25 +0100

> 
> * David Miller <davem@davemloft.net> wrote:
> 
> > While working on an NMI watchdog implementation on sparc64 I noticed 
> > what seems to be a peculiar behavior of the NMI timer int oprofile 
> > support on x86.
> > 
> > When the NMI watchdog tests itself at boot timer we start with nmi_hz 
> > equal to HZ.
> > 
> > After the NMI watchdog self-test passes, nmi_hz is reduced down to '1'.
> > 
> > The NMI timer int oprofile support simply uses DIE_NMI notifiers for 
> > it's implementation.  But I don't see anything in the code of 
> > arch/x86/oprofile/nmi_timer_int.c nor the NMI watchdog infrastructure 
> > which will re-adjust nmi_hz back to HZ or something similar.
> > 
> > Am I missing something?
> 
> Reducing it to 1 HZ was kind of a performance hack: running NMIs at HZ 
> needlessly interrupts the CPU HZ times a second. It's more than enough to 
> have 1 nmi-watchdog tick per second to notice deadlocks that take longer 
> than 5 seconds.

For the NMI watchdog's purposes I understand the intent, and this
is perfectly fine.

The problem is that it stays at '1' when oprofile starts using the NMI
watchdog, and we certainly want more than one oprofile tick per second
:-)

Re: x86's nmi_hz wrt. oprofile's nmi_timer_int.c

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Fri, 30 Jan 2009 13:54:09 -0800 (PST)

> From: Ingo Molnar <mingo@elte.hu>
> Date: Fri, 30 Jan 2009 16:01:25 +0100
> 
> > 
> > * David Miller <davem@davemloft.net> wrote:
> > 
> > Reducing it to 1 HZ was kind of a performance hack: running NMIs at HZ 
> > needlessly interrupts the CPU HZ times a second. It's more than enough to 
> > have 1 nmi-watchdog tick per second to notice deadlocks that take longer 
> > than 5 seconds.
> 
> For the NMI watchdog's purposes I understand the intent, and this
> is perfectly fine.
> 
> The problem is that it stays at '1' when oprofile starts using the NMI
> watchdog, and we certainly want more than one oprofile tick per second
> :-)

Just making sure you understand the problem, here is the
sequence of events:

1) At bootup, the NMI watchdog is tested.

   It is tested with nmi_hz=HZ

2) If the test passes, nmi_hz is reduced down to '1'

As I stated, everything up to this point is fine.  Next:

3) oprofile initializes and if we choose to use the NMI
   timer for oprofile profiling it is implemented using
   a simple DIE_NMI notifier.

   However, nmi_hz is still just '1' which means that oprofile
   will only receive one sample per-second.  And this is definitely
   not what we want.

Somehow the code in arch/x86/oprofile/nmi_timer_int.c needs to
have an interface into the NMI watchdog core so that it can
increase nmi_hz back up to "HZ" when the NMI timer profiling is
enabled and back down to "1" when such profiling stops.

Re: x86's nmi_hz wrt. oprofile's nmi_timer_int.c

[Ingo Molnar]

* David Miller <davem@davemloft.net> wrote:

> From: David Miller <davem@davemloft.net>
> Date: Fri, 30 Jan 2009 13:54:09 -0800 (PST)
> 
> > From: Ingo Molnar <mingo@elte.hu>
> > Date: Fri, 30 Jan 2009 16:01:25 +0100
> > 
> > > 
> > > * David Miller <davem@davemloft.net> wrote:
> > > 
> > > Reducing it to 1 HZ was kind of a performance hack: running NMIs at HZ 
> > > needlessly interrupts the CPU HZ times a second. It's more than enough to 
> > > have 1 nmi-watchdog tick per second to notice deadlocks that take longer 
> > > than 5 seconds.
> > 
> > For the NMI watchdog's purposes I understand the intent, and this
> > is perfectly fine.
> > 
> > The problem is that it stays at '1' when oprofile starts using the NMI
> > watchdog, and we certainly want more than one oprofile tick per second
> > :-)
> 
> Just making sure you understand the problem, here is the
> sequence of events:
> 
> 1) At bootup, the NMI watchdog is tested.
> 
>    It is tested with nmi_hz=HZ
> 
> 2) If the test passes, nmi_hz is reduced down to '1'
> 
> As I stated, everything up to this point is fine.  Next:
> 
> 3) oprofile initializes and if we choose to use the NMI
>    timer for oprofile profiling it is implemented using
>    a simple DIE_NMI notifier.
> 
>    However, nmi_hz is still just '1' which means that oprofile
>    will only receive one sample per-second.  And this is definitely
>    not what we want.
> 
> Somehow the code in arch/x86/oprofile/nmi_timer_int.c needs to have an 
> interface into the NMI watchdog core so that it can increase nmi_hz back 
> up to "HZ" when the NMI timer profiling is enabled and back down to "1" 
> when such profiling stops.

btw., these types of interactions will be solved in a natural way via 
perfcounters: in that model the NMI watchdog is a set of per-CPU counters 
running on each CPU [with a NMI watchdog callback in the IRQ handling 
routine] - and oprofile uses its own perfcounter - what is left of the PMU 
hardware.

I.e. each PMU using facility can just use performance counters transparently 
and interactions will be solved naturally by perfcounters resource 
management.

	Ingo

Re: x86's nmi_hz wrt. oprofile's nmi_timer_int.c

[Andi Kleen]

David Miller <davem@davemloft.net> writes:

Really old mail, but I was very behind. I didn't see an 
correct answer, so let's answer it.

> While working on an NMI watchdog implementation on sparc64
> I noticed what seems to be a peculiar behavior of the NMI
> timer int oprofile support on x86.
>
> When the NMI watchdog tests itself at boot timer we start
> with nmi_hz equal to HZ.
>
> After the NMI watchdog self-test passes, nmi_hz is reduced
> down to '1'.
>
> The NMI timer int oprofile support simply uses DIE_NMI notifiers for
> it's implementation.  But I don't see anything in the code of
> arch/x86/oprofile/nmi_timer_int.c nor the NMI watchdog infrastructure
> which will re-adjust nmi_hz back to HZ or something similar.
>
> Am I missing something?

oprofile generates its own NMIs, it does not rely on 
the ones from the nmi watchdog.

In timer mode it does not use nmis or die notifiers, but relies on the 
regular non nmi timer interrupt.

Does that answer your question?

-Andi
-- 
ak@linux.intel.com -- Speaking for myself only.

Re: x86's nmi_hz wrt. oprofile's nmi_timer_int.c

[David Miller]

From: Andi Kleen <andi@firstfloor.org>
Date: Sun, 22 Feb 2009 18:06:45 +0100

> David Miller <davem@davemloft.net> writes:
> 
> Really old mail, but I was very behind. I didn't see an 
> correct answer, so let's answer it.
> 
> > While working on an NMI watchdog implementation on sparc64
> > I noticed what seems to be a peculiar behavior of the NMI
> > timer int oprofile support on x86.
> >
> > When the NMI watchdog tests itself at boot timer we start
> > with nmi_hz equal to HZ.
> >
> > After the NMI watchdog self-test passes, nmi_hz is reduced
> > down to '1'.
> >
> > The NMI timer int oprofile support simply uses DIE_NMI notifiers for
> > it's implementation.  But I don't see anything in the code of
> > arch/x86/oprofile/nmi_timer_int.c nor the NMI watchdog infrastructure
> > which will re-adjust nmi_hz back to HZ or something similar.
> >
> > Am I missing something?
> 
> oprofile generates its own NMIs, it does not rely on 
> the ones from the nmi watchdog.

The code in nmi_timer_int.c doesn't.

> In timer mode it does not use nmis or die notifiers, but relies on the 
> regular non nmi timer interrupt.

Again, the code in nmi_timer_int.c doesn't.

It uses the NMI watchdog timer interrupts, it catches DIE_NMI
events.

> Does that answer your question?

Not really.

Re: x86's nmi_hz wrt. oprofile's nmi_timer_int.c

[Andi Kleen]

> Again, the code in nmi_timer_int.c doesn't.
> 
> It uses the NMI watchdog timer interrupts, it catches DIE_NMI
> events.
> 
> > Does that answer your question?
> 
> Not really.

Ah see what you mean now. The nmi_timer_int code can only
be active ever when the cpu is not known to nmi_int.c and 
when the nmi watchdog is in io apic mode. But IO apic mode
doesn't use the fast check/slowdown because it always runs
at HZ frequency. That only happens in LAPIC mode.

The standard fallback mode for unknown CPU is the non NMI timer 
fallback in oprofile_init, the IO APIC mode happens near never in practice.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.

Re: x86's nmi_hz wrt. oprofile's nmi_timer_int.c

[David Miller]

From: Andi Kleen <andi@firstfloor.org>
Date: Mon, 23 Feb 2009 05:52:00 +0100

> > Again, the code in nmi_timer_int.c doesn't.
> > 
> > It uses the NMI watchdog timer interrupts, it catches DIE_NMI
> > events.
> > 
> > > Does that answer your question?
> > 
> > Not really.
> 
> Ah see what you mean now. The nmi_timer_int code can only
> be active ever when the cpu is not known to nmi_int.c and 
> when the nmi watchdog is in io apic mode. But IO apic mode
> doesn't use the fast check/slowdown because it always runs
> at HZ frequency. That only happens in LAPIC mode.
> 
> The standard fallback mode for unknown CPU is the non NMI timer 
> fallback in oprofile_init, the IO APIC mode happens near never in practice.

Look at the fallback logic, the pure NMI profiler can fail for
a number of reasons, not just because the watchdog is in
I/O APIC mode.

No matter what the failure reason, nmi_int.c is used.

And in some of those cases, the nmi_hz has been decreased to '1'.

Re: x86's nmi_hz wrt. oprofile's nmi_timer_int.c

[Andi Kleen]

> Look at the fallback logic, the pure NMI profiler can fail for
> a number of reasons, not just because the watchdog is in
> I/O APIC mode.

I assume you mean nmi_init? It doesn't check for the watchdog at all,
just if it knows the CPU and if it can profile
All the reasons it fails on (unknown CPU, no APIC) will imply that the lapic 
based watchdog won't run either, because it relies on the same
perfctr hardware. The only case where it could fall into 
this path is in IO-APIC nmi watchdog mode (and unknown CPU) and then again 
the IO-APIC watchdog doesn't do the multiple frequencies thing, it always
runs with HZ.

Admittedly the logic is quite obscure.

-Andi
-- 
ak@linux.intel.com -- Speaking for myself only.