Re: [PATCH] Re: UP APIC reenabling vs. cpu type detection o

Re: [PATCH] Re: UP APIC reenabling vs. cpu type detection o

[Petr Vandrovec]

On  8 Feb 01 at 6:04, Mikael Pettersson wrote:

> ordering and offsets in processor.h and head.S. The resulting
> kernel works ok on my UP P6.
> (Petr: can you check that it still works on your K7?)

I'll try.

I have another question for UP APIC NMI: As I reported some time ago,
if performance counters overflow when LVTPC has 'disabled' bit set,
NMI is lost forever. This causes problems with VMware - it has to
disable NMI deliveries during CR3 (memory mapping) switching, and if 
performance counter overflows at that time, you'll not receive another 
NMI for couple of days on K7 (4.1 * 65536 seconds on fully loaded 1GHz 
Athlon. And 410 * 65536 seconds on idle Athlon)...

So it came to my mind - why (on K7 we easy can, as counter has 48 bits)
we do not reload NMI watchdog in each timer interrupt with 5sec timeout,
and if we receive even one NMI, we are locked up? It should increase
performance, as we'll do same number of MSR writes anyway (100/s), but
we will not receive any NMI during normal operation, so we save time
spent in processing this. Or do I miss something?

It may be problem on P6, as it has only 32bit perfctrs, so we are limited
to 4.1s watchdog timeout on 1GHz PIII :-(
                                    Thanks,
                                            Petr Vandrovec
                                            vandrove@vc.cvut.cz
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [PATCH] Re: UP APIC reenabling vs. cpu type detection o

[Maciej W. Rozycki]

On Thu, 8 Feb 2001, Petr Vandrovec wrote:

> So it came to my mind - why (on K7 we easy can, as counter has 48 bits)
> we do not reload NMI watchdog in each timer interrupt with 5sec timeout,
> and if we receive even one NMI, we are locked up? It should increase
> performance, as we'll do same number of MSR writes anyway (100/s), but
> we will not receive any NMI during normal operation, so we save time
> spent in processing this. Or do I miss something?

 I guess it's the external watchdog heritage.  The code is common for both
kinds of the watchdog at the moment.  It might get separated, I suppose. 

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [PATCH] Re: UP APIC reenabling vs. cpu type detection o

[Mikael Pettersson]

On Thu, 8 Feb 2001 12:32:01 MET-1, Petr Vandrovec wrote:

>I have another question for UP APIC NMI: As I reported some time ago,
>if performance counters overflow when LVTPC has 'disabled' bit set,
>NMI is lost forever. This causes problems with VMware - it has to
>disable NMI deliveries during CR3 (memory mapping) switching, and if 
>performance counter overflows at that time, you'll not receive another 
>NMI for couple of days on K7 (4.1 * 65536 seconds on fully loaded 1GHz 
>Athlon. And 410 * 65536 seconds on idle Athlon)...

How long do you need to keep NMIs blocked?

The watchdog (re-)initialises the perfctr to a negative value, but
after overflow the counter will read as positive. (You'll have to
use rdpmc() and sign-extend the low bits of the high word.)
So if, after your critical section, the counter reads as positive
you know that you missed an NMI. In that case, just write the restart
value to the counter.

Alternatively, if you block the NMI watchdog for a long time, say a
couple of thousand cycles or more, you can unconditionally restart it.

(I had a theory about inspecting the APIC_LVTPC "Delivery Status"
field, but unfortunately it doesn't seem to get set if a counter
overflowed while LVTPC was masked. Perhaps it's because NMIs are
edge-triggered.)

>So it came to my mind - why (on K7 we easy can, as counter has 48 bits)
>we do not reload NMI watchdog in each timer interrupt with 5sec timeout,
>and if we receive even one NMI, we are locked up? It should increase
>performance, as we'll do same number of MSR writes anyway (100/s), but
>we will not receive any NMI during normal operation, so we save time
>spent in processing this. Or do I miss something?

The 100 NMI/s rate was just to match the original IO-APIC driven watchdog;
it's certainly not a design goal to have the rate this high. In fact,
we'll probably reduce it to 1Hz soon. (I'm running it in that mode now.)

/Mikael
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [PATCH] Re: UP APIC reenabling vs. cpu type detection o

[Petr Vandrovec]

On  9 Feb 01 at 0:06, Mikael Pettersson wrote:
> On Thu, 8 Feb 2001 12:32:01 MET-1, Petr Vandrovec wrote:
> 
> >I have another question for UP APIC NMI: As I reported some time ago,
> >if performance counters overflow when LVTPC has 'disabled' bit set,
> >NMI is lost forever. This causes problems with VMware - it has to
> >disable NMI deliveries during CR3 (memory mapping) switching, and if 
> >performance counter overflows at that time, you'll not receive another 
> >NMI for couple of days on K7 (4.1 * 65536 seconds on fully loaded 1GHz 
> >Athlon. And 410 * 65536 seconds on idle Athlon)...
> 
> How long do you need to keep NMIs blocked?

Under some circumstances until `real' (usually timer) interrupt happens :-(
It is up to 10ms (on ia32).
 
> The watchdog (re-)initialises the perfctr to a negative value, but
> after overflow the counter will read as positive. (You'll have to
> use rdpmc() and sign-extend the low bits of the high word.)
> So if, after your critical section, the counter reads as positive
> you know that you missed an NMI. In that case, just write the restart
> value to the counter.
> 
> Alternatively, if you block the NMI watchdog for a long time, say a
> couple of thousand cycles or more, you can unconditionally restart it.

Unfortunately both these ways needs intimate knowledge of how UP NMI
watchdog works in each kernel, and it is incompatible with other
perfctr uses. Probably I'll switch perfctr delivery to some real
maskable interrupt while VMware VM owns CPU - if it is possible.
Then interrupt should be still pending after VM does __sti().
                                                    Petr Vandrovec
                                                    vandrove@vc.cvut.cz
                                                    
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [PATCH] Re: UP APIC reenabling vs. cpu type detection o

[Maciej W. Rozycki]

On Fri, 9 Feb 2001, Mikael Pettersson wrote:

> (I had a theory about inspecting the APIC_LVTPC "Delivery Status"
> field, but unfortunately it doesn't seem to get set if a counter
> overflowed while LVTPC was masked. Perhaps it's because NMIs are
> edge-triggered.)

 The delivery status bit gets only set if an interrupt got accepted (that
implies it was unmasked at the moment) but its delivery is held pending
due to some reason.  It may happen when the target CPU has its interrupts
masked or the target local APIC cannot accept it due to a higher task
priority or no free slots.  The semantics of the bit is the same for both
local and I/O APIC interrupts.

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [PATCH] Re: UP APIC reenabling vs. cpu type detection o

[Maciej W. Rozycki]

On Fri, 9 Feb 2001, Petr Vandrovec wrote:

> Unfortunately both these ways needs intimate knowledge of how UP NMI
> watchdog works in each kernel, and it is incompatible with other
> perfctr uses. Probably I'll switch perfctr delivery to some real
> maskable interrupt while VMware VM owns CPU - if it is possible.
> Then interrupt should be still pending after VM does __sti().

 Why do you need to mask NMI at all? 

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [PATCH] Re: UP APIC reenabling vs. cpu type detection o

[Petr Vandrovec]

On  9 Feb 01 at 13:10, Maciej W. Rozycki wrote:
> On Fri, 9 Feb 2001, Petr Vandrovec wrote:
> 
> > Unfortunately both these ways needs intimate knowledge of how UP NMI
> > watchdog works in each kernel, and it is incompatible with other
> > perfctr uses. Probably I'll switch perfctr delivery to some real
> > maskable interrupt while VMware VM owns CPU - if it is possible.
> > Then interrupt should be still pending after VM does __sti().
> 
>  Why do you need to mask NMI at all? 

Because of you must provide some function which handles NMI, and as
you cannot switch IDT and CR3 atomically together, NMI handler has
to be on same address in both address spaces - at least temporary. 
And in addition NMI handler in VM would have to switch address spaces 
back, execute NMI handler, and return CPU/MMU back to previous state - 
which may be just in the middle of normal VM<->Linux transition, so 
this context switching cannot use any global variable, it must 
save complete CPU/MMU state on stack. And it must not use any spinlock.

If you have any idea how it can be done with NMI unmasked all the way
around...
                                    Thanks,
                                            Petr Vandrovec
                                            vandrove@vc.cvut.cz
                                            
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [PATCH] Re: UP APIC reenabling vs. cpu type detection o

[Maciej W. Rozycki]

On Fri, 9 Feb 2001, Petr Vandrovec wrote:

> >  Why do you need to mask NMI at all? 
> 
> Because of you must provide some function which handles NMI, and as
> you cannot switch IDT and CR3 atomically together, NMI handler has
> to be on same address in both address spaces - at least temporary. 

 Can't it be?

> And in addition NMI handler in VM would have to switch address spaces 
> back, execute NMI handler, and return CPU/MMU back to previous state - 
> which may be just in the middle of normal VM<->Linux transition, so 
> this context switching cannot use any global variable, it must 
> save complete CPU/MMU state on stack. And it must not use any spinlock.

 Do you need to pass NMIs to VM at all?  NMIs as defined by the PC/AT
architecture are delivered as a result of memory parity errors or ISA
IOCHK errors.  Is that functionality really needed in VM?

> If you have any idea how it can be done with NMI unmasked all the way
> around...

 It depends on the application -- you may avoid problems by careful coding
and a nested NMI will never happen -- the CPU masks the NMI line
internally, from accepting an NMI till a subsequent iret. 

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/