NMI is broken!

NMI is broken!

[george anzinger]

It appears that the NMI generation set up code is NOT correctly setting
up the system to generate NMIs if the interrupt system is off!  Kernel
version 2.4.2 NMI worked correctly, 2.4.7 and later do not.  I have not
tested kernels 2.4.3 thru 2.4.6.  If the interrupt system is on, NMIs
are being correctly generated.

IMHO the check_nmi_watchdog() test is flawed in that it explicitly
enables interrupts prior to the test.  Below is a patch to fix this
issue.  When this patch is applied to the 2.4.2 kernel, the test works
correctly, however, 2.4.7 and above all report "NMI appears to be
stuck".  (Note, the check_nmi_watchdog() code is in io_apic.c in the
2.4.2 kernel and is called "nmi_irq_works()".)

Note that this is not just a testing issue, NMI is no longer pulling the
system out of deadlocks with any of the irq spinlocks.

I am afraid that I don't know enough or have the right documentation to
figure out what changed, but it appears that the change was about the
same time that the nmi.c module was introduced.

Here is the recommended patch for nmi.c:

--- linux-2.4.13-org/arch/i386/kernel/nmi.c	Tue Sep 25 00:34:57 2001
+++ linux/arch/i386/kernel/nmi.c	Fri Nov 30 00:31:03 2001
@@ -46,28 +54,30 @@
 int __init check_nmi_watchdog (void)
 {
 	irq_cpustat_t tmp[NR_CPUS];
-	int j, cpu;
+	int j, cpu, err = 0;
 
 	printk(KERN_INFO "testing NMI watchdog ... ");
 
 	memcpy(tmp, irq_stat, sizeof(tmp));
-	sti();
+	cli();
 	mdelay((10*1000)/nmi_hz); // wait 10 ticks
 
 	for (j = 0; j < smp_num_cpus; j++) {
 		cpu = cpu_logical_map(j);
 		if (nmi_count(cpu) - tmp[cpu].__nmi_count <= 5) {
 			printk("CPU#%d: NMI appears to be stuck!\n", cpu);
-			return -1;
+			err = -1;
 		}
 	}
-	printk("OK.\n");
+        if (! err ){
+                printk("OK.\n");
+        }
 
 	/* now that we know it works we can reduce NMI frequency to
 	   something more reasonable; makes a difference in some configs */
 	if (nmi_watchdog == NMI_LOCAL_APIC)
 		nmi_hz = 1;
-
+        sti();
 	return 0;
 }
 

-- 
George           george@mvista.com
High-res-timers: http://sourceforge.net/projects/high-res-timers/
Real time sched: http://sourceforge.net/projects/rtsched/

Re: NMI is broken!

[george anzinger]

george anzinger wrote:
> 
> It appears that the NMI generation set up code is NOT correctly setting
> up the system to generate NMIs if the interrupt system is off!  Kernel
> version 2.4.2 NMI worked correctly, 2.4.7 and later do not.  I have not
> tested kernels 2.4.3 thru 2.4.6.  If the interrupt system is on, NMIs
> are being correctly generated.

Oops!  I did it!  It appears that PIT interrupt requests are somehow
involved in the NMI generation.  Since the high-res-timers patch
programs the PIT to interrupt once at the end of each PIT interrupt, it
stops interrupting if the interrupt system is off.  At least this
appears to be the problem.  Now if someone could explain just how this
is done, or at least point me to the the appropriate documentation...

So, sorry for the noise, but still, I think the patch below is a good
thing(tm).

George

> 
> IMHO the check_nmi_watchdog() test is flawed in that it explicitly
> enables interrupts prior to the test.  Below is a patch to fix this
> issue.  When this patch is applied to the 2.4.2 kernel, the test works
> correctly, however, 2.4.7 and above all report "NMI appears to be
> stuck".  (Note, the check_nmi_watchdog() code is in io_apic.c in the
> 2.4.2 kernel and is called "nmi_irq_works()".)
> 
> Note that this is not just a testing issue, NMI is no longer pulling the
> system out of deadlocks with any of the irq spinlocks.
> 
> I am afraid that I don't know enough or have the right documentation to
> figure out what changed, but it appears that the change was about the
> same time that the nmi.c module was introduced.
> 
> Here is the recommended patch for nmi.c:
> 
> --- linux-2.4.13-org/arch/i386/kernel/nmi.c     Tue Sep 25 00:34:57 2001
> +++ linux/arch/i386/kernel/nmi.c        Fri Nov 30 00:31:03 2001
> @@ -46,28 +54,30 @@
>  int __init check_nmi_watchdog (void)
>  {
>         irq_cpustat_t tmp[NR_CPUS];
> -       int j, cpu;
> +       int j, cpu, err = 0;
> 
>         printk(KERN_INFO "testing NMI watchdog ... ");
> 
>         memcpy(tmp, irq_stat, sizeof(tmp));
> -       sti();
> +       cli();
>         mdelay((10*1000)/nmi_hz); // wait 10 ticks
> 
>         for (j = 0; j < smp_num_cpus; j++) {
>                 cpu = cpu_logical_map(j);
>                 if (nmi_count(cpu) - tmp[cpu].__nmi_count <= 5) {
>                         printk("CPU#%d: NMI appears to be stuck!\n", cpu);
> -                       return -1;
> +                       err = -1;
>                 }
>         }
> -       printk("OK.\n");
> +        if (! err ){
> +                printk("OK.\n");
> +        }
> 
>         /* now that we know it works we can reduce NMI frequency to
>            something more reasonable; makes a difference in some configs */
>         if (nmi_watchdog == NMI_LOCAL_APIC)
>                 nmi_hz = 1;
> -
> +        sti();
>         return 0;
>  }
> 
> 
> --
> George           george@mvista.com
> High-res-timers: http://sourceforge.net/projects/high-res-timers/
> Real time sched: http://sourceforge.net/projects/rtsched/
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
George           george@mvista.com
High-res-timers: http://sourceforge.net/projects/high-res-timers/
Real time sched: http://sourceforge.net/projects/rtsched/

nmi is broken?

[OGAWA Hirofumi]

Hi,

I noticed recently NMI on guest kernel is not working well.  host/guest
kernel is 2.6.39-rc4, and using vmx.

And test code is something like the following:

	local_irq_disable();
	for (i = 0; i < 10; i++) {
		int cpu = get_cpu();
		printk("%s: nmi %u, lapic %u\n", __FUNCTION__,
			nmi_count(cpu), irq_stat[cpu].apic_timer_irqs);
		mdelay(1000);
		put_cpu();
	}

the result is both of nmi and lapic are not increased. If I used
-no-kvm-irqchip, it works fine (increase nmi only). So, it seems to be
the bug of kvm driver side.

Ideas?
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: nmi is broken?

[OGAWA Hirofumi]

OGAWA Hirofumi <hirofumi@mail.parknet.co.jp> writes:

> I noticed recently NMI on guest kernel is not working well.  host/guest
> kernel is 2.6.39-rc4, and using vmx.
>
> And test code is something like the following:
>
> 	local_irq_disable();
> 	for (i = 0; i < 10; i++) {
> 		int cpu = get_cpu();
> 		printk("%s: nmi %u, lapic %u\n", __FUNCTION__,
> 			nmi_count(cpu), irq_stat[cpu].apic_timer_irqs);
> 		mdelay(1000);
> 		put_cpu();
> 	}
>
> the result is both of nmi and lapic are not increased. If I used
> -no-kvm-irqchip, it works fine (increase nmi only). So, it seems to be
> the bug of kvm driver side.

With some debug, the cause seems to be in pit_do_work(). With the
following patch, NMI watchdog seems to be working correctly (if irq
disabled for long time, NMI watchdog can detect it).

Is the following patch right?
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>


[PATCH] kvm: Fix NMI on irq disabled state

On irq disabled state, there is no ->irq_ask from quest kernel. But
NMI shouldn't be affected by it.

This fixes inject NMI unconditionally.

Signed-off-by: OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>
---

 arch/x86/kvm/i8254.c |   26 +++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)

diff -puN arch/x86/kvm/i8254.c~kvm-nmi-fix arch/x86/kvm/i8254.c
--- linux-2.6/arch/x86/kvm/i8254.c~kvm-nmi-fix	2011-04-24 07:42:23.000000000 +0900
+++ linux-2.6-hirofumi/arch/x86/kvm/i8254.c	2011-04-24 07:42:23.000000000 +0900
@@ -305,20 +305,20 @@ static void pit_do_work(struct work_stru
 	if (inject) {
 		kvm_set_irq(kvm, kvm->arch.vpit->irq_source_id, 0, 1);
 		kvm_set_irq(kvm, kvm->arch.vpit->irq_source_id, 0, 0);
-
-		/*
-		 * Provides NMI watchdog support via Virtual Wire mode.
-		 * The route is: PIT -> PIC -> LVT0 in NMI mode.
-		 *
-		 * Note: Our Virtual Wire implementation is simplified, only
-		 * propagating PIT interrupts to all VCPUs when they have set
-		 * LVT0 to NMI delivery. Other PIC interrupts are just sent to
-		 * VCPU0, and only if its LVT0 is in EXTINT mode.
-		 */
-		if (kvm->arch.vapics_in_nmi_mode > 0)
-			kvm_for_each_vcpu(i, vcpu, kvm)
-				kvm_apic_nmi_wd_deliver(vcpu);
 	}
+
+	/*
+	 * Provides NMI watchdog support via Virtual Wire mode.
+	 * The route is: PIT -> PIC -> LVT0 in NMI mode.
+	 *
+	 * Note: Our Virtual Wire implementation is simplified, only
+	 * propagating PIT interrupts to all VCPUs when they have set
+	 * LVT0 to NMI delivery. Other PIC interrupts are just sent to
+	 * VCPU0, and only if its LVT0 is in EXTINT mode.
+	 */
+	if (kvm->arch.vapics_in_nmi_mode > 0)
+		kvm_for_each_vcpu(i, vcpu, kvm)
+			kvm_apic_nmi_wd_deliver(vcpu);
 }
 
 static enum hrtimer_restart pit_timer_fn(struct hrtimer *data)
_

Re: nmi is broken?

[Avi Kivity]

On 04/24/2011 01:50 AM, OGAWA Hirofumi wrote:
> OGAWA Hirofumi<hirofumi@mail.parknet.co.jp>  writes:
>
> >  I noticed recently NMI on guest kernel is not working well.  host/guest
> >  kernel is 2.6.39-rc4, and using vmx.
> >
> >  And test code is something like the following:
> >
> >  	local_irq_disable();
> >  	for (i = 0; i<  10; i++) {
> >  		int cpu = get_cpu();
> >  		printk("%s: nmi %u, lapic %u\n", __FUNCTION__,
> >  			nmi_count(cpu), irq_stat[cpu].apic_timer_irqs);
> >  		mdelay(1000);
> >  		put_cpu();
> >  	}
> >
> >  the result is both of nmi and lapic are not increased. If I used
> >  -no-kvm-irqchip, it works fine (increase nmi only). So, it seems to be
> >  the bug of kvm driver side.
>
> With some debug, the cause seems to be in pit_do_work(). With the
> following patch, NMI watchdog seems to be working correctly (if irq
> disabled for long time, NMI watchdog can detect it).
>
> Is the following patch right?

This would cause IRQs to be delivered even if the PIT is masked, no?

Are you in fact using the PIT?  Linux prefers the HPET, and in my 
experience the -no-hpet option makes NMIs work.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: nmi is broken?

[OGAWA Hirofumi]

Avi Kivity <avi@redhat.com> writes:

>> With some debug, the cause seems to be in pit_do_work(). With the
>> following patch, NMI watchdog seems to be working correctly (if irq
>> disabled for long time, NMI watchdog can detect it).
>>
>> Is the following patch right?
>
> This would cause IRQs to be delivered even if the PIT is masked, no?

In here, mask means pit_mask_notifer()? If masked, ->pending doesn't
prevent to fire it? I'm not sure.

> Are you in fact using the PIT?  Linux prefers the HPET, and in my 
> experience the -no-hpet option makes NMIs work.

Yes, this is old kernel (2.6.9), and is not using HPET
(CONFIG_HPET_TIMER=n), using IO-APIC as NMI source (PMU seems to be
unsupported on kvm for now).

Unfortunately, -no-hpet didn't change situation. irq_stat.apic_timer_irqs
and nmi_count() are same while irq is disabled.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: nmi is broken?

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1181 bytes --]

On 2011-04-24 11:17, OGAWA Hirofumi wrote:
> Avi Kivity <avi@redhat.com> writes:
> 
>>> With some debug, the cause seems to be in pit_do_work(). With the
>>> following patch, NMI watchdog seems to be working correctly (if irq
>>> disabled for long time, NMI watchdog can detect it).
>>>
>>> Is the following patch right?
>>
>> This would cause IRQs to be delivered even if the PIT is masked, no?
> 
> In here, mask means pit_mask_notifer()? If masked, ->pending doesn't
> prevent to fire it? I'm not sure.
> 
>> Are you in fact using the PIT?  Linux prefers the HPET, and in my 
>> experience the -no-hpet option makes NMIs work.
> 
> Yes, this is old kernel (2.6.9), and is not using HPET
> (CONFIG_HPET_TIMER=n), using IO-APIC as NMI source (PMU seems to be
> unsupported on kvm for now).
> 
> Unfortunately, -no-hpet didn't change situation. irq_stat.apic_timer_irqs
> and nmi_count() are same while irq is disabled.

Maybe it's a regression of latest NMI injection patches.

This worked before? Can you check if commits
c2dd554dc61173ecb6b3741b680d2ae4c245d2ba and
f86368493ec038218e8663cc1b6e5393cd8e008a have any impact on your problem?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: nmi is broken?

[OGAWA Hirofumi]

Jan Kiszka <jan.kiszka@web.de> writes:

>> Yes, this is old kernel (2.6.9), and is not using HPET
>> (CONFIG_HPET_TIMER=n), using IO-APIC as NMI source (PMU seems to be
>> unsupported on kvm for now).
>> 
>> Unfortunately, -no-hpet didn't change situation. irq_stat.apic_timer_irqs
>> and nmi_count() are same while irq is disabled.
>
> Maybe it's a regression of latest NMI injection patches.
>
> This worked before? Can you check if commits
> c2dd554dc61173ecb6b3741b680d2ae4c245d2ba and
> f86368493ec038218e8663cc1b6e5393cd8e008a have any impact on your problem?

Unfortunately, no. I've checked kvm_inject_nmi() was not called if irq
was disabled on guest.

The path is following (without my patch, kvm_apic_nmi_wd_deliver is not
called),

pit_do_work()
    kvm_apic_nmi_wd_deliver()
        kvm_apic_local_deliver()
            __apic_accept_irq()
                kvm_inject_nmi()

so it is not meaningful to change after kvm_inject_nmi() (if I'm
understanding commits correctly).

Thanks.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: nmi is broken?

[OGAWA Hirofumi]

OGAWA Hirofumi <hirofumi@mail.parknet.co.jp> writes:

>> Maybe it's a regression of latest NMI injection patches.
>>
>> This worked before? Can you check if commits
>> c2dd554dc61173ecb6b3741b680d2ae4c245d2ba and
>> f86368493ec038218e8663cc1b6e5393cd8e008a have any impact on your problem?
>
> Unfortunately, no. I've checked kvm_inject_nmi() was not called if irq
> was disabled on guest.

To make sure situation, NMI is working if irq is enabled.

> The path is following (without my patch, kvm_apic_nmi_wd_deliver is not
> called),
>
> pit_do_work()
>     kvm_apic_nmi_wd_deliver()
>         kvm_apic_local_deliver()
>             __apic_accept_irq()
>                 kvm_inject_nmi()
>
> so it is not meaningful to change after kvm_inject_nmi() (if I'm
> understanding commits correctly).
>
> Thanks.

-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: nmi is broken?

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1473 bytes --]

On 2011-04-24 08:44, Avi Kivity wrote:
> On 04/24/2011 01:50 AM, OGAWA Hirofumi wrote:
>> OGAWA Hirofumi<hirofumi@mail.parknet.co.jp>  writes:
>>
>> >  I noticed recently NMI on guest kernel is not working well. 
>> host/guest
>> >  kernel is 2.6.39-rc4, and using vmx.
>> >
>> >  And test code is something like the following:
>> >
>> >      local_irq_disable();
>> >      for (i = 0; i<  10; i++) {
>> >          int cpu = get_cpu();
>> >          printk("%s: nmi %u, lapic %u\n", __FUNCTION__,
>> >              nmi_count(cpu), irq_stat[cpu].apic_timer_irqs);
>> >          mdelay(1000);
>> >          put_cpu();
>> >      }
>> >
>> >  the result is both of nmi and lapic are not increased. If I used
>> >  -no-kvm-irqchip, it works fine (increase nmi only). So, it seems to be
>> >  the bug of kvm driver side.
>>
>> With some debug, the cause seems to be in pit_do_work(). With the
>> following patch, NMI watchdog seems to be working correctly (if irq
>> disabled for long time, NMI watchdog can detect it).
>>
>> Is the following patch right?
> 
> This would cause IRQs to be delivered even if the PIT is masked, no?

I checked the patch and our code again: NMI watchdog masking is managed
via arch.vapics_in_nmi_mode and by re-checking the per-APIC mask
situation in kvm_apic_local_deliver when delivering the NMI.

So the patch looks correct - NMIs aren't acked like timer IRQs, the
current logic is definitely wrong.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: nmi is broken?

[Avi Kivity]

On 04/24/2011 03:24 PM, Jan Kiszka wrote:
> >
> >  This would cause IRQs to be delivered even if the PIT is masked, no?
>
> I checked the patch and our code again: NMI watchdog masking is managed
> via arch.vapics_in_nmi_mode and by re-checking the per-APIC mask
> situation in kvm_apic_local_deliver when delivering the NMI.
>
> So the patch looks correct - NMIs aren't acked like timer IRQs, the
> current logic is definitely wrong.

Can you elaborate?  Why aren't NMIs acked (if delivered via the PIC)?  
Is the PIC programmed into auto-EOI mode or something?

-- 
error compiling committee.c: too many arguments to function

Re: nmi is broken?

[Jan Kiszka]

On 2011-04-27 10:55, Avi Kivity wrote:
> On 04/24/2011 03:24 PM, Jan Kiszka wrote:
>> >
>> >  This would cause IRQs to be delivered even if the PIT is masked, no?
>>
>> I checked the patch and our code again: NMI watchdog masking is managed
>> via arch.vapics_in_nmi_mode and by re-checking the per-APIC mask
>> situation in kvm_apic_local_deliver when delivering the NMI.
>>
>> So the patch looks correct - NMIs aren't acked like timer IRQs, the
>> current logic is definitely wrong.
> 
> Can you elaborate?  Why aren't NMIs acked (if delivered via the PIC)? 
> Is the PIC programmed into auto-EOI mode or something?

At least there are no traces of 8259 interaction in the NMI handler of
old kernels when configured to IOAPIC mode. This wouldn't work anyway as
you can't synchronize with other PIC-accessing functions from the NMI
handler.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: nmi is broken?

[OGAWA Hirofumi]

Avi Kivity <avi@redhat.com> writes:

> On 04/24/2011 03:24 PM, Jan Kiszka wrote:
>> >
>> >  This would cause IRQs to be delivered even if the PIT is masked, no?
>>
>> I checked the patch and our code again: NMI watchdog masking is managed
>> via arch.vapics_in_nmi_mode and by re-checking the per-APIC mask
>> situation in kvm_apic_local_deliver when delivering the NMI.
>>
>> So the patch looks correct - NMIs aren't acked like timer IRQs, the
>> current logic is definitely wrong.
>
> Can you elaborate?  Why aren't NMIs acked (if delivered via the PIC)?  
> Is the PIC programmed into auto-EOI mode or something?

This seems to be complex stuff depending on hardware configurations. I'm
not fully understanding though, current state of it is,

Yes, PIC is in AEOI mode if linux is using IO-APIC. Um.., kvm says
irq == 0 is mp_INT mode in MADT, not mp_ExtINT. So I guess system does
automatically INTA cycle (and AEOI because of PIC config), or not
connected via 8259A? (like in mpspec figure 5-2.)

To checking it, I've tested in check_timer() of linux on the physical
machine (irq==0 and mp_INT). The test is something like,

	if (pin1 != -1) {
		/*
		 * Ok, does IRQ0 through the IOAPIC work?
		 */
		unmask_IO_APIC_irq(0);
                disable_8259_irq(0);
		if (timer_irq_works()) {

even if I called disable_8259_irq(0), timer was still working via
IO-APIC.

Would this explain why?

Thanks.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: nmi is broken?

[Avi Kivity]

On 04/28/2011 04:28 AM, OGAWA Hirofumi wrote:
> Avi Kivity<avi@redhat.com>  writes:
>
> >  On 04/24/2011 03:24 PM, Jan Kiszka wrote:
> >>  >
> >>  >   This would cause IRQs to be delivered even if the PIT is masked, no?
> >>
> >>  I checked the patch and our code again: NMI watchdog masking is managed
> >>  via arch.vapics_in_nmi_mode and by re-checking the per-APIC mask
> >>  situation in kvm_apic_local_deliver when delivering the NMI.
> >>
> >>  So the patch looks correct - NMIs aren't acked like timer IRQs, the
> >>  current logic is definitely wrong.
> >
> >  Can you elaborate?  Why aren't NMIs acked (if delivered via the PIC)?
> >  Is the PIC programmed into auto-EOI mode or something?
>
> This seems to be complex stuff depending on hardware configurations. I'm
> not fully understanding though, current state of it is,
>
> Yes, PIC is in AEOI mode if linux is using IO-APIC. Um.., kvm says
> irq == 0 is mp_INT mode in MADT, not mp_ExtINT.

That is correct, kvm doesn't connect the master 8259 output to the 
IOAPIC.  Instead the 8259 is connected to LINT0 (which is configured for 
ExtInt when the IOAPIC is disabled, or for NMI which the NMI watchdog is 
enabled).

However, now I can't see how it would work. auto EOI works on the INTA 
cycle, which would only occur if LINT0 is configured for ExtInt.  If it 
is configured for NMI, I don't think it would issue the INTA cycle.  So 
the NMI watchdog not working is actually correct for our hardware 
configuration!

But I may be misunderstanding something here.

> So I guess system does
> automatically INTA cycle (and AEOI because of PIC config), or not
> connected via 8259A? (like in mpspec figure 5-2.)
>
> To checking it, I've tested in check_timer() of linux on the physical
> machine (irq==0 and mp_INT). The test is something like,
>
> 	if (pin1 != -1) {
> 		/*
> 		 * Ok, does IRQ0 through the IOAPIC work?
> 		 */
> 		unmask_IO_APIC_irq(0);
>                  disable_8259_irq(0);
> 		if (timer_irq_works()) {
>
> even if I called disable_8259_irq(0), timer was still working via
> IO-APIC.
>
> Would this explain why?

Sorry, I got lost - what does this explain?

-- 
error compiling committee.c: too many arguments to function

Re: nmi is broken?

[OGAWA Hirofumi]

Avi Kivity <avi@redhat.com> writes:

>> This seems to be complex stuff depending on hardware configurations. I'm
>> not fully understanding though, current state of it is,
>>
>> Yes, PIC is in AEOI mode if linux is using IO-APIC. Um.., kvm says
>> irq == 0 is mp_INT mode in MADT, not mp_ExtINT.
>
> That is correct, kvm doesn't connect the master 8259 output to the 
> IOAPIC.  Instead the 8259 is connected to LINT0 (which is configured for 
> ExtInt when the IOAPIC is disabled, or for NMI which the NMI watchdog is 
> enabled).
>
> However, now I can't see how it would work. auto EOI works on the INTA 
> cycle, which would only occur if LINT0 is configured for ExtInt.  If it 
> is configured for NMI, I don't think it would issue the INTA cycle.  So 
> the NMI watchdog not working is actually correct for our hardware 
> configuration!
>
> But I may be misunderstanding something here.

I see. If the physical machine was configured as above, I guess (not
pretty sure, I don't have this configuration machine), IOAPIC test
(check_timer() in io_apic.c) should fail, and IOAPIC wouldn't have any
effect. And I think MADT should tell mp_ExtINT.

Yes, I also guess the above configuration wouldn't work NMI watchdog of
IOAPIC mode, and linux will report as NMI watchdog can't work in
check_timer().

>> So I guess system does
>> automatically INTA cycle (and AEOI because of PIC config), or not
>> connected via 8259A? (like in mpspec figure 5-2.)
>>
>> To checking it, I've tested in check_timer() of linux on the physical
>> machine (irq==0 and mp_INT). The test is something like,
>>
>> 	if (pin1 != -1) {
>> 		/*
>> 		 * Ok, does IRQ0 through the IOAPIC work?
>> 		 */
>> 		unmask_IO_APIC_irq(0);
>>                  disable_8259_irq(0);
>> 		if (timer_irq_works()) {
>>
>> even if I called disable_8259_irq(0), timer was still working via
>> IO-APIC.
>>
>> Would this explain why?
>
> Sorry, I got lost - what does this explain?

I think this explains irq == 0 and mp_INT configuration tell PIT is
connected to both of IOAPIC pin2 and PIC pin0, and it is why timer
interrupt is working even when PIC pin0 was disabled.

Thanks.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: nmi is broken?

[OGAWA Hirofumi]

OGAWA Hirofumi <hirofumi@mail.parknet.co.jp> writes:

> Avi Kivity <avi@redhat.com> writes:
>
>>> This seems to be complex stuff depending on hardware configurations. I'm
>>> not fully understanding though, current state of it is,
>>>
>>> Yes, PIC is in AEOI mode if linux is using IO-APIC. Um.., kvm says
>>> irq == 0 is mp_INT mode in MADT, not mp_ExtINT.
>>
>> That is correct, kvm doesn't connect the master 8259 output to the 
>> IOAPIC.  Instead the 8259 is connected to LINT0 (which is configured for 
>> ExtInt when the IOAPIC is disabled, or for NMI which the NMI watchdog is 
>> enabled).
>>
>> However, now I can't see how it would work. auto EOI works on the INTA 
>> cycle, which would only occur if LINT0 is configured for ExtInt.  If it 
>> is configured for NMI, I don't think it would issue the INTA cycle.  So 
>> the NMI watchdog not working is actually correct for our hardware 
>> configuration!
>>
>> But I may be misunderstanding something here.
>
> I see. If the physical machine was configured as above, I guess (not
> pretty sure, I don't have this configuration machine), IOAPIC test
> (check_timer() in io_apic.c) should fail, and IOAPIC wouldn't have any
> effect. And I think MADT should tell mp_ExtINT.
>
> Yes, I also guess the above configuration wouldn't work NMI watchdog of
> IOAPIC mode, and linux will report as NMI watchdog can't work in
> check_timer().

Hm.., if smp was enabled, what configuration model is used by kvm? I
think this configuration model can't work on smp.

Thanks.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: nmi is broken?

[OGAWA Hirofumi]

OGAWA Hirofumi <hirofumi@mail.parknet.co.jp> writes:

>>> That is correct, kvm doesn't connect the master 8259 output to the 
>>> IOAPIC.  Instead the 8259 is connected to LINT0 (which is configured for 
>>> ExtInt when the IOAPIC is disabled, or for NMI which the NMI watchdog is 
>>> enabled).
>>>
>>> However, now I can't see how it would work. auto EOI works on the INTA 
>>> cycle, which would only occur if LINT0 is configured for ExtInt.  If it 
>>> is configured for NMI, I don't think it would issue the INTA cycle.  So 
>>> the NMI watchdog not working is actually correct for our hardware 
>>> configuration!
>>>
>>> But I may be misunderstanding something here.
>>
>> I see. If the physical machine was configured as above, I guess (not
>> pretty sure, I don't have this configuration machine), IOAPIC test
>> (check_timer() in io_apic.c) should fail, and IOAPIC wouldn't have any
>> effect. And I think MADT should tell mp_ExtINT.
>>
>> Yes, I also guess the above configuration wouldn't work NMI watchdog of
>> IOAPIC mode, and linux will report as NMI watchdog can't work in
>> check_timer().
>
> Hm.., if smp was enabled, what configuration model is used by kvm? I
> think this configuration model can't work on smp.

As far as I can see, kvm is not configured (from MADT and some of
behaviors) like you said.

So, I think there are some solutions, a) current behavior is right (I
don't know why it's right though), b) fix the behavior of IO-APIC and
MADT like you said, then linux can detect it, c) change the model to
like mpspec figure 5-2, d) other.

My suggestion is c) if there is no good d). Because current behavior
looks like almost c), and non-legacy chipsets are using c) model as far
as I know.

Thanks.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: nmi is broken?

[Avi Kivity]

On 05/01/2011 04:45 AM, OGAWA Hirofumi wrote:
> >
> >  Hm.., if smp was enabled, what configuration model is used by kvm? I
> >  think this configuration model can't work on smp.
>
> As far as I can see, kvm is not configured (from MADT and some of
> behaviors) like you said.

We may well have an error there; and our NMI-from-PIT emulation bypasses 
all the wiring I described, so we may be emulating a configuration that 
can't possibly exist on hardware.

> So, I think there are some solutions, a) current behavior is right (I
> don't know why it's right though), b) fix the behavior of IO-APIC and
> MADT like you said, then linux can detect it, c) change the model to
> like mpspec figure 5-2, d) other.
>
> My suggestion is c) if there is no good d). Because current behavior
> looks like almost c), and non-legacy chipsets are using c) model as far
> as I know.

You're probably right.  However we can't just change it, we need to make 
it an option, keeping the current configuration as the default.  This is 
so that live migration can work, and because 5-2 requires a new 
kernel/user interface, to set IMCR.E0.

Looking at figures 3-3 and 3-4 of the mpspec, the current model supports 
3-3 but not 3-4.  Do we report that IMCR exists in the mptables?

-- 
error compiling committee.c: too many arguments to function

Re: nmi is broken?

[OGAWA Hirofumi]

Avi Kivity <avi@redhat.com> writes:

> On 05/01/2011 04:45 AM, OGAWA Hirofumi wrote:
>> >
>> >  Hm.., if smp was enabled, what configuration model is used by kvm? I
>> >  think this configuration model can't work on smp.
>>
>> As far as I can see, kvm is not configured (from MADT and some of
>> behaviors) like you said.
>
> We may well have an error there; and our NMI-from-PIT emulation bypasses 
> all the wiring I described, so we may be emulating a configuration that 
> can't possibly exist on hardware.
>
>> So, I think there are some solutions, a) current behavior is right (I
>> don't know why it's right though), b) fix the behavior of IO-APIC and
>> MADT like you said, then linux can detect it, c) change the model to
>> like mpspec figure 5-2, d) other.
>>
>> My suggestion is c) if there is no good d). Because current behavior
>> looks like almost c), and non-legacy chipsets are using c) model as far
>> as I know.
>
> You're probably right.  However we can't just change it, we need to make 
> it an option, keeping the current configuration as the default.  This is 
> so that live migration can work, and because 5-2 requires a new 
> kernel/user interface, to set IMCR.E0.
>
> Looking at figures 3-3 and 3-4 of the mpspec, the current model supports 
> 3-3 but not 3-4.  Do we report that IMCR exists in the mptables?

I don't think we have to implement IMCR, because it seems to be an
optional. In fact, physical hardwares which I have don't report IMCR in
mptables. (I don't see the benefit to implement it on recent chipsets
even if it's possible. The virtual wire mode seems to be enough.)

I don't know about live migration of kvm. If we said the wiring is like
figure 5-2, what is required for the live migration? It was required
only if IMCR was required?

Thanks.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: nmi is broken?

[Avi Kivity]

On 05/02/2011 05:30 PM, OGAWA Hirofumi wrote:
> >
> >>  So, I think there are some solutions, a) current behavior is right (I
> >>  don't know why it's right though), b) fix the behavior of IO-APIC and
> >>  MADT like you said, then linux can detect it, c) change the model to
> >>  like mpspec figure 5-2, d) other.
> >>
> >>  My suggestion is c) if there is no good d). Because current behavior
> >>  looks like almost c), and non-legacy chipsets are using c) model as far
> >>  as I know.
> >
> >  You're probably right.  However we can't just change it, we need to make
> >  it an option, keeping the current configuration as the default.  This is
> >  so that live migration can work, and because 5-2 requires a new
> >  kernel/user interface, to set IMCR.E0.
> >
> >  Looking at figures 3-3 and 3-4 of the mpspec, the current model supports
> >  3-3 but not 3-4.  Do we report that IMCR exists in the mptables?
>
> I don't think we have to implement IMCR, because it seems to be an
> optional. In fact, physical hardwares which I have don't report IMCR in
> mptables. (I don't see the benefit to implement it on recent chipsets
> even if it's possible. The virtual wire mode seems to be enough.)

Okay.

> I don't know about live migration of kvm. If we said the wiring is like
> figure 5-2, what is required for the live migration? It was required
> only if IMCR was required?

The issue with live migration is that we can't change the running 
configuration while the system is running, like adding the IMCR or 
changing the wiring.  The hardware will be programmed for the old 
configuration and will likely fail with the new one.  For example, the 
current wiring has the PIT output wired to PIC IRQ0 and IOAPIC INTI0; we 
need to change it to IOAPIC INTI2 instead.

-- 
error compiling committee.c: too many arguments to function

Re: nmi is broken?

[Avi Kivity]

On 05/03/2011 12:36 PM, Avi Kivity wrote:
>
>> I don't know about live migration of kvm. If we said the wiring is like
>> figure 5-2, what is required for the live migration? It was required
>> only if IMCR was required?
>
> The issue with live migration is that we can't change the running 
> configuration while the system is running, like adding the IMCR or 
> changing the wiring.  The hardware will be programmed for the old 
> configuration and will likely fail with the new one.  For example, the 
> current wiring has the PIT output wired to PIC IRQ0 and IOAPIC INTI0; 
> we need to change it to IOAPIC INTI2 instead.
>

btw, I believe that the configuration currently implemented is legal 
(it's similar to config 7 in table 5-2 of the mpspec); the only problem 
is that it can't support the NMI watchdog through the I/O APIC trick, 
yet we allow it through a hack.

Something we can do is connect the 8259A output to the I/O APIC INTIN2; 
it should be masked so live migration will continue to work.  We just 
have to make sure that the guest is able to find that it is connected there.

-- 
error compiling committee.c: too many arguments to function

Re: nmi is broken?

[OGAWA Hirofumi]

Avi Kivity <avi@redhat.com> writes:

> On 05/03/2011 12:36 PM, Avi Kivity wrote:
>>
>>> I don't know about live migration of kvm. If we said the wiring is like
>>> figure 5-2, what is required for the live migration? It was required
>>> only if IMCR was required?
>>
>> The issue with live migration is that we can't change the running 
>> configuration while the system is running, like adding the IMCR or 
>> changing the wiring.  The hardware will be programmed for the old 
>> configuration and will likely fail with the new one.  For example, the 
>> current wiring has the PIT output wired to PIC IRQ0 and IOAPIC INTI0; 
>> we need to change it to IOAPIC INTI2 instead.
>>
>
> btw, I believe that the configuration currently implemented is legal 
> (it's similar to config 7 in table 5-2 of the mpspec); the only problem 
> is that it can't support the NMI watchdog through the I/O APIC trick, 
> yet we allow it through a hack.

I'm confused a bit. "config 7 in table 5-2" says PIT output wired to
IOAPIC INTIN2. So, we don't need to change it?

> Something we can do is connect the 8259A output to the I/O APIC INTIN2; 
> it should be masked so live migration will continue to work.  We just 
> have to make sure that the guest is able to find that it is connected there.

8259A INTR to IOAPIC INTIN2 sounds strange. Isn't it IOAPIC INTIN0?

Thanks.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: nmi is broken?

[Avi Kivity]

On 05/03/2011 04:25 PM, OGAWA Hirofumi wrote:
> Avi Kivity<avi@redhat.com>  writes:
>
> >  On 05/03/2011 12:36 PM, Avi Kivity wrote:
> >>
> >>>  I don't know about live migration of kvm. If we said the wiring is like
> >>>  figure 5-2, what is required for the live migration? It was required
> >>>  only if IMCR was required?
> >>
> >>  The issue with live migration is that we can't change the running
> >>  configuration while the system is running, like adding the IMCR or
> >>  changing the wiring.  The hardware will be programmed for the old
> >>  configuration and will likely fail with the new one.  For example, the
> >>  current wiring has the PIT output wired to PIC IRQ0 and IOAPIC INTI0;
> >>  we need to change it to IOAPIC INTI2 instead.
> >>
> >
> >  btw, I believe that the configuration currently implemented is legal
> >  (it's similar to config 7 in table 5-2 of the mpspec); the only problem
> >  is that it can't support the NMI watchdog through the I/O APIC trick,
> >  yet we allow it through a hack.
>
> I'm confused a bit. "config 7 in table 5-2" says PIT output wired to
> IOAPIC INTIN2. So, we don't need to change it?

We're like config 7 in that the 8259A output isn't wired to the IOAPIC.  
However we're unlike config 7 in that the PIT output is wired to IOAPIC 
INTIN0.  I think we can keep it that way.

> >  Something we can do is connect the 8259A output to the I/O APIC INTIN2;
> >  it should be masked so live migration will continue to work.  We just
> >  have to make sure that the guest is able to find that it is connected there.
>
> 8259A INTR to IOAPIC INTIN2 sounds strange. Isn't it IOAPIC INTIN0?

Usually yes, but we already have the PIT wired to INTIN0.  I saw that 
the kernel consults the mptable to see which pin to use, so with the 
right BIOS magic we can get things to work.

-- 
error compiling committee.c: too many arguments to function

Re: nmi is broken?

[OGAWA Hirofumi]

Avi Kivity <avi@redhat.com> writes:

>> >>  The issue with live migration is that we can't change the running
>> >>  configuration while the system is running, like adding the IMCR or
>> >>  changing the wiring.  The hardware will be programmed for the old
>> >>  configuration and will likely fail with the new one.  For example, the
>> >>  current wiring has the PIT output wired to PIC IRQ0 and IOAPIC INTI0;
>> >>  we need to change it to IOAPIC INTI2 instead.
>> >>
>> >
>> >  btw, I believe that the configuration currently implemented is legal
>> >  (it's similar to config 7 in table 5-2 of the mpspec); the only problem
>> >  is that it can't support the NMI watchdog through the I/O APIC trick,
>> >  yet we allow it through a hack.
>>
>> I'm confused a bit. "config 7 in table 5-2" says PIT output wired to
>> IOAPIC INTIN2. So, we don't need to change it?
>
> We're like config 7 in that the 8259A output isn't wired to the IOAPIC.  
> However we're unlike config 7 in that the PIT output is wired to IOAPIC 
> INTIN0.  I think we can keep it that way.
>
>> >  Something we can do is connect the 8259A output to the I/O APIC INTIN2;
>> >  it should be masked so live migration will continue to work.  We just
>> >  have to make sure that the guest is able to find that it is
>> > connected there.
>>
>> 8259A INTR to IOAPIC INTIN2 sounds strange. Isn't it IOAPIC INTIN0?
>
> Usually yes, but we already have the PIT wired to INTIN0.  I saw that 
> the kernel consults the mptable to see which pin to use, so with the 
> right BIOS magic we can get things to work.

Um..., I'm confused more. If so, MADT doesn't say it. MADT says irq == 0
is pin == 2, this is one of reasons why linux is quite silent in
check_timer(). And I can't see why it is working by pin == 2 for IOAPIC.

If I can make time, I'll see what happens by pin == 2 and pin == 0 of
IOAPIC in kvm.

Thanks.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: nmi is broken?

[Avi Kivity]

On 05/03/2011 07:57 PM, OGAWA Hirofumi wrote:
> >
> >  Usually yes, but we already have the PIT wired to INTIN0.  I saw that
> >  the kernel consults the mptable to see which pin to use, so with the
> >  right BIOS magic we can get things to work.
>
> Um..., I'm confused more. If so, MADT doesn't say it. MADT says irq == 0
> is pin == 2, this is one of reasons why linux is quite silent in
> check_timer(). And I can't see why it is working by pin == 2 for IOAPIC.
>
> If I can make time, I'll see what happens by pin == 2 and pin == 0 of
> IOAPIC in kvm.

You're right.  The default routing is INTIN0, but qemu changes it to 
INTIN2 and tells kvm.

So INTIN0 is free for the 8259A output.

-- 
error compiling committee.c: too many arguments to function

Re: nmi is broken?

[OGAWA Hirofumi]

Avi Kivity <avi@redhat.com> writes:

> On 05/03/2011 07:57 PM, OGAWA Hirofumi wrote:
>> >
>> >  Usually yes, but we already have the PIT wired to INTIN0.  I saw that
>> >  the kernel consults the mptable to see which pin to use, so with the
>> >  right BIOS magic we can get things to work.
>>
>> Um..., I'm confused more. If so, MADT doesn't say it. MADT says irq == 0
>> is pin == 2, this is one of reasons why linux is quite silent in
>> check_timer(). And I can't see why it is working by pin == 2 for IOAPIC.
>>
>> If I can make time, I'll see what happens by pin == 2 and pin == 0 of
>> IOAPIC in kvm.
>
> You're right.  The default routing is INTIN0, but qemu changes it to 
> INTIN2 and tells kvm.
>
> So INTIN0 is free for the 8259A output.

I see. Did it mean qemu changes the wiring, so kvm can't work for live
migration with it?

Thanks.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: nmi is broken?

[Avi Kivity]

On 05/03/2011 08:45 PM, OGAWA Hirofumi wrote:
> Avi Kivity<avi@redhat.com>  writes:
>
> >  On 05/03/2011 07:57 PM, OGAWA Hirofumi wrote:
> >>  >
> >>  >   Usually yes, but we already have the PIT wired to INTIN0.  I saw that
> >>  >   the kernel consults the mptable to see which pin to use, so with the
> >>  >   right BIOS magic we can get things to work.
> >>
> >>  Um..., I'm confused more. If so, MADT doesn't say it. MADT says irq == 0
> >>  is pin == 2, this is one of reasons why linux is quite silent in
> >>  check_timer(). And I can't see why it is working by pin == 2 for IOAPIC.
> >>
> >>  If I can make time, I'll see what happens by pin == 2 and pin == 0 of
> >>  IOAPIC in kvm.
> >
> >  You're right.  The default routing is INTIN0, but qemu changes it to
> >  INTIN2 and tells kvm.
> >
> >  So INTIN0 is free for the 8259A output.
>
> I see. Did it mean qemu changes the wiring, so kvm can't work for live
> migration with it?

qemu changes the wiring, but it does so on both the migration source and 
the migration target (before either is launched), so they are migration 
compatible.

-- 
error compiling committee.c: too many arguments to function

Re: nmi is broken?

[OGAWA Hirofumi]

Avi Kivity <avi@redhat.com> writes:

> On 05/03/2011 08:45 PM, OGAWA Hirofumi wrote:
>> Avi Kivity<avi@redhat.com>  writes:
>>
>> >  On 05/03/2011 07:57 PM, OGAWA Hirofumi wrote:
>> >>  >
>> >>  >   Usually yes, but we already have the PIT wired to INTIN0.  I saw that
>> >>  >   the kernel consults the mptable to see which pin to use, so with the
>> >>  >   right BIOS magic we can get things to work.
>> >>
>> >>  Um..., I'm confused more. If so, MADT doesn't say it. MADT says irq == 0
>> >>  is pin == 2, this is one of reasons why linux is quite silent in
>> >>  check_timer(). And I can't see why it is working by pin == 2 for IOAPIC.
>> >>
>> >>  If I can make time, I'll see what happens by pin == 2 and pin == 0 of
>> >>  IOAPIC in kvm.
>> >
>> >  You're right.  The default routing is INTIN0, but qemu changes it to
>> >  INTIN2 and tells kvm.
>> >
>> >  So INTIN0 is free for the 8259A output.
>>
>> I see. Did it mean qemu changes the wiring, so kvm can't work for live
>> migration with it?
>
> qemu changes the wiring, but it does so on both the migration source and 
> the migration target (before either is launched), so they are migration 
> compatible.

OK. So, kvm have to detect current wiring to change behavior.  (E.g. nmi
watchdog of IOAPIC mode without irq_ack.) IOW, if kvm can detect it, we
would be able to solve issues.

Would my thought be true?

Thanks.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: nmi is broken?

[Avi Kivity]

On 05/04/2011 06:40 PM, OGAWA Hirofumi wrote:
> >
> >  qemu changes the wiring, but it does so on both the migration source and
> >  the migration target (before either is launched), so they are migration
> >  compatible.
>
> OK. So, kvm have to detect current wiring to change behavior.  (E.g. nmi
> watchdog of IOAPIC mode without irq_ack.) IOW, if kvm can detect it, we
> would be able to solve issues.
>
> Would my thought be true?
>

I think so.

It is also likely that all guests mask INTIN0 in the redirection table 
entry (except when the watchdog is active).  In that case the wiring 
different would make no change, and compatibility would be maintained.

-- 
error compiling committee.c: too many arguments to function

Re: nmi is broken?

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1718 bytes --]

On 2011-04-24 08:44, Avi Kivity wrote:
> On 04/24/2011 01:50 AM, OGAWA Hirofumi wrote:
>> OGAWA Hirofumi<hirofumi@mail.parknet.co.jp>  writes:
>>
>> >  I noticed recently NMI on guest kernel is not working well. 
>> host/guest
>> >  kernel is 2.6.39-rc4, and using vmx.
>> >
>> >  And test code is something like the following:
>> >
>> >      local_irq_disable();
>> >      for (i = 0; i<  10; i++) {
>> >          int cpu = get_cpu();
>> >          printk("%s: nmi %u, lapic %u\n", __FUNCTION__,
>> >              nmi_count(cpu), irq_stat[cpu].apic_timer_irqs);
>> >          mdelay(1000);
>> >          put_cpu();
>> >      }
>> >
>> >  the result is both of nmi and lapic are not increased. If I used
>> >  -no-kvm-irqchip, it works fine (increase nmi only). So, it seems to be
>> >  the bug of kvm driver side.
>>
>> With some debug, the cause seems to be in pit_do_work(). With the
>> following patch, NMI watchdog seems to be working correctly (if irq
>> disabled for long time, NMI watchdog can detect it).
>>
>> Is the following patch right?
> 
> This would cause IRQs to be delivered even if the PIT is masked, no?
> 
> Are you in fact using the PIT?  Linux prefers the HPET, and in my
> experience the -no-hpet option makes NMIs work.

BTW, that's another bug of the in-kernel PIT model: It disables the
timer in HPET legacy mode even if we are aware of NMI watchdog
receivers. Actually, the whole legacy disabling looks a bit strange in
the PIT (mode hackery + flag testing...).

While this should be fixed/refactored, adding basic perf support to KVM
will be the only option long-term as Linux dropped virtual-wire NMI
watchdog support some releases ago.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: nmi is broken?

[Avi Kivity]

On 04/24/2011 05:08 PM, Jan Kiszka wrote:
> On 2011-04-24 08:44, Avi Kivity wrote:
> >  On 04/24/2011 01:50 AM, OGAWA Hirofumi wrote:
> >>  OGAWA Hirofumi<hirofumi@mail.parknet.co.jp>   writes:
> >>
> >>  >   I noticed recently NMI on guest kernel is not working well.
> >>  host/guest
> >>  >   kernel is 2.6.39-rc4, and using vmx.
> >>  >
> >>  >   And test code is something like the following:
> >>  >
> >>  >       local_irq_disable();
> >>  >       for (i = 0; i<   10; i++) {
> >>  >           int cpu = get_cpu();
> >>  >           printk("%s: nmi %u, lapic %u\n", __FUNCTION__,
> >>  >               nmi_count(cpu), irq_stat[cpu].apic_timer_irqs);
> >>  >           mdelay(1000);
> >>  >           put_cpu();
> >>  >       }
> >>  >
> >>  >   the result is both of nmi and lapic are not increased. If I used
> >>  >   -no-kvm-irqchip, it works fine (increase nmi only). So, it seems to be
> >>  >   the bug of kvm driver side.
> >>
> >>  With some debug, the cause seems to be in pit_do_work(). With the
> >>  following patch, NMI watchdog seems to be working correctly (if irq
> >>  disabled for long time, NMI watchdog can detect it).
> >>
> >>  Is the following patch right?
> >
> >  This would cause IRQs to be delivered even if the PIT is masked, no?
> >
> >  Are you in fact using the PIT?  Linux prefers the HPET, and in my
> >  experience the -no-hpet option makes NMIs work.
>
> BTW, that's another bug of the in-kernel PIT model: It disables the
> timer in HPET legacy mode even if we are aware of NMI watchdog
> receivers. Actually, the whole legacy disabling looks a bit strange in
> the PIT (mode hackery + flag testing...).
>
> While this should be fixed/refactored, adding basic perf support to KVM
> will be the only option long-term as Linux dropped virtual-wire NMI
> watchdog support some releases ago.

Yes.  Unfortunately that is very vendor and model specific.  The 
architectural PMU is supported, but that is only available on Intel.

Perhaps we could emulate the architectural PMU on AMD as well, and make 
the detection code in the guest vendor agnostic.  Since it's based on a 
cpuid bit, it should be safe.

-- 
error compiling committee.c: too many arguments to function

Re: nmi is broken?

[Jan Kiszka]

On 2011-05-03 11:43, Avi Kivity wrote:
> On 04/24/2011 05:08 PM, Jan Kiszka wrote:
>> On 2011-04-24 08:44, Avi Kivity wrote:
>> >  On 04/24/2011 01:50 AM, OGAWA Hirofumi wrote:
>> >>  OGAWA Hirofumi<hirofumi@mail.parknet.co.jp>   writes:
>> >>
>> >>  >   I noticed recently NMI on guest kernel is not working well.
>> >>  host/guest
>> >>  >   kernel is 2.6.39-rc4, and using vmx.
>> >>  >
>> >>  >   And test code is something like the following:
>> >>  >
>> >>  >       local_irq_disable();
>> >>  >       for (i = 0; i<   10; i++) {
>> >>  >           int cpu = get_cpu();
>> >>  >           printk("%s: nmi %u, lapic %u\n", __FUNCTION__,
>> >>  >               nmi_count(cpu), irq_stat[cpu].apic_timer_irqs);
>> >>  >           mdelay(1000);
>> >>  >           put_cpu();
>> >>  >       }
>> >>  >
>> >>  >   the result is both of nmi and lapic are not increased. If I used
>> >>  >   -no-kvm-irqchip, it works fine (increase nmi only). So, it
>> seems to be
>> >>  >   the bug of kvm driver side.
>> >>
>> >>  With some debug, the cause seems to be in pit_do_work(). With the
>> >>  following patch, NMI watchdog seems to be working correctly (if irq
>> >>  disabled for long time, NMI watchdog can detect it).
>> >>
>> >>  Is the following patch right?
>> >
>> >  This would cause IRQs to be delivered even if the PIT is masked, no?
>> >
>> >  Are you in fact using the PIT?  Linux prefers the HPET, and in my
>> >  experience the -no-hpet option makes NMIs work.
>>
>> BTW, that's another bug of the in-kernel PIT model: It disables the
>> timer in HPET legacy mode even if we are aware of NMI watchdog
>> receivers. Actually, the whole legacy disabling looks a bit strange in
>> the PIT (mode hackery + flag testing...).
>>
>> While this should be fixed/refactored, adding basic perf support to KVM
>> will be the only option long-term as Linux dropped virtual-wire NMI
>> watchdog support some releases ago.
> 
> Yes.  Unfortunately that is very vendor and model specific.  The
> architectural PMU is supported, but that is only available on Intel.

Is it supposed to have any practical value already? I did not yet find a
magic -cpu switch to let Linux detect anything, not to speak of perf or
watchdog support.

> 
> Perhaps we could emulate the architectural PMU on AMD as well, and make
> the detection code in the guest vendor agnostic.  Since it's based on a
> cpuid bit, it should be safe.
> 

We may only make Linux happy this way, no?

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: nmi is broken?

[Avi Kivity]

On 05/03/2011 01:37 PM, Jan Kiszka wrote:
> >
> >  Yes.  Unfortunately that is very vendor and model specific.  The
> >  architectural PMU is supported, but that is only available on Intel.
>
> Is it supposed to have any practical value already? I did not yet find a
> magic -cpu switch to let Linux detect anything, not to speak of perf or
> watchdog support.

On the guest side it is supported for the watchdog 
(arch/x86/kernel/cpu/perfctr-watchdog.c, look for 
X86_FEATURE_ARCH_PERFMON).  It's also mentioned in perf_event_intel.c, 
but I don't know if it will work without the other PMU features being 
present.

> >
> >  Perhaps we could emulate the architectural PMU on AMD as well, and make
> >  the detection code in the guest vendor agnostic.  Since it's based on a
> >  cpuid bit, it should be safe.
> >
>
> We may only make Linux happy this way, no?

I would argue that if a feature is discoverable by a cpuid bit it 
shouldn't need to be qualified by vendor.  No idea how other OSes work 
this out (or even if they make use of the architectural PMU).

-- 
error compiling committee.c: too many arguments to function

Re: nmi is broken?

[Jan Kiszka]

On 2011-05-03 15:31, Avi Kivity wrote:
> On 05/03/2011 01:37 PM, Jan Kiszka wrote:
>>>
>>>  Yes.  Unfortunately that is very vendor and model specific.  The
>>>  architectural PMU is supported, but that is only available on Intel.
>>
>> Is it supposed to have any practical value already? I did not yet find a
>> magic -cpu switch to let Linux detect anything, not to speak of perf or
>> watchdog support.
> 
> On the guest side it is supported for the watchdog 
> (arch/x86/kernel/cpu/perfctr-watchdog.c, look for 
> X86_FEATURE_ARCH_PERFMON).  It's also mentioned in perf_event_intel.c, 
> but I don't know if it will work without the other PMU features being 
> present.

I've tested with some SUSE 2.6.38 guest kernel, and it complained like
this:

(-cpu kvm64)
Performance Events: unsupported Netburst CPU model 6 no PMU driver, software events only.
NMI watchdog disabled (cpu0): hardware events not enabled

or
(-cpu host)
Performance Events: unsupported p6 CPU model 37 no PMU driver, software events only.
NMI watchdog disabled (cpu0): hardware events not enabled

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: nmi is broken?

[Avi Kivity]

On 05/03/2011 05:29 PM, Jan Kiszka wrote:
> On 2011-05-03 15:31, Avi Kivity wrote:
> >  On 05/03/2011 01:37 PM, Jan Kiszka wrote:
> >>>
> >>>   Yes.  Unfortunately that is very vendor and model specific.  The
> >>>   architectural PMU is supported, but that is only available on Intel.
> >>
> >>  Is it supposed to have any practical value already? I did not yet find a
> >>  magic -cpu switch to let Linux detect anything, not to speak of perf or
> >>  watchdog support.
> >
> >  On the guest side it is supported for the watchdog
> >  (arch/x86/kernel/cpu/perfctr-watchdog.c, look for
> >  X86_FEATURE_ARCH_PERFMON).  It's also mentioned in perf_event_intel.c,
> >  but I don't know if it will work without the other PMU features being
> >  present.
>
> I've tested with some SUSE 2.6.38 guest kernel, and it complained like
> this:
>
> (-cpu kvm64)
> Performance Events: unsupported Netburst CPU model 6 no PMU driver, software events only.
> NMI watchdog disabled (cpu0): hardware events not enabled
>

Sorry, I meant to write, but forgot, that on the host side it is 
completely unsupported.  It shouldn't be too hard to use perf_events to 
emulate the architectural PMU.  Once we do that we can expose the 
architectural pmu bit and the guest will use it.

-- 
error compiling committee.c: too many arguments to function

Re: nmi is broken?

[Jan Kiszka]

On 2011-05-03 16:37, Avi Kivity wrote:
> On 05/03/2011 05:29 PM, Jan Kiszka wrote:
>> On 2011-05-03 15:31, Avi Kivity wrote:
>>>  On 05/03/2011 01:37 PM, Jan Kiszka wrote:
>>>>>
>>>>>   Yes.  Unfortunately that is very vendor and model specific.  The
>>>>>   architectural PMU is supported, but that is only available on Intel.
>>>>
>>>>  Is it supposed to have any practical value already? I did not yet find a
>>>>  magic -cpu switch to let Linux detect anything, not to speak of perf or
>>>>  watchdog support.
>>>
>>>  On the guest side it is supported for the watchdog
>>>  (arch/x86/kernel/cpu/perfctr-watchdog.c, look for
>>>  X86_FEATURE_ARCH_PERFMON).  It's also mentioned in perf_event_intel.c,
>>>  but I don't know if it will work without the other PMU features being
>>>  present.
>>
>> I've tested with some SUSE 2.6.38 guest kernel, and it complained like
>> this:
>>
>> (-cpu kvm64)
>> Performance Events: unsupported Netburst CPU model 6 no PMU driver, software events only.
>> NMI watchdog disabled (cpu0): hardware events not enabled
>>
> 
> Sorry, I meant to write, but forgot, that on the host side it is 
> completely unsupported.  It shouldn't be too hard to use perf_events to 
> emulate the architectural PMU.  Once we do that we can expose the 
> architectural pmu bit and the guest will use it.

Oh, and I already thought I would have missed some thrilling KVM patches...

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux