VMX: Host NMI triggering on NMI vmexit

VMX: Host NMI triggering on NMI vmexit

[Jan Kiszka]

Sheng,

out of curiosity: vmx_vcpu_run invokes 'int $2' to trigger a host NMI if
the VM exited due to an external NMI event. According to Intel specs I
have, software-triggered NMIs do not block hardware NMIs. So are we
facing the risk to receive another NMI while running the first handler?
Or will the VM be left with the hardware blocking logic armed? Or does
Linux not care about NMI handler re-entrance?

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: VMX: Host NMI triggering on NMI vmexit

[Avi Kivity]

Jan Kiszka wrote:
> Sheng,
>
> out of curiosity: vmx_vcpu_run invokes 'int $2' to trigger a host NMI if
> the VM exited due to an external NMI event. According to Intel specs I
> have, software-triggered NMIs do not block hardware NMIs. So are we
> facing the risk to receive another NMI while running the first handler?
> Or will the VM be left with the hardware blocking logic armed? Or does
> Linux not care about NMI handler re-entrance?
>   

All good questions.  Usually this doesn't happen since NMI sources are
far apart (oprofile, watchdog).

Maybe the answer is to generate the local nmi via an IPI-to-self command
to the local apic.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: VMX: Host NMI triggering on NMI vmexit

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1035 bytes --]

Avi Kivity wrote:
> Jan Kiszka wrote:
>> Sheng,
>>
>> out of curiosity: vmx_vcpu_run invokes 'int $2' to trigger a host NMI if
>> the VM exited due to an external NMI event. According to Intel specs I
>> have, software-triggered NMIs do not block hardware NMIs. So are we
>> facing the risk to receive another NMI while running the first handler?
>> Or will the VM be left with the hardware blocking logic armed? Or does
>> Linux not care about NMI handler re-entrance?
>>   
> 
> All good questions.  Usually this doesn't happen since NMI sources are
> far apart (oprofile, watchdog).

Only true until you have multiple unsynchronized NMI sources, e.g.
inter-CPU NMIs of kgdb + a watchdog. I just stumbled over several bugs
in kvm's and my own NMI code that were triggered by such a scenario
(sigh...).

> 
> Maybe the answer is to generate the local nmi via an IPI-to-self command
> to the local apic.
> 

That sounds like a good idea, will look into this right after fixing the
other NMI issues.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: VMX: Host NMI triggering on NMI vmexit

[Jan Kiszka]

Avi Kivity wrote:
> Jan Kiszka wrote:
>> Sheng,
>>
>> out of curiosity: vmx_vcpu_run invokes 'int $2' to trigger a host NMI if
>> the VM exited due to an external NMI event. According to Intel specs I
>> have, software-triggered NMIs do not block hardware NMIs. So are we
>> facing the risk to receive another NMI while running the first handler?
>> Or will the VM be left with the hardware blocking logic armed? Or does
>> Linux not care about NMI handler re-entrance?
>>   
> 
> All good questions.  Usually this doesn't happen since NMI sources are
> far apart (oprofile, watchdog).
> 
> Maybe the answer is to generate the local nmi via an IPI-to-self command
> to the local apic.

Going this way leaves me with a few questions: Will it be OK for the
related mainainers to export the required service? And is it safe to
assume VMX == LAPIC available and usable?

However, this is how it would look like. Yet untested, /me has to
replace his host kernel first...

Jan

---
 arch/x86/kernel/genapic_64.c |    1 +
 arch/x86/kernel/ipi.c        |    1 +
 arch/x86/kvm/vmx.c           |    6 +++---
 3 files changed, 5 insertions(+), 3 deletions(-)

Index: b/arch/x86/kernel/genapic_64.c
===================================================================
--- a/arch/x86/kernel/genapic_64.c
+++ b/arch/x86/kernel/genapic_64.c
@@ -65,6 +65,7 @@ void send_IPI_self(int vector)
 {
 	__send_IPI_shortcut(APIC_DEST_SELF, vector, APIC_DEST_PHYSICAL);
 }
+EXPORT_SYMBOL_GPL(send_IPI_self);
 
 int __init acpi_madt_oem_check(char *oem_id, char *oem_table_id)
 {
Index: b/arch/x86/kernel/ipi.c
===================================================================
--- a/arch/x86/kernel/ipi.c
+++ b/arch/x86/kernel/ipi.c
@@ -77,6 +77,7 @@ void send_IPI_self(int vector)
 {
 	__send_IPI_shortcut(APIC_DEST_SELF, vector);
 }
+EXPORT_SYMBOL_GPL(send_IPI_self);
 
 /*
  * This is used to send an IPI with no shorthand notation (the destination is
Index: b/arch/x86/kvm/vmx.c
===================================================================
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -2488,7 +2488,7 @@ static int handle_exception(struct kvm_v
 		set_bit(irq / BITS_PER_LONG, &vcpu->arch.irq_summary);
 	}
 
-	if ((intr_info & INTR_INFO_INTR_TYPE_MASK) == 0x200) /* nmi */
+	if ((intr_info & INTR_INFO_INTR_TYPE_MASK) == INTR_TYPE_NMI_INTR)
 		return 1;  /* already handled by vmx_vcpu_run() */
 
 	if (is_no_device(intr_info)) {
@@ -3321,10 +3321,10 @@ static void vmx_vcpu_run(struct kvm_vcpu
 	intr_info = vmcs_read32(VM_EXIT_INTR_INFO);
 
 	/* We need to handle NMIs before interrupts are enabled */
-	if ((intr_info & INTR_INFO_INTR_TYPE_MASK) == 0x200 &&
+	if ((intr_info & INTR_INFO_INTR_TYPE_MASK) == INTR_TYPE_NMI_INTR &&
 	    (intr_info & INTR_INFO_VALID_MASK)) {
 		KVMTRACE_0D(NMI, vcpu, handler);
-		asm("int $2");
+		send_IPI_self(APIC_DM_NMI);
 	}
 
 	vmx_complete_interrupts(vmx);

Re: VMX: Host NMI triggering on NMI vmexit

[Avi Kivity]

Jan Kiszka wrote:
>> Maybe the answer is to generate the local nmi via an IPI-to-self command
>> to the local apic.
>>     
>
> Going this way leaves me with a few questions: Will it be OK for the
> related mainainers to export the required service? 

If we can make a case for it (I think we can), then I don't see why not.

Sheng, can you confirm that 'int 2' is problematic, and that 
nmi-via-lapic is the best workaround?

> And is it safe to
> assume VMX == LAPIC available and usable?
>
>   

Yes.

> However, this is how it would look like. 

I'd define a send_nmi_self() instead, to allow the implementation to 
change (x2apic/etc).

> Yet untested, /me has to
> replace his host kernel first...
>   

You could test it in a VM, if someone implements nested vmx :)

btw, looks like svm is not affected by this.

-- 
error compiling committee.c: too many arguments to function

Re: VMX: Host NMI triggering on NMI vmexit

[Yang, Sheng]

On Monday 22 September 2008 19:00:38 Avi Kivity wrote:
> Jan Kiszka wrote:
> >> Maybe the answer is to generate the local nmi via an IPI-to-self command
> >> to the local apic.
> >
> > Going this way leaves me with a few questions: Will it be OK for the
> > related mainainers to export the required service?
>
> If we can make a case for it (I think we can), then I don't see why not.
>
> Sheng, can you confirm that 'int 2' is problematic, and that
> nmi-via-lapic is the best workaround?

Just back from vacation... :)

Jan said is true, "int 2" itself won't block subsequent NMIs. But I think it's 
too obviously as a hardware issue when using with NMI exiting=1 in vmx 
nonroot mode, so I have checked it with my colleague, finally found these in 
SDM 3B 23-2:

The following bullets detail when architectural state is and is not updated in 
response to VM exits:
•   If an event causes a VM exit *directly*, it does not update architectural 
state as it would have if it had it not caused the VM exit:
[...]
— *An NMI causes subsequent NMIs to be blocked*, but only after the VM exit
  completes.

So we needn't worry about that, and this shouldn't cause any trouble AFAIK...

Jan, seems we need to do more investigating on the issues you met...

-- 
regards
Yang, Sheng

> > And is it safe to
> > assume VMX == LAPIC available and usable?
>
> Yes.
>
> > However, this is how it would look like.
>
> I'd define a send_nmi_self() instead, to allow the implementation to
> change (x2apic/etc).
>
> > Yet untested, /me has to
> > replace his host kernel first...
>
> You could test it in a VM, if someone implements nested vmx :)
>
> btw, looks like svm is not affected by this.
>
> --
> error compiling committee.c: too many arguments to function

Re: VMX: Host NMI triggering on NMI vmexit

[Jan Kiszka]

Yang, Sheng wrote:
> On Monday 22 September 2008 19:00:38 Avi Kivity wrote:
>> Jan Kiszka wrote:
>>>> Maybe the answer is to generate the local nmi via an IPI-to-self command
>>>> to the local apic.
>>> Going this way leaves me with a few questions: Will it be OK for the
>>> related mainainers to export the required service?
>> If we can make a case for it (I think we can), then I don't see why not.
>>
>> Sheng, can you confirm that 'int 2' is problematic, and that
>> nmi-via-lapic is the best workaround?
> 
> Just back from vacation... :)
> 
> Jan said is true, "int 2" itself won't block subsequent NMIs. But I think it's 
> too obviously as a hardware issue when using with NMI exiting=1 in vmx 
> nonroot mode, so I have checked it with my colleague, finally found these in 
> SDM 3B 23-2:
> 
> The following bullets detail when architectural state is and is not updated in 
> response to VM exits:
> •   If an event causes a VM exit *directly*, it does not update architectural 
> state as it would have if it had it not caused the VM exit:
> [...]
> — *An NMI causes subsequent NMIs to be blocked*, but only after the VM exit
>   completes.
> 
> So we needn't worry about that, and this shouldn't cause any trouble AFAIK...

Fine, problems--. :)

> 
> Jan, seems we need to do more investigating on the issues you met...

Sorry, which one do you mean now?

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: VMX: Host NMI triggering on NMI vmexit

[Yang, Sheng]

On Tuesday 23 September 2008 16:47:54 Jan Kiszka wrote:
> Yang, Sheng wrote:
> > On Monday 22 September 2008 19:00:38 Avi Kivity wrote:
> >> Jan Kiszka wrote:
> >>>> Maybe the answer is to generate the local nmi via an IPI-to-self
> >>>> command to the local apic.
> >>>
> >>> Going this way leaves me with a few questions: Will it be OK for the
> >>> related mainainers to export the required service?
> >>
> >> If we can make a case for it (I think we can), then I don't see why not.
> >>
> >> Sheng, can you confirm that 'int 2' is problematic, and that
> >> nmi-via-lapic is the best workaround?
> >
> > Just back from vacation... :)
> >
> > Jan said is true, "int 2" itself won't block subsequent NMIs. But I think
> > it's too obviously as a hardware issue when using with NMI exiting=1 in
> > vmx nonroot mode, so I have checked it with my colleague, finally found
> > these in SDM 3B 23-2:
> >
> > The following bullets detail when architectural state is and is not
> > updated in response to VM exits:
> > •   If an event causes a VM exit *directly*, it does not update
> > architectural state as it would have if it had it not caused the VM exit:
> > [...]
> > — *An NMI causes subsequent NMIs to be blocked*, but only after the VM
> > exit completes.
> >
> > So we needn't worry about that, and this shouldn't cause any trouble
> > AFAIK...
>
> Fine, problems--. :)
>
> > Jan, seems we need to do more investigating on the issues you met...
>
> Sorry, which one do you mean now?

You said 
"Only true until you have multiple unsynchronized NMI sources, e.g.
inter-CPU NMIs of kgdb + a watchdog. I just stumbled over several bugs
in kvm's and my own NMI code that were triggered by such a scenario
(sigh...)." ?

If it's not related to this one. That's fine. :)

-- 
regards
Yang, Sheng


>
> Jan
>
> --
> Siemens AG, Corporate Technology, CT SE 2
> Corporate Competence Center Embedded Linux

Re: VMX: Host NMI triggering on NMI vmexit

[Jan Kiszka]

Yang, Sheng wrote:
> On Tuesday 23 September 2008 16:47:54 Jan Kiszka wrote:
>> Yang, Sheng wrote:
>>> On Monday 22 September 2008 19:00:38 Avi Kivity wrote:
>>>> Jan Kiszka wrote:
>>>>>> Maybe the answer is to generate the local nmi via an IPI-to-self
>>>>>> command to the local apic.
>>>>> Going this way leaves me with a few questions: Will it be OK for the
>>>>> related mainainers to export the required service?
>>>> If we can make a case for it (I think we can), then I don't see why not.
>>>>
>>>> Sheng, can you confirm that 'int 2' is problematic, and that
>>>> nmi-via-lapic is the best workaround?
>>> Just back from vacation... :)
>>>
>>> Jan said is true, "int 2" itself won't block subsequent NMIs. But I think
>>> it's too obviously as a hardware issue when using with NMI exiting=1 in
>>> vmx nonroot mode, so I have checked it with my colleague, finally found
>>> these in SDM 3B 23-2:
>>>
>>> The following bullets detail when architectural state is and is not
>>> updated in response to VM exits:
>>> •   If an event causes a VM exit *directly*, it does not update
>>> architectural state as it would have if it had it not caused the VM exit:
>>> [...]
>>> — *An NMI causes subsequent NMIs to be blocked*, but only after the VM
>>> exit completes.
>>>
>>> So we needn't worry about that, and this shouldn't cause any trouble
>>> AFAIK...
>> Fine, problems--. :)
>>
>>> Jan, seems we need to do more investigating on the issues you met...
>> Sorry, which one do you mean now?
> 
> You said 
> "Only true until you have multiple unsynchronized NMI sources, e.g.
> inter-CPU NMIs of kgdb + a watchdog. I just stumbled over several bugs
> in kvm's and my own NMI code that were triggered by such a scenario
> (sigh...)." ?
> 
> If it's not related to this one. That's fine. :)

Nope, the actual issues were related to guests facing "NMI storms" (and
should be fixed by my series). That made me wonder what would happen on
the host side if int 2 wasn't safe. But it is, as we know now.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux