VMX: NMI injection without virtual NMI support

VMX: NMI injection without virtual NMI support

[Jan Kiszka]

Hi Sheng,

we had parts of this discussion privately a while back, but now I need
to dig deeper. I finally have to complete and push out user space NMI
injection that bitrots over and over again in my queue.

Intel CPUs with virtual NMI support can easily be programmed to inject
NMIs only when the guest CPU is ready or tell the guest to exit as soon
as it changes its NMI readiness. But we are also facing a lot of CPUs
that were produced without this feature, and we need to find some way to
achieve virtual NMI support for them, even if the interruptibility of
the guest is not as good as with true virtual NMIs.

Now my questions to you or anyone else with VMX expert knowledge:

The System Programming Guide, Volume 3B, 27.2 says that "Without
NMI-window exiting support, the VMM will need to poll and check the
interruptibility state of the guest to deliver virtual NMIs."
Interruptibility involves, besides "blocked by MOV SS" and "blocked by
STI", the NMI nesting. So, how can the VMM track if the guest is still
inside its NMI handler after event injection?

According to 22.6.1, NMI blocking is active as long as the guest runs if
the "NMI-exiting" control bit is 1 (and setting it appears to be
required to avoid that the guest can block NMIs for the host, right?).
But what does this mean for bit 3 of the interruptibility state? Can I
still use it after some guest exit for whatever reason (like a hard IRQ)
to poll if the guest can now accept pending virtual NMIs? In that case,
virtual NMI injections would only be delayed a bit on older CPUs, but
could still work reliably, right? Or is there some other way to track
the NMI interruptibility on such CPUs?

TiA,
Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: VMX: NMI injection without virtual NMI support

[Yang, Sheng]

On Thursday 11 September 2008 23:11:14 Jan Kiszka wrote:
> Hi Sheng,
>
> we had parts of this discussion privately a while back, but now I need
> to dig deeper. I finally have to complete and push out user space NMI
> injection that bitrots over and over again in my queue.
>
> Intel CPUs with virtual NMI support can easily be programmed to inject
> NMIs only when the guest CPU is ready or tell the guest to exit as soon
> as it changes its NMI readiness. But we are also facing a lot of CPUs
> that were produced without this feature, and we need to find some way to
> achieve virtual NMI support for them, even if the interruptibility of
> the guest is not as good as with true virtual NMIs.
>
> Now my questions to you or anyone else with VMX expert knowledge:
>
> The System Programming Guide, Volume 3B, 27.2 says that "Without
> NMI-window exiting support, the VMM will need to poll and check the
> interruptibility state of the guest to deliver virtual NMIs."
> Interruptibility involves, besides "blocked by MOV SS" and "blocked by
> STI", the NMI nesting. So, how can the VMM track if the guest is still
> inside its NMI handler after event injection?

I think, we have no direct idea about that...

> According to 22.6.1, NMI blocking is active as long as the guest runs if
> the "NMI-exiting" control bit is 1 (and setting it appears to be
> required to avoid that the guest can block NMIs for the host, right?).

Yes.

> But what does this mean for bit 3 of the interruptibility state? Can I
> still use it after some guest exit for whatever reason (like a hard IRQ)
> to poll if the guest can now accept pending virtual NMIs? 

No. Though the public spec is a little ambiguous, I just checked some document 
and found that, if "virtual NMI" is not enabled, "Block by NMI" just 
indicated block *host physical* NMI rather than virtual NMI.

> In that case, virtual NMI injections would only be delayed a bit on older 
>CPUs, but could still work reliably, right? Or is there some other way to 
>track the NMI interruptibility on such CPUs?

I hope so, but sadly seems we can't know if we can inject NMI to guest without 
enable "virtual NMI".

After some brainstorming with my colleague Haitao, we found one way *may* can 
work on non-virtual nmi supported hardware, but it's very tricky and 
untested(just a theoretical idea), also sacrifice host physical NMI, and 
depends on guest NMI implement. 

The basic idea is:
1. Disable "NMI Exiting" feature, so that guest would handle any host physical 
NMI.
2. Set "Blocking by NMI" to true. So if guest execute "iret", this bit should 
be cleaned. This also block physical NMIs. 
3. Enable "IRQ Window" and ensure IDT vector 2 is a interrupt gate(so that IF 
bit is cleared when executing the handler). So we use IRQ window to replace 
NMI window here, otherwise for the period between IRET and VMEXIT, host 
physical NMIs would be handled by guest handler.
4. Check "IRQ Window" exit, if "Blocking by NMI" is cleared(also STI and SS), 
we can inject NMI.

It's just untested idea, and I think it's quite tricky. If you still have 
interest, you may try it, but I can't guarantee the result...

Thanks.

-- 
regards
Yang, Sheng

>
> TiA,
> Jan

Re: VMX: NMI injection without virtual NMI support

[Jan Kiszka]

Hi Sheng,

thanks for your thoughts!

Yang, Sheng wrote:
> On Thursday 11 September 2008 23:11:14 Jan Kiszka wrote:
>> Hi Sheng,
>>
>> we had parts of this discussion privately a while back, but now I need
>> to dig deeper. I finally have to complete and push out user space NMI
>> injection that bitrots over and over again in my queue.
>>
>> Intel CPUs with virtual NMI support can easily be programmed to inject
>> NMIs only when the guest CPU is ready or tell the guest to exit as soon
>> as it changes its NMI readiness. But we are also facing a lot of CPUs
>> that were produced without this feature, and we need to find some way to
>> achieve virtual NMI support for them, even if the interruptibility of
>> the guest is not as good as with true virtual NMIs.
>>
>> Now my questions to you or anyone else with VMX expert knowledge:
>>
>> The System Programming Guide, Volume 3B, 27.2 says that "Without
>> NMI-window exiting support, the VMM will need to poll and check the
>> interruptibility state of the guest to deliver virtual NMIs."
>> Interruptibility involves, besides "blocked by MOV SS" and "blocked by
>> STI", the NMI nesting. So, how can the VMM track if the guest is still
>> inside its NMI handler after event injection?
> 
> I think, we have no direct idea about that...
> 
>> According to 22.6.1, NMI blocking is active as long as the guest runs if
>> the "NMI-exiting" control bit is 1 (and setting it appears to be
>> required to avoid that the guest can block NMIs for the host, right?).
> 
> Yes.
> 
>> But what does this mean for bit 3 of the interruptibility state? Can I
>> still use it after some guest exit for whatever reason (like a hard IRQ)
>> to poll if the guest can now accept pending virtual NMIs? 
> 
> No. Though the public spec is a little ambiguous, I just checked some document 
> and found that, if "virtual NMI" is not enabled, "Block by NMI" just 
> indicated block *host physical* NMI rather than virtual NMI.

Grmbl.

> 
>> In that case, virtual NMI injections would only be delayed a bit on older 
>> CPUs, but could still work reliably, right? Or is there some other way to 
>> track the NMI interruptibility on such CPUs?
> 
> I hope so, but sadly seems we can't know if we can inject NMI to guest without 
> enable "virtual NMI".
> 
> After some brainstorming with my colleague Haitao, we found one way *may* can 
> work on non-virtual nmi supported hardware, but it's very tricky and 
> untested(just a theoretical idea), also sacrifice host physical NMI, and 
> depends on guest NMI implement. 
> 
> The basic idea is:
> 1. Disable "NMI Exiting" feature, so that guest would handle any host physical 
> NMI.
> 2. Set "Blocking by NMI" to true. So if guest execute "iret", this bit should 
> be cleaned. This also block physical NMIs. 
> 3. Enable "IRQ Window" and ensure IDT vector 2 is a interrupt gate(so that IF 
> bit is cleared when executing the handler). So we use IRQ window to replace 
> NMI window here, otherwise for the period between IRET and VMEXIT, host 
> physical NMIs would be handled by guest handler.
> 4. Check "IRQ Window" exit, if "Blocking by NMI" is cleared(also STI and SS), 
> we can inject NMI.
> 
> It's just untested idea, and I think it's quite tricky. If you still have 
> interest, you may try it, but I can't guarantee the result...

Well, I thought in this direction already as well. But I wasn't sure if,
while the guest is in NMI context, hard IRQs will also be blocked and
won't cause guest exists anymore. Can you comment on this?

However, even if that is no issue, I do not really like this workaround.
Specifically the need to fiddle with the guest's IDT and, of course,
that we may delays host NMIs.

I'm now playing with this idea, basically a "light" version of yours:
After we injected an NMI, consider the guest being in NMI context until
the next IRQ window opens. That may cause lost NMIs if the guest blocks
IRQ delivery infinitely, but I would say this is rather untypical and
still much better than the current situation (no NMIs at all!). And it
is easier to implement. Comments?

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: VMX: NMI injection without virtual NMI support

[Avi Kivity]

Jan Kiszka wrote:
> Well, I thought in this direction already as well. But I wasn't sure if,
> while the guest is in NMI context, hard IRQs will also be blocked and
> won't cause guest exists anymore. Can you comment on this?
>
> However, even if that is no issue, I do not really like this workaround.
> Specifically the need to fiddle with the guest's IDT and, of course,
> that we may delays host NMIs.
>
> I'm now playing with this idea, basically a "light" version of yours:
> After we injected an NMI, consider the guest being in NMI context until
> the next IRQ window opens. That may cause lost NMIs if the guest blocks
> IRQ delivery infinitely, but I would say this is rather untypical and
> still much better than the current situation (no NMIs at all!). And it
> is easier to implement. Comments?
>
>
>   


In some cases misbehaving NMIs are worse than no NMIs.  For example, a
software watchdog may use NMIs to monitor a system.  But if the guest
spins with interrupts disabled, the irq window will never open, and NMIs
will never be delivered, so the watchdog will deliver a false negative.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: VMX: NMI injection without virtual NMI support

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1709 bytes --]

Avi Kivity wrote:
> Jan Kiszka wrote:
>> Well, I thought in this direction already as well. But I wasn't sure if,
>> while the guest is in NMI context, hard IRQs will also be blocked and
>> won't cause guest exists anymore. Can you comment on this?
>>
>> However, even if that is no issue, I do not really like this workaround.
>> Specifically the need to fiddle with the guest's IDT and, of course,
>> that we may delays host NMIs.
>>
>> I'm now playing with this idea, basically a "light" version of yours:
>> After we injected an NMI, consider the guest being in NMI context until
>> the next IRQ window opens. That may cause lost NMIs if the guest blocks
>> IRQ delivery infinitely, but I would say this is rather untypical and
>> still much better than the current situation (no NMIs at all!). And it
>> is easier to implement. Comments?
>>
>>
>>   
> 
> 
> In some cases misbehaving NMIs are worse than no NMIs.  For example, a
> software watchdog may use NMIs to monitor a system.  But if the guest
> spins with interrupts disabled, the irq window will never open, and NMIs
> will never be delivered, so the watchdog will deliver a false negative.
> 

I fail to see the regression. Currently that watchdog would _always_
deliver false positives and pull the trigger immediately (in fact, this
is precisely the situation we face @work with some special board
emulation where we have to provide an NMI-based watchdog).

Moreover, only the second and succeeding NMIs under the same
interrupts-disabled window need to get lost: Along with injecting the
first NMI we could request the IRQ window unconditionally, using it to
reset the virtual NMI-blocked state.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: VMX: NMI injection without virtual NMI support

[Avi Kivity]

Jan Kiszka wrote:
>> In some cases misbehaving NMIs are worse than no NMIs.  For example, a
>> software watchdog may use NMIs to monitor a system.  But if the guest
>> spins with interrupts disabled, the irq window will never open, and NMIs
>> will never be delivered, so the watchdog will deliver a false negative.
>>
>>     
>
> I fail to see the regression. Currently that watchdog would _always_
> deliver false positives and pull the trigger immediately (in fact, this
> is precisely the situation we face @work with some special board
> emulation where we have to provide an NMI-based watchdog).
>
>   

Linux checks whether nmi works and enables it only if it does (I think
-- not sure).  So for Linux, there would be a regression.

> Moreover, only the second and succeeding NMIs under the same
> interrupts-disabled window need to get lost: Along with injecting the
> first NMI we could request the IRQ window unconditionally, using it to
> reset the virtual NMI-blocked state.
>   

But the interrupt window would never open.  Consider a spin_lock()
executing with interrupts disabled, on a spin lock that is already
locked by the current cpu.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.