Create VM exits when guest executes IRET on Intel CPU ?

Create VM exits when guest executes IRET on Intel CPU ?

[sci sci]

Dear KVM experts,

I am trying to create a VM exit whenever the guest OS executes a IRET
instruction. I am using a Intel i5,3.3Ghz,VT-x/EPT processor and KVM
version 3.10.1.

I tried several combinations of the following settings but with little
success.

1) Set/Unset Virtual NMIs which is bit 5 in pin based vm execution control
2) Set/Unset bit 3 in guest interruptibility state (Blocking by NMI)
3) Set/Unset "NMI-blocking" which is bit 3 in pin based NMI exiting
4) Set/Unset "NMI-window exiting" which is bit 22 in primary processor
based VM execution control.

The intel manual says the following about creating vm exits when IRET is
executed : "NMI-window exiting. If the “virtual NMIs” VM-execution is set,
the processor tracks virtual-NMI blocking. The “NMI-window exiting”
VM-execution control (Section 24.6.2) causes VM exits when there is no
virtual-NMI blocking. For example, after execution of the IRET instruction,
a VM exit occurs if the “NMI-window exiting” VMexecution control is 1. This
feature allows a VMM to queue a virtual NMI to a guest when the guest is
not ready to receive NMIs. The VMM can set the “NMI-window exiting”
VM-execution control for the guest and depend on a VM exit to know when the
guest becomes ready for NMIs (and, therefore, when it can inject a virtual
NMI). The VMM can detect such VM exits by checking for the basic exit
reason “NMI window” (value = 8). If this feature is not used, the VMM will
need to poll and check the interruptibility state of the guest to deliver
virtual NMIs."

I will be very thankful for any help from you to allow intercepting (cause
VM exits) when guest executes IRET.

Thanks!

Re: Create VM exits when guest executes IRET on Intel CPU ?

[Paolo Bonzini]

On 24/02/2017 11:32, sci sci wrote:
> I tried several combinations of the following settings but with little
> success.
> 
> 1) Set/Unset Virtual NMIs which is bit 5 in pin based vm execution control
> 2) Set/Unset bit 3 in guest interruptibility state (Blocking by NMI)
> 3) Set/Unset "NMI-blocking" which is bit 3 in pin based NMI exiting
> 4) Set/Unset "NMI-window exiting" which is bit 22 in primary processor
> based VM execution control.

Doing all four of these should do it:

- set bit 3 in pin-based controls to cause a VMEXIT for host NMIs and
enable special processing of IRET (paragraph 25.3)

- set bit 5 in pin-based controls so that guest interruptibility state
tracks virtual-NMI blocking

- set bit 22 of VM executino control to enable the NMI-window vmexit

- set bit 3 in guest interruptibility state to delay the NMI-window exit
until the next IRET

Paolo

Re: Create VM exits when guest executes IRET on Intel CPU ?

[sci sci]

Thanks very much for the reply.

I set all the four that you mentioned and am still having trouble with
intecepting guest IRET.

I print out the following right before the assembly instructions for
entering guest in vmx_vcpu_run() function in vmx.c  file.

"vmcs config pin based 3f cpu based b6e0edfa cpu based2 fb interruptibility 8"

The printf statement producing this is :
printk(KERN_ERR " vmcs config pin based %x cpu based %x cpu based2 %x
interruptibility %x\n", vmcs_config.pin_based_exec_ctrl,
vmcs_config.cpu_based_exec_ctrl,
vmcs_config.cpu_based_2nd_exec_ctrl,vmcs_read32(GUEST_INTERRUPTIBILITY_INFO));

You can see that I am setting all four bits that you mentioned.

I am testing for vmexits due to  IRET in the vmx_handle_exit()
function with the following test:

   // this check is from Section 31.7.1.2 of the intel manual
"Resuming Guest Software after Handling an Exception"
    if( ((vectoring_info & 2147483648) == 0) && ((vectoring_info &
255)!=8) && ((vectoring_info & 4096)==1) ){
        printk(KERN_ERR "kvm nmi blocking exit complex check caught
using hw interception due to IRET \n");
    }

    if( exit_reason == EXIT_REASON_NMI_WINDOW ){
        printk(KERN_ERR "kvm nmi window exit simple exit reason caught
using hw interception due to IRET \n");
    }

Within the guest I run a BSD guest OS that returns from certain system
calls using the IRET instruction.


Do you have any thoughts on where I could be wrong?


Thanks !


On Fri, Feb 24, 2017 at 6:40 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>
> On 24/02/2017 11:32, sci sci wrote:
>> I tried several combinations of the following settings but with little
>> success.
>>
>> 1) Set/Unset Virtual NMIs which is bit 5 in pin based vm execution control
>> 2) Set/Unset bit 3 in guest interruptibility state (Blocking by NMI)
>> 3) Set/Unset "NMI-blocking" which is bit 3 in pin based NMI exiting
>> 4) Set/Unset "NMI-window exiting" which is bit 22 in primary processor
>> based VM execution control.
>
> Doing all four of these should do it:
>
> - set bit 3 in pin-based controls to cause a VMEXIT for host NMIs and
> enable special processing of IRET (paragraph 25.3)
>
> - set bit 5 in pin-based controls so that guest interruptibility state
> tracks virtual-NMI blocking
>
> - set bit 22 of VM executino control to enable the NMI-window vmexit
>
> - set bit 3 in guest interruptibility state to delay the NMI-window exit
> until the next IRET
>
> Paolo

Re: Create VM exits when guest executes IRET on Intel CPU ?

[Steve Rutherford]

Are you trying to get NMI IRETs? or generic IRETs?
I think SVM has a specific IRET control bit (but obviously, based on
your question, you are on an Intel chip).
If I'm reading the manual right, Paolo's suggestion should work. That
said, I have some hacky suggestions that won't get you every IRET and
will take more work.

1: Enable interrupt window exits whenever you inject an interrupt. It
won't work every time (you won't get irets from nested interrupts, or
if the guest clears IF), but you should get pretty much every other
one.

2: If you want to do a lot of work, you could set the GDT limit to 0
and enable GP exiting whenever you inject an interrupt (I think you'll
also have to disable posted-interrupts). You'll have to keep count of
how many interrupts you've nested so you know when you are resuming a
normal non-interrupt context. And make sure your GPs aren't normal
GPs.

On Fri, Feb 24, 2017 at 12:20 PM, sci sci <scicomplete@gmail.com> wrote:
> Thanks very much for the reply.
>
> I set all the four that you mentioned and am still having trouble with
> intecepting guest IRET.
>
> I print out the following right before the assembly instructions for
> entering guest in vmx_vcpu_run() function in vmx.c  file.
>
> "vmcs config pin based 3f cpu based b6e0edfa cpu based2 fb interruptibility 8"
>
> The printf statement producing this is :
> printk(KERN_ERR " vmcs config pin based %x cpu based %x cpu based2 %x
> interruptibility %x\n", vmcs_config.pin_based_exec_ctrl,
> vmcs_config.cpu_based_exec_ctrl,
> vmcs_config.cpu_based_2nd_exec_ctrl,vmcs_read32(GUEST_INTERRUPTIBILITY_INFO));
>
> You can see that I am setting all four bits that you mentioned.
>
> I am testing for vmexits due to  IRET in the vmx_handle_exit()
> function with the following test:
>
>    // this check is from Section 31.7.1.2 of the intel manual
> "Resuming Guest Software after Handling an Exception"
>     if( ((vectoring_info & 2147483648) == 0) && ((vectoring_info &
> 255)!=8) && ((vectoring_info & 4096)==1) ){
>         printk(KERN_ERR "kvm nmi blocking exit complex check caught
> using hw interception due to IRET \n");
>     }
>
>     if( exit_reason == EXIT_REASON_NMI_WINDOW ){
>         printk(KERN_ERR "kvm nmi window exit simple exit reason caught
> using hw interception due to IRET \n");
>     }
>
> Within the guest I run a BSD guest OS that returns from certain system
> calls using the IRET instruction.
>
>
> Do you have any thoughts on where I could be wrong?
>
>
> Thanks !
>
>
> On Fri, Feb 24, 2017 at 6:40 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>
>>
>> On 24/02/2017 11:32, sci sci wrote:
>>> I tried several combinations of the following settings but with little
>>> success.
>>>
>>> 1) Set/Unset Virtual NMIs which is bit 5 in pin based vm execution control
>>> 2) Set/Unset bit 3 in guest interruptibility state (Blocking by NMI)
>>> 3) Set/Unset "NMI-blocking" which is bit 3 in pin based NMI exiting
>>> 4) Set/Unset "NMI-window exiting" which is bit 22 in primary processor
>>> based VM execution control.
>>
>> Doing all four of these should do it:
>>
>> - set bit 3 in pin-based controls to cause a VMEXIT for host NMIs and
>> enable special processing of IRET (paragraph 25.3)
>>
>> - set bit 5 in pin-based controls so that guest interruptibility state
>> tracks virtual-NMI blocking
>>
>> - set bit 22 of VM executino control to enable the NMI-window vmexit
>>
>> - set bit 3 in guest interruptibility state to delay the NMI-window exit
>> until the next IRET
>>
>> Paolo

Re: Create VM exits when guest executes IRET on Intel CPU ?

[Steve Rutherford]

(Wait, that first suggestion doesn't make any sense. Somehow forgot
interrupt window exiting isn't blocked by other interrupts.)

On Fri, Feb 24, 2017 at 1:21 PM, Steve Rutherford
<srutherford@google.com> wrote:
> Are you trying to get NMI IRETs? or generic IRETs?
> I think SVM has a specific IRET control bit (but obviously, based on
> your question, you are on an Intel chip).
> If I'm reading the manual right, Paolo's suggestion should work. That
> said, I have some hacky suggestions that won't get you every IRET and
> will take more work.
>
> 1: Enable interrupt window exits whenever you inject an interrupt. It
> won't work every time (you won't get irets from nested interrupts, or
> if the guest clears IF), but you should get pretty much every other
> one.
>
> 2: If you want to do a lot of work, you could set the GDT limit to 0
> and enable GP exiting whenever you inject an interrupt (I think you'll
> also have to disable posted-interrupts). You'll have to keep count of
> how many interrupts you've nested so you know when you are resuming a
> normal non-interrupt context. And make sure your GPs aren't normal
> GPs.
>
> On Fri, Feb 24, 2017 at 12:20 PM, sci sci <scicomplete@gmail.com> wrote:
>> Thanks very much for the reply.
>>
>> I set all the four that you mentioned and am still having trouble with
>> intecepting guest IRET.
>>
>> I print out the following right before the assembly instructions for
>> entering guest in vmx_vcpu_run() function in vmx.c  file.
>>
>> "vmcs config pin based 3f cpu based b6e0edfa cpu based2 fb interruptibility 8"
>>
>> The printf statement producing this is :
>> printk(KERN_ERR " vmcs config pin based %x cpu based %x cpu based2 %x
>> interruptibility %x\n", vmcs_config.pin_based_exec_ctrl,
>> vmcs_config.cpu_based_exec_ctrl,
>> vmcs_config.cpu_based_2nd_exec_ctrl,vmcs_read32(GUEST_INTERRUPTIBILITY_INFO));
>>
>> You can see that I am setting all four bits that you mentioned.
>>
>> I am testing for vmexits due to  IRET in the vmx_handle_exit()
>> function with the following test:
>>
>>    // this check is from Section 31.7.1.2 of the intel manual
>> "Resuming Guest Software after Handling an Exception"
>>     if( ((vectoring_info & 2147483648) == 0) && ((vectoring_info &
>> 255)!=8) && ((vectoring_info & 4096)==1) ){
>>         printk(KERN_ERR "kvm nmi blocking exit complex check caught
>> using hw interception due to IRET \n");
>>     }
>>
>>     if( exit_reason == EXIT_REASON_NMI_WINDOW ){
>>         printk(KERN_ERR "kvm nmi window exit simple exit reason caught
>> using hw interception due to IRET \n");
>>     }
>>
>> Within the guest I run a BSD guest OS that returns from certain system
>> calls using the IRET instruction.
>>
>>
>> Do you have any thoughts on where I could be wrong?
>>
>>
>> Thanks !
>>
>>
>> On Fri, Feb 24, 2017 at 6:40 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>>
>>>
>>> On 24/02/2017 11:32, sci sci wrote:
>>>> I tried several combinations of the following settings but with little
>>>> success.
>>>>
>>>> 1) Set/Unset Virtual NMIs which is bit 5 in pin based vm execution control
>>>> 2) Set/Unset bit 3 in guest interruptibility state (Blocking by NMI)
>>>> 3) Set/Unset "NMI-blocking" which is bit 3 in pin based NMI exiting
>>>> 4) Set/Unset "NMI-window exiting" which is bit 22 in primary processor
>>>> based VM execution control.
>>>
>>> Doing all four of these should do it:
>>>
>>> - set bit 3 in pin-based controls to cause a VMEXIT for host NMIs and
>>> enable special processing of IRET (paragraph 25.3)
>>>
>>> - set bit 5 in pin-based controls so that guest interruptibility state
>>> tracks virtual-NMI blocking
>>>
>>> - set bit 22 of VM executino control to enable the NMI-window vmexit
>>>
>>> - set bit 3 in guest interruptibility state to delay the NMI-window exit
>>> until the next IRET
>>>
>>> Paolo

Re: Create VM exits when guest executes IRET on Intel CPU ?

[sci sci]

Thanks for your reply and suggestions.
I am trying to intercept generic IRETs.
Yes, SVM has a IRET control bit (which I think I tested out a while
back - if my recollection is right).

Thanks!

On Fri, Feb 24, 2017 at 3:21 PM, Steve Rutherford
<srutherford@google.com> wrote:
> Are you trying to get NMI IRETs? or generic IRETs?
> I think SVM has a specific IRET control bit (but obviously, based on
> your question, you are on an Intel chip).
> If I'm reading the manual right, Paolo's suggestion should work. That
> said, I have some hacky suggestions that won't get you every IRET and
> will take more work.
>
> 1: Enable interrupt window exits whenever you inject an interrupt. It
> won't work every time (you won't get irets from nested interrupts, or
> if the guest clears IF), but you should get pretty much every other
> one.
>
> 2: If you want to do a lot of work, you could set the GDT limit to 0
> and enable GP exiting whenever you inject an interrupt (I think you'll
> also have to disable posted-interrupts). You'll have to keep count of
> how many interrupts you've nested so you know when you are resuming a
> normal non-interrupt context. And make sure your GPs aren't normal
> GPs.
>
> On Fri, Feb 24, 2017 at 12:20 PM, sci sci <scicomplete@gmail.com> wrote:
>> Thanks very much for the reply.
>>
>> I set all the four that you mentioned and am still having trouble with
>> intecepting guest IRET.
>>
>> I print out the following right before the assembly instructions for
>> entering guest in vmx_vcpu_run() function in vmx.c  file.
>>
>> "vmcs config pin based 3f cpu based b6e0edfa cpu based2 fb interruptibility 8"
>>
>> The printf statement producing this is :
>> printk(KERN_ERR " vmcs config pin based %x cpu based %x cpu based2 %x
>> interruptibility %x\n", vmcs_config.pin_based_exec_ctrl,
>> vmcs_config.cpu_based_exec_ctrl,
>> vmcs_config.cpu_based_2nd_exec_ctrl,vmcs_read32(GUEST_INTERRUPTIBILITY_INFO));
>>
>> You can see that I am setting all four bits that you mentioned.
>>
>> I am testing for vmexits due to  IRET in the vmx_handle_exit()
>> function with the following test:
>>
>>    // this check is from Section 31.7.1.2 of the intel manual
>> "Resuming Guest Software after Handling an Exception"
>>     if( ((vectoring_info & 2147483648) == 0) && ((vectoring_info &
>> 255)!=8) && ((vectoring_info & 4096)==1) ){
>>         printk(KERN_ERR "kvm nmi blocking exit complex check caught
>> using hw interception due to IRET \n");
>>     }
>>
>>     if( exit_reason == EXIT_REASON_NMI_WINDOW ){
>>         printk(KERN_ERR "kvm nmi window exit simple exit reason caught
>> using hw interception due to IRET \n");
>>     }
>>
>> Within the guest I run a BSD guest OS that returns from certain system
>> calls using the IRET instruction.
>>
>>
>> Do you have any thoughts on where I could be wrong?
>>
>>
>> Thanks !
>>
>>
>> On Fri, Feb 24, 2017 at 6:40 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>>
>>>
>>> On 24/02/2017 11:32, sci sci wrote:
>>>> I tried several combinations of the following settings but with little
>>>> success.
>>>>
>>>> 1) Set/Unset Virtual NMIs which is bit 5 in pin based vm execution control
>>>> 2) Set/Unset bit 3 in guest interruptibility state (Blocking by NMI)
>>>> 3) Set/Unset "NMI-blocking" which is bit 3 in pin based NMI exiting
>>>> 4) Set/Unset "NMI-window exiting" which is bit 22 in primary processor
>>>> based VM execution control.
>>>
>>> Doing all four of these should do it:
>>>
>>> - set bit 3 in pin-based controls to cause a VMEXIT for host NMIs and
>>> enable special processing of IRET (paragraph 25.3)
>>>
>>> - set bit 5 in pin-based controls so that guest interruptibility state
>>> tracks virtual-NMI blocking
>>>
>>> - set bit 22 of VM executino control to enable the NMI-window vmexit
>>>
>>> - set bit 3 in guest interruptibility state to delay the NMI-window exit
>>> until the next IRET
>>>
>>> Paolo

Re: Create VM exits when guest executes IRET on Intel CPU ?

[Paolo Bonzini]

> Within the guest I run a BSD guest OS that returns from certain system
> calls using the IRET instruction.
> 
> Do you have any thoughts on where I could be wrong?

No, but I suggest using a small unit test in kvm-unit-tests
https://www.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git/
(either vmx.flat to test the VMX settings, or a test using IRET
to test your modified KVM).  Tracing the unit test with trace-cmd
can provide clues more easily.

Paolo