Re: [PATCH kvm-unit-tests 0/8]: x86: vmx: Test INIT processing in various CPU VMX states

[Sean Christopherson <sean.j.christopherson@intel.com>]

On Tue, Oct 01, 2019 at 11:40:34AM -0700, Sean Christopherson wrote:
> On Mon, Sep 30, 2019 at 06:29:52PM -0700, Nadav Amit wrote:
> > > On Sep 30, 2019, at 6:23 PM, Liran Alon <liran.alon@oracle.com> wrote:

...

> > > I also remembered to verify this behaviour against some discussions made online:
> > > 1) https://software.intel.com/en-us/forums/virtualization-software-development/topic/355484
> > > * "When the 16-bit guest issues an INIT IPI to itself using the APIC, I run into an infinite VMExit situation that my hypervisor cannot seem to recover from.”
> > > * "In response to the VMExit with a reason of 3 (which is expected), the hypervisor resets the 16-bit guest's registers, limits, access rights, etc. to simulate starting execution from a known initialization point.  However, it seems that as soon as the hypervisor resumes guest execution, the VMExit occurs again, repeatedly.”
> > > 2) https://patchwork.kernel.org/patch/2244311/
> > > "I actually find it very useful. On INIT vmexit hypervisor may call vmxoff and do proper reset."
> > > 
> > > Anyway, Sean, can you assist verifying inside Intel what should be the expected behaviour?
> > 
> > It might always be (yet) another kvm-unit-tests bug that is only apparent on
> > bare-metal. But if Sean can confirm what the expected behavior is, it would
> > save time.
> > 
> > I do not have an ITP, so debugging on bare-metal is not fun at all...
> 
> My understanding of the architecture is that the INIT should be consumed
> on VM-Exit.  The only scenario where an event is not consumed/acknowledge
> is when a vanilla interrupt occurs without VM_EXIT_ACK_INTR_ON_EXIT set,
> in which case the VM-Exit is technically considered a "pending" interrupt.
> For all other cases (NMI, SMI, INIT, and INTR w/ ACK-ON-EXIT), the VM-Exit
> is the end result of delivering the event.
> 
> INITs are indeed blocked and not dropped in VMX root mode.  But entering
> non-root (guest) mode should unblock INITs and cause a VM-Exit, and thus
> clear the INIT that was pended while in VMX root mode.  This behavior does
> not conflict with the whitepaper[*] referenced by link (2) above, and in
> fact the whitepaper explicitly covers guest mode behavior in a footnote:
> 
>   When the processor is in VMX guest mode, delivery of INIT causes a
>   normal VMEXIT, of course.
> 
> The INIT attack described uses "VMX mode" to refer to VMX root mode, and
> other than the footnote, doesn't mention VMX guest mode.  My reading of it
> is that they're showing a proof of concept of based on getting the OS into
> VMX root mode but not actually running a guest, e.g. this can be done
> in KVM by creating a VM (KVM_CREATE_VM) but not running it (KVM_RUN).
> 
> Anyways, I'll double check that the INIT should indeed be consumed as part
> of the VM-Exit.

I couldn't help but run a few tests before reaching out to the architecture
folks...

I modified KVM to have the CPU send an INIT IPI to itself in vmx_vcpu_run(),
with a bit of delay to ensure the INIT is pending prior to VM-Enter.  On an
INIT VM-Exit, KVM immediately resumes the guest.  On Haswell client system,
the INIT does indeed appear to be consumed when it's handled by VM-Exit,
i.e. KVM doesn't get stuck in an infinite INIT VM-Exits loop.

One possible explanation for the infinite loop observed in (1) above, is
that the developer didn't properly reconfigure guest state when emulating
INIT and hit a VM-Fail.  Because vmcs.EXIT_REASON isn't written on VM-Fail,
if the VMM isn't checking for VM-Fail it will think it's getting endless
INIT VM-Exits.  I did exactly this when tweaking KVM to handle INIT (forgot
to mark the VMCS as launched redoing VM-Enter), so I even inadvertantly
confirmed that it's plausible :-)