Re: KVM_EXIT_FAIL_ENTRY with hardware_entry_failure_reason = 7

[Sean Christopherson <seanjc@google.com>]

On Wed, Jul 26, 2023, Yahya Sohail wrote:
> Hi,
> 
> I'm trying to copy the state of an x86 emulator into a KVM VM.
> 
> I've loaded the relevant state (i.e. registers and memory) into a KVM VM and
> VCPU, and tried to do a KVM_RUN on the VCPU, but it fails with
> KVM_EXIT_FAIL_ENTRY and hardware_entry_failure_reason = 7. I looked through
> the KVM source and Intel manuals to determine that this either means that
> the CPU is in an interrupt window and the VM was setup to exit on an
> interrupt window, or that a VM entry occurred with invalid control fields.
> The former is not possible because my RFLAGS.IF = 0, meaning interrupts are
> currently disabled, so I think it's the latter.

No, there are far, far more possible problems.  Error code 7 is "invalid control
field", which is a gigantic bin for any failed consistency check that is related
to one or more VMCS control fields.

> Is it possible for someone using the KVM API to set the VMCS to an invalid
> state?

Yes.  Ideally it _shouldn't_ be possible[*], but practically speaking I don't think
there's ever been a version of KVM that prevents userspace from coercing KVM into
loading invalid state.  E.g. see https://lore.kernel.org/all/20230613203037.1968489-1-seanjc@google.com

[*] For VMCS control fields specifically.  Preventing userspace from loading
    invalid guest state is extremely difficult, and not something I realistically
    expect KVM to get 100% right anytime soon.

> If so, what fields in the kvm_run struct should I check that could cause such
> an issue?

Heh, all of them.  I'm only somewhat joking.  Root causing "invalid control field"
errors on bare metal is painfully difficult, bordering on impossible if you don't
have something to give you a hint as to what might be going wrong.

If you can, try running a nested setup, i.e. run a normal Linux guest as your L1
VM (L0 is bare metal), and then run your problematic x86 emulator VM within that
L1 guest (that's your L2).  Then, in L0 (your bare metal host), enable the
kvm_nested_vmenter_failed tracepoint.

The kvm_nested_vmenter_failed tracepoint logs all VM-Enter failures that _KVM_
detects when L1 attempts a nested VM-Enter from L1 to L2.  If you're at all lucky,
KVM in L0 (acting a the CPU from L1's perspective) will detect the invalid state
and explicitly log which consistency check failed.