Re: [PATCH v3 6/6] KVM: selftests: test KVM_GUESTDBG_BLOCKIRQ

[Sean Christopherson <seanjc@google.com>]

On Tue, Nov 02, 2021, Vitaly Kuznetsov wrote:
> Sean Christopherson <seanjc@google.com> writes:
> > I haven't verified on hardware, but my guess is that this code in vmx_vcpu_run()
> >
> > 	/* When single-stepping over STI and MOV SS, we must clear the
> > 	 * corresponding interruptibility bits in the guest state. Otherwise
> > 	 * vmentry fails as it then expects bit 14 (BS) in pending debug
> > 	 * exceptions being set, but that's not correct for the guest debugging
> > 	 * case. */
> > 	if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP)
> > 		vmx_set_interrupt_shadow(vcpu, 0);
> >
> > interacts badly with APICv=1.  It will kill the STI shadow and cause the IRQ in
> > vmcs.GUEST_RVI to be recognized when it (micro-)architecturally should not.  My
> > head is going in circles trying to sort out what would actually happen.  Maybe
> > comment out that and/or disable APICv to see if either one makes the test pass?
> >
> 
> Interestingly,
> 
> loading 'kvm-intel' with 'enable_apicv=0' makes the test pass, however,
> commenting out "vmx_set_interrupt_shadow()" as suggested gives a
> different result (with enable_apicv=1):
> 
> # ./x86_64/debug_regs 
> ==== Test Assertion Failure ====
>   x86_64/debug_regs.c:179: run->exit_reason == KVM_EXIT_DEBUG && run->debug.arch.exception == DB_VECTOR && run->debug.arch.pc == target_rip && run->debug.arch.dr6 == target_dr6
>   pid=16352 tid=16352 errno=0 - Success
>      1	0x0000000000402b33: main at debug_regs.c:179 (discriminator 10)
>      2	0x00007f36401bd554: ?? ??:0
>      3	0x00000000004023a9: _start at ??:?
>   SINGLE_STEP[1]: exit 9 exception -2147483615 rip 0x1 (should be 0x4024d9) dr6 0xffff4ff0 (should be 0xffff4ff0)

Exit 9 is KVM_EXIT_FAIL_ENTRY, which in this case VM-Entry likely failed due to
invalid guest state because there was STI blocking with single-step enabled but
no pending BS #DB:

  Bit 14 (BS) must be 1 if the TF flag (bit 8) in the RFLAGS field is 1 and the
  BTF flag (bit 1) in the IA32_DEBUGCTL field is 0.

Which is precisely what that hack-a-fix avoids.  There isn't really a clean
solution for legacy single-step, AFAIK the only way to avoid this would be to
switch KVM_GUESTDBG_SINGLESTEP to use MTF.

But that mess is a red herring, the test fails with the same signature with APICv=1
if the STI is replaced by PUSHF+BTS+POPFD (to avoid the STI shadow).  We all missed
this key detail from Vitaly's report:

SINGLE_STEP[1]: exit 8 exception 1 rip 0x402a25 (should be 0x402a27) dr6 0xffff4ff0 (should be 0xffff4ff0)
                ^^^^^^

Exit '8' is KVM_EXIT_SHUTDOWN, i.e. the arrival of the IRQ hosed the guest because
the test doesn't invoke vm_init_descriptor_tables() to install event handlers.
The "exception 1" shows up because the run page isn't sanitized by the test, i.e.
it's stale data that happens to match.

So I would fully expect this test to fail with AVIC=1.  The problem is that
KVM_GUESTDBG_BLOCKIRQ does absolutely nothing to handle APICv interrupts.  And
even if KVM does something to fudge that behavior in the emulated local APIC, the
test will then fail miserably virtual IPIs (currently AVIC only).

I stand by my original comment that "Deviating this far from architectural behavior
will end in tears at some point."  Rather than try to "fix" APICv, I vote to instead
either reject KVM_GUESTDBG_BLOCKIRQ if APICv=1, or log a debug message saying that
KVM_GUESTDBG_BLOCKIRQ is ineffective with APICv=1.