Re: [FYI PATCH] Revert "KVM: x86/mmu: Zap only TDP MMU leafs in kvm_zap_gfn_range()"

[Sean Christopherson <seanjc@google.com>]

On Mon, Mar 21, 2022, Paolo Bonzini wrote:
> On 3/18/22 17:48, Paolo Bonzini wrote:
> > This reverts commit cf3e26427c08ad9015956293ab389004ac6a338e.
> > 
> > Multi-vCPU Hyper-V guests started crashing randomly on boot with the
> > latest kvm/queue and the problem can be bisected the problem to this
> > particular patch. Basically, I'm not able to boot e.g. 16-vCPU guest
> > successfully anymore. Both Intel and AMD seem to be affected. Reverting
> > the commit saves the day.
> > 
> > Reported-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> > Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> 
> This is not enough, the following is also needed to account
> for "KVM: x86/mmu: Defer TLB flush to caller when freeing TDP MMU shadow
> pages":
> 
> ------------------- 8< ----------------
> From: Paolo Bonzini <pbonzini@redhat.com>
> Subject: [PATCH] kvm: x86/mmu: Flush TLB before zap_gfn_range releases RCU
> 
> Since "KVM: x86/mmu: Zap only TDP MMU leafs in kvm_zap_gfn_range()"
> is going to be reverted, it's not going to be true anymore that
> the zap-page flow does not free any 'struct kvm_mmu_page'.  Introduce
> an early flush before tdp_mmu_zap_leafs() returns, to preserve
> bisectability.

Can I have 1-2 weeks to try and root cause and fix the underlying issue before
sending reverts to Linus?  I really don't want to paper over a TLB flushing bug
or an off-by-one bug, and I really, really don't want to end up with another
scenario where KVM zaps everything just because.

Vitaly, can you provide repro instructions?  A nearly-complete QEMU command line
would be wonderful :-)  Is the issue unique to any particular guest kernel?  I've
been unable to repro with a 112 vCPU Linux guest with these Hyper-V enlightenments:

$ : dm | grep -i hyper-v
[    0.000000] Hypervisor detected: Microsoft Hyper-V
[    0.000000] Hyper-V: privilege flags low 0x2aff, high 0x830, hints 0x4e2c, misc 0x80d12
[    0.000000] Hyper-V Host Build:14393-10.0-0-0.0
[    0.000000] Hyper-V: Nested features: 0x80101
[    0.000000] Hyper-V: LAPIC Timer Frequency: 0x3d0900
[    0.000000] Hyper-V: Using hypercall for remote TLB flush
[    0.000004] tsc: Marking TSC unstable due to running on Hyper-V
[    0.129376] Booting paravirtualized kernel on Hyper-V
[    0.140419] Hyper-V: PV spinlocks disabled
[    0.247500] Hyper-V: Using IPI hypercalls
[    0.247502] Hyper-V: Using enlightened APIC (x2apic mode)

Actually, since this is apparently specific to kvm_zap_gfn_range(), can you add
printk "tracing" in update_mtrr(), kvm_post_set_cr0(), and __kvm_request_apicv_update()
to see what is actually triggering zaps?  Capturing the start and end GFNs would be very
helpful for the MTRR case.

The APICv update seems unlikely to affect only Hyper-V guests, though there is the auto
EOI crud.  And the other two only come into play with non-coherent DMA.  In other words,
figuring out exactly what sequence leads to failure should be straightforward.