Re: [PATCH] KVM: SVM: make svm_flush_tlb_gva do a full asid flush if NPT enabled

Re: [PATCH] KVM: SVM: make svm_flush_tlb_gva do a full asid flush if NPT enabled

[Sean Christopherson]

+Yosry, who has been digging deep on SVM TLB crud.

On Thu, Jul 23, 2026, Paolo Bonzini wrote:
> The flush is issued from kvm_hv_vcpu_flush_tlb(), which receives the
> cross-CPU requests from the Hyper-V TLB flush hypercalls via a kfifo
> and is invoked by the KVM_REQ_HV_TLB_FLUSH request.  The mechanism is
> the same for both Intel and AMD, and the handler for both vendors is
> a simple INVVPID(ADDR)/INVLPGA instruction.
> 
> Because the request is handled on the destination CPU, there is a question
> of what happens if the VM is migrated across physical CPUs.  In that case,
> the INVLPGA instruction would use a stale svm->vmcb->control.asid; but
> if anything that might do an *unnecessary* flush (on an asid that's being
> used for another VM) and then pre_svm_run() would force a full TLB rebuild.
> 
> So, for lack of better ideas, this patch forces a full ASID bump in
> svm_flush_tlb_gva().  To avoid paying the price on Intel and also to
> avoid unnecessary loops on AMD, the flush_tlb_gva op now returns whether
> it did a full flush or not; kvm_hv_vcpu_flush_tlb() takes note and exits
> its loops immediately.  While there is an obvious performance impact,
> about half of the benefit from Hyper-V tlbflush is preserved (10% vs. 20%
> on the SQL Server workload).
> 
> kvm_mmu_invalidate_addr() is the only other caller of the flush_tlb_gva op.
> The change would have a performance impact on every intercepted INVLPG and,
> for nested SVM, on every L1 INVLPGA.  For INVLPGA specifically, this covers
> the same suspected issue but for nested hypervisors, so it is correct to
> apply the workaround; for INVLPG on shadow paging, instead, the impact
> would be stronger and, due to lack of data, for now the use of INVLPGA is
> left in place in svm_flush_tlb_gva().
> 
> Analyzed-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> Analyzed-by: Alexander Lougovski <alougovsk@redhat.com>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>  arch/x86/include/asm/kvm_host.h |  2 +-
>  arch/x86/kvm/hyperv.c           |  7 ++++---
>  arch/x86/kvm/mmu/mmu.c          |  2 +-
>  arch/x86/kvm/svm/svm.c          | 27 ++++++++++++++++++++-------
>  arch/x86/kvm/vmx/main.c         |  4 ++--
>  arch/x86/kvm/vmx/vmx.c          |  2 +-
>  arch/x86/kvm/vmx/x86_ops.h      |  2 +-
>  7 files changed, 30 insertions(+), 16 deletions(-)
> 
> diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
> index b517257a6315..eca04d4b974e 100644
> --- a/arch/x86/include/asm/kvm_host.h
> +++ b/arch/x86/include/asm/kvm_host.h
> @@ -1751,7 +1751,7 @@ struct kvm_x86_ops {
>  	 * Can potentially get non-canonical addresses through INVLPGs, which
>  	 * the implementation may choose to ignore if appropriate.
>  	 */
> -	void (*flush_tlb_gva)(struct kvm_vcpu *vcpu, gva_t addr);
> +	void (*flush_tlb_gva)(struct kvm_vcpu *vcpu, gva_t addr, bool *full);

LOL, why on earth are you using an out-param?  If we do this at runtime, just
return a bool, at least that way we don't have to churn every call-site.

But I would much rather handle this by nuking .flush_tlb_gva at setup, e.g.


diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h
index 3776cf5382a2..4235331b71a0 100644
--- a/arch/x86/include/asm/kvm-x86-ops.h
+++ b/arch/x86/include/asm/kvm-x86-ops.h
@@ -61,7 +61,7 @@ KVM_X86_OP(flush_tlb_current)
 KVM_X86_OP_OPTIONAL(flush_remote_tlbs)
 KVM_X86_OP_OPTIONAL(flush_remote_tlbs_range)
 #endif
-KVM_X86_OP(flush_tlb_gva)
+KVM_X86_OP_OPTIONAL(flush_tlb_gva)
 KVM_X86_OP(flush_tlb_guest)
 KVM_X86_OP(vcpu_pre_run)
 KVM_X86_OP(vcpu_run)
diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index 4438ecac9a89..7b1c6391f878 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -1978,7 +1978,8 @@ int kvm_hv_vcpu_flush_tlb(struct kvm_vcpu *vcpu)
        count = kfifo_out(&tlb_flush_fifo->entries, entries, KVM_HV_TLB_FLUSH_FIFO_SIZE);
 
        for (i = 0; i < count; i++) {
-               if (entries[i] == KVM_HV_TLB_FLUSHALL_ENTRY)
+               if (entries[i] == KVM_HV_TLB_FLUSHALL_ENTRY ||
+                   !kvm_x86_ops.flush_tlb_gva)
                        goto out_flush_all;
 
                /*
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index f0144ae8d891..bf16119640dc 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -6555,7 +6555,10 @@ void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,
                if (is_noncanonical_invlpg_address(addr, vcpu))
                        return;
 
-               kvm_x86_call(flush_tlb_gva)(vcpu, addr);
+               if (kvm_x86_ops.flush_tlb_gva)
+                       kvm_x86_call(flush_tlb_gva)(vcpu, addr);
+               else
+                       kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
        }
 
        if (!mmu->sync_spte)
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index c46a34aeb3df..9e04f016d939 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -5688,6 +5688,15 @@ static __init int svm_hardware_setup(void)
        if (!enable_pmu)
                pr_info("PMU virtualization is disabled\n");
 
+       /*
+        * INVLPGA has had errata on Genoa and Turin, and even on older
+        * generations there were reports of Windows BSODs if INVLPGA
+        * was used for Hyper-V tlbflush.  Use it only for shadow paging
+        * where it seems to be okay.
+        */
+       if (npt_enabled)
+               svm_x86_ops.flush_tlb_gva = NULL;
+
        svm_set_cpu_caps();
 
        kvm_caps.inapplicable_quirks &= ~KVM_X86_QUIRK_CD_NW_CLEARED;

Re: [PATCH] KVM: SVM: make svm_flush_tlb_gva do a full asid flush if NPT enabled

[Yosry Ahmed]

On Thu, Jul 23, 2026 at 2:44 AM Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> Red Hat is seeing multiple reports of Windows memory corruptions
> (and consequent BSODs) with hv-tlbflush=on, on AMD processors only.
> The crashes, while extremely rare, happen even with a stock configuration,
> but with Driver Verifier enabled they can be detected after approximately
> 200 VM hours.  In particular, Alexander Lougovski measured the following:
>
> - on AMD Turin, 15 crashes in 3300 VM hours
>
> - on AMD Milan, 2 crashes in 500 VM hours (there are fewer hours
>   here due to the host being smaller)
>
> - on Intel Sapphire Rapids, 0 crashes in 8000 VM hours
>
> - on AMD Turin with full TLB flush (not exactly this patch but
>   similar), no crashes in ~2 weeks of run time which should also
>   be ~7000 VM hours
>
> For Turin, the microcode version was 0x0b002162, which (assuming
> this is the same issue) should not be affected by the problem listed in
> https://knowledge.broadcom.com/external/article/419026/bsod-on-virtual-machines-running-on-amd.html;
> on the other hand that problem should not apply to earlier processors.
> AMD has not provided any information or analysis yet, and when we asked
> we didn't know yet that it reproduced on Milan as well.
>
> As to the workload, Alexander threw more or less everything at the same
> time at the VM:
>
> - a full Windows Defender scan every 30 minutes
>
> - a disk I/O job
>
> - a loop doing repeated mmap of system files (mostly to hope that
>   it triggers some consistency check in the Windows memory manager)
>
> - SQL Express 2022 + StressDB (1.6M rows), with the host doing queries
>   (75% write/25% read) via sqlcmd
>
> Driver Verifier is able to detect BSODs more or less at the same time as
> the pages are freed.  They mostly happen in the Windows Defender filter
> driver, but occasionally also in the networking stack (e.g., afd.sys)
> or elsewhere in the filesystem stack (e.g., fltmgr.sys).
>
> The flush is issued from kvm_hv_vcpu_flush_tlb(), which receives the
> cross-CPU requests from the Hyper-V TLB flush hypercalls via a kfifo
> and is invoked by the KVM_REQ_HV_TLB_FLUSH request.  The mechanism is
> the same for both Intel and AMD, and the handler for both vendors is
> a simple INVVPID(ADDR)/INVLPGA instruction.
>
> Because the request is handled on the destination CPU, there is a question
> of what happens if the VM is migrated across physical CPUs.  In that case,
> the INVLPGA instruction would use a stale svm->vmcb->control.asid; but
> if anything that might do an *unnecessary* flush (on an asid that's being
> used for another VM) and then pre_svm_run() would force a full TLB rebuild.
>
> So, for lack of better ideas, this patch forces a full ASID bump in
> svm_flush_tlb_gva().  To avoid paying the price on Intel and also to
> avoid unnecessary loops on AMD, the flush_tlb_gva op now returns whether
> it did a full flush or not; kvm_hv_vcpu_flush_tlb() takes note and exits
> its loops immediately.  While there is an obvious performance impact,
> about half of the benefit from Hyper-V tlbflush is preserved (10% vs. 20%
> on the SQL Server workload).
>
> kvm_mmu_invalidate_addr() is the only other caller of the flush_tlb_gva op.
> The change would have a performance impact on every intercepted INVLPG and,
> for nested SVM, on every L1 INVLPGA.  For INVLPGA specifically, this covers
> the same suspected issue but for nested hypervisors, so it is correct to
> apply the workaround; for INVLPG on shadow paging, instead, the impact
> would be stronger and, due to lack of data, for now the use of INVLPGA is
> left in place in svm_flush_tlb_gva().
>
> Analyzed-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> Analyzed-by: Alexander Lougovski <alougovsk@redhat.com>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
[..]
> +static void svm_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t gva, bool *full)
> +{
> +       struct vcpu_svm *svm = to_svm(vcpu);
> +
> +       /*
> +        * INVLPGA has had errata on Genoa and Turin, and even on older
> +        * generations there were reports of Windows BSODs if INVLPGA
> +        * was used for Hyper-V tlbflush.  Use it only for shadow paging
> +        * where it seems to be okay.

Is this an actual errata documented by AMD, or is this just an
empirical observation?

I ask because the APM says:
---
The input address is always interpreted as a guest virtual address, so
INVLPGA is typically meaningful only when used with shadow page
tables; it does not provide a means to invalidate a nested translation
by guest physical address
---

While this is terrible wording, it seems like KVM should *not* be
using INVLPGA when TDP is enabled. Looks like
kvm_mmu_invalidate_addr() might be doing the right thing, but it seems
like kvm_hv_vcpu_flush_tlb() shouldn't be calling flush_tlb_gva() with
TDP enabled to begin with, at least on AMD?

I don't have enough context about what kvm_hv_vcpu_flush_tlb() is
doing to know if flush_tlb_gva() makes sense on Intel. But at least on
AMD, looks like it should always just do a full ASID flush (since it
falls back to a full flush with TDP disabled anyway)?

So maybe something like:

int kvm_hv_vcpu_flush_tlb(struct kvm_vcpu *vcpu)
{
       ...
       if (AMD CPU)
              goto out_flush_all;
       ...
}


I also love Sean's idea, I think it's good to harden against this by
nullifying flush_tlb_gva, and maybe add a helper that does the
fallback:

static void kvm_vcpu_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t gva)
{
       if (kvm_x86_ops.flush_tlb_gva)
              kvm_x86_call(flush_tlb_gva)(vcpu, addr);
       else
              kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
}

Hmm actually we check KVM_REQ_TLB_FLUSH_GUEST before
KVM_REQ_HV_TLB_FLUSH, so maybe just call kvm_vcpu_flush_tlb_guest()
directly for the fallback:

static void kvm_vcpu_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t gva)
{
       if (kvm_x86_ops.flush_tlb_gva)
              kvm_x86_call(flush_tlb_gva)(vcpu, addr);
       else
              kvm_vcpu_flush_tlb_guest(vcpu);
}

And if we go this route, I think we can key off the presence of
flush_tlb_gva in kvm_hv_vcpu_flush_tlb() instead of checking for an
AMD CPU. We can probably break it down into a stable-friendly fix that
just jumps to out_flush_all on AMD CPUs, then the hardening on top.

Re: [PATCH] KVM: SVM: make svm_flush_tlb_gva do a full asid flush if NPT enabled

[Yosry Ahmed]

> I also love Sean's idea, I think it's good to harden against this by
> nullifying flush_tlb_gva, and maybe add a helper that does the
> fallback:
>
> static void kvm_vcpu_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t gva)
> {
>        if (kvm_x86_ops.flush_tlb_gva)
>               kvm_x86_call(flush_tlb_gva)(vcpu, addr);
>        else
>               kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
> }
>
> Hmm actually we check KVM_REQ_TLB_FLUSH_GUEST before
> KVM_REQ_HV_TLB_FLUSH, so maybe just call kvm_vcpu_flush_tlb_guest()
> directly for the fallback:
>
> static void kvm_vcpu_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t gva)
> {
>        if (kvm_x86_ops.flush_tlb_gva)
>               kvm_x86_call(flush_tlb_gva)(vcpu, addr);
>        else
>               kvm_vcpu_flush_tlb_guest(vcpu);
> }
>
> And if we go this route, I think we can key off the presence of
> flush_tlb_gva in kvm_hv_vcpu_flush_tlb() instead of checking for an
> AMD CPU. We can probably break it down into a stable-friendly fix that
> just jumps to out_flush_all on AMD CPUs, then the hardening on top.

Although it could be less crud if we just made svm_flush_tlb_gva() do
the fallback. kvm_hv_vcpu_flush_tlb() would need to key-off AMD CPU
though instead of the presence of flush_tlb_gva. Pick your poison, I
guess.