Re: Some options abort RFC: KVM: SVM: Rework ASID management

Re: Some options abort RFC: KVM: SVM: Rework ASID management

[Yosry Ahmed]

On Wed, Oct 28, 2026 at 6:37 AM Jackie Yu <ufo19890607@gmail.com> wrote:
>
> Hi, guys!
>
> Jackie Yu <ufo19890607@gmail.com> 于2026年10月28日周三 21:36写道：
>>
>> Hi Yosry and KVM folks,
>>
>> We would appreciate your guidance on supporting INVLPGB in ordinary (non-SEV)
>> L1 guests alongside the proposed static per-vCPU ASIDs.
>>
>> The 2025 ASID rework RFC proposed one static ASID per L1 VM and listed
>> "enabling INVLPGB virtualization" among its motivations [1]. The June
>> 2026 RFC v2 changed the design from per-VM to per-vCPU ASIDs [2]. In
>> the October v2 series, patches 2-11 establish static per-vCPU ASIDs,
>> while the remaining patches primarily address nested TLB flushes [3].
>> Our question is how guest INVLPGB should be supported under the
>> per-vCPU design.
>>
>> Some of our workloads frequently discard and repopulate memory using
>> patterns involving madvise() and memset(). Broadcast IPI-based TLB
>> shootdowns are expensive for these workloads, so we are interested in
>> making AMD INVLPGB available to guests. We use a small benchmark to
>> approximate the resulting shared-address-space pressure:
>>
>>     region = mmap(256 MiB, MAP_SHARED | MAP_ANONYMOUS);
>>     memset(region, 0xab, 256 MiB);       // Initial population
>>     start N workers, pinned across 64 allowed CPUs;
>>     each worker repeatedly reads pages in its own slice of region;
>>
>>     repeat 500 warmup + 2,000 measured iterations:
>>         start = clock_gettime(CLOCK_MONOTONIC);
>>         madvise(region, 256 MiB, MADV_DONTNEED);
>>         record elapsed madvise() time;
>>         write one page in every 64 pages to repopulate part of region;
>>
>>     stop and join the workers;
>>
>> The benchmark uses memset() for initial population; subsequent
>> iterations re-touch pages rather than repeatedly calling memset().
>> The numbers below measure the advisor thread's madvise() syscall
>> latency, not worker throughput.
>>
>> In our experimental KVM4 configuration with a VM-shared ASID and our
>> separate guest INVLPGB passthrough changes, we observed:
>>
>>                          INVLPGB off     INVLPGB on    Difference
>>     64 workers  mean       21.624 ms       12.505 ms     -42.17%
>>                 P99        32.760 ms       21.518 ms     -34.32%
>>     128 workers mean       29.553 ms       20.459 ms     -30.77%
>>                 P99        39.628 ms       28.160 ms     -28.94%
>>
>> These are single, sequential A/B runs for each worker count, with
>> 500 warmup and 2,000 measured calls per run. The process had 64
>> allowed CPUs, so the 128-worker case was oversubscribed. The
>> measurements show a useful latency difference, but do not by
>> themselves establish INVLPGB instruction counts or IPI reductions.
>>
>> We then experimentally combined the October per-vCPU ASID changes for
>> ordinary guests with our existing, out-of-tree L1 INVLPGB/TLBSYNC
>> passthrough. With guest INVLPGB enabled, a one-vCPU guest boots, but
>> otherwise comparable two-vCPU and 64-vCPU guests fail early during
>> boot. Keeping QEMU's +invlpgb setting while adding clearcpuid=419
>> to the same guest kernel allows the multi-vCPU guest to boot; the
>> guest logs "Clearing CPUID bits: invlpgb".

I imagine for INVLPGB to work correctly with per-vCPU ASIDs, you need
to intercept INVLPGB and basically issue an INVLPGB for each ASID used
by this VM (nr_vcpus or 2 * nr_vcpus with nested). I can't imagine
this would perform well on large VMs, not sure if virtualizing INVLPGB
would still make sense.

Regarding the boot failure, I would guess there are some missing cases
due to the complexity of emulating INVLPGB without a shared per-VM
ASID.

>> These controlled boot comparisons identify a problem with combining
>> static per-vCPU ASIDs and direct guest INVLPGB/TLBSYNC passthrough:
>> the multi-vCPU guest fails to boot when it uses INVLPGB, but boots
>> when its INVLPGB CPUID bit is cleared. A one-vCPU guest also boots
>> with INVLPGB enabled.
>>
>> In this configuration, an unintercepted guest INVLPGB targets the
>> invoking vCPU's hardware ASID, while sibling vCPUs have distinct
>> ASIDs.

It's neat that INVLPGB will automatically use the current ASID and not
ASID=0 for guest usage.

>> The October ASID rework does not add handling to extend that
>> guest broadcast invalidation to sibling ASIDs. Consequently, direct
>> passthrough does not preserve the required multi-vCPU guest
>> invalidation semantics. This problem arises when our out-of-tree
>> passthrough is combined with the per-vCPU ASID scheme; it is not a
>> claim that the upstream ASID series alone causes a boot failure.
>> The earlier discussion of guest INVLPGB and global ASIDs is relevant
>> here [4].
>>
>> Could you advise on the preferred upstream direction?

This is a question for Sean and Paolo.

It is possible to use one ASID per-VM, however:
1. It adds complexity as we need to track what vCPU in a VM ran last
on a CPU, so that we can flush the shared ASID in between different
vCPUs running on the same CPU.
2. It adds some TLB flushes due to scenario (1), not sure how much
this matters in practice.
3. It will re-establish a divergence in ASID/VPID management between
AMD and Intel, unless we also switch VMX to per-VM ASIDs, but I am not
sure if there's any benefit to this for now.

>>
>>   1. Would an opt-in, VM-scoped ASID mode for ordinary VMs that expose
>>      guest INVLPGB be acceptable, while other VMs retain per-vCPU
>>      ASIDs? This would require a properly managed shared-ASID lifetime
>>      and flush/reuse semantics, rather than simply giving vCPUs the
>>      same VMCB ASID value.
>>
>>   2. Alternatively, is the intended direction to retain per-vCPU
>>      ASIDs and intercept/virtualize guest INVLPGB and TLBSYNC, so that
>>      a guest broadcast invalidation correctly covers the relevant
>>      sibling vCPU ASIDs and preserves the required ordering?
>>
>>   3. If neither approach is currently planned, should ordinary
>>      multi-vCPU guests be prevented from enabling INVLPGB until the
>>      required semantics exist?
>>
>> We can share the madvise benchmark and its measurement methodology,
>> and would be happy to compare the same benchmark under both proposed
>> designs. We would appreciate feedback on the preferred upstream
>> architecture before pursuing a downstream implementation.
>>
>> References:
>> [1] 2025 ASID rework RFC (per-VM):
>>     https://patchew.org/linux/20250326193619.3714986-1-yosry.ahmed@linux.dev
>> [2] June 2026 RFC v2 (per-VM to per-vCPU):
>>     https://lore.kernel.org/kvm/20260616004155.1435766-1-yosry@kernel.org/
>> [3] October 2026 v2 series:
>>     https://lore.kernel.org/kvm/20261008001425.2458927-1-yosry@kernel.org/
>> [4] Discussion of non-SEV guest INVLPGB and global ASIDs:
>>     https://patchew.org/linux/20250223194943.3518952-1-riel@surriel.com/20250223194943.3518952-7-riel@surriel.com
>>
>> Best regards,
>> Wind Yu

Re: Some options abort RFC: KVM: SVM: Rework ASID management

[Yosry Ahmed]

On Wed, Oct 28, 2026 at 1:15 PM Yosry Ahmed <yosry@kernel.org> wrote:
>
> On Wed, Oct 28, 2026 at 6:37 AM Jackie Yu <ufo19890607@gmail.com> wrote:
> >
> > Hi, guys!
> >
> > Jackie Yu <ufo19890607@gmail.com> 于2026年10月28日周三 21:36写道：
> >>
> >> Hi Yosry and KVM folks,
> >>
> >> We would appreciate your guidance on supporting INVLPGB in ordinary (non-SEV)
> >> L1 guests alongside the proposed static per-vCPU ASIDs.
> >>
> >> The 2025 ASID rework RFC proposed one static ASID per L1 VM and listed
> >> "enabling INVLPGB virtualization" among its motivations [1]. The June
> >> 2026 RFC v2 changed the design from per-VM to per-vCPU ASIDs [2]. In
> >> the October v2 series, patches 2-11 establish static per-vCPU ASIDs,
> >> while the remaining patches primarily address nested TLB flushes [3].
> >> Our question is how guest INVLPGB should be supported under the
> >> per-vCPU design.
> >>
> >> Some of our workloads frequently discard and repopulate memory using
> >> patterns involving madvise() and memset(). Broadcast IPI-based TLB
> >> shootdowns are expensive for these workloads, so we are interested in
> >> making AMD INVLPGB available to guests. We use a small benchmark to
> >> approximate the resulting shared-address-space pressure:
> >>
> >>     region = mmap(256 MiB, MAP_SHARED | MAP_ANONYMOUS);
> >>     memset(region, 0xab, 256 MiB);       // Initial population
> >>     start N workers, pinned across 64 allowed CPUs;
> >>     each worker repeatedly reads pages in its own slice of region;
> >>
> >>     repeat 500 warmup + 2,000 measured iterations:
> >>         start = clock_gettime(CLOCK_MONOTONIC);
> >>         madvise(region, 256 MiB, MADV_DONTNEED);
> >>         record elapsed madvise() time;
> >>         write one page in every 64 pages to repopulate part of region;
> >>
> >>     stop and join the workers;
> >>
> >> The benchmark uses memset() for initial population; subsequent
> >> iterations re-touch pages rather than repeatedly calling memset().
> >> The numbers below measure the advisor thread's madvise() syscall
> >> latency, not worker throughput.
> >>
> >> In our experimental KVM4 configuration with a VM-shared ASID and our
> >> separate guest INVLPGB passthrough changes, we observed:
> >>
> >>                          INVLPGB off     INVLPGB on    Difference
> >>     64 workers  mean       21.624 ms       12.505 ms     -42.17%
> >>                 P99        32.760 ms       21.518 ms     -34.32%
> >>     128 workers mean       29.553 ms       20.459 ms     -30.77%
> >>                 P99        39.628 ms       28.160 ms     -28.94%
> >>
> >> These are single, sequential A/B runs for each worker count, with
> >> 500 warmup and 2,000 measured calls per run. The process had 64
> >> allowed CPUs, so the 128-worker case was oversubscribed. The
> >> measurements show a useful latency difference, but do not by
> >> themselves establish INVLPGB instruction counts or IPI reductions.
> >>
> >> We then experimentally combined the October per-vCPU ASID changes for
> >> ordinary guests with our existing, out-of-tree L1 INVLPGB/TLBSYNC
> >> passthrough. With guest INVLPGB enabled, a one-vCPU guest boots, but
> >> otherwise comparable two-vCPU and 64-vCPU guests fail early during
> >> boot. Keeping QEMU's +invlpgb setting while adding clearcpuid=419
> >> to the same guest kernel allows the multi-vCPU guest to boot; the
> >> guest logs "Clearing CPUID bits: invlpgb".
>
> I imagine for INVLPGB to work correctly with per-vCPU ASIDs, you need
> to intercept INVLPGB and basically issue an INVLPGB for each ASID used
> by this VM (nr_vcpus or 2 * nr_vcpus with nested). I can't imagine
> this would perform well on large VMs, not sure if virtualizing INVLPGB
> would still make sense.
>
> Regarding the boot failure, I would guess there are some missing cases
> due to the complexity of emulating INVLPGB without a shared per-VM
> ASID.
>
> >> These controlled boot comparisons identify a problem with combining
> >> static per-vCPU ASIDs and direct guest INVLPGB/TLBSYNC passthrough:
> >> the multi-vCPU guest fails to boot when it uses INVLPGB, but boots
> >> when its INVLPGB CPUID bit is cleared. A one-vCPU guest also boots
> >> with INVLPGB enabled.
> >>
> >> In this configuration, an unintercepted guest INVLPGB targets the
> >> invoking vCPU's hardware ASID, while sibling vCPUs have distinct
> >> ASIDs.
>
> It's neat that INVLPGB will automatically use the current ASID and not
> ASID=0 for guest usage.

Hmm, actually, how is this supposed to work with nested
virtualization? The APM says:

    A guest that executes a legal INVLPGB that is not intercepted will have
    the requested ASID field replaced by the current ASID and the valid
    ASID bit set before doing the broadcast invalidation.

If any ASID passed by an L1 guest is replaced by its own ASID, then if
L1 is flushing L2's ASID (i.e. a non-zero ASID), passing through
INVLPGB would do the wrong thing and flush L1's ASID instead, right?

Seems like if the VM is using nested (on any vCPU), KVM must intercept
INVLPGB anyway to correctly handle the ASID?

The APM also says:

    Two forms of INVLPGB intercepts, conditional and unconditional, are
    available to the hypervisor. The unconditional intercept traps all guest
    usage of INVLPGB. The conditional intercept traps only
    illegally-specified INVLPGB instructions.

When I read "conditional", I was really hoping it was conditional on
the ASID being zero or non-zero, as KVM really only needs to intercept
INVLPGB with non-zero ASIDs (assuming the ASID is shared to begin
with).

Am I missing something?

Re: Some options abort RFC: KVM: SVM: Rework ASID management

[Yosry Ahmed]

On Thu, Oct 29, 2026 at 3:57 AM Jackie Yu <ufo19890607@gmail.com> wrote:
>
> Subject: Re: [PATCH v2 00/29] KVM: nSVM: Optimize nSVM TLB flushes
>
> Hi Yosry,
>
> Thank you for the quick response. Two clarifications from our side:
>
> 1. About the boot failures: no emulation is involved in our experiment. Our out-of-tree changes enable direct guest INVLPGB/TLBSYNC execution by clearing the KVM intercepts, without any software emulation of the instruction. The failures occur with this unintercepted passthrough combined with static per-vCPU ASIDs, so they are not missing cases in an emulation path. Under the per-vCPU design we did not attempt to intercept and emulate INVLPGB at all.

That's what I meant, the missing emulation of INVLPGB needed with
different per-vCPU ASIDs.

>
> 2. We do not use nested virtualization in our deployments. L1 guests in our production scenarios run without nested guests. Therefore, the ASID-replacement behavior you quoted from the APM does not affect our current use case: with one shared ASID per non-nested VM, an unintercepted guest INVLPGB would be scoped to that VM's ASID, which matches what our per-VM configuration relies on today.
>
> Regarding the direction: we understand that with per-vCPU ASIDs, correct handling would require intercepting INVLPGB and re-issuing a flush for every ASID in use by the VM, which you note is unlikely to perform well on large VMs. We also note your point that as long as any vCPU uses nested, KVM must intercept INVLPGB anyway because the hardware replaces the requested ASID with the current ASID.
>
> If we read the situation correctly, the practical options seem to be either (a) a per-VM shared ASID mode restricted to non-nested VMs, with unintercepted guest INVLPGB, plus mandatory interception whenever nested is in use, or (b) always intercepting and virtualizing INVLPGB under per-vCPU ASIDs. Either way, direct passthrough would at most cover the non-nested case. We would appreciate Sean's and Paolo's input on which direction upstream prefers; we are glad to provide more details of our configuration or run tests that would help decide between these options.

I don't think (b) makes a lot of sense, but if we do (a) we should
always use per-VM ASIDs, supporting both implementations based on
nested vs. non-nested would be a huge pain. I think what KVM can do is
disable INVLPGB intercepts until one vCPU sets EFER.SVME (or maybe on
first VMRUN, because with enable_virt_at_load KVM will always set
EFER.SVME on once it's loaded).

>
> Thanks,
>
> Jackie
>
>
> Yosry Ahmed <yosry@kernel.org> 于2026年10月29日周四 14:08写道：
>>
>> On Wed, Oct 28, 2026 at 1:15 PM Yosry Ahmed <yosry@kernel.org> wrote:
>> >
>> > On Wed, Oct 28, 2026 at 6:37 AM Jackie Yu <ufo19890607@gmail.com> wrote:
>> > >
>> > > Hi, guys!
>> > >
>> > > Jackie Yu <ufo19890607@gmail.com> 于2026年10月28日周三 21:36写道：
>> > >>
>> > >> Hi Yosry and KVM folks,
>> > >>
>> > >> We would appreciate your guidance on supporting INVLPGB in ordinary (non-SEV)
>> > >> L1 guests alongside the proposed static per-vCPU ASIDs.
>> > >>
>> > >> The 2025 ASID rework RFC proposed one static ASID per L1 VM and listed
>> > >> "enabling INVLPGB virtualization" among its motivations [1]. The June
>> > >> 2026 RFC v2 changed the design from per-VM to per-vCPU ASIDs [2]. In
>> > >> the October v2 series, patches 2-11 establish static per-vCPU ASIDs,
>> > >> while the remaining patches primarily address nested TLB flushes [3].
>> > >> Our question is how guest INVLPGB should be supported under the
>> > >> per-vCPU design.
>> > >>
>> > >> Some of our workloads frequently discard and repopulate memory using
>> > >> patterns involving madvise() and memset(). Broadcast IPI-based TLB
>> > >> shootdowns are expensive for these workloads, so we are interested in
>> > >> making AMD INVLPGB available to guests. We use a small benchmark to
>> > >> approximate the resulting shared-address-space pressure:
>> > >>
>> > >>     region = mmap(256 MiB, MAP_SHARED | MAP_ANONYMOUS);
>> > >>     memset(region, 0xab, 256 MiB);       // Initial population
>> > >>     start N workers, pinned across 64 allowed CPUs;
>> > >>     each worker repeatedly reads pages in its own slice of region;
>> > >>
>> > >>     repeat 500 warmup + 2,000 measured iterations:
>> > >>         start = clock_gettime(CLOCK_MONOTONIC);
>> > >>         madvise(region, 256 MiB, MADV_DONTNEED);
>> > >>         record elapsed madvise() time;
>> > >>         write one page in every 64 pages to repopulate part of region;
>> > >>
>> > >>     stop and join the workers;
>> > >>
>> > >> The benchmark uses memset() for initial population; subsequent
>> > >> iterations re-touch pages rather than repeatedly calling memset().
>> > >> The numbers below measure the advisor thread's madvise() syscall
>> > >> latency, not worker throughput.
>> > >>
>> > >> In our experimental KVM4 configuration with a VM-shared ASID and our
>> > >> separate guest INVLPGB passthrough changes, we observed:
>> > >>
>> > >>                          INVLPGB off     INVLPGB on    Difference
>> > >>     64 workers  mean       21.624 ms       12.505 ms     -42.17%
>> > >>                 P99        32.760 ms       21.518 ms     -34.32%
>> > >>     128 workers mean       29.553 ms       20.459 ms     -30.77%
>> > >>                 P99        39.628 ms       28.160 ms     -28.94%
>> > >>
>> > >> These are single, sequential A/B runs for each worker count, with
>> > >> 500 warmup and 2,000 measured calls per run. The process had 64
>> > >> allowed CPUs, so the 128-worker case was oversubscribed. The
>> > >> measurements show a useful latency difference, but do not by
>> > >> themselves establish INVLPGB instruction counts or IPI reductions.
>> > >>
>> > >> We then experimentally combined the October per-vCPU ASID changes for
>> > >> ordinary guests with our existing, out-of-tree L1 INVLPGB/TLBSYNC
>> > >> passthrough. With guest INVLPGB enabled, a one-vCPU guest boots, but
>> > >> otherwise comparable two-vCPU and 64-vCPU guests fail early during
>> > >> boot. Keeping QEMU's +invlpgb setting while adding clearcpuid=419
>> > >> to the same guest kernel allows the multi-vCPU guest to boot; the
>> > >> guest logs "Clearing CPUID bits: invlpgb".
>> >
>> > I imagine for INVLPGB to work correctly with per-vCPU ASIDs, you need
>> > to intercept INVLPGB and basically issue an INVLPGB for each ASID used
>> > by this VM (nr_vcpus or 2 * nr_vcpus with nested). I can't imagine
>> > this would perform well on large VMs, not sure if virtualizing INVLPGB
>> > would still make sense.
>> >
>> > Regarding the boot failure, I would guess there are some missing cases
>> > due to the complexity of emulating INVLPGB without a shared per-VM
>> > ASID.
>> >
>> > >> These controlled boot comparisons identify a problem with combining
>> > >> static per-vCPU ASIDs and direct guest INVLPGB/TLBSYNC passthrough:
>> > >> the multi-vCPU guest fails to boot when it uses INVLPGB, but boots
>> > >> when its INVLPGB CPUID bit is cleared. A one-vCPU guest also boots
>> > >> with INVLPGB enabled.
>> > >>
>> > >> In this configuration, an unintercepted guest INVLPGB targets the
>> > >> invoking vCPU's hardware ASID, while sibling vCPUs have distinct
>> > >> ASIDs.
>> >
>> > It's neat that INVLPGB will automatically use the current ASID and not
>> > ASID=0 for guest usage.
>>
>> Hmm, actually, how is this supposed to work with nested
>> virtualization? The APM says:
>>
>>     A guest that executes a legal INVLPGB that is not intercepted will have
>>     the requested ASID field replaced by the current ASID and the valid
>>     ASID bit set before doing the broadcast invalidation.
>>
>> If any ASID passed by an L1 guest is replaced by its own ASID, then if
>> L1 is flushing L2's ASID (i.e. a non-zero ASID), passing through
>> INVLPGB would do the wrong thing and flush L1's ASID instead, right?
>>
>> Seems like if the VM is using nested (on any vCPU), KVM must intercept
>> INVLPGB anyway to correctly handle the ASID?
>>
>> The APM also says:
>>
>>     Two forms of INVLPGB intercepts, conditional and unconditional, are
>>     available to the hypervisor. The unconditional intercept traps all guest
>>     usage of INVLPGB. The conditional intercept traps only
>>     illegally-specified INVLPGB instructions.
>>
>> When I read "conditional", I was really hoping it was conditional on
>> the ASID being zero or non-zero, as KVM really only needs to intercept
>> INVLPGB with non-zero ASIDs (assuming the ASID is shared to begin
>> with).
>>
>> Am I missing something?

Re: Some options abort RFC: KVM: SVM: Rework ASID management

[Sean Christopherson]

Please send mails as plain text, otherwise they get ignored by the mailing lists
and don't get archived by lore.

On Thu, Oct 29, 2026, Jackie Yu wrote:
> Subject: Re: [PATCH v2 00/29] KVM: nSVM: Optimize nSVM TLB flushes
> 
> Hi Yosry,
> 
> Thank you for the quick response. Two clarifications from our side:
> 
> 1. About the boot failures: no emulation is involved in our
> experiment. Our out-of-tree changes enable direct guest
> INVLPGB/TLBSYNC execution by clearing the KVM intercepts, without any
> software emulation of the instruction. The failures occur with this
> unintercepted passthrough combined with static per-vCPU ASIDs, so they
> are not missing cases in an emulation path. Under the per-vCPU design
> we did not attempt to intercept and emulate INVLPGB at all.
> 
> 2. We do not use nested virtualization in our deployments. L1 guests
> in our production scenarios run without nested guests. Therefore, the
> ASID-replacement behavior you quoted from the APM does not affect our
> current use case: with one shared ASID per non-nested VM, an
> unintercepted guest INVLPGB would be scoped to that VM's ASID, which
> matches what our per-VM configuration relies on today.
> 
> Regarding the direction: we understand that with per-vCPU ASIDs,
> correct handling would require intercepting INVLPGB and re-issuing a
> flush for every ASID in use by the VM, which you note is unlikely to
> perform well on large VMs. We also note your point that as long as any
> vCPU uses nested, KVM must intercept INVLPGB anyway because the
> hardware replaces the requested ASID with the current ASID.
> 
> If we read the situation correctly, the practical options seem to be
> either (a) a per-VM shared ASID mode restricted to non-nested VMs,
> with unintercepted guest INVLPGB, plus mandatory interception whenever
> nested is in use, or (b) always intercepting and virtualizing INVLPGB
> under per-vCPU ASIDs. Either way, direct passthrough would at most
> cover the non-nested case. We would appreciate Sean's and Paolo's
> input on which direction upstream prefers; we are glad to provide more
> details of our configuration or run tests that would help decide
> between these options.
> 
> Thanks,
> 
> Jackie
> 
> 
> Yosry Ahmed <yosry@kernel.org> 于2026年10月29日周四 14:08写道：
> 
> > On Wed, Oct 28, 2026 at 1:15 PM Yosry Ahmed <yosry@kernel.org> wrote:
> > >
> > > On Wed, Oct 28, 2026 at 6:37 AM Jackie Yu <ufo19890607@gmail.com> wrote:
> > > >
> > > > Hi, guys!
> > > >
> > > > Jackie Yu <ufo19890607@gmail.com> 于2026年10月28日周三 21:36写道：
> > > >>
> > > >> Hi Yosry and KVM folks,
> > > >>
> > > >> We would appreciate your guidance on supporting INVLPGB in ordinary
> > (non-SEV)
> > > >> L1 guests alongside the proposed static per-vCPU ASIDs.
> > > >>
> > > >> The 2025 ASID rework RFC proposed one static ASID per L1 VM and listed
> > > >> "enabling INVLPGB virtualization" among its motivations [1]. The June
> > > >> 2026 RFC v2 changed the design from per-VM to per-vCPU ASIDs [2]. In
> > > >> the October v2 series, patches 2-11 establish static per-vCPU ASIDs,
> > > >> while the remaining patches primarily address nested TLB flushes [3].
> > > >> Our question is how guest INVLPGB should be supported under the
> > > >> per-vCPU design.
> > > >>
> > > >> Some of our workloads frequently discard and repopulate memory using
> > > >> patterns involving madvise() and memset(). Broadcast IPI-based TLB
> > > >> shootdowns are expensive for these workloads, so we are interested in
> > > >> making AMD INVLPGB available to guests. We use a small benchmark to
> > > >> approximate the resulting shared-address-space pressure:
> > > >>
> > > >>     region = mmap(256 MiB, MAP_SHARED | MAP_ANONYMOUS);
> > > >>     memset(region, 0xab, 256 MiB);       // Initial population
> > > >>     start N workers, pinned across 64 allowed CPUs;
> > > >>     each worker repeatedly reads pages in its own slice of region;
> > > >>
> > > >>     repeat 500 warmup + 2,000 measured iterations:
> > > >>         start = clock_gettime(CLOCK_MONOTONIC);
> > > >>         madvise(region, 256 MiB, MADV_DONTNEED);
> > > >>         record elapsed madvise() time;
> > > >>         write one page in every 64 pages to repopulate part of region;
> > > >>
> > > >>     stop and join the workers;
> > > >>
> > > >> The benchmark uses memset() for initial population; subsequent
> > > >> iterations re-touch pages rather than repeatedly calling memset().
> > > >> The numbers below measure the advisor thread's madvise() syscall
> > > >> latency, not worker throughput.
> > > >>
> > > >> In our experimental KVM4 configuration with a VM-shared ASID and our
> > > >> separate guest INVLPGB passthrough changes, we observed:
> > > >>
> > > >>                          INVLPGB off     INVLPGB on    Difference
> > > >>     64 workers  mean       21.624 ms       12.505 ms     -42.17%
> > > >>                 P99        32.760 ms       21.518 ms     -34.32%
> > > >>     128 workers mean       29.553 ms       20.459 ms     -30.77%
> > > >>                 P99        39.628 ms       28.160 ms     -28.94%
> > > >>
> > > >> These are single, sequential A/B runs for each worker count, with
> > > >> 500 warmup and 2,000 measured calls per run. The process had 64
> > > >> allowed CPUs, so the 128-worker case was oversubscribed. The
> > > >> measurements show a useful latency difference, but do not by
> > > >> themselves establish INVLPGB instruction counts or IPI reductions.
> > > >>
> > > >> We then experimentally combined the October per-vCPU ASID changes for
> > > >> ordinary guests with our existing, out-of-tree L1 INVLPGB/TLBSYNC
> > > >> passthrough. With guest INVLPGB enabled, a one-vCPU guest boots, but
> > > >> otherwise comparable two-vCPU and 64-vCPU guests fail early during
> > > >> boot. Keeping QEMU's +invlpgb setting while adding clearcpuid=419
> > > >> to the same guest kernel allows the multi-vCPU guest to boot; the
> > > >> guest logs "Clearing CPUID bits: invlpgb".
> > >
> > > I imagine for INVLPGB to work correctly with per-vCPU ASIDs, you need
> > > to intercept INVLPGB and basically issue an INVLPGB for each ASID used
> > > by this VM (nr_vcpus or 2 * nr_vcpus with nested). I can't imagine
> > > this would perform well on large VMs, not sure if virtualizing INVLPGB
> > > would still make sense.
> > >
> > > Regarding the boot failure, I would guess there are some missing cases
> > > due to the complexity of emulating INVLPGB without a shared per-VM
> > > ASID.
> > >
> > > >> These controlled boot comparisons identify a problem with combining
> > > >> static per-vCPU ASIDs and direct guest INVLPGB/TLBSYNC passthrough:
> > > >> the multi-vCPU guest fails to boot when it uses INVLPGB, but boots
> > > >> when its INVLPGB CPUID bit is cleared. A one-vCPU guest also boots
> > > >> with INVLPGB enabled.
> > > >>
> > > >> In this configuration, an unintercepted guest INVLPGB targets the
> > > >> invoking vCPU's hardware ASID, while sibling vCPUs have distinct
> > > >> ASIDs.
> > >
> > > It's neat that INVLPGB will automatically use the current ASID and not
> > > ASID=0 for guest usage.
> >
> > Hmm, actually, how is this supposed to work with nested
> > virtualization? The APM says:
> >
> >     A guest that executes a legal INVLPGB that is not intercepted will have
> >     the requested ASID field replaced by the current ASID and the valid
> >     ASID bit set before doing the broadcast invalidation.
> >
> > If any ASID passed by an L1 guest is replaced by its own ASID, then if
> > L1 is flushing L2's ASID (i.e. a non-zero ASID), passing through
> > INVLPGB would do the wrong thing and flush L1's ASID instead, right?
> >
> > Seems like if the VM is using nested (on any vCPU), KVM must intercept
> > INVLPGB anyway to correctly handle the ASID?
> >
> > The APM also says:
> >
> >     Two forms of INVLPGB intercepts, conditional and unconditional, are
> >     available to the hypervisor. The unconditional intercept traps all
> > guest
> >     usage of INVLPGB. The conditional intercept traps only
> >     illegally-specified INVLPGB instructions.
> >
> > When I read "conditional", I was really hoping it was conditional on
> > the ASID being zero or non-zero, as KVM really only needs to intercept
> > INVLPGB with non-zero ASIDs (assuming the ASID is shared to begin
> > with).
> >
> > Am I missing something?
> >

Re: Some options abort RFC: KVM: SVM: Rework ASID management

[Jackie Yu]

Hi Sean, Yosry, and KVM folks,

Sean, thank you for the reminder. Sorry about the earlier formatting.
We will use plain text for future mailing-list replies.

Yosry, thanks for clarifying. We understand that by "missing
emulation" you meant the INVLPGB handling needed with distinct
per-vCPU ASIDs, not a bug in an existing emulation path in our tree.

We agree that intercepting each guest INVLPGB and issuing a separate
invalidation for every ASID in a large VM may undermine the benefit
we are seeking. Using per-VM ASIDs consistently, rather than
switching ASID allocation models based on nested usage, sounds like
a promising direction. Our production guests do not run nested VMs,
so direct execution before nested use would cover our use case.

Your suggestion to change the INVLPGB interception policy when
nested virtualization is first used is helpful. If EFER.SVME can be
set before a guest runs a nested VM, the first VMRUN seems like a
more precise trigger. Would KVM catch that VMRUN before entering L2
and enable the necessary intercepts across the VM before any vCPU
can execute an INVLPGB requiring nested handling? Could those
intercepts then remain enabled for the VM's lifetime, rather than
switching back when the guest leaves nested mode? We realize that
interception would still require correct nested instruction handling.

For context, our experimental KVM4 configuration used a VM-shared
ASID and separate, out-of-tree guest INVLPGB passthrough. The
madvise() latency measurements were:

  Workers             INVLPGB off      INVLPGB on
  64 mean              21.624 ms        12.505 ms
  64 P99               32.760 ms        21.518 ms
  128 mean             29.553 ms        20.459 ms
  128 P99              39.628 ms        28.160 ms

These were single sequential A/B runs with 64 allowed CPUs; the
128-worker run was oversubscribed. The results measure madvise()
syscall latency, not IPI counts, and do not quantify any additional
flushes from vCPU switches on a physical CPU under per-VM ASIDs.

We can share the benchmark source and run further comparisons to
help evaluate that trade-off. Would the maintainers consider
consistent per-VM ASIDs, with direct guest INVLPGB execution before
nested use and interception once nested use begins, a direction
worth pursuing?

Best regards,


Sean Christopherson <seanjc@google.com> 于2026年10月30日周五 00:09写道：
>
> Please send mails as plain text, otherwise they get ignored by the mailing lists
> and don't get archived by lore.
>
> On Thu, Oct 29, 2026, Jackie Yu wrote:
> > Subject: Re: [PATCH v2 00/29] KVM: nSVM: Optimize nSVM TLB flushes
> >
> > Hi Yosry,
> >
> > Thank you for the quick response. Two clarifications from our side:
> >
> > 1. About the boot failures: no emulation is involved in our
> > experiment. Our out-of-tree changes enable direct guest
> > INVLPGB/TLBSYNC execution by clearing the KVM intercepts, without any
> > software emulation of the instruction. The failures occur with this
> > unintercepted passthrough combined with static per-vCPU ASIDs, so they
> > are not missing cases in an emulation path. Under the per-vCPU design
> > we did not attempt to intercept and emulate INVLPGB at all.
> >
> > 2. We do not use nested virtualization in our deployments. L1 guests
> > in our production scenarios run without nested guests. Therefore, the
> > ASID-replacement behavior you quoted from the APM does not affect our
> > current use case: with one shared ASID per non-nested VM, an
> > unintercepted guest INVLPGB would be scoped to that VM's ASID, which
> > matches what our per-VM configuration relies on today.
> >
> > Regarding the direction: we understand that with per-vCPU ASIDs,
> > correct handling would require intercepting INVLPGB and re-issuing a
> > flush for every ASID in use by the VM, which you note is unlikely to
> > perform well on large VMs. We also note your point that as long as any
> > vCPU uses nested, KVM must intercept INVLPGB anyway because the
> > hardware replaces the requested ASID with the current ASID.
> >
> > If we read the situation correctly, the practical options seem to be
> > either (a) a per-VM shared ASID mode restricted to non-nested VMs,
> > with unintercepted guest INVLPGB, plus mandatory interception whenever
> > nested is in use, or (b) always intercepting and virtualizing INVLPGB
> > under per-vCPU ASIDs. Either way, direct passthrough would at most
> > cover the non-nested case. We would appreciate Sean's and Paolo's
> > input on which direction upstream prefers; we are glad to provide more
> > details of our configuration or run tests that would help decide
> > between these options.
> >
> > Thanks,
> >
> > Jackie
> >
> >
> > Yosry Ahmed <yosry@kernel.org> 于2026年10月29日周四 14:08写道：
> >
> > > On Wed, Oct 28, 2026 at 1:15 PM Yosry Ahmed <yosry@kernel.org> wrote:
> > > >
> > > > On Wed, Oct 28, 2026 at 6:37 AM Jackie Yu <ufo19890607@gmail.com> wrote:
> > > > >
> > > > > Hi, guys!
> > > > >
> > > > > Jackie Yu <ufo19890607@gmail.com> 于2026年10月28日周三 21:36写道：
> > > > >>
> > > > >> Hi Yosry and KVM folks,
> > > > >>
> > > > >> We would appreciate your guidance on supporting INVLPGB in ordinary
> > > (non-SEV)
> > > > >> L1 guests alongside the proposed static per-vCPU ASIDs.
> > > > >>
> > > > >> The 2025 ASID rework RFC proposed one static ASID per L1 VM and listed
> > > > >> "enabling INVLPGB virtualization" among its motivations [1]. The June
> > > > >> 2026 RFC v2 changed the design from per-VM to per-vCPU ASIDs [2]. In
> > > > >> the October v2 series, patches 2-11 establish static per-vCPU ASIDs,
> > > > >> while the remaining patches primarily address nested TLB flushes [3].
> > > > >> Our question is how guest INVLPGB should be supported under the
> > > > >> per-vCPU design.
> > > > >>
> > > > >> Some of our workloads frequently discard and repopulate memory using
> > > > >> patterns involving madvise() and memset(). Broadcast IPI-based TLB
> > > > >> shootdowns are expensive for these workloads, so we are interested in
> > > > >> making AMD INVLPGB available to guests. We use a small benchmark to
> > > > >> approximate the resulting shared-address-space pressure:
> > > > >>
> > > > >>     region = mmap(256 MiB, MAP_SHARED | MAP_ANONYMOUS);
> > > > >>     memset(region, 0xab, 256 MiB);       // Initial population
> > > > >>     start N workers, pinned across 64 allowed CPUs;
> > > > >>     each worker repeatedly reads pages in its own slice of region;
> > > > >>
> > > > >>     repeat 500 warmup + 2,000 measured iterations:
> > > > >>         start = clock_gettime(CLOCK_MONOTONIC);
> > > > >>         madvise(region, 256 MiB, MADV_DONTNEED);
> > > > >>         record elapsed madvise() time;
> > > > >>         write one page in every 64 pages to repopulate part of region;
> > > > >>
> > > > >>     stop and join the workers;
> > > > >>
> > > > >> The benchmark uses memset() for initial population; subsequent
> > > > >> iterations re-touch pages rather than repeatedly calling memset().
> > > > >> The numbers below measure the advisor thread's madvise() syscall
> > > > >> latency, not worker throughput.
> > > > >>
> > > > >> In our experimental KVM4 configuration with a VM-shared ASID and our
> > > > >> separate guest INVLPGB passthrough changes, we observed:
> > > > >>
> > > > >>                          INVLPGB off     INVLPGB on    Difference
> > > > >>     64 workers  mean       21.624 ms       12.505 ms     -42.17%
> > > > >>                 P99        32.760 ms       21.518 ms     -34.32%
> > > > >>     128 workers mean       29.553 ms       20.459 ms     -30.77%
> > > > >>                 P99        39.628 ms       28.160 ms     -28.94%
> > > > >>
> > > > >> These are single, sequential A/B runs for each worker count, with
> > > > >> 500 warmup and 2,000 measured calls per run. The process had 64
> > > > >> allowed CPUs, so the 128-worker case was oversubscribed. The
> > > > >> measurements show a useful latency difference, but do not by
> > > > >> themselves establish INVLPGB instruction counts or IPI reductions.
> > > > >>
> > > > >> We then experimentally combined the October per-vCPU ASID changes for
> > > > >> ordinary guests with our existing, out-of-tree L1 INVLPGB/TLBSYNC
> > > > >> passthrough. With guest INVLPGB enabled, a one-vCPU guest boots, but
> > > > >> otherwise comparable two-vCPU and 64-vCPU guests fail early during
> > > > >> boot. Keeping QEMU's +invlpgb setting while adding clearcpuid=419
> > > > >> to the same guest kernel allows the multi-vCPU guest to boot; the
> > > > >> guest logs "Clearing CPUID bits: invlpgb".
> > > >
> > > > I imagine for INVLPGB to work correctly with per-vCPU ASIDs, you need
> > > > to intercept INVLPGB and basically issue an INVLPGB for each ASID used
> > > > by this VM (nr_vcpus or 2 * nr_vcpus with nested). I can't imagine
> > > > this would perform well on large VMs, not sure if virtualizing INVLPGB
> > > > would still make sense.
> > > >
> > > > Regarding the boot failure, I would guess there are some missing cases
> > > > due to the complexity of emulating INVLPGB without a shared per-VM
> > > > ASID.
> > > >
> > > > >> These controlled boot comparisons identify a problem with combining
> > > > >> static per-vCPU ASIDs and direct guest INVLPGB/TLBSYNC passthrough:
> > > > >> the multi-vCPU guest fails to boot when it uses INVLPGB, but boots
> > > > >> when its INVLPGB CPUID bit is cleared. A one-vCPU guest also boots
> > > > >> with INVLPGB enabled.
> > > > >>
> > > > >> In this configuration, an unintercepted guest INVLPGB targets the
> > > > >> invoking vCPU's hardware ASID, while sibling vCPUs have distinct
> > > > >> ASIDs.
> > > >
> > > > It's neat that INVLPGB will automatically use the current ASID and not
> > > > ASID=0 for guest usage.
> > >
> > > Hmm, actually, how is this supposed to work with nested
> > > virtualization? The APM says:
> > >
> > >     A guest that executes a legal INVLPGB that is not intercepted will have
> > >     the requested ASID field replaced by the current ASID and the valid
> > >     ASID bit set before doing the broadcast invalidation.
> > >
> > > If any ASID passed by an L1 guest is replaced by its own ASID, then if
> > > L1 is flushing L2's ASID (i.e. a non-zero ASID), passing through
> > > INVLPGB would do the wrong thing and flush L1's ASID instead, right?
> > >
> > > Seems like if the VM is using nested (on any vCPU), KVM must intercept
> > > INVLPGB anyway to correctly handle the ASID?
> > >
> > > The APM also says:
> > >
> > >     Two forms of INVLPGB intercepts, conditional and unconditional, are
> > >     available to the hypervisor. The unconditional intercept traps all
> > > guest
> > >     usage of INVLPGB. The conditional intercept traps only
> > >     illegally-specified INVLPGB instructions.
> > >
> > > When I read "conditional", I was really hoping it was conditional on
> > > the ASID being zero or non-zero, as KVM really only needs to intercept
> > > INVLPGB with non-zero ASIDs (assuming the ASID is shared to begin
> > > with).
> > >
> > > Am I missing something?
> > >

Re: Some options abort RFC: KVM: SVM: Rework ASID management

[Yosry Ahmed]

On Mon, Nov 2, 2026 at 3:20 AM Jackie Yu <ufo19890607@gmail.com> wrote:
>
> Hi Sean, Yosry, and KVM folks,
>
> Sean, thank you for the reminder. Sorry about the earlier formatting.
> We will use plain text for future mailing-list replies.
>
> Yosry, thanks for clarifying. We understand that by "missing
> emulation" you meant the INVLPGB handling needed with distinct
> per-vCPU ASIDs, not a bug in an existing emulation path in our tree.
>
> We agree that intercepting each guest INVLPGB and issuing a separate
> invalidation for every ASID in a large VM may undermine the benefit
> we are seeking. Using per-VM ASIDs consistently, rather than
> switching ASID allocation models based on nested usage, sounds like
> a promising direction. Our production guests do not run nested VMs,
> so direct execution before nested use would cover our use case.
>
> Your suggestion to change the INVLPGB interception policy when
> nested virtualization is first used is helpful. If EFER.SVME can be
> set before a guest runs a nested VM, the first VMRUN seems like a
> more precise trigger. Would KVM catch that VMRUN before entering L2
> and enable the necessary intercepts across the VM before any vCPU
> can execute an INVLPGB requiring nested handling?

Yeah we can do this. For example, we can have a static branch in the
VMRUN emulation path that flips between a synchronized path that
enables the INVLPGB intercept, and the normal path. The static branch
might be an overkill.

> Could those
> intercepts then remain enabled for the VM's lifetime, rather than
> switching back when the guest leaves nested mode?

Probably? I think we can only switch back when all vCPUs clear
EFER.SVME. I doubt it's worth optimizing.

> We realize that
> interception would still require correct nested instruction handling.
>
> For context, our experimental KVM4 configuration used a VM-shared
> ASID and separate, out-of-tree guest INVLPGB passthrough. The
> madvise() latency measurements were:
>
>   Workers             INVLPGB off      INVLPGB on
>   64 mean              21.624 ms        12.505 ms
>   64 P99               32.760 ms        21.518 ms
>   128 mean             29.553 ms        20.459 ms
>   128 P99              39.628 ms        28.160 ms
>
> These were single sequential A/B runs with 64 allowed CPUs; the
> 128-worker run was oversubscribed. The results measure madvise()
> syscall latency, not IPI counts, and do not quantify any additional
> flushes from vCPU switches on a physical CPU under per-VM ASIDs.
>
> We can share the benchmark source and run further comparisons to
> help evaluate that trade-off. Would the maintainers consider
> consistent per-VM ASIDs, with direct guest INVLPGB execution before
> nested use and interception once nested use begins, a direction
> worth pursuing?

As I mentioned, I (not the maintainer) have no problem with per-VM
ASIDs. We probably want to make sure we don't regress workloads due to
the extra TLB flushes when multiple vCPUs run on the same pCPU tho.

Other than that, I don't want to rework my series to go back to per-VM
ASIDs. So if Sean and Paolo are okay with per-VM ASIDs, I think this
should be done as a follow up work on top. Just my 2c.

Re: Some options abort RFC: KVM: SVM: Rework ASID management

[Sean Christopherson]

On Mon, Nov 02, 2026, Yosry Ahmed wrote:
> On Mon, Nov 2, 2026 at 3:20 AM Jackie Yu <ufo19890607@gmail.com> wrote:
> >
> > Hi Sean, Yosry, and KVM folks,
> >
> > Sean, thank you for the reminder. Sorry about the earlier formatting.
> > We will use plain text for future mailing-list replies.
> >
> > Yosry, thanks for clarifying. We understand that by "missing
> > emulation" you meant the INVLPGB handling needed with distinct
> > per-vCPU ASIDs, not a bug in an existing emulation path in our tree.
> >
> > We agree that intercepting each guest INVLPGB and issuing a separate
> > invalidation for every ASID in a large VM may undermine the benefit
> > we are seeking. Using per-VM ASIDs consistently, rather than
> > switching ASID allocation models based on nested usage, sounds like
> > a promising direction. Our production guests do not run nested VMs,
> > so direct execution before nested use would cover our use case.
> >
> > Your suggestion to change the INVLPGB interception policy when
> > nested virtualization is first used is helpful. If EFER.SVME can be
> > set before a guest runs a nested VM, the first VMRUN seems like a
> > more precise trigger. Would KVM catch that VMRUN before entering L2
> > and enable the necessary intercepts across the VM before any vCPU
> > can execute an INVLPGB requiring nested handling?
> 
> Yeah we can do this. For example, we can have a static branch in the
> VMRUN emulation path that flips between a synchronized path that
> enables the INVLPGB intercept, and the normal path. The static branch
> might be an overkill.
> 
> > Could those
> > intercepts then remain enabled for the VM's lifetime, rather than
> > switching back when the guest leaves nested mode?
> 
> Probably? I think we can only switch back when all vCPUs clear
> EFER.SVME. I doubt it's worth optimizing.
> 
> > We realize that interception would still require correct nested instruction
> > handling.
> >
> > For context, our experimental KVM4 configuration used a VM-shared
> > ASID and separate, out-of-tree guest INVLPGB passthrough. The
> > madvise() latency measurements were:
> >
> >   Workers             INVLPGB off      INVLPGB on
> >   64 mean              21.624 ms        12.505 ms
> >   64 P99               32.760 ms        21.518 ms
> >   128 mean             29.553 ms        20.459 ms
> >   128 P99              39.628 ms        28.160 ms
> >
> > These were single sequential A/B runs with 64 allowed CPUs; the
> > 128-worker run was oversubscribed. The results measure madvise()
> > syscall latency, not IPI counts, and do not quantify any additional
> > flushes from vCPU switches on a physical CPU under per-VM ASIDs.
> >
> > We can share the benchmark source and run further comparisons to
> > help evaluate that trade-off. Would the maintainers consider
> > consistent per-VM ASIDs, with direct guest INVLPGB execution before
> > nested use and interception once nested use begins, a direction
> > worth pursuing?
> 
> As I mentioned, I (not the maintainer) have no problem with per-VM
> ASIDs. We probably want to make sure we don't regress workloads due to
> the extra TLB flushes when multiple vCPUs run on the same pCPU tho.

Short answer: no, I don't think per-VM ASIDs is a direction worth pursing.

I actually really like the *idea* of per-VM ASIDs, but in practice they just
don't work well.  Having to flush when running a different vCPU with the same
ASID is generally acceptable for SEV+, because those types of VMs aren't often
CPU overcommited, and they don't support running as nested VMs[1].  Neither of
those holds true for "regular" VMs.

E.g. when running KVM as L1, having to flush on every vCPU migration becomes a
major pain point, because L0 typically has to intercept the flushes.  With per-vCPU
ASIDs, we have to line of sight to eliding those flushes[2].  That should be a
win for KVM-as-L0 as well, as it'll remove any "artificial" TLB misses due to
technically-unnecessary flushing.  CPU overcommited vCPUs will still suffer more
TLB misses due to cross-vCPU TLB pressure, but they won't get hit with a pile of
misses if a vCPU is bouncing between a small set of CPUs (many setups affine vCPUs
to a pool, versus doing a 1:1 pin).

And despite ASIDs being per-VM for SEV+, that's more of a clever hack to avoid
having to add new ISA to manage encryption keys than anything else.  Maybe AMD
intended ASIDs to be per-VM, but the reality is they can't truly be per-VM since
there is no separate tag and invalidation mechanism for host-controlled mappings.
I.e. the ASID is basically VMX's EPT root and VPID rolled into one, and as a result,
it takes on the properties of both a per-VM thing and a per-vCPU thing.

INVLPGB itself also has problems.  It's a broadcast to the *entire* system.  Even
if there's no work to be done, *all* other pCPUs still need to respond.  I haven't
done benchmarking, but I wouldn't be at all surprised if there are serious noisy
neighbor issues if a misbehaving guest is spamming INVLPGB in a tight loop.  E.g.
Maxim measured the impact of ringing the AVIC doorbell in a tight loop, and found
he could degrade CPU througput by something like 40-50%.

And then if the guest does something that forces KVM to intercept INVLPGB to do
the right thing, I suspect Yosry is right and performance will tank.

So, outside of very specific setups, per-VM ASIDs and INVLPGB passthrough seem
like a lot of complexity for little-to-no benefit.  Add in that per-VM ASIDs will
likely do more harm than good for most other use cases, and I'm not inclinded to
take on maintaining per-VM ASIDs in upstream.

Sorry :-/

[1] It's actually possible to get KVM functional when running as an SEV guest,
    but it's hilariously stupid and completely defeats the value added by SEV.

[2] https://lore.kernel.org/all/aHWjPSIdp5B-2UBl@google.com

> Other than that, I don't want to rework my series to go back to per-VM
> ASIDs. So if Sean and Paolo are okay with per-VM ASIDs, I think this
> should be done as a follow up work on top. Just my 2c.
>