Re: Some options abort RFC: KVM: SVM: Rework ASID management

Re: Some options abort RFC: KVM: SVM: Rework ASID management

[Yosry Ahmed]

On Wed, Oct 28, 2026 at 6:37 AM Jackie Yu <ufo19890607@gmail.com> wrote:
>
> Hi, guys!
>
> Jackie Yu <ufo19890607@gmail.com> 于2026年10月28日周三 21:36写道：
>>
>> Hi Yosry and KVM folks,
>>
>> We would appreciate your guidance on supporting INVLPGB in ordinary (non-SEV)
>> L1 guests alongside the proposed static per-vCPU ASIDs.
>>
>> The 2025 ASID rework RFC proposed one static ASID per L1 VM and listed
>> "enabling INVLPGB virtualization" among its motivations [1]. The June
>> 2026 RFC v2 changed the design from per-VM to per-vCPU ASIDs [2]. In
>> the October v2 series, patches 2-11 establish static per-vCPU ASIDs,
>> while the remaining patches primarily address nested TLB flushes [3].
>> Our question is how guest INVLPGB should be supported under the
>> per-vCPU design.
>>
>> Some of our workloads frequently discard and repopulate memory using
>> patterns involving madvise() and memset(). Broadcast IPI-based TLB
>> shootdowns are expensive for these workloads, so we are interested in
>> making AMD INVLPGB available to guests. We use a small benchmark to
>> approximate the resulting shared-address-space pressure:
>>
>>     region = mmap(256 MiB, MAP_SHARED | MAP_ANONYMOUS);
>>     memset(region, 0xab, 256 MiB);       // Initial population
>>     start N workers, pinned across 64 allowed CPUs;
>>     each worker repeatedly reads pages in its own slice of region;
>>
>>     repeat 500 warmup + 2,000 measured iterations:
>>         start = clock_gettime(CLOCK_MONOTONIC);
>>         madvise(region, 256 MiB, MADV_DONTNEED);
>>         record elapsed madvise() time;
>>         write one page in every 64 pages to repopulate part of region;
>>
>>     stop and join the workers;
>>
>> The benchmark uses memset() for initial population; subsequent
>> iterations re-touch pages rather than repeatedly calling memset().
>> The numbers below measure the advisor thread's madvise() syscall
>> latency, not worker throughput.
>>
>> In our experimental KVM4 configuration with a VM-shared ASID and our
>> separate guest INVLPGB passthrough changes, we observed:
>>
>>                          INVLPGB off     INVLPGB on    Difference
>>     64 workers  mean       21.624 ms       12.505 ms     -42.17%
>>                 P99        32.760 ms       21.518 ms     -34.32%
>>     128 workers mean       29.553 ms       20.459 ms     -30.77%
>>                 P99        39.628 ms       28.160 ms     -28.94%
>>
>> These are single, sequential A/B runs for each worker count, with
>> 500 warmup and 2,000 measured calls per run. The process had 64
>> allowed CPUs, so the 128-worker case was oversubscribed. The
>> measurements show a useful latency difference, but do not by
>> themselves establish INVLPGB instruction counts or IPI reductions.
>>
>> We then experimentally combined the October per-vCPU ASID changes for
>> ordinary guests with our existing, out-of-tree L1 INVLPGB/TLBSYNC
>> passthrough. With guest INVLPGB enabled, a one-vCPU guest boots, but
>> otherwise comparable two-vCPU and 64-vCPU guests fail early during
>> boot. Keeping QEMU's +invlpgb setting while adding clearcpuid=419
>> to the same guest kernel allows the multi-vCPU guest to boot; the
>> guest logs "Clearing CPUID bits: invlpgb".

I imagine for INVLPGB to work correctly with per-vCPU ASIDs, you need
to intercept INVLPGB and basically issue an INVLPGB for each ASID used
by this VM (nr_vcpus or 2 * nr_vcpus with nested). I can't imagine
this would perform well on large VMs, not sure if virtualizing INVLPGB
would still make sense.

Regarding the boot failure, I would guess there are some missing cases
due to the complexity of emulating INVLPGB without a shared per-VM
ASID.

>> These controlled boot comparisons identify a problem with combining
>> static per-vCPU ASIDs and direct guest INVLPGB/TLBSYNC passthrough:
>> the multi-vCPU guest fails to boot when it uses INVLPGB, but boots
>> when its INVLPGB CPUID bit is cleared. A one-vCPU guest also boots
>> with INVLPGB enabled.
>>
>> In this configuration, an unintercepted guest INVLPGB targets the
>> invoking vCPU's hardware ASID, while sibling vCPUs have distinct
>> ASIDs.

It's neat that INVLPGB will automatically use the current ASID and not
ASID=0 for guest usage.

>> The October ASID rework does not add handling to extend that
>> guest broadcast invalidation to sibling ASIDs. Consequently, direct
>> passthrough does not preserve the required multi-vCPU guest
>> invalidation semantics. This problem arises when our out-of-tree
>> passthrough is combined with the per-vCPU ASID scheme; it is not a
>> claim that the upstream ASID series alone causes a boot failure.
>> The earlier discussion of guest INVLPGB and global ASIDs is relevant
>> here [4].
>>
>> Could you advise on the preferred upstream direction?

This is a question for Sean and Paolo.

It is possible to use one ASID per-VM, however:
1. It adds complexity as we need to track what vCPU in a VM ran last
on a CPU, so that we can flush the shared ASID in between different
vCPUs running on the same CPU.
2. It adds some TLB flushes due to scenario (1), not sure how much
this matters in practice.
3. It will re-establish a divergence in ASID/VPID management between
AMD and Intel, unless we also switch VMX to per-VM ASIDs, but I am not
sure if there's any benefit to this for now.

>>
>>   1. Would an opt-in, VM-scoped ASID mode for ordinary VMs that expose
>>      guest INVLPGB be acceptable, while other VMs retain per-vCPU
>>      ASIDs? This would require a properly managed shared-ASID lifetime
>>      and flush/reuse semantics, rather than simply giving vCPUs the
>>      same VMCB ASID value.
>>
>>   2. Alternatively, is the intended direction to retain per-vCPU
>>      ASIDs and intercept/virtualize guest INVLPGB and TLBSYNC, so that
>>      a guest broadcast invalidation correctly covers the relevant
>>      sibling vCPU ASIDs and preserves the required ordering?
>>
>>   3. If neither approach is currently planned, should ordinary
>>      multi-vCPU guests be prevented from enabling INVLPGB until the
>>      required semantics exist?
>>
>> We can share the madvise benchmark and its measurement methodology,
>> and would be happy to compare the same benchmark under both proposed
>> designs. We would appreciate feedback on the preferred upstream
>> architecture before pursuing a downstream implementation.
>>
>> References:
>> [1] 2025 ASID rework RFC (per-VM):
>>     https://patchew.org/linux/20250326193619.3714986-1-yosry.ahmed@linux.dev
>> [2] June 2026 RFC v2 (per-VM to per-vCPU):
>>     https://lore.kernel.org/kvm/20260616004155.1435766-1-yosry@kernel.org/
>> [3] October 2026 v2 series:
>>     https://lore.kernel.org/kvm/20261008001425.2458927-1-yosry@kernel.org/
>> [4] Discussion of non-SEV guest INVLPGB and global ASIDs:
>>     https://patchew.org/linux/20250223194943.3518952-1-riel@surriel.com/20250223194943.3518952-7-riel@surriel.com
>>
>> Best regards,
>> Wind Yu

Re: Some options abort RFC: KVM: SVM: Rework ASID management

[Yosry Ahmed]

On Wed, Oct 28, 2026 at 1:15 PM Yosry Ahmed <yosry@kernel.org> wrote:
>
> On Wed, Oct 28, 2026 at 6:37 AM Jackie Yu <ufo19890607@gmail.com> wrote:
> >
> > Hi, guys!
> >
> > Jackie Yu <ufo19890607@gmail.com> 于2026年10月28日周三 21:36写道：
> >>
> >> Hi Yosry and KVM folks,
> >>
> >> We would appreciate your guidance on supporting INVLPGB in ordinary (non-SEV)
> >> L1 guests alongside the proposed static per-vCPU ASIDs.
> >>
> >> The 2025 ASID rework RFC proposed one static ASID per L1 VM and listed
> >> "enabling INVLPGB virtualization" among its motivations [1]. The June
> >> 2026 RFC v2 changed the design from per-VM to per-vCPU ASIDs [2]. In
> >> the October v2 series, patches 2-11 establish static per-vCPU ASIDs,
> >> while the remaining patches primarily address nested TLB flushes [3].
> >> Our question is how guest INVLPGB should be supported under the
> >> per-vCPU design.
> >>
> >> Some of our workloads frequently discard and repopulate memory using
> >> patterns involving madvise() and memset(). Broadcast IPI-based TLB
> >> shootdowns are expensive for these workloads, so we are interested in
> >> making AMD INVLPGB available to guests. We use a small benchmark to
> >> approximate the resulting shared-address-space pressure:
> >>
> >>     region = mmap(256 MiB, MAP_SHARED | MAP_ANONYMOUS);
> >>     memset(region, 0xab, 256 MiB);       // Initial population
> >>     start N workers, pinned across 64 allowed CPUs;
> >>     each worker repeatedly reads pages in its own slice of region;
> >>
> >>     repeat 500 warmup + 2,000 measured iterations:
> >>         start = clock_gettime(CLOCK_MONOTONIC);
> >>         madvise(region, 256 MiB, MADV_DONTNEED);
> >>         record elapsed madvise() time;
> >>         write one page in every 64 pages to repopulate part of region;
> >>
> >>     stop and join the workers;
> >>
> >> The benchmark uses memset() for initial population; subsequent
> >> iterations re-touch pages rather than repeatedly calling memset().
> >> The numbers below measure the advisor thread's madvise() syscall
> >> latency, not worker throughput.
> >>
> >> In our experimental KVM4 configuration with a VM-shared ASID and our
> >> separate guest INVLPGB passthrough changes, we observed:
> >>
> >>                          INVLPGB off     INVLPGB on    Difference
> >>     64 workers  mean       21.624 ms       12.505 ms     -42.17%
> >>                 P99        32.760 ms       21.518 ms     -34.32%
> >>     128 workers mean       29.553 ms       20.459 ms     -30.77%
> >>                 P99        39.628 ms       28.160 ms     -28.94%
> >>
> >> These are single, sequential A/B runs for each worker count, with
> >> 500 warmup and 2,000 measured calls per run. The process had 64
> >> allowed CPUs, so the 128-worker case was oversubscribed. The
> >> measurements show a useful latency difference, but do not by
> >> themselves establish INVLPGB instruction counts or IPI reductions.
> >>
> >> We then experimentally combined the October per-vCPU ASID changes for
> >> ordinary guests with our existing, out-of-tree L1 INVLPGB/TLBSYNC
> >> passthrough. With guest INVLPGB enabled, a one-vCPU guest boots, but
> >> otherwise comparable two-vCPU and 64-vCPU guests fail early during
> >> boot. Keeping QEMU's +invlpgb setting while adding clearcpuid=419
> >> to the same guest kernel allows the multi-vCPU guest to boot; the
> >> guest logs "Clearing CPUID bits: invlpgb".
>
> I imagine for INVLPGB to work correctly with per-vCPU ASIDs, you need
> to intercept INVLPGB and basically issue an INVLPGB for each ASID used
> by this VM (nr_vcpus or 2 * nr_vcpus with nested). I can't imagine
> this would perform well on large VMs, not sure if virtualizing INVLPGB
> would still make sense.
>
> Regarding the boot failure, I would guess there are some missing cases
> due to the complexity of emulating INVLPGB without a shared per-VM
> ASID.
>
> >> These controlled boot comparisons identify a problem with combining
> >> static per-vCPU ASIDs and direct guest INVLPGB/TLBSYNC passthrough:
> >> the multi-vCPU guest fails to boot when it uses INVLPGB, but boots
> >> when its INVLPGB CPUID bit is cleared. A one-vCPU guest also boots
> >> with INVLPGB enabled.
> >>
> >> In this configuration, an unintercepted guest INVLPGB targets the
> >> invoking vCPU's hardware ASID, while sibling vCPUs have distinct
> >> ASIDs.
>
> It's neat that INVLPGB will automatically use the current ASID and not
> ASID=0 for guest usage.

Hmm, actually, how is this supposed to work with nested
virtualization? The APM says:

    A guest that executes a legal INVLPGB that is not intercepted will have
    the requested ASID field replaced by the current ASID and the valid
    ASID bit set before doing the broadcast invalidation.

If any ASID passed by an L1 guest is replaced by its own ASID, then if
L1 is flushing L2's ASID (i.e. a non-zero ASID), passing through
INVLPGB would do the wrong thing and flush L1's ASID instead, right?

Seems like if the VM is using nested (on any vCPU), KVM must intercept
INVLPGB anyway to correctly handle the ASID?

The APM also says:

    Two forms of INVLPGB intercepts, conditional and unconditional, are
    available to the hypervisor. The unconditional intercept traps all guest
    usage of INVLPGB. The conditional intercept traps only
    illegally-specified INVLPGB instructions.

When I read "conditional", I was really hoping it was conditional on
the ASID being zero or non-zero, as KVM really only needs to intercept
INVLPGB with non-zero ASIDs (assuming the ASID is shared to begin
with).

Am I missing something?

Re: Some options abort RFC: KVM: SVM: Rework ASID management

[Yosry Ahmed]

On Thu, Oct 29, 2026 at 3:57 AM Jackie Yu <ufo19890607@gmail.com> wrote:
>
> Subject: Re: [PATCH v2 00/29] KVM: nSVM: Optimize nSVM TLB flushes
>
> Hi Yosry,
>
> Thank you for the quick response. Two clarifications from our side:
>
> 1. About the boot failures: no emulation is involved in our experiment. Our out-of-tree changes enable direct guest INVLPGB/TLBSYNC execution by clearing the KVM intercepts, without any software emulation of the instruction. The failures occur with this unintercepted passthrough combined with static per-vCPU ASIDs, so they are not missing cases in an emulation path. Under the per-vCPU design we did not attempt to intercept and emulate INVLPGB at all.

That's what I meant, the missing emulation of INVLPGB needed with
different per-vCPU ASIDs.

>
> 2. We do not use nested virtualization in our deployments. L1 guests in our production scenarios run without nested guests. Therefore, the ASID-replacement behavior you quoted from the APM does not affect our current use case: with one shared ASID per non-nested VM, an unintercepted guest INVLPGB would be scoped to that VM's ASID, which matches what our per-VM configuration relies on today.
>
> Regarding the direction: we understand that with per-vCPU ASIDs, correct handling would require intercepting INVLPGB and re-issuing a flush for every ASID in use by the VM, which you note is unlikely to perform well on large VMs. We also note your point that as long as any vCPU uses nested, KVM must intercept INVLPGB anyway because the hardware replaces the requested ASID with the current ASID.
>
> If we read the situation correctly, the practical options seem to be either (a) a per-VM shared ASID mode restricted to non-nested VMs, with unintercepted guest INVLPGB, plus mandatory interception whenever nested is in use, or (b) always intercepting and virtualizing INVLPGB under per-vCPU ASIDs. Either way, direct passthrough would at most cover the non-nested case. We would appreciate Sean's and Paolo's input on which direction upstream prefers; we are glad to provide more details of our configuration or run tests that would help decide between these options.

I don't think (b) makes a lot of sense, but if we do (a) we should
always use per-VM ASIDs, supporting both implementations based on
nested vs. non-nested would be a huge pain. I think what KVM can do is
disable INVLPGB intercepts until one vCPU sets EFER.SVME (or maybe on
first VMRUN, because with enable_virt_at_load KVM will always set
EFER.SVME on once it's loaded).

>
> Thanks,
>
> Jackie
>
>
> Yosry Ahmed <yosry@kernel.org> 于2026年10月29日周四 14:08写道：
>>
>> On Wed, Oct 28, 2026 at 1:15 PM Yosry Ahmed <yosry@kernel.org> wrote:
>> >
>> > On Wed, Oct 28, 2026 at 6:37 AM Jackie Yu <ufo19890607@gmail.com> wrote:
>> > >
>> > > Hi, guys!
>> > >
>> > > Jackie Yu <ufo19890607@gmail.com> 于2026年10月28日周三 21:36写道：
>> > >>
>> > >> Hi Yosry and KVM folks,
>> > >>
>> > >> We would appreciate your guidance on supporting INVLPGB in ordinary (non-SEV)
>> > >> L1 guests alongside the proposed static per-vCPU ASIDs.
>> > >>
>> > >> The 2025 ASID rework RFC proposed one static ASID per L1 VM and listed
>> > >> "enabling INVLPGB virtualization" among its motivations [1]. The June
>> > >> 2026 RFC v2 changed the design from per-VM to per-vCPU ASIDs [2]. In
>> > >> the October v2 series, patches 2-11 establish static per-vCPU ASIDs,
>> > >> while the remaining patches primarily address nested TLB flushes [3].
>> > >> Our question is how guest INVLPGB should be supported under the
>> > >> per-vCPU design.
>> > >>
>> > >> Some of our workloads frequently discard and repopulate memory using
>> > >> patterns involving madvise() and memset(). Broadcast IPI-based TLB
>> > >> shootdowns are expensive for these workloads, so we are interested in
>> > >> making AMD INVLPGB available to guests. We use a small benchmark to
>> > >> approximate the resulting shared-address-space pressure:
>> > >>
>> > >>     region = mmap(256 MiB, MAP_SHARED | MAP_ANONYMOUS);
>> > >>     memset(region, 0xab, 256 MiB);       // Initial population
>> > >>     start N workers, pinned across 64 allowed CPUs;
>> > >>     each worker repeatedly reads pages in its own slice of region;
>> > >>
>> > >>     repeat 500 warmup + 2,000 measured iterations:
>> > >>         start = clock_gettime(CLOCK_MONOTONIC);
>> > >>         madvise(region, 256 MiB, MADV_DONTNEED);
>> > >>         record elapsed madvise() time;
>> > >>         write one page in every 64 pages to repopulate part of region;
>> > >>
>> > >>     stop and join the workers;
>> > >>
>> > >> The benchmark uses memset() for initial population; subsequent
>> > >> iterations re-touch pages rather than repeatedly calling memset().
>> > >> The numbers below measure the advisor thread's madvise() syscall
>> > >> latency, not worker throughput.
>> > >>
>> > >> In our experimental KVM4 configuration with a VM-shared ASID and our
>> > >> separate guest INVLPGB passthrough changes, we observed:
>> > >>
>> > >>                          INVLPGB off     INVLPGB on    Difference
>> > >>     64 workers  mean       21.624 ms       12.505 ms     -42.17%
>> > >>                 P99        32.760 ms       21.518 ms     -34.32%
>> > >>     128 workers mean       29.553 ms       20.459 ms     -30.77%
>> > >>                 P99        39.628 ms       28.160 ms     -28.94%
>> > >>
>> > >> These are single, sequential A/B runs for each worker count, with
>> > >> 500 warmup and 2,000 measured calls per run. The process had 64
>> > >> allowed CPUs, so the 128-worker case was oversubscribed. The
>> > >> measurements show a useful latency difference, but do not by
>> > >> themselves establish INVLPGB instruction counts or IPI reductions.
>> > >>
>> > >> We then experimentally combined the October per-vCPU ASID changes for
>> > >> ordinary guests with our existing, out-of-tree L1 INVLPGB/TLBSYNC
>> > >> passthrough. With guest INVLPGB enabled, a one-vCPU guest boots, but
>> > >> otherwise comparable two-vCPU and 64-vCPU guests fail early during
>> > >> boot. Keeping QEMU's +invlpgb setting while adding clearcpuid=419
>> > >> to the same guest kernel allows the multi-vCPU guest to boot; the
>> > >> guest logs "Clearing CPUID bits: invlpgb".
>> >
>> > I imagine for INVLPGB to work correctly with per-vCPU ASIDs, you need
>> > to intercept INVLPGB and basically issue an INVLPGB for each ASID used
>> > by this VM (nr_vcpus or 2 * nr_vcpus with nested). I can't imagine
>> > this would perform well on large VMs, not sure if virtualizing INVLPGB
>> > would still make sense.
>> >
>> > Regarding the boot failure, I would guess there are some missing cases
>> > due to the complexity of emulating INVLPGB without a shared per-VM
>> > ASID.
>> >
>> > >> These controlled boot comparisons identify a problem with combining
>> > >> static per-vCPU ASIDs and direct guest INVLPGB/TLBSYNC passthrough:
>> > >> the multi-vCPU guest fails to boot when it uses INVLPGB, but boots
>> > >> when its INVLPGB CPUID bit is cleared. A one-vCPU guest also boots
>> > >> with INVLPGB enabled.
>> > >>
>> > >> In this configuration, an unintercepted guest INVLPGB targets the
>> > >> invoking vCPU's hardware ASID, while sibling vCPUs have distinct
>> > >> ASIDs.
>> >
>> > It's neat that INVLPGB will automatically use the current ASID and not
>> > ASID=0 for guest usage.
>>
>> Hmm, actually, how is this supposed to work with nested
>> virtualization? The APM says:
>>
>>     A guest that executes a legal INVLPGB that is not intercepted will have
>>     the requested ASID field replaced by the current ASID and the valid
>>     ASID bit set before doing the broadcast invalidation.
>>
>> If any ASID passed by an L1 guest is replaced by its own ASID, then if
>> L1 is flushing L2's ASID (i.e. a non-zero ASID), passing through
>> INVLPGB would do the wrong thing and flush L1's ASID instead, right?
>>
>> Seems like if the VM is using nested (on any vCPU), KVM must intercept
>> INVLPGB anyway to correctly handle the ASID?
>>
>> The APM also says:
>>
>>     Two forms of INVLPGB intercepts, conditional and unconditional, are
>>     available to the hypervisor. The unconditional intercept traps all guest
>>     usage of INVLPGB. The conditional intercept traps only
>>     illegally-specified INVLPGB instructions.
>>
>> When I read "conditional", I was really hoping it was conditional on
>> the ASID being zero or non-zero, as KVM really only needs to intercept
>> INVLPGB with non-zero ASIDs (assuming the ASID is shared to begin
>> with).
>>
>> Am I missing something?

Re: Some options abort RFC: KVM: SVM: Rework ASID management

[Sean Christopherson]

Please send mails as plain text, otherwise they get ignored by the mailing lists
and don't get archived by lore.

On Thu, Oct 29, 2026, Jackie Yu wrote:
> Subject: Re: [PATCH v2 00/29] KVM: nSVM: Optimize nSVM TLB flushes
> 
> Hi Yosry,
> 
> Thank you for the quick response. Two clarifications from our side:
> 
> 1. About the boot failures: no emulation is involved in our
> experiment. Our out-of-tree changes enable direct guest
> INVLPGB/TLBSYNC execution by clearing the KVM intercepts, without any
> software emulation of the instruction. The failures occur with this
> unintercepted passthrough combined with static per-vCPU ASIDs, so they
> are not missing cases in an emulation path. Under the per-vCPU design
> we did not attempt to intercept and emulate INVLPGB at all.
> 
> 2. We do not use nested virtualization in our deployments. L1 guests
> in our production scenarios run without nested guests. Therefore, the
> ASID-replacement behavior you quoted from the APM does not affect our
> current use case: with one shared ASID per non-nested VM, an
> unintercepted guest INVLPGB would be scoped to that VM's ASID, which
> matches what our per-VM configuration relies on today.
> 
> Regarding the direction: we understand that with per-vCPU ASIDs,
> correct handling would require intercepting INVLPGB and re-issuing a
> flush for every ASID in use by the VM, which you note is unlikely to
> perform well on large VMs. We also note your point that as long as any
> vCPU uses nested, KVM must intercept INVLPGB anyway because the
> hardware replaces the requested ASID with the current ASID.
> 
> If we read the situation correctly, the practical options seem to be
> either (a) a per-VM shared ASID mode restricted to non-nested VMs,
> with unintercepted guest INVLPGB, plus mandatory interception whenever
> nested is in use, or (b) always intercepting and virtualizing INVLPGB
> under per-vCPU ASIDs. Either way, direct passthrough would at most
> cover the non-nested case. We would appreciate Sean's and Paolo's
> input on which direction upstream prefers; we are glad to provide more
> details of our configuration or run tests that would help decide
> between these options.
> 
> Thanks,
> 
> Jackie
> 
> 
> Yosry Ahmed <yosry@kernel.org> 于2026年10月29日周四 14:08写道：
> 
> > On Wed, Oct 28, 2026 at 1:15 PM Yosry Ahmed <yosry@kernel.org> wrote:
> > >
> > > On Wed, Oct 28, 2026 at 6:37 AM Jackie Yu <ufo19890607@gmail.com> wrote:
> > > >
> > > > Hi, guys!
> > > >
> > > > Jackie Yu <ufo19890607@gmail.com> 于2026年10月28日周三 21:36写道：
> > > >>
> > > >> Hi Yosry and KVM folks,
> > > >>
> > > >> We would appreciate your guidance on supporting INVLPGB in ordinary
> > (non-SEV)
> > > >> L1 guests alongside the proposed static per-vCPU ASIDs.
> > > >>
> > > >> The 2025 ASID rework RFC proposed one static ASID per L1 VM and listed
> > > >> "enabling INVLPGB virtualization" among its motivations [1]. The June
> > > >> 2026 RFC v2 changed the design from per-VM to per-vCPU ASIDs [2]. In
> > > >> the October v2 series, patches 2-11 establish static per-vCPU ASIDs,
> > > >> while the remaining patches primarily address nested TLB flushes [3].
> > > >> Our question is how guest INVLPGB should be supported under the
> > > >> per-vCPU design.
> > > >>
> > > >> Some of our workloads frequently discard and repopulate memory using
> > > >> patterns involving madvise() and memset(). Broadcast IPI-based TLB
> > > >> shootdowns are expensive for these workloads, so we are interested in
> > > >> making AMD INVLPGB available to guests. We use a small benchmark to
> > > >> approximate the resulting shared-address-space pressure:
> > > >>
> > > >>     region = mmap(256 MiB, MAP_SHARED | MAP_ANONYMOUS);
> > > >>     memset(region, 0xab, 256 MiB);       // Initial population
> > > >>     start N workers, pinned across 64 allowed CPUs;
> > > >>     each worker repeatedly reads pages in its own slice of region;
> > > >>
> > > >>     repeat 500 warmup + 2,000 measured iterations:
> > > >>         start = clock_gettime(CLOCK_MONOTONIC);
> > > >>         madvise(region, 256 MiB, MADV_DONTNEED);
> > > >>         record elapsed madvise() time;
> > > >>         write one page in every 64 pages to repopulate part of region;
> > > >>
> > > >>     stop and join the workers;
> > > >>
> > > >> The benchmark uses memset() for initial population; subsequent
> > > >> iterations re-touch pages rather than repeatedly calling memset().
> > > >> The numbers below measure the advisor thread's madvise() syscall
> > > >> latency, not worker throughput.
> > > >>
> > > >> In our experimental KVM4 configuration with a VM-shared ASID and our
> > > >> separate guest INVLPGB passthrough changes, we observed:
> > > >>
> > > >>                          INVLPGB off     INVLPGB on    Difference
> > > >>     64 workers  mean       21.624 ms       12.505 ms     -42.17%
> > > >>                 P99        32.760 ms       21.518 ms     -34.32%
> > > >>     128 workers mean       29.553 ms       20.459 ms     -30.77%
> > > >>                 P99        39.628 ms       28.160 ms     -28.94%
> > > >>
> > > >> These are single, sequential A/B runs for each worker count, with
> > > >> 500 warmup and 2,000 measured calls per run. The process had 64
> > > >> allowed CPUs, so the 128-worker case was oversubscribed. The
> > > >> measurements show a useful latency difference, but do not by
> > > >> themselves establish INVLPGB instruction counts or IPI reductions.
> > > >>
> > > >> We then experimentally combined the October per-vCPU ASID changes for
> > > >> ordinary guests with our existing, out-of-tree L1 INVLPGB/TLBSYNC
> > > >> passthrough. With guest INVLPGB enabled, a one-vCPU guest boots, but
> > > >> otherwise comparable two-vCPU and 64-vCPU guests fail early during
> > > >> boot. Keeping QEMU's +invlpgb setting while adding clearcpuid=419
> > > >> to the same guest kernel allows the multi-vCPU guest to boot; the
> > > >> guest logs "Clearing CPUID bits: invlpgb".
> > >
> > > I imagine for INVLPGB to work correctly with per-vCPU ASIDs, you need
> > > to intercept INVLPGB and basically issue an INVLPGB for each ASID used
> > > by this VM (nr_vcpus or 2 * nr_vcpus with nested). I can't imagine
> > > this would perform well on large VMs, not sure if virtualizing INVLPGB
> > > would still make sense.
> > >
> > > Regarding the boot failure, I would guess there are some missing cases
> > > due to the complexity of emulating INVLPGB without a shared per-VM
> > > ASID.
> > >
> > > >> These controlled boot comparisons identify a problem with combining
> > > >> static per-vCPU ASIDs and direct guest INVLPGB/TLBSYNC passthrough:
> > > >> the multi-vCPU guest fails to boot when it uses INVLPGB, but boots
> > > >> when its INVLPGB CPUID bit is cleared. A one-vCPU guest also boots
> > > >> with INVLPGB enabled.
> > > >>
> > > >> In this configuration, an unintercepted guest INVLPGB targets the
> > > >> invoking vCPU's hardware ASID, while sibling vCPUs have distinct
> > > >> ASIDs.
> > >
> > > It's neat that INVLPGB will automatically use the current ASID and not
> > > ASID=0 for guest usage.
> >
> > Hmm, actually, how is this supposed to work with nested
> > virtualization? The APM says:
> >
> >     A guest that executes a legal INVLPGB that is not intercepted will have
> >     the requested ASID field replaced by the current ASID and the valid
> >     ASID bit set before doing the broadcast invalidation.
> >
> > If any ASID passed by an L1 guest is replaced by its own ASID, then if
> > L1 is flushing L2's ASID (i.e. a non-zero ASID), passing through
> > INVLPGB would do the wrong thing and flush L1's ASID instead, right?
> >
> > Seems like if the VM is using nested (on any vCPU), KVM must intercept
> > INVLPGB anyway to correctly handle the ASID?
> >
> > The APM also says:
> >
> >     Two forms of INVLPGB intercepts, conditional and unconditional, are
> >     available to the hypervisor. The unconditional intercept traps all
> > guest
> >     usage of INVLPGB. The conditional intercept traps only
> >     illegally-specified INVLPGB instructions.
> >
> > When I read "conditional", I was really hoping it was conditional on
> > the ASID being zero or non-zero, as KVM really only needs to intercept
> > INVLPGB with non-zero ASIDs (assuming the ASID is shared to begin
> > with).
> >
> > Am I missing something?
> >