Re: [RFC 00/12] arm64: Add support for TLBI domains

Re: [RFC 00/12] arm64: Add support for TLBI domains

[Zi Yan]

On Thu Oct 1, 2026 at 7:06 AM EDT, Kristina Martšenko wrote:
> Hi,
>
> This series allows the kernel to use TLBI domains. TLBI domains (FEAT_TLBID) is
> an Arm architecture feature that enables a TLB invalidation to target a subset
> of CPUs instead of the whole system. It can improve performance on systems with
> many CPUs. TLBID is not yet part of the Arm Architecture Reference Manual but
> some of it is described in the latest XML [1].
>
> The goal of this series is to send TLBI commands only to the CPUs where a
> process has run (i.e. where its page tables have been installed). The last

IIUC, mm_cpumask() is used to store this information on x86 and other
archs. I see you put this information in tlbid_domains of mm_context_t
from patch 9, although in the form of tlb id domains. I wonder why not
make arm64 also update mm_cpumask() like x86 and other archs and derive
the tlbid domains based on mm_cpumask() and a cpu-to-domain mapping
(added in patch 7).

Thanks.

> patch explains a bit about how it works. This is an RFC since I'm not familiar
> with the kernel's TLB/mm code and I'd like feedback on whether the general
> approach is correct.
>
> A few notes and open questions:
>
>   * Domains are described in a new "TLBI" ACPI table. This is not yet part of
>     the ACPI specification. There was an initial proposal [2] but this series
>     uses a newer WIP version of the table that aims to be published by the end
>     of the year.
>
>   * There is currently no device tree binding since the feature is mainly meant
>     for server-class systems.
>
>   * It's also possible to use TLBI domains to limit a KVM guest's TLB
>     invalidations to CPUs where the guest has run. I'll try to include that in
>     the next version. This series only disables domains for guests.
>
>   * With this series a long-running process may eventually run on all CPUs and
>     just issue broadcast TLBIs. It may be worth periodically reducing the set
>     of CPUs but it's not clear when.
>
>   * There are currently no performance numbers since there is no hardware with
>     this feature yet.
>
>   * The series has been tested on an Arm FVP which supports the TLBID
>     architecture but does not have any domains (except domain 0).
>
> Any feedback is welcome.
>
> Thanks,
> Kristina
>
> [1] https://support.arm.com/architectures/a-profile-architecture#Downloads
> [2] https://github.com/tianocore/edk2/issues/11999
>
>
> Kristina Martšenko (12):
>   arm64: sysreg: Add definitions for FEAT_TLBID
>   arm64: Detect FEAT_TLBID
>   KVM: arm64: Hide TLBID from guest sysregs
>   KVM: arm64: Hide TLBID from guest instructions
>   ACPICA: Add TLBI table definition
>   ACPI: TLBI: Parse domains from table
>   arm64: tlbid: Set up CPU domain bitmaps
>   efi/arm: Check return value of init_new_context()
>   arm64: tlbid: Track the TLBI domain of a task
>   arm64: Support TLBIP instructions
>   arm64: tlbid: Pass domain to TLBI instructions
>   arm64: tlbid: Add documentation
>
>  Documentation/arch/arm64/booting.rst    |  10 ++
>  arch/arm64/Kconfig                      |  12 ++
>  arch/arm64/include/asm/acpi.h           |  10 ++
>  arch/arm64/include/asm/cpu.h            |   1 +
>  arch/arm64/include/asm/cpufeature.h     |  15 ++
>  arch/arm64/include/asm/el2_setup.h      |  76 +++++++-
>  arch/arm64/include/asm/kvm_emulate.h    |   5 +
>  arch/arm64/include/asm/mmu.h            |  13 ++
>  arch/arm64/include/asm/mmu_context.h    |   9 +-
>  arch/arm64/include/asm/tlbflush.h       | 219 +++++++++++++++++-------
>  arch/arm64/kernel/cpufeature.c          |  25 +++
>  arch/arm64/kernel/cpuinfo.c             |   3 +
>  arch/arm64/kvm/config.c                 |   6 +
>  arch/arm64/kvm/emulate-nested.c         |   1 +
>  arch/arm64/kvm/hyp/include/hyp/switch.h |   7 +-
>  arch/arm64/kvm/hyp/nvhe/tlb.c           |   2 +-
>  arch/arm64/kvm/hyp/vhe/tlb.c            |   4 +-
>  arch/arm64/kvm/sys_regs.c               |   3 +
>  arch/arm64/mm/Makefile                  |   1 +
>  arch/arm64/mm/context.c                 |   6 +-
>  arch/arm64/mm/tlbid.c                   | 210 +++++++++++++++++++++++
>  arch/arm64/tools/cpucaps                |   1 +
>  arch/arm64/tools/sysreg                 |  53 +++++-
>  drivers/acpi/arm64/Kconfig              |   3 +
>  drivers/acpi/arm64/Makefile             |   1 +
>  drivers/acpi/arm64/init.c               |   2 +
>  drivers/acpi/arm64/init.h               |   1 +
>  drivers/acpi/arm64/tlbi.c               | 196 +++++++++++++++++++++
>  drivers/firmware/efi/arm-runtime.c      |   3 +-
>  include/acpi/actbl2.h                   |  32 ++++
>  30 files changed, 858 insertions(+), 72 deletions(-)
>  create mode 100644 arch/arm64/mm/tlbid.c
>  create mode 100644 drivers/acpi/arm64/tlbi.c
>
>
> base-commit: 93f51579e7df248780214094418f205253383cc5




-- 
Best Regards,
Yan, Zi

Re: [RFC 00/12] arm64: Add support for TLBI domains

[Will Deacon]

On Sun, Oct 04, 2026 at 02:06:48PM -0400, Zi Yan wrote:
> On Thu Oct 1, 2026 at 7:06 AM EDT, Kristina Martšenko wrote:
> > This series allows the kernel to use TLBI domains. TLBI domains (FEAT_TLBID) is
> > an Arm architecture feature that enables a TLB invalidation to target a subset
> > of CPUs instead of the whole system. It can improve performance on systems with
> > many CPUs. TLBID is not yet part of the Arm Architecture Reference Manual but
> > some of it is described in the latest XML [1].
> >
> > The goal of this series is to send TLBI commands only to the CPUs where a
> > process has run (i.e. where its page tables have been installed). The last
> 
> IIUC, mm_cpumask() is used to store this information on x86 and other
> archs. I see you put this information in tlbid_domains of mm_context_t
> from patch 9, although in the form of tlb id domains. I wonder why not
> make arm64 also update mm_cpumask() like x86 and other archs and derive
> the tlbid domains based on mm_cpumask() and a cpu-to-domain mapping
> (added in patch 7).

Historically, maintaining mm_cpumask() was a horrible source of contention
when we looked at it in the past. Maybe that's no longer true, but it
would be good to see some numbers if we want to start keeping that thing
accurate on arm64.

Will

Re: [RFC 00/12] arm64: Add support for TLBI domains

[Catalin Marinas]

On Sun, Oct 04, 2026 at 02:06:48PM -0400, Zi Yan wrote:
> On Thu Oct 1, 2026 at 7:06 AM EDT, Kristina Martšenko wrote:
> > This series allows the kernel to use TLBI domains. TLBI domains (FEAT_TLBID) is
> > an Arm architecture feature that enables a TLB invalidation to target a subset
> > of CPUs instead of the whole system. It can improve performance on systems with
> > many CPUs. TLBID is not yet part of the Arm Architecture Reference Manual but
> > some of it is described in the latest XML [1].
> >
> > The goal of this series is to send TLBI commands only to the CPUs where a
> > process has run (i.e. where its page tables have been installed). The last
> 
> IIUC, mm_cpumask() is used to store this information on x86 and other
> archs. I see you put this information in tlbid_domains of mm_context_t
> from patch 9, although in the form of tlb id domains. I wonder why not
> make arm64 also update mm_cpumask() like x86 and other archs and derive
> the tlbid domains based on mm_cpumask() and a cpu-to-domain mapping
> (added in patch 7).

One reason is that it's cheaper to compute the tlbi domains at context
switch time (bitmap-and IIRC) than working from mm_cpumask() every time
we do a TLBI.

That said, suggestions for a quick algorithm are welcome. Subject to the
race conditions Will mentioned, we could use the mm_cpumask() for a
quick check if we ever ran on that CPU and avoid recomputing the tlbi
domains.

-- 
Catalin

Re: [RFC 00/12] arm64: Add support for TLBI domains

[Zi Yan]

On 5 Oct 2026, at 2:29, Will Deacon wrote:

> On Sun, Oct 04, 2026 at 02:06:48PM -0400, Zi Yan wrote:
>> On Thu Oct 1, 2026 at 7:06 AM EDT, Kristina Martšenko wrote:
>>> This series allows the kernel to use TLBI domains. TLBI domains (FEAT_TLBID) is
>>> an Arm architecture feature that enables a TLB invalidation to target a subset
>>> of CPUs instead of the whole system. It can improve performance on systems with
>>> many CPUs. TLBID is not yet part of the Arm Architecture Reference Manual but
>>> some of it is described in the latest XML [1].
>>>
>>> The goal of this series is to send TLBI commands only to the CPUs where a
>>> process has run (i.e. where its page tables have been installed). The last
>>
>> IIUC, mm_cpumask() is used to store this information on x86 and other
>> archs. I see you put this information in tlbid_domains of mm_context_t
>> from patch 9, although in the form of tlb id domains. I wonder why not
>> make arm64 also update mm_cpumask() like x86 and other archs and derive
>> the tlbid domains based on mm_cpumask() and a cpu-to-domain mapping
>> (added in patch 7).
>
> Historically, maintaining mm_cpumask() was a horrible source of contention
> when we looked at it in the past. Maybe that's no longer true, but it

Do you have a pointer to the cost you have in mind? LLM can only finds some
overhead reduction work done by Rik for x86[1][2].

[1] https://lore.kernel.org/all/20180716190337.26133-8-riel@surriel.com/T/#u
[2] https://lore.kernel.org/all/20241114152723.1294686-2-riel@surriel.com/

BTW, patch 9 is updating tlbid domains unconditionally. It can be optimized
in the same way as Rik did for x86 mm_cpumask(), read the tlbid domain first
and only update if CPU is in a new domain.

> would be good to see some numbers if we want to start keeping that thing
> accurate on arm64.


Best Regards,
Yan, Zi