Re: [RFC PATCH 00/18] mm: arm64: Add kernel replication feature

[Yang Shi <yang@os.amperecomputing.com>]

On 8/27/26 9:11 AM, Nikita Panov wrote:
> Current status:
>
> There were several prior submissions with some sort of replication
> for NUMA systems, including one from our side for the x86_64 platform.
> In the last couple years, several research articles related to solving
> locality issues on NUMA machines through replication emerged as well.

Hi Nikita,

Thank you for the effort. As Christopher said we are working on the same 
feature. Please see the below inline comments.

>
> [1]  - arm64 kernel text replication
> [2]  - x86 NUMA-aware kernel replication
> [3]  - x86 kernel text replication
> [4]  - NUMA replication of user data
> [5]  - Mitosis: Transparently Self-Replicating Page-Tables for Large-Memory Machines
> [6]  - WASP: Workload-Aware Self-Replicating Page-Tables for NUMA Servers
> [7]  - PaCaR: Improved Buffered I/O Locality on NUMA Systems with Page Cache Replication
> [8]  - Memory page replication for Linux on X86 processors
> [12] - Optimize this_cpu_*() ops for non-x86 (ARM64 for this series)
>
> As of today, none of it was merged into mainline.
>
> Patch set description:
>
> This patchset implements initial support of the kernel
> text and ro-data replication for arm64 platform.
> Linux-next was used as a baseline.
> (903c1cf6dff9 Add linux-next specific files for 20260821).
> This patch set heavily relies on our previous submission [2],
> their generic part is the same, these solutions might be merged into a single one.
> However, after thorough re-evaluation, we were not able to observe
> performance improvement for the x86 platform, so we have decided to stop this
> direction and switch on arm64.
>
> Current implementation supports the next functionality:
>
> 1. Replicated kernel text and rodata per NUMA node
> 2. Vmalloc is able to work with replicated areas, so
>     kernel modules text and rodata are also replicated during
>     modules loading stage.
> 3. KPROBES, KGDB and all functionality that depends on
>     kernel text patching work without any limitation.
> 4. KPTI, KASLR, and KASAN are supported.
> 5. Only part of a translation table related to
>     replicated text and rodata is replicated
>     (up to 2 pgd entries if kaslr is enabled)
> 6. 4K + 48bit and 64K + 48/52bit are supported and tested.

I think 16K + 47 VA bits should work too. I guess it is because there 
are just 2 top level entries with 16K + 48 VA bits. That requires some 
extra effort to sync up kernel page tables.

> 7. New cmdline option: "kernel_replication=on|off",
>     to configure on the boot stage
> 8. For verification check dmesg output or /sys/kernel/debug/numa_replication/
>
> In general, setting up a new pgd entry in kernel translation table
> not often scenario, most of them are covered by current patch set.
>
> TT overview:
>                     NODE 0                   NODE 1
>                     KERNEL                   KERNEL
>             ---------------------    ---------------------
>       PGD   |@| | |@|   | | | |*|    |@| | |@|   | | | |*|
>             ---------------------    ---------------------
>                                |                        |
>              -------------------      -------------------
>              |                        |
>             ---------------------    ---------------------
>       PUD   | | | | |   |@| |*|*|    | | | | |   |@| |*|*|
>             ---------------------    ---------------------
>                                |                        |
>              -------------------      -------------------
>              |                        |
>             ---------------------    ---------------------
>       PMD   |READ-ONLY|MUTABLE  |    |READ-ONLY|MUTABLE  |
>             ---------------------    ---------------------
>                    |       |                  |     |
>                    |       --------------------------
>                    |               |          |
>                  --------       -------      --------
>     PHYS         |      |       |     |      |      |
>      MEM         --------       -------      --------
>                  <------>                    <------>
>                   NODE 0        Shared        NODE 1
>                                 between
>                                 nodes
> * - entries unique in each table
> @ - same entries accross replicated tables
>
> Since for kernel space and user space different tables are used,
> user space tables are not replicated at all, so synchronization
> is not required.
>
> Known problems:
>
> 1. Other combinations of base page size and va size (especially with 16K pages)
>     should be adapted and verified.
> 2. Replicated translation tables for the vmalloc region are not local right now.
>     Allocation performed with default memory policy, so translation tables
>     for kernel modules will not be local. However,
>     replicated text and rodata of the modules are local.
>     In general, vmalloc patch should be cleaned up.
> 3. Any modifications of kernel PGD level. These modifications
>     should be synchronized across all replicated tables.
>     Right now, for example, memory hotplug/hotunplug
>     lacks this support, vmemmap and kasan regions for
>     added memory might not be observed correctly. This could be fixed
>     by patching all places in the kernel where swapper_pg_dir
>     is modified, or by "lazy" propagation on kernel faults in the pgd-level.
>     Propagation approach will not help in the case of pgd_clear()
>     on swapper_pg_dir though.

My percpu patchset (I saw you mentioned it above) already had these 
problems solved. I have not looked into too much detail yet, but it 
seems like you have replicate kernel page tables per node. My patchset 
added percpu kernel page tables. It should be able to support kernel 
text replication as well without too much extra effort. And it can 
support multiple usecases, for example, this_cpu optimization 
implemented in my series, kernel text replication and some potential 
security features. Multiple usecases should be able to make it more 
attractive and convincing. So as Christopher suggested it may be better 
to combine the effort.

>
> Overall, this patch set in an early PoC stage and require some improvements.
>
> Overhead:
>
> Memory overhead for the kernel itself is about 30MB per NUMA node
>    on our deployment. For kernel modules - depends on their sizes, but text
>    and ro-data are not that big.
> CPU overhead - replication performed on the boot stage. After boot
>    only "rare" operations are slowed down -
>    module loading, text patching, kernel table pgd-level modifications.
>
> Performance evaluation:
>
> Our local testing was performed on
> Kunpeng 920, 128 CPU, 4 nodes, 100Gb for each node.
>
> Microbenchmark:
> Kernel module with a huge text section (~50MB) filled with CPU-bound
> instructions. For each NUMA node thread is spawned, each thread in a loop
> executes isntructions. Total execution time of each thread is measured.
> The insmod call bound to node 0 through numactl (less time is better).
>
> node                 0        1        2        3
> Before time, s     5.567    7.598   13.294   18.905
> After  time, s     5.469    6.960    6.777    5.531
>
> Diff               ~0%      -8.5%    -49%     -70%
> In this benchmark, interconnect was not used by any other actors,
> so microbenchmark numbers might be significantly improved.
>
> Customer's evaluation:
> We were provided with the following feedback on this patch set
> directly from our customers. Unfortunately, we do not have details
> regarding how these measurements were done other than it was
> a production setup.
> Evaulation was performed on Kunpeng 920 and 920B platforms:
> CEPH distributed storage +5%
> StarRocksDB              +5%
>
> Couple more words about patch set and technology:
>
> This patchset was merged into the innovative branch of
> the openEuler distributive 1.5 year ago (openEuler-25.03)
> and was actively tested in production environment [9], [10].
> In addition, besides the kernel part, we have published
> user space replication (for translation tables and rodata) as well,
> but it is very complex and experimental
> even compared to this patch set [11]. With replication in user
> space, we were able to achieve the following numbers in
> performance improvement:
> MySQL + sysbench      1-6%
> Spark TPC-H           4-20%
> Phoronix test-suite   0-25%

Thank you for sharing the performance data. Does the SUT with 4 nodes 
have 4 real sockets? Nowadays the CPU design is moving to multiple 
chiplets. Subnuma configuration may be more and more popular in the 
future, so we thought kernel text replication can help performance for 
more usecases other than multi sockets machines.

Thanks,
Yang

>
> Discussion:
>
> The main question we'd like to discuss is the following:
> Should the kernel replication feature be merged into the Linux
> somewhere in the future? In any form, not specifically this patch set,
> but the core concept itself.
>
> If the answer is yes, please share your thoughts on this patch set. What else
> should be fixed (or reimplemented and redsigned completly) in this patch
> for mainline in your opinion? We'd be glad to do it, and in that case
> I'll send an updated version in the near future.
>
> [1]  https://lwn.net/ml/linux-doc/ZHYCUVa8fzmB4XZV@shell.armlinux.org.uk/
> [2]  https://lwn.net/ml/linux-mm/20231228131056.602411-1-artem.kuzin@huawei.com/
> [3]  https://lwn.net/Articles/36602/
> [4]  https://lwn.net/Articles/45082/
> [5]  https://github.com/mitosis-project/mitosis-asplos20-artifact
> [6]  https://dl.acm.org/doi/10.1145/3620665.3640369
> [7]  https://dl.acm.org/doi/10.1145/3767295.3769359
> [8]  https://github.com/Carrefour/linux-replication
> [9]  https://mailweb.openeuler.org/archives/list/kernel@openeuler.org/message/C7M5E2K2UD7FV7XYPWPZREJBCOCBIVRN/
> [10] https://www.openeuler.org/whitepaper/en/openEuler%2025.03%20Technical%20White%20Paper.pdf
> [11] https://mailweb.openeuler.org/archives/list/kernel@openeuler.org/message/73MAUDM6WCGSSOKPZGPNZAYRNQGUR6DE/
> [12] https://lore.kernel.org/linux-mm/20260715180455.515692-1-yang@os.amperecomputing.com/
>
> Nikita Panov (18):
>    mm: arm64 add Kconfig option for kernel replication
>    arm64: align kernel text and rodata
>    mm: allow per-NUMA node local P4D/PUD/PMD/PTE allocation
>    arm64: add arch callbacks for kernel replication
>    mm: per-NUMA node replication core infrastructure
>    mm: add support of memory protection for NUMA replicas
>    arm64: add support of memory protection for NUMA replicas
>    mm: set memory permissions for BPF handlers replicas
>    mm: add replicas allocation support for vmalloc
>    arm64: enable per-NUMA node kernel text and rodata replication
>    mm: enable per-NUMA node kernel text and rodata replication
>    arm64: make power management aware about kernel replication
>    arm64: make kernel text patching aware about replicas
>    arm64: add correct alignment to kimage in efi code
>    arm64: add support of NUMA replication for ptdump
>    arm64: add kernel modules text and rodata replication support
>    mm: init kernel modules with replication support
>    mm: introduce kernel cmdline option "kernel_replication="
>
>   .../admin-guide/kernel-parameters.txt         |   7 +
>   arch/arm64/include/asm/efi.h                  |  18 +-
>   arch/arm64/include/asm/mmu_context.h          |   4 +
>   arch/arm64/include/asm/numa_replication.h     |  54 ++
>   arch/arm64/include/asm/pgtable.h              |   4 +
>   arch/arm64/kernel/alternative.c               |  33 +-
>   arch/arm64/kernel/hibernate.c                 |   5 +
>   arch/arm64/kernel/module.c                    |  11 +
>   arch/arm64/kernel/patching.c                  |  96 ++
>   arch/arm64/kernel/sleep.S                     |   8 +
>   arch/arm64/kernel/smp.c                       |   8 +
>   arch/arm64/kernel/vmlinux.lds.S               |  22 +
>   arch/arm64/mm/init.c                          |  49 ++
>   arch/arm64/mm/kasan_init.c                    |   2 +
>   arch/arm64/mm/mmu.c                           |  42 +-
>   arch/arm64/mm/pageattr.c                      |  72 +-
>   arch/arm64/mm/ptdump.c                        |  24 +-
>   arch/arm64/net/bpf_jit_comp.c                 |   4 +-
>   include/asm-generic/pgalloc.h                 |  90 ++
>   include/asm-generic/pgtable-nop4d.h           |   5 +
>   include/asm-generic/pgtable-nopmd.h           |   5 +
>   include/asm-generic/pgtable-nopud.h           |   5 +
>   include/asm-generic/set_memory.h              |  14 +
>   include/linux/mm.h                            |  92 +-
>   include/linux/mm_types.h                      |   3 +
>   include/linux/moduleloader.h                  |   4 +
>   include/linux/numa_kernel_replication.h       | 112 +++
>   include/linux/set_memory.h                    |  21 +
>   include/linux/vmalloc.h                       |  18 +
>   init/main.c                                   |  17 +
>   kernel/bpf/core.c                             |   4 +-
>   kernel/bpf/trampoline.c                       |   2 +-
>   kernel/module/main.c                          |  20 +
>   kernel/module/strict_rwx.c                    |  12 +-
>   mm/Kconfig                                    |  10 +
>   mm/Makefile                                   |   2 +
>   mm/execmem.c                                  |  39 +-
>   mm/memory.c                                   | 129 +++
>   mm/mm_init.c                                  |   3 +
>   mm/numa_kernel_replication.c                  | 821 ++++++++++++++++++
>   mm/vmalloc.c                                  | 454 ++++++++--
>   41 files changed, 2246 insertions(+), 99 deletions(-)
>   create mode 100644 arch/arm64/include/asm/numa_replication.h
>   create mode 100644 include/linux/numa_kernel_replication.h
>   create mode 100644 mm/numa_kernel_replication.c
>
> --
> 2.34.1
>
>