* [RFC PATCH 01/18] mm: arm64 add Kconfig option for kernel replication
2026-08-27 16:11 [RFC PATCH 00/18] mm: arm64: Add kernel replication feature Nikita Panov
@ 2026-08-27 16:11 ` Nikita Panov
2026-08-27 16:11 ` [RFC PATCH 02/18] arm64: align kernel text and rodata Nikita Panov
` (18 subsequent siblings)
19 siblings, 0 replies; 23+ messages in thread
From: Nikita Panov @ 2026-08-27 16:11 UTC (permalink / raw)
To: catalin.marinas, akpm, david, ljs, vbabka, cl, linux, will,
mark.rutland, liam, rppt, surenb, mhocko
Cc: linux-mm, linux-kernel, linux-arm-kernel, wangkefeng.wang,
artem.kuzin, panov.nikita
Acked-by: Artem Kuzin <artem.kuzin@huawei.com>
Acked-by: Alexander Grubnikov <alexander.grubnikov@huawei.com>
Acked-by: Ilya Hanov <ilya.hanov@huawei-partners.com>
Acked-by: Denis Darvish <darvish.denis@huawei.com>
Signed-off-by: Nikita Panov <panov.nikita@huawei.com>
---
mm/Kconfig | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/mm/Kconfig b/mm/Kconfig
index 604c58199acb..1302268b72b6 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -1446,6 +1446,16 @@ config LOCK_MM_AND_FIND_VMA
bool
depends on !STACK_GROWSUP
+config KERNEL_REPLICATION
+ bool "Enable kernel text and ro-data replication across NUMA nodes"
+ default n
+ depends on ARM64 && MMU && NUMA && !MAXSMP
+
+ help
+ Creates per-NUMA node replicas of kernel text and ro-data sections.
+ Page tables are replicated partially, according to replicated kernel memory range.
+ If unsure, say "n".
+
config IOMMU_MM_DATA
bool
--
2.34.1
^ permalink raw reply related [flat|nested] 23+ messages in thread* [RFC PATCH 02/18] arm64: align kernel text and rodata
2026-08-27 16:11 [RFC PATCH 00/18] mm: arm64: Add kernel replication feature Nikita Panov
2026-08-27 16:11 ` [RFC PATCH 01/18] mm: arm64 add Kconfig option for kernel replication Nikita Panov
@ 2026-08-27 16:11 ` Nikita Panov
2026-08-27 17:35 ` Lorenzo Stoakes (ARM)
2026-08-27 16:11 ` [RFC PATCH 03/18] mm: allow per-NUMA node local P4D/PUD/PMD/PTE allocation Nikita Panov
` (17 subsequent siblings)
19 siblings, 1 reply; 23+ messages in thread
From: Nikita Panov @ 2026-08-27 16:11 UTC (permalink / raw)
To: catalin.marinas, akpm, david, ljs, vbabka, cl, linux, will,
mark.rutland, liam, rppt, surenb, mhocko
Cc: linux-mm, linux-kernel, linux-arm-kernel, wangkefeng.wang,
artem.kuzin, panov.nikita
In order to comply with AArch64 page table entries type -
table and block descriptor format, we need to use correct alignment
for replicated memory areas. Physical blocks allocated for replicated areas
must have the same offset for all replicas and same relative offset
to the virtual address.
For simplicity, we just use an explicit alignment for kernel sections.
If 4K pages are used, 2MB alignment should be applied.
Otherwise CONT_PTE_SIZE alignment is used.
Acked-by: Artem Kuzin <artem.kuzin@huawei.com>
Acked-by: Alexander Grubnikov <alexander.grubnikov@huawei.com>
Acked-by: Ilya Hanov <ilya.hanov@huawei-partners.com>
Acked-by: Denis Darvish <darvish.denis@huawei.com>
Signed-off-by: Nikita Panov <panov.nikita@huawei.com>
---
arch/arm64/kernel/vmlinux.lds.S | 22 ++++++++++++++++++++++
1 file changed, 22 insertions(+)
diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
index af1d72020976..aaf07e926c54 100644
--- a/arch/arm64/kernel/vmlinux.lds.S
+++ b/arch/arm64/kernel/vmlinux.lds.S
@@ -192,6 +192,13 @@ SECTIONS
_text = .;
HEAD_TEXT
}
+#ifdef CONFIG_KERNEL_REPLICATION
+#ifdef CONFIG_ARM64_4K_PAGES
+ . = ALIGN(PMD_SIZE);
+#else
+ . = ALIGN(CONT_PTE_SIZE);
+#endif
+#endif
.text : ALIGN(SEGMENT_ALIGN) { /* Real text segment */
_stext = .; /* Text and read-only data */
IRQENTRY_TEXT
@@ -207,10 +214,25 @@ SECTIONS
}
. = ALIGN(SEGMENT_ALIGN);
+#ifdef CONFIG_KERNEL_REPLICATION
+#ifdef CONFIG_ARM64_4K_PAGES
+ . = ALIGN(PMD_SIZE);
+#else
+ . = ALIGN(CONT_PTE_SIZE);
+#endif
+#endif
_etext = .; /* End of text section */
/* everything from this point to __init_begin will be marked RO NX */
+#ifdef CONFIG_KERNEL_REPLICATION
+#ifdef CONFIG_ARM64_4K_PAGES
+ RO_DATA(PMD_SIZE)
+#else
+ RO_DATA(CONT_PTE_SIZE)
+#endif
+#else
RO_DATA(PAGE_SIZE)
+#endif
HYPERVISOR_RODATA_SECTIONS
--
2.34.1
^ permalink raw reply related [flat|nested] 23+ messages in thread* Re: [RFC PATCH 02/18] arm64: align kernel text and rodata
2026-08-27 16:11 ` [RFC PATCH 02/18] arm64: align kernel text and rodata Nikita Panov
@ 2026-08-27 17:35 ` Lorenzo Stoakes (ARM)
0 siblings, 0 replies; 23+ messages in thread
From: Lorenzo Stoakes (ARM) @ 2026-08-27 17:35 UTC (permalink / raw)
To: artem.kuzin
Cc: catalin.marinas, akpm, david, vbabka, cl, linux, will,
mark.rutland, liam, rppt, surenb, mhocko, linux-mm, linux-kernel,
linux-arm-kernel, wangkefeng.wang, panov.nikita
On Fri, Aug 28, 2026 at 12:11:42AM +0800, Nikita Panov wrote:
> Acked-by: Artem Kuzin <artem.kuzin@huawei.com>
> Acked-by: Alexander Grubnikov <alexander.grubnikov@huawei.com>
> Acked-by: Ilya Hanov <ilya.hanov@huawei-partners.com>
> Acked-by: Denis Darvish <darvish.denis@huawei.com>
One small note on procedure: it's not really the done thing to pre-Ack
patches in series sent upstream like this :)
--
Cheers, Lorenzo
^ permalink raw reply [flat|nested] 23+ messages in thread
* [RFC PATCH 03/18] mm: allow per-NUMA node local P4D/PUD/PMD/PTE allocation
2026-08-27 16:11 [RFC PATCH 00/18] mm: arm64: Add kernel replication feature Nikita Panov
2026-08-27 16:11 ` [RFC PATCH 01/18] mm: arm64 add Kconfig option for kernel replication Nikita Panov
2026-08-27 16:11 ` [RFC PATCH 02/18] arm64: align kernel text and rodata Nikita Panov
@ 2026-08-27 16:11 ` Nikita Panov
2026-08-27 16:11 ` [RFC PATCH 04/18] arm64: add arch callbacks for kernel replication Nikita Panov
` (16 subsequent siblings)
19 siblings, 0 replies; 23+ messages in thread
From: Nikita Panov @ 2026-08-27 16:11 UTC (permalink / raw)
To: catalin.marinas, akpm, david, ljs, vbabka, cl, linux, will,
mark.rutland, liam, rppt, surenb, mhocko
Cc: linux-mm, linux-kernel, linux-arm-kernel, wangkefeng.wang,
artem.kuzin, panov.nikita
Acked-by: Artem Kuzin <artem.kuzin@huawei.com>
Acked-by: Alexander Grubnikov <alexander.grubnikov@huawei.com>
Acked-by: Ilya Hanov <ilya.hanov@huawei-partners.com>
Acked-by: Denis Darvish <darvish.denis@huawei.com>
Signed-off-by: Nikita Panov <panov.nikita@huawei.com>
---
include/asm-generic/pgalloc.h | 90 +++++++++++++++++++++++++++++
include/asm-generic/pgtable-nop4d.h | 5 ++
include/asm-generic/pgtable-nopmd.h | 5 ++
include/asm-generic/pgtable-nopud.h | 5 ++
include/linux/mm.h | 87 +++++++++++++++++++++++++++-
mm/memory.c | 68 ++++++++++++++++++++++
6 files changed, 259 insertions(+), 1 deletion(-)
diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h
index 051aa1331051..79d10806c0fc 100644
--- a/include/asm-generic/pgalloc.h
+++ b/include/asm-generic/pgalloc.h
@@ -85,6 +85,26 @@ static inline pgtable_t __pte_alloc_one_noprof(struct mm_struct *mm, gfp_t gfp)
}
#define __pte_alloc_one(...) alloc_hooks(__pte_alloc_one_noprof(__VA_ARGS__))
+#ifdef CONFIG_KERNEL_REPLICATION
+static inline pgtable_t __pte_alloc_one_node_noprof(unsigned int nid,
+ struct mm_struct *mm, gfp_t gfp)
+{
+ struct ptdesc *ptdesc;
+
+ ptdesc = pagetable_alloc_node_noprof(nid, gfp, 0);
+ if (!ptdesc)
+ return NULL;
+ if (!pagetable_pte_ctor(mm, ptdesc)) {
+ pagetable_free(ptdesc);
+ return NULL;
+ }
+
+ return ptdesc_page(ptdesc);
+}
+
+#define __pte_alloc_one_node(...) alloc_hooks(__pte_alloc_one_node_noprof(__VA_ARGS__))
+#endif
+
#ifndef __HAVE_ARCH_PTE_ALLOC_ONE
/**
* pte_alloc_one - allocate a page for PTE-level user page table
@@ -99,6 +119,17 @@ static inline pgtable_t pte_alloc_one_noprof(struct mm_struct *mm)
return __pte_alloc_one_noprof(mm, GFP_PGTABLE_USER);
}
#define pte_alloc_one(...) alloc_hooks(pte_alloc_one_noprof(__VA_ARGS__))
+
+#ifdef CONFIG_KERNEL_REPLICATION
+static inline pgtable_t pte_alloc_one_node_noprof(unsigned int nid,
+ struct mm_struct *mm)
+{
+ return __pte_alloc_one_node(nid, mm, GFP_PGTABLE_USER | __GFP_THISNODE);
+}
+
+#define pte_alloc_one_node(...) alloc_hooks(pte_alloc_one_node_noprof(__VA_ARGS__))
+#endif
+
#endif
/*
@@ -154,6 +185,33 @@ static inline pmd_t *pmd_alloc_one_noprof(struct mm_struct *mm, unsigned long ad
return ptdesc_address(ptdesc);
}
#define pmd_alloc_one(...) alloc_hooks(pmd_alloc_one_noprof(__VA_ARGS__))
+
+#ifdef CONFIG_KERNEL_REPLICATION
+static inline pmd_t *pmd_alloc_one_node_noprof(unsigned int nid,
+ struct mm_struct *mm,
+ unsigned long addr)
+{
+ struct ptdesc *ptdesc;
+ gfp_t gfp = GFP_PGTABLE_USER;
+
+ if (mm == &init_mm)
+ gfp = GFP_PGTABLE_KERNEL;
+
+ gfp |= __GFP_THISNODE;
+
+ ptdesc = pagetable_alloc_node_noprof(nid, gfp, 0);
+ if (!ptdesc)
+ return NULL;
+ if (!pagetable_pmd_ctor(mm, ptdesc)) {
+ pagetable_free(ptdesc);
+ return NULL;
+ }
+ return ptdesc_address(ptdesc);
+}
+
+#define pmd_alloc_one_node(...) alloc_hooks(pmd_alloc_one_node_noprof(__VA_ARGS__))
+#endif /* CONFIG_KERNEL_REPLICATION */
+
#endif
#ifndef __HAVE_ARCH_PMD_FREE
@@ -191,6 +249,27 @@ static inline pud_t *__pud_alloc_one_noprof(struct mm_struct *mm, unsigned long
}
#define __pud_alloc_one(...) alloc_hooks(__pud_alloc_one_noprof(__VA_ARGS__))
+#ifdef CONFIG_KERNEL_REPLICATION
+static inline pud_t *__pud_alloc_one_node_noprof(unsigned int nid,
+ struct mm_struct *mm,
+ unsigned long addr)
+{
+ gfp_t gfp = GFP_PGTABLE_USER;
+ struct ptdesc *ptdesc;
+
+ if (mm == &init_mm)
+ gfp = GFP_PGTABLE_KERNEL;
+
+ gfp |= __GFP_THISNODE;
+ ptdesc = pagetable_alloc_node_noprof(nid, gfp, 0);
+ if (!ptdesc)
+ return NULL;
+ return ptdesc_address(ptdesc);
+}
+
+#define __pud_alloc_one_node(...) alloc_hooks(__pud_alloc_one_node_noprof(__VA_ARGS__))
+#endif /* CONFIG_KERNEL_REPLICATION */
+
#ifndef __HAVE_ARCH_PUD_ALLOC_ONE
/**
* pud_alloc_one - allocate memory for a PUD-level page table
@@ -206,6 +285,17 @@ static inline pud_t *pud_alloc_one_noprof(struct mm_struct *mm, unsigned long ad
return __pud_alloc_one_noprof(mm, addr);
}
#define pud_alloc_one(...) alloc_hooks(pud_alloc_one_noprof(__VA_ARGS__))
+
+#ifdef CONFIG_KERNEL_REPLICATION
+static inline pud_t *pud_alloc_one_node_noprof(unsigned int nid,
+ struct mm_struct *mm, unsigned long addr)
+{
+ return __pud_alloc_one_node(nid, mm, addr);
+}
+
+#define pud_alloc_one_node(...) alloc_hooks(pud_alloc_one_node_noprof(__VA_ARGS__))
+#endif /* CONFIG_KERNEL_REPLICATION */
+
#endif
static inline void __pud_free(struct mm_struct *mm, pud_t *pud)
diff --git a/include/asm-generic/pgtable-nop4d.h b/include/asm-generic/pgtable-nop4d.h
index 89c21f84cffb..95c03ceeeea0 100644
--- a/include/asm-generic/pgtable-nop4d.h
+++ b/include/asm-generic/pgtable-nop4d.h
@@ -48,6 +48,11 @@ static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address)
* inside the pgd, so has no extra memory associated with it.
*/
#define p4d_alloc_one(mm, address) NULL
+
+#ifdef CONFIG_KERNEL_REPLICATION
+#define p4d_alloc_one_node(nid, mm, address) NULL
+#endif
+
#define p4d_free(mm, x) do { } while (0)
#define p4d_free_tlb(tlb, x, a) do { } while (0)
diff --git a/include/asm-generic/pgtable-nopmd.h b/include/asm-generic/pgtable-nopmd.h
index 36b6490ed180..9849760b3751 100644
--- a/include/asm-generic/pgtable-nopmd.h
+++ b/include/asm-generic/pgtable-nopmd.h
@@ -60,6 +60,11 @@ static inline pmd_t * pmd_offset(pud_t * pud, unsigned long address)
* inside the pud, so has no extra memory associated with it.
*/
#define pmd_alloc_one(mm, address) NULL
+
+#ifdef CONFIG_KERNEL_REPLICATION
+#define pmd_alloc_one_node(nid, mm, address) NULL
+#endif
+
static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)
{
}
diff --git a/include/asm-generic/pgtable-nopud.h b/include/asm-generic/pgtable-nopud.h
index 356cbfbaab24..0acbac9961d0 100644
--- a/include/asm-generic/pgtable-nopud.h
+++ b/include/asm-generic/pgtable-nopud.h
@@ -56,6 +56,11 @@ static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address)
* inside the p4d, so has no extra memory associated with it.
*/
#define pud_alloc_one(mm, address) NULL
+
+#ifdef CONFIG_KERNEL_REPLICATION
+#define pud_alloc_one_node(nid, mm, address) NULL
+#endif
+
#define pud_free(mm, x) do { } while (0)
#define pud_free_tlb(tlb, x, a) do { } while (0)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index dd09c438fa23..95213a81907c 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -3595,8 +3595,24 @@ static inline int __p4d_alloc(struct mm_struct *mm, pgd_t *pgd,
{
return 0;
}
-#else
+
+#ifdef CONFIG_KERNEL_REPLICATION
+static inline int __p4d_alloc_node(unsigned int nid,
+ struct mm_struct *mm,
+ pgd_t *pgd, unsigned long address)
+{
+ return 0;
+}
+#endif
+
+#else /* !__PAGETABLE_P4D_FOLDED */
int __p4d_alloc(struct mm_struct *mm, pgd_t *pgd, unsigned long address);
+
+#ifdef CONFIG_KERNEL_REPLICATION
+int __p4d_alloc_node(unsigned int nid, struct mm_struct *mm,
+ pgd_t *pgd, unsigned long address);
+#endif
+
#endif
#if defined(__PAGETABLE_PUD_FOLDED) || !defined(CONFIG_MMU)
@@ -3605,12 +3621,27 @@ static inline int __pud_alloc(struct mm_struct *mm, p4d_t *p4d,
{
return 0;
}
+
+#ifdef CONFIG_KERNEL_REPLICATION
+static inline int __pud_alloc_node(unsigned int nid,
+ struct mm_struct *mm,
+ p4d_t *p4d, unsigned long address)
+{
+ return 0;
+}
+#endif /* CONFIG_KERNEL_REPLICATION */
+
static inline void mm_inc_nr_puds(struct mm_struct *mm) {}
static inline void mm_dec_nr_puds(struct mm_struct *mm) {}
#else
int __pud_alloc(struct mm_struct *mm, p4d_t *p4d, unsigned long address);
+#ifdef CONFIG_KERNEL_REPLICATION
+int __pud_alloc_node(unsigned int nid,
+ struct mm_struct *mm,
+ p4d_t *p4d, unsigned long address);
+#endif /* CONFIG_KERNEL_REPLICATION */
static inline void mm_inc_nr_puds(struct mm_struct *mm)
{
if (mm_pud_folded(mm))
@@ -3633,12 +3664,27 @@ static inline int __pmd_alloc(struct mm_struct *mm, pud_t *pud,
return 0;
}
+#ifdef CONFIG_KERNEL_REPLICATION
+static inline int __pmd_alloc_node(unsigned int nid,
+ struct mm_struct *mm,
+ pud_t *pud, unsigned long address)
+{
+ return 0;
+}
+#endif /* CONFIG_KERNEL_REPLICATION */
+
static inline void mm_inc_nr_pmds(struct mm_struct *mm) {}
static inline void mm_dec_nr_pmds(struct mm_struct *mm) {}
#else
int __pmd_alloc(struct mm_struct *mm, pud_t *pud, unsigned long address);
+#ifdef CONFIG_KERNEL_REPLICATION
+int __pmd_alloc_node(unsigned int nid,
+ struct mm_struct *mm,
+ pud_t *pud, unsigned long address);
+#endif /* CONFIG_KERNEL_REPLICATION */
+
static inline void mm_inc_nr_pmds(struct mm_struct *mm)
{
if (mm_pmd_folded(mm))
@@ -3710,6 +3756,32 @@ static inline pmd_t *pmd_alloc(struct mm_struct *mm, pud_t *pud, unsigned long a
return (unlikely(pud_none(*pud)) && __pmd_alloc(mm, pud, address))?
NULL: pmd_offset(pud, address);
}
+
+#ifdef CONFIG_KERNEL_REPLICATION
+static inline p4d_t *p4d_alloc_node(unsigned int nid,
+ struct mm_struct *mm,
+ pgd_t *pgd, unsigned long address)
+{
+ return (unlikely(pgd_none(*pgd)) && __p4d_alloc_node(nid, mm, pgd, address)) ?
+ NULL : p4d_offset(pgd, address);
+}
+
+static inline pud_t *pud_alloc_node(unsigned int nid,
+ struct mm_struct *mm,
+ p4d_t *p4d, unsigned long address)
+{
+ return (unlikely(p4d_none(*p4d)) && __pud_alloc_node(nid, mm, p4d, address)) ?
+ NULL : pud_offset(p4d, address);
+}
+
+static inline pmd_t *pmd_alloc_node(unsigned int nid,
+ struct mm_struct *mm,
+ pud_t *pud, unsigned long address)
+{
+ return (unlikely(pud_none(*pud)) && __pmd_alloc_node(nid, mm, pud, address)) ?
+ NULL : pmd_offset(pud, address);
+}
+#endif /* CONFIG_KERNEL_REPLICATION */
#endif /* CONFIG_MMU */
enum pt_flags {
@@ -3812,6 +3884,19 @@ static inline void pagetable_free_kernel(struct ptdesc *pt)
__pagetable_free(pt);
}
#endif
+
+#ifdef CONFIG_KERNEL_REPLICATION
+
+static inline struct ptdesc *pagetable_alloc_node_noprof(int nid, gfp_t gfp,
+ unsigned int order)
+{
+ struct page *page = alloc_pages_node_noprof(nid, gfp | __GFP_COMP, order);
+
+ return page_ptdesc(page);
+}
+
+#endif
+
/**
* pagetable_free - Free pagetables
* @pt: The page table descriptor
diff --git a/mm/memory.c b/mm/memory.c
index 8b0c2c735d3d..05853e4fb266 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -6906,6 +6906,28 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
EXPORT_SYMBOL_GPL(handle_mm_fault);
#ifndef __PAGETABLE_P4D_FOLDED
+
+#ifdef CONFIG_KERNEL_REPLICATION
+int __p4d_alloc_node(unsigned int nid,
+ struct mm_struct *mm,
+ pgd_t *pgd, unsigned long address)
+{
+ p4d_t *new = p4d_alloc_one_node(nid, mm, address);
+ if (!new)
+ return -ENOMEM;
+
+ spin_lock(&mm->page_table_lock);
+ if (pgd_present(*pgd)) { /* Another has populated it */
+ p4d_free(mm, new);
+ } else {
+ smp_wmb(); /* See comment in pmd_install() */
+ pgd_populate(mm, pgd, new);
+ }
+ spin_unlock(&mm->page_table_lock);
+ return 0;
+}
+#endif /* CONFIG_KERNEL_REPLICATION */
+
/*
* Allocate p4d page table.
* We've already handled the fast-path in-line.
@@ -6929,6 +6951,28 @@ int __p4d_alloc(struct mm_struct *mm, pgd_t *pgd, unsigned long address)
#endif /* __PAGETABLE_P4D_FOLDED */
#ifndef __PAGETABLE_PUD_FOLDED
+
+#ifdef CONFIG_KERNEL_REPLICATION
+int __pud_alloc_node(unsigned int nid,
+ struct mm_struct *mm,
+ p4d_t *p4d, unsigned long address)
+{
+ pud_t *new = pud_alloc_one_node(nid, mm, address);
+ if (!new)
+ return -ENOMEM;
+
+ spin_lock(&mm->page_table_lock);
+ if (!p4d_present(*p4d)) {
+ mm_inc_nr_puds(mm);
+ smp_wmb(); /* See comment in pmd_install() */
+ p4d_populate(mm, p4d, new);
+ } else /* Another has populated it */
+ pud_free(mm, new);
+ spin_unlock(&mm->page_table_lock);
+ return 0;
+}
+#endif /* CONFIG_KERNEL_REPLICATION */
+
/*
* Allocate page upper directory.
* We've already handled the fast-path in-line.
@@ -6974,6 +7018,30 @@ int __pmd_alloc(struct mm_struct *mm, pud_t *pud, unsigned long address)
spin_unlock(ptl);
return 0;
}
+
+#ifdef CONFIG_KERNEL_REPLICATION
+int __pmd_alloc_node(unsigned int nid,
+ struct mm_struct *mm,
+ pud_t *pud, unsigned long address)
+{
+ spinlock_t *ptl;
+ pmd_t *new = pmd_alloc_one_node(nid, mm, address);
+ if (!new)
+ return -ENOMEM;
+
+ ptl = pud_lock(mm, pud);
+ if (!pud_present(*pud)) {
+ mm_inc_nr_pmds(mm);
+ smp_wmb(); /* See comment in pmd_install() */
+ pud_populate(mm, pud, new);
+ } else { /* Another has populated it */
+ pmd_free(mm, new);
+ }
+ spin_unlock(ptl);
+ return 0;
+}
+#endif /* CONFIG_KERNEL_REPLICATION */
+
#endif /* __PAGETABLE_PMD_FOLDED */
static inline void pfnmap_args_setup(struct follow_pfnmap_args *args,
--
2.34.1
^ permalink raw reply related [flat|nested] 23+ messages in thread* [RFC PATCH 04/18] arm64: add arch callbacks for kernel replication
2026-08-27 16:11 [RFC PATCH 00/18] mm: arm64: Add kernel replication feature Nikita Panov
` (2 preceding siblings ...)
2026-08-27 16:11 ` [RFC PATCH 03/18] mm: allow per-NUMA node local P4D/PUD/PMD/PTE allocation Nikita Panov
@ 2026-08-27 16:11 ` Nikita Panov
2026-08-27 16:11 ` [RFC PATCH 05/18] mm: per-NUMA node replication core infrastructure Nikita Panov
` (15 subsequent siblings)
19 siblings, 0 replies; 23+ messages in thread
From: Nikita Panov @ 2026-08-27 16:11 UTC (permalink / raw)
To: catalin.marinas, akpm, david, ljs, vbabka, cl, linux, will,
mark.rutland, liam, rppt, surenb, mhocko
Cc: linux-mm, linux-kernel, linux-arm-kernel, wangkefeng.wang,
artem.kuzin, panov.nikita
Acked-by: Artem Kuzin <artem.kuzin@huawei.com>
Acked-by: Alexander Grubnikov <alexander.grubnikov@huawei.com>
Acked-by: Ilya Hanov <ilya.hanov@huawei-partners.com>
Acked-by: Denis Darvish <darvish.denis@huawei.com>
Signed-off-by: Nikita Panov <panov.nikita@huawei.com>
---
arch/arm64/include/asm/numa_replication.h | 54 +++++++++++++++++++++++
1 file changed, 54 insertions(+)
create mode 100644 arch/arm64/include/asm/numa_replication.h
diff --git a/arch/arm64/include/asm/numa_replication.h b/arch/arm64/include/asm/numa_replication.h
new file mode 100644
index 000000000000..441a399f9c51
--- /dev/null
+++ b/arch/arm64/include/asm/numa_replication.h
@@ -0,0 +1,54 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef __ASM_NUMA_REPLICATION_H
+#define __ASM_NUMA_REPLICATION_H
+
+#ifdef CONFIG_KERNEL_REPLICATION
+#include <asm/pgtable.h>
+#include <asm/tlbflush.h>
+#include <asm/pgalloc.h>
+#include <asm/memory.h>
+#include <asm/mmu_context.h>
+#include <linux/mm.h>
+#include <linux/seq_file.h>
+
+#define PAGE_TABLE_REPLICATION_LEFT ((max((u64)_end - SZ_2G, (u64)MODULES_VADDR)) & PGDIR_MASK)
+#define PAGE_TABLE_REPLICATION_RIGHT ((((u64)_end + SZ_2G) & PGDIR_MASK) + PGDIR_SIZE - 1)
+
+static inline pgd_t *numa_replicate_pgt_pgd(int nid)
+{
+ pgd_t *new_pgd;
+ struct page *pgd_page;
+
+ pgd_page = alloc_pages_node_noprof(nid, GFP_PGTABLE_KERNEL, 2);
+ BUG_ON(pgd_page == NULL);
+
+ new_pgd = (pgd_t *)page_address(pgd_page);
+ new_pgd += (PAGE_SIZE * 2 / sizeof(pgd_t)); //Extra pages for KPTI
+ copy_page(new_pgd, swapper_pg_dir);
+
+ return new_pgd;
+}
+
+static inline void numa_load_replicated_pgd(pgd_t *pgd)
+{
+ cpu_replace_ttbr1(pgd);
+ local_flush_tlb_all();
+}
+
+static inline ssize_t numa_cpu_dump(struct seq_file *m)
+{
+ seq_printf(m, "NODE: #%02d, CPU: #%04d, ttbr1_el1: 0x%p, COMM: %s\n",
+ numa_node_id(),
+ smp_processor_id(),
+ (void *)read_sysreg(ttbr1_el1),
+ current->group_leader->comm);
+ return 0;
+}
+
+static inline void numa_sync_text_replicas(unsigned long start, unsigned long end)
+{
+ caches_clean_inval_pou(start, end);
+ icache_inval_all_pou();
+}
+#endif /* CONFIG_KERNEL_REPLICATION */
+#endif /* __ASM_NUMA_REPLICATION_H */
--
2.34.1
^ permalink raw reply related [flat|nested] 23+ messages in thread* [RFC PATCH 05/18] mm: per-NUMA node replication core infrastructure
2026-08-27 16:11 [RFC PATCH 00/18] mm: arm64: Add kernel replication feature Nikita Panov
` (3 preceding siblings ...)
2026-08-27 16:11 ` [RFC PATCH 04/18] arm64: add arch callbacks for kernel replication Nikita Panov
@ 2026-08-27 16:11 ` Nikita Panov
2026-08-27 16:11 ` [RFC PATCH 06/18] mm: add support of memory protection for NUMA replicas Nikita Panov
` (14 subsequent siblings)
19 siblings, 0 replies; 23+ messages in thread
From: Nikita Panov @ 2026-08-27 16:11 UTC (permalink / raw)
To: catalin.marinas, akpm, david, ljs, vbabka, cl, linux, will,
mark.rutland, liam, rppt, surenb, mhocko
Cc: linux-mm, linux-kernel, linux-arm-kernel, wangkefeng.wang,
artem.kuzin, panov.nikita
In current design mutable kernel data modifications don't require
synchronization between translation tables due to on 64-bit platforms
all physical memory already mapped in kernel space and this mapping
is persistent.
No need to synchronize userspace at all. Due to separate ttbrs for
userspace and kernel are used
TT overview:
NODE 0 NODE 1
USER KERNEL USER KERNEL
--------------------- ---------------------
PGD | | | | | | | | |*| | | | | | | | | |*|
--------------------- ---------------------
| |
------------------- -------------------
| |
--------------------- ---------------------
PUD | | | | | | | |*|*| | | | | | | | |*|*|
--------------------- ---------------------
| |
------------------- -------------------
| |
--------------------- ---------------------
PMD |READ-ONLY|MUTABLE | |READ-ONLY|MUTABLE |
--------------------- ---------------------
| | | |
| --------------------------
| | |
-------- ------- --------
PHYS | | | | | |
MEM -------- ------- --------
<------> <------>
NODE 0 Shared NODE 1
between
nodes
* - entries unique in each table
Acked-by: Alexander Grubnikov <alexander.grubnikov@huawei.com>
Acked-by: Ilya Hanov <ilya.hanov@huawei-partners.com>
Acked-by: Denis Darvish <darvish.denis@huawei.com>
Co-developed-by: Artem Kuzin <artem.kuzin@huawei.com>
Signed-off-by: Artem Kuzin <artem.kuzin@huawei.com>
Co-developed-by: Nikita Panov <panov.nikita@huawei.com>
Signed-off-by: Nikita Panov <panov.nikita@huawei.com>
---
include/linux/mm_types.h | 3 +
include/linux/numa_kernel_replication.h | 89 +++
mm/Makefile | 2 +
mm/numa_kernel_replication.c | 740 ++++++++++++++++++++++++
4 files changed, 834 insertions(+)
create mode 100644 include/linux/numa_kernel_replication.h
create mode 100644 mm/numa_kernel_replication.c
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 6d815f6440c9..06c42b7918a5 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -1419,6 +1419,9 @@ struct mm_struct {
#ifdef CONFIG_MM_ID
mm_id_t mm_id;
#endif /* CONFIG_MM_ID */
+#ifdef CONFIG_KERNEL_REPLICATION
+ pgd_t **pgd_numa;
+#endif
} __randomize_layout;
/*
diff --git a/include/linux/numa_kernel_replication.h b/include/linux/numa_kernel_replication.h
new file mode 100644
index 000000000000..34aa063d42e6
--- /dev/null
+++ b/include/linux/numa_kernel_replication.h
@@ -0,0 +1,89 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef _LINUX_NUMA_REPLICATION_H
+#define _LINUX_NUMA_REPLICATION_H
+
+#ifdef CONFIG_KERNEL_REPLICATION
+
+#include <linux/mm_types.h>
+#include <linux/nodemask.h>
+#include <linux/module.h>
+#include <linux/mm.h>
+
+#include <asm/numa_replication.h>
+
+extern nodemask_t replica_nodes;
+
+#define for_each_memory_node(nid) \
+ for (nid = first_node(replica_nodes); \
+ nid != MAX_NUMNODES; \
+ nid = next_node(nid, replica_nodes))
+
+#define this_node_pgd(mm) ((mm)->pgd_numa[numa_node_id()])
+#define per_node_pgd(mm, nid) ((mm)->pgd_numa[nid])
+
+static inline bool numa_addr_has_replica(const void *addr)
+{
+ return ((unsigned long)addr >= PAGE_TABLE_REPLICATION_LEFT) &&
+ ((unsigned long)addr <= PAGE_TABLE_REPLICATION_RIGHT);
+}
+
+void __init numa_replication_init(void);
+void __init numa_replicate_kernel_text(void);
+void numa_replicate_kernel_rodata(void);
+void numa_replication_fini(void);
+
+void numa_setup_pgd(void);
+void __init_or_module *numa_get_replica(void *vaddr, int nid);
+int numa_get_memory_node(int nid);
+void dump_mm_pgtables(struct mm_struct *mm,
+ unsigned long start, unsigned long end);
+
+#else
+
+#include <linux/mm.h>
+
+#define this_node_pgd(mm) ((mm)->pgd)
+#define per_node_pgd(mm, nid) ((mm)->pgd)
+
+static inline void numa_setup_pgd(void)
+{
+}
+
+static inline void __init numa_replication_init(void)
+{
+}
+
+static inline void __init numa_replicate_kernel_text(void)
+{
+}
+
+static inline void numa_replicate_kernel_rodata(void)
+{
+}
+
+static inline void numa_replication_fini(void)
+{
+}
+
+static inline bool numa_addr_has_replica(const void *addr)
+{
+ return false;
+}
+
+static inline bool is_text_replicated(void)
+{
+ return false;
+}
+
+static inline void *numa_get_replica(void *vaddr, int nid)
+{
+ return lm_alias(vaddr);
+}
+
+static inline void dump_mm_pgtables(struct mm_struct *mm,
+ unsigned long start, unsigned long end)
+{
+}
+
+#endif /*CONFIG_KERNEL_REPLICATION*/
+#endif /*_LINUX_NUMA_REPLICATION_H*/
diff --git a/mm/Makefile b/mm/Makefile
index e7245cb88c66..e2bbc04a48d3 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -148,3 +148,5 @@ obj-$(CONFIG_EXECMEM) += execmem.o
obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o
obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o
obj-$(CONFIG_MEM_ALLOC_PROFILING) += alloc_tag.o
+obj-$(CONFIG_KERNEL_REPLICATION) += numa_kernel_replication.o
+
diff --git a/mm/numa_kernel_replication.c b/mm/numa_kernel_replication.c
new file mode 100644
index 000000000000..082aabc6b8bc
--- /dev/null
+++ b/mm/numa_kernel_replication.c
@@ -0,0 +1,740 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <linux/kernel.h>
+#include <linux/pagewalk.h>
+#include <linux/numa_kernel_replication.h>
+#include <linux/memblock.h>
+#include <linux/pgtable.h>
+#include <linux/hugetlb.h>
+#include <linux/kobject.h>
+#include <linux/debugfs.h>
+
+#include <asm/sections.h>
+#include <asm/tlbflush.h>
+#include <asm/mmu_context.h>
+
+#define KERNEL_TEXT_START ((unsigned long)&_stext)
+#define KERNEL_TEXT_END ((unsigned long)&_etext)
+
+#define KERNEL_RODATA_START ((unsigned long)&__start_rodata)
+#define KERNEL_RODATA_END ((unsigned long)&__end_rodata)
+
+#define PMD_ALLOC_ORDER (PMD_SHIFT-PAGE_SHIFT)
+#define PAGES_PER_PMD (1 << PMD_ALLOC_ORDER)
+
+#define replication_log(data, fmt, args...) \
+({ \
+ if (data && data->m) \
+ seq_printf(data->m, fmt, ##args); \
+ else \
+ pr_info(KERN_CONT fmt, ##args); \
+})
+
+struct numa_node_desc {
+ pgd_t *pgd;
+ void *text_vaddr;
+ void *rodata_vaddr;
+};
+
+static struct numa_node_desc __initdata_or_module node_desc[MAX_NUMNODES];
+
+struct dump_data {
+ struct seq_file *m;
+};
+
+struct dump_config {
+ int pgd_extra_info:1;
+ int p4d_extra_info:1;
+ int pud_extra_info:1;
+ int pmd_extra_info:1;
+ int pte_extra_info:1;
+ struct dump_data *data;
+};
+
+static bool text_replicated;
+/*
+ * The first ready NUMA node, used as a source node
+ * for kernel text and rodata replication
+ */
+static unsigned int master_node = INT_MAX;
+/*
+ * The case when machine has memoryless nodes is rare
+ * but possible. To handle memoryless nodes properly
+ * kernel replication maintains mapping node -> node with memory
+ * for all NUMA nodes.
+ */
+static int node_to_memory_node[MAX_NUMNODES];
+
+static bool pgtables_extra;
+static DEFINE_SPINLOCK(debugfs_lock);
+
+bool is_text_replicated(void)
+{
+ return text_replicated;
+}
+
+static void binary_dump(struct dump_data *data, unsigned long value)
+{
+ int i;
+
+ for (i = BITS_PER_LONG - 1; i >= 0; i--) {
+ if ((BITS_PER_LONG - 1 - i) % BITS_PER_BYTE == 0)
+ replication_log(data, "%-9d", i);
+ }
+ replication_log(data, "%d\n", 0);
+
+ for (i = BITS_PER_LONG - 1; i >= 0; i--) {
+ if ((BITS_PER_LONG - 1 - i) % BITS_PER_BYTE == 0)
+ replication_log(data, "|");
+
+ replication_log(data, "%d", (1UL << i) & value ? 1 : 0);
+ }
+ replication_log(data, "|\n");
+}
+
+static int pgd_callback(pgd_t *pgd,
+ unsigned long addr, unsigned long next,
+ struct mm_walk *walk)
+{
+ unsigned long val = pgd_val(*pgd);
+ struct dump_config *c = (struct dump_config *)walk->private;
+
+ if (!val)
+ return 0;
+
+ addr = addr & PGDIR_MASK;
+ next = (addr & PGDIR_MASK) - 1 + PGDIR_SIZE;
+
+ replication_log(c->data,
+ "PGD ADDR: 0x%p PGD VAL: 0x%016lx [%p --- %p]\n",
+ pgd, val, (void *)addr, (void *)next);
+
+ if (c->pgd_extra_info)
+ binary_dump(c->data, val);
+
+ return 0;
+}
+
+static int p4d_callback(p4d_t *p4d,
+ unsigned long addr, unsigned long next,
+ struct mm_walk *walk)
+{
+ unsigned long val = p4d_val(*p4d);
+ struct dump_config *c = (struct dump_config *)walk->private;
+
+ if (!val)
+ return 0;
+
+ addr = addr & P4D_MASK;
+ next = (addr & P4D_MASK) - 1 + P4D_SIZE;
+
+ replication_log(c->data,
+ "P4D ADDR: 0x%p P4D VAL: 0x%016lx [%p --- %p]\n",
+ p4d, val, (void *)addr, (void *)next);
+
+ if (c->p4d_extra_info)
+ binary_dump(c->data, val);
+
+ return 0;
+}
+
+static int pud_callback(pud_t *pud,
+ unsigned long addr, unsigned long next,
+ struct mm_walk *walk)
+{
+ unsigned long val = pud_val(*pud);
+ struct dump_config *c = (struct dump_config *)walk->private;
+
+ if (!val)
+ return 0;
+
+ addr = addr & PUD_MASK;
+ next = (addr & PUD_MASK) - 1 + PUD_SIZE;
+
+ replication_log(c->data,
+ "PUD ADDR: 0x%p PUD VAL: 0x%016lx huge(%d) [%p --- %p]\n",
+ pud, val, !pud_table(*pud), (void *)addr, (void *)next);
+
+ if (c->pud_extra_info)
+ binary_dump(c->data, val);
+
+ return 0;
+}
+
+static int pmd_callback(pmd_t *pmd,
+ unsigned long addr, unsigned long next,
+ struct mm_walk *walk)
+{
+ unsigned long val = pmd_val(*pmd);
+ unsigned long paddr = pmd_pfn(*pmd) << PAGE_SHIFT;
+ struct dump_config *c = (struct dump_config *)walk->private;
+
+ if (!val)
+ return 0;
+
+ addr = addr & PMD_MASK;
+ next = (addr & PMD_MASK) - 1 + PMD_SIZE;
+
+ replication_log(c->data,
+ "PMD ADDR: 0x%p PMD VAL: 0x%016lx huge(%d) [%p --- %p] to %p\n",
+ pmd, val, !pmd_table(*pmd), (void *)addr, (void *)next, (void *)paddr);
+
+ if (c->pmd_extra_info)
+ binary_dump(c->data, val);
+
+ return 0;
+}
+
+static int pte_callback(pte_t *pte,
+ unsigned long addr, unsigned long next,
+ struct mm_walk *walk)
+{
+ unsigned long val = pte_val(*pte);
+ unsigned long paddr = pte_pfn(*pte) << PAGE_SHIFT;
+ struct dump_config *c = (struct dump_config *)walk->private;
+
+ if (!val)
+ return 0;
+
+ addr = addr & PAGE_MASK;
+ next = (addr & PAGE_MASK) - 1 + PAGE_SIZE;
+
+ replication_log(c->data,
+ "PTE ADDR: 0x%p PTE VAL: 0x%016lx [%p --- %p] to %p\n",
+ pte, val, (void *)addr, (void *)next, (void *)paddr);
+
+ if (c->pte_extra_info)
+ binary_dump(c->data, val);
+
+ return 0;
+}
+
+static int pte_hole_callback(unsigned long addr, unsigned long next,
+ int depth, struct mm_walk *walk)
+{
+ struct dump_config *c = (struct dump_config *)walk->private;
+
+ replication_log(c->data, "%*chole\n", depth * 2, ' ');
+
+ return 0;
+}
+
+static void dump_pgtables(struct mm_struct *mm,
+ struct dump_data *data,
+ unsigned long start, unsigned long end)
+{
+ int nid = 0;
+ int extra = pgtables_extra ? 1 : 0;
+ bool locked = false;
+ struct dump_config conf = {
+ .pgd_extra_info = extra,
+ .p4d_extra_info = extra,
+ .pud_extra_info = extra,
+ .pmd_extra_info = extra,
+ .pte_extra_info = extra,
+ .data = data,
+ };
+
+ const struct mm_walk_ops ops = {
+ .pgd_entry = pgd_callback,
+ .p4d_entry = p4d_callback,
+ .pud_entry = pud_callback,
+ .pmd_entry = pmd_callback,
+ .pte_entry = pte_callback,
+ .pte_hole = pte_hole_callback
+ };
+
+ BUG_ON(data && data->m == NULL);
+
+ start = start & PAGE_MASK;
+ end = (end & PAGE_MASK) - 1 + PAGE_SIZE;
+
+ replication_log(data,
+ "----PER-NUMA NODE KERNEL REPLICATION ENABLED----\n");
+
+ if (rwsem_is_locked(&mm->mmap_lock))
+ locked = true;
+ else
+ mmap_read_lock(mm);
+
+ for_each_memory_node(nid) {
+ replication_log(data, "NUMA node id #%d\n", nid);
+ replication_log(data, "PGD: %p PGD phys: %p\n",
+ mm->pgd_numa[nid], (void *)virt_to_phys(mm->pgd_numa[nid]));
+ walk_kernel_page_table_range(start, end, &ops, mm->pgd_numa[nid], &conf);
+ }
+
+ if (!locked)
+ mmap_read_unlock(mm);
+
+ replication_log(data,
+ "----PER-NUMA NODE KERNEL REPLICATION ENABLED----\n");
+}
+
+static void dump_kernel_pgtables(struct dump_data *data,
+ unsigned long start, unsigned long end)
+{
+ dump_pgtables(&init_mm, data, start, end);
+}
+
+void dump_mm_pgtables(struct mm_struct *mm,
+ unsigned long start, unsigned long end)
+{
+ dump_pgtables(mm, NULL, start, end);
+}
+
+static void cpu_dump(void *info)
+{
+ struct dump_data *data = (struct dump_data *)info;
+
+ spin_lock(&debugfs_lock);
+ numa_cpu_dump(data->m);
+ spin_unlock(&debugfs_lock);
+}
+
+static int stats_show(struct seq_file *m, void *v)
+{
+ int cpu;
+ struct dump_data data = {
+ .m = m,
+ };
+
+ for_each_online_cpu(cpu)
+ smp_call_function_single(cpu, cpu_dump, &data, 1);
+
+ return 0;
+}
+
+DEFINE_SHOW_ATTRIBUTE(stats);
+
+static int pgtables_show(struct seq_file *m, void *v)
+{
+ struct dump_data data = {
+ .m = m,
+ };
+
+ dump_kernel_pgtables(&data,
+ KERNEL_TEXT_START, KERNEL_RODATA_END - 1);
+
+ return 0;
+}
+
+DEFINE_SHOW_ATTRIBUTE(pgtables);
+
+static void debugfs_init(void)
+{
+ struct dentry *dir;
+ static struct dentry *debugfs_dir;
+
+ debugfs_dir = debugfs_create_dir("numa_replication", NULL);
+ if (IS_ERR(debugfs_dir)) {
+ pr_err("Failed to create debugfs entry for NUMA"
+ " replication: %ld\n",
+ PTR_ERR(debugfs_dir));
+ return;
+ }
+ dir = debugfs_create_file("stats", 0400, debugfs_dir,
+ NULL, &stats_fops);
+ if (IS_ERR(dir)) {
+ pr_err("Failed to create debugfs entry for NUMA"
+ " replication stats: %ld\n",
+ PTR_ERR(dir));
+ return;
+ }
+
+ dir = debugfs_create_file("pgtables_kernel", 0400, debugfs_dir,
+ NULL, &pgtables_fops);
+ if (IS_ERR(dir)) {
+ pr_err("Failed to create debugfs entry for NUMA"
+ " replication pgtables: %ld\n",
+ PTR_ERR(dir));
+ return;
+ }
+
+ debugfs_create_bool("pgtables_kernel_extra", 0600, debugfs_dir,
+ &pgtables_extra);
+}
+
+/*
+ * The case, when machine has memoryless NUMA nodes
+ * should be handled in a special way. To do this we
+ * create node<->memory mapping to have an information
+ * about the node with memory that memoryless node can use.
+ */
+static void init_node_to_memory_mapping(void)
+{
+ int nid;
+
+ for_each_online_node(nid) {
+ int memory_nid;
+ int min_dist = INT_MAX;
+
+ node_to_memory_node[nid] = nid;
+ for_each_memory_node(memory_nid) {
+ int dist = node_distance(nid, memory_nid);
+
+ if (dist < min_dist) {
+ min_dist = dist;
+ node_to_memory_node[nid] = memory_nid;
+ }
+ }
+ pr_info("For node %d memory is on the node - %d\n",
+ nid, node_to_memory_node[nid]);
+ }
+}
+
+int numa_get_memory_node(int nid)
+{
+ return node_to_memory_node[nid];
+}
+
+/*
+ * The function creates replica of particular memory area
+ * and install replicated memory in translation table of
+ * required NUMA node.
+ */
+static void replicate_memory(void *dst, unsigned long start, unsigned long end, int nid)
+{
+ pgd_t *pgd;
+ p4d_t *p4d;
+ pud_t *pud;
+ pmd_t *pmd;
+ pte_t *pte;
+ pgprot_t prot;
+ unsigned int offset_in_pages = 0;
+ unsigned long vaddr = start;
+ struct page *pages = virt_to_page(dst);
+
+ memcpy(dst, lm_alias(start), end - start);
+ while (vaddr < end) {
+ pgd = pgd_offset_pgd(node_desc[nid].pgd, vaddr);
+ p4d = p4d_offset(pgd, vaddr);
+ pud = pud_offset(p4d, vaddr);
+ pmd = pmd_offset(pud, vaddr);
+
+ if (pmd_leaf(*pmd)) {
+ prot = pmd_pgprot(*pmd);
+
+ set_pmd(pmd, pfn_pmd(page_to_pfn(pages) + offset_in_pages, prot));
+ offset_in_pages += PAGES_PER_PMD;
+ vaddr += PMD_SIZE;
+ continue;
+ }
+ pte = pte_offset_kernel(pmd, vaddr);
+ prot = pte_pgprot(*pte);
+ __set_pte(pte, pfn_pte(page_to_pfn(pages) + offset_in_pages, prot));
+ offset_in_pages++;
+ vaddr += PAGE_SIZE;
+ }
+}
+
+static void __init replicate_kernel_text(int nid)
+{
+ replicate_memory(node_desc[nid].text_vaddr,
+ KERNEL_TEXT_START, KERNEL_TEXT_END, nid);
+ numa_sync_text_replicas((unsigned long)node_desc[nid].text_vaddr,
+ (unsigned long)node_desc[nid].text_vaddr + (KERNEL_TEXT_END - KERNEL_TEXT_START));
+}
+
+static void replicate_kernel_rodata(int nid)
+{
+ replicate_memory(node_desc[nid].rodata_vaddr,
+ KERNEL_RODATA_START, KERNEL_RODATA_END, nid);
+}
+
+//'-1' in next functions have only one purpose - prevent unsgined long overflow
+static void replicate_pgt_pte(pud_t *dst, pud_t *src,
+ unsigned long start, unsigned long end,
+ unsigned int nid)
+{
+ unsigned long left = start & PMD_MASK;
+ unsigned long right = (end & PMD_MASK) - 1 + PMD_SIZE;
+ unsigned long addr;
+
+ pmd_t *clone_pmd = pmd_offset(dst, left);
+ pmd_t *orig_pmd = pmd_offset(src, left);
+
+ for (addr = left;
+ (addr >= left && addr < right); addr += PMD_SIZE) {
+ pgtable_t new_pte;
+
+ if (pmd_none(*orig_pmd) || !pmd_table(*orig_pmd) ||
+ pmd_val(*orig_pmd) == 0)
+ goto skip;
+
+ pmd_clear(clone_pmd);
+ new_pte = pte_alloc_one_node(nid, &init_mm);
+ pmd_populate_kernel(&init_mm, clone_pmd, page_to_virt(new_pte));
+ BUG_ON(new_pte == NULL);
+
+ copy_page(page_to_virt(pmd_pgtable(*clone_pmd)),
+ page_to_virt(pmd_pgtable(*orig_pmd)));
+skip:
+ clone_pmd++;
+ orig_pmd++;
+ }
+}
+
+//'-1' in next functions have only one purpose - prevent unsgined long overflow
+static void replicate_pgt_pmd(p4d_t *dst, p4d_t *src,
+ unsigned long start, unsigned long end,
+ unsigned int nid)
+{
+ unsigned long left = start & PUD_MASK;
+ unsigned long right = (end & PUD_MASK) - 1 + PUD_SIZE;
+
+ pud_t *clone_pud = pud_offset(dst, left);
+ pud_t *orig_pud = pud_offset(src, left);
+
+ for (unsigned long addr = left;
+ (addr >= left && addr < right); addr += PUD_SIZE) {
+ pmd_t *new_pmd;
+
+ if (pud_none(*orig_pud) || !pud_table(*orig_pud) ||
+ pud_val(*orig_pud) == 0)
+ goto skip;
+
+ pud_clear(clone_pud);
+ new_pmd = pmd_alloc_node(nid, &init_mm, clone_pud, addr);
+ BUG_ON(new_pmd == NULL);
+
+ copy_page(pud_pgtable(*clone_pud), pud_pgtable(*orig_pud));
+
+ replicate_pgt_pte(clone_pud, orig_pud, max(addr, start),
+ min(addr - 1 + PUD_SIZE, end), nid);
+skip:
+ clone_pud++;
+ orig_pud++;
+ }
+}
+
+static void replicate_pgt_pud(pgd_t *dst, pgd_t *src,
+ unsigned long start, unsigned long end,
+ unsigned int nid)
+{
+ unsigned long left = start & P4D_MASK;
+ unsigned long right = (end & P4D_MASK) - 1 + P4D_SIZE;
+
+ p4d_t *clone_p4d = p4d_offset(dst, left);
+ p4d_t *orig_p4d = p4d_offset(src, left);
+
+ for (unsigned long addr = left;
+ (addr >= left && addr < right); addr += P4D_SIZE) {
+ pud_t *new_pud;
+
+ if (p4d_none(*orig_p4d) || p4d_val(*orig_p4d) == 0)
+ goto skip;
+
+ p4d_clear(clone_p4d);
+ new_pud = pud_alloc_node(nid, &init_mm, clone_p4d, addr);
+ BUG_ON(new_pud == NULL);
+
+ copy_page(p4d_pgtable(*clone_p4d), p4d_pgtable(*orig_p4d));
+ /*
+ * start and end passed to the next function must be in
+ * range of p4ds, so min and max are used here
+ */
+ replicate_pgt_pmd(clone_p4d, orig_p4d, max(addr, start),
+ min(addr - 1 + P4D_SIZE, end), nid);
+skip:
+ clone_p4d++;
+ orig_p4d++;
+ }
+}
+
+static void replicate_pgt_p4d(pgd_t *dst, pgd_t *src,
+ unsigned long start, unsigned long end,
+ unsigned int nid)
+{
+ unsigned long left = start & PGDIR_MASK;
+ unsigned long right = (end & PGDIR_MASK) - 1 + PGDIR_SIZE;
+
+ pgd_t *clone_pgd = pgd_offset_pgd(dst, left);
+ pgd_t *orig_pgd = pgd_offset_pgd(src, left);
+
+ for (unsigned long addr = left;
+ (addr >= left && addr < right); addr += PGDIR_SIZE) {
+ p4d_t *new_p4d;
+
+ /* TODO: remove last condition and do something better
+ * In the case of a folded P4D level, pgd_none and pgd_huge
+ * always return 0, so we might start to replicate empty entries.
+ * We obviously want to avoid this, so the last check is performed here.
+ */
+ if (pgd_none(*orig_pgd) || pgd_val(*orig_pgd) == 0)
+ goto skip;
+
+ pgd_clear(clone_pgd);
+ new_p4d = p4d_alloc_node(nid, &init_mm, clone_pgd, addr);
+ BUG_ON(new_p4d == NULL);
+
+ copy_page((void *)pgd_page_vaddr(*clone_pgd),
+ (void *)pgd_page_vaddr(*orig_pgd));
+ replicate_pgt_pud(clone_pgd, orig_pgd, max(addr, start),
+ min(addr - 1 + PGDIR_SIZE, end), nid);
+skip:
+ clone_pgd++;
+ orig_pgd++;
+ }
+}
+
+static void replicate_pgt(int nid, unsigned long start, unsigned long end)
+{
+ replicate_pgt_p4d(node_desc[nid].pgd, init_mm.pgd, start, end, nid);
+}
+
+/*
+ * Page tables replication works in a way when first
+ * pgd level replicated and then the replication of the
+ * left part if done. The only part of pagetable that
+ * contains text and rodata is replicated. Obviously a
+ * part of upper layer entries of page table should be
+ * replicated too. As result, the pgd, p4d, pud and pmd
+ * layers are touched by replication. In particular, the
+ * page table sub-tree that cover kernel text and rodata.
+ */
+static void replicate_pgtables(void)
+{
+ int nid;
+
+ init_mm.pgd_numa = (pgd_t **)kmalloc(sizeof(pgd_t *) * MAX_NUMNODES, GFP_PGTABLE_KERNEL);
+ BUG_ON(!init_mm.pgd_numa);
+
+ for_each_memory_node(nid) {
+ node_desc[nid].pgd = numa_replicate_pgt_pgd(nid);
+ replicate_pgt(nid, PAGE_TABLE_REPLICATION_LEFT,
+ PAGE_TABLE_REPLICATION_RIGHT);
+ }
+
+ init_mm.pgd = node_desc[numa_get_memory_node(master_node)].pgd;
+
+ for_each_online_node(nid) {
+ int memory_nid = numa_get_memory_node(nid);
+
+ init_mm.pgd_numa[nid] = node_desc[memory_nid].pgd;
+ }
+}
+
+/*
+ * Kernel text replication includes two steps:
+ * 1. page tables replication for init_mm
+ * 2. kernel text pages replication and
+ * corresponding page table update.
+ * 3. setup page table, related to
+ * current NUMA node on current cpu,
+ * for other NUMA cpus page tables will
+ * be updated later, during cpu initialization.
+ * Master node - the first NUMA node, used as
+ * a source for replicas. Memory for master node
+ * is expected to be already local.
+ */
+void __init numa_replicate_kernel_text(void)
+{
+ int nid;
+
+ replicate_pgtables();
+
+ for_each_memory_node(nid) {
+ if (nid == master_node)
+ continue;
+ replicate_kernel_text(nid);
+ }
+
+ text_replicated = true;
+
+ numa_setup_pgd();
+}
+
+void numa_replicate_kernel_rodata(void)
+{
+ int nid;
+
+ for_each_memory_node(nid) {
+ if (nid == master_node)
+ continue;
+ replicate_kernel_rodata(nid);
+ }
+
+ flush_tlb_all();
+}
+
+void numa_setup_pgd(void)
+{
+ numa_load_replicated_pgd(init_mm.pgd_numa[numa_node_id()]);
+}
+
+void __init_or_module *numa_get_replica(void *vaddr, int nid)
+{
+ unsigned long addr = (unsigned long)vaddr;
+ unsigned long offset = addr - KERNEL_TEXT_START;
+
+ BUG_ON(addr < KERNEL_TEXT_START || addr >= KERNEL_TEXT_END);
+ BUG_ON(node_desc[nid].text_vaddr == NULL);
+ BUG_ON(numa_get_memory_node(nid) != nid);
+
+ return node_desc[nid].text_vaddr + offset;
+}
+
+nodemask_t __ro_after_init replica_nodes = { { [0] = 1UL } };
+
+void __init numa_replication_init(void)
+{
+ int nid;
+
+ unsigned long align = PAGE_SIZE;
+#ifdef CONFIG_ARM64_4K_PAGES
+ align = HPAGE_SIZE;
+#else
+ align = CONT_PTE_SIZE;
+#endif
+ nodes_clear(replica_nodes);
+
+ for_each_node_state(nid, N_MEMORY) {
+ __node_set(nid, &replica_nodes);
+ }
+
+ for_each_memory_node(nid)
+ pr_info("Memory node: %d\n", nid);
+
+ init_node_to_memory_mapping();
+ master_node = page_to_nid(virt_to_page(lm_alias((void *)KERNEL_TEXT_START)));
+
+ pr_info("Master Node: #%d\n", master_node);
+ for_each_memory_node(nid) {
+ if (nid == master_node) {
+ node_desc[nid].text_vaddr = lm_alias((void *)KERNEL_TEXT_START);
+ node_desc[nid].rodata_vaddr = lm_alias((void *)KERNEL_RODATA_START);
+ } else {
+ node_desc[nid].text_vaddr = memblock_alloc_try_nid(
+ (KERNEL_TEXT_END - KERNEL_TEXT_START),
+ align, 0, MEMBLOCK_ALLOC_ANYWHERE, nid);
+
+ node_desc[nid].rodata_vaddr = memblock_alloc_try_nid(
+ (KERNEL_RODATA_END - KERNEL_RODATA_START),
+ align, 0, MEMBLOCK_ALLOC_ANYWHERE, nid);
+ }
+
+ BUG_ON(node_desc[nid].text_vaddr == NULL);
+ BUG_ON(node_desc[nid].rodata_vaddr == NULL);
+ }
+}
+
+void numa_replication_fini(void)
+{
+ int nid;
+
+ /*
+ * Clear addresses form linear space
+ */
+ for_each_memory_node(nid) {
+ node_desc[nid].text_vaddr = NULL;
+ node_desc[nid].rodata_vaddr = NULL;
+ }
+
+ debugfs_init();
+
+ pr_info("Replicated page table : [%p --- %p]\n",
+ (void *)PAGE_TABLE_REPLICATION_LEFT,
+ (void *)PAGE_TABLE_REPLICATION_RIGHT);
+
+ dump_kernel_pgtables(NULL, KERNEL_TEXT_START, KERNEL_RODATA_END - 1);
+}
--
2.34.1
^ permalink raw reply related [flat|nested] 23+ messages in thread* [RFC PATCH 06/18] mm: add support of memory protection for NUMA replicas
2026-08-27 16:11 [RFC PATCH 00/18] mm: arm64: Add kernel replication feature Nikita Panov
` (4 preceding siblings ...)
2026-08-27 16:11 ` [RFC PATCH 05/18] mm: per-NUMA node replication core infrastructure Nikita Panov
@ 2026-08-27 16:11 ` Nikita Panov
2026-08-27 16:11 ` [RFC PATCH 07/18] arm64: " Nikita Panov
` (13 subsequent siblings)
19 siblings, 0 replies; 23+ messages in thread
From: Nikita Panov @ 2026-08-27 16:11 UTC (permalink / raw)
To: catalin.marinas, akpm, david, ljs, vbabka, cl, linux, will,
mark.rutland, liam, rppt, surenb, mhocko
Cc: linux-mm, linux-kernel, linux-arm-kernel, wangkefeng.wang,
artem.kuzin, panov.nikita
Acked-by: Artem Kuzin <artem.kuzin@huawei.com>
Acked-by: Alexander Grubnikov <alexander.grubnikov@huawei.com>
Acked-by: Ilya Hanov <ilya.hanov@huawei-partners.com>
Acked-by: Denis Darvish <darvish.denis@huawei.com>
Signed-off-by: Nikita Panov <panov.nikita@huawei.com>
---
include/linux/set_memory.h | 35 +++++++++++++++++++++++++++++++++++
1 file changed, 35 insertions(+)
diff --git a/include/linux/set_memory.h b/include/linux/set_memory.h
index 3030d9245f5a..03fccacef037 100644
--- a/include/linux/set_memory.h
+++ b/include/linux/set_memory.h
@@ -7,11 +7,32 @@
#ifdef CONFIG_ARCH_HAS_SET_MEMORY
#include <asm/set_memory.h>
+
+#ifdef CONFIG_KERNEL_REPLICATION
+int numa_set_memory_ro(unsigned long addr, int numpages);
+int numa_set_memory_rw(unsigned long addr, int numpages);
+int numa_set_memory_x(unsigned long addr, int numpages);
+int numa_set_memory_nx(unsigned long addr, int numpages);
+#else
+
+#define numa_set_memory_ro set_memory_ro
+#define numa_set_memory_rw set_memory_rw
+#define numa_set_memory_x set_memory_x
+#define numa_set_memory_nx set_memory_nx
+
+#endif /* CONFIG_KERNEL_REPLICATION */
+
#else
static inline int __must_check set_memory_ro(unsigned long addr, int numpages) { return 0; }
static inline int __must_check set_memory_rw(unsigned long addr, int numpages) { return 0; }
static inline int __must_check set_memory_x(unsigned long addr, int numpages) { return 0; }
static inline int __must_check set_memory_nx(unsigned long addr, int numpages) { return 0; }
+
+#define numa_set_memory_ro set_memory_ro
+#define numa_set_memory_rw set_memory_rw
+#define numa_set_memory_x set_memory_x
+#define numa_set_memory_nx set_memory_nx
+
#endif
#ifndef set_memory_rox
@@ -24,6 +45,20 @@ static inline int set_memory_rox(unsigned long addr, int numpages)
}
#endif
+#ifndef numa_set_memory_rox
+#ifdef CONFIG_KERNEL_REPLICATION
+static inline int numa_set_memory_rox(unsigned long addr, int numpages)
+{
+ int ret = numa_set_memory_ro(addr, numpages);
+ if (ret)
+ return ret;
+ return numa_set_memory_x(addr, numpages);
+}
+#else
+#define numa_set_memory_rox set_memory_rox
+#endif
+#endif
+
#ifndef CONFIG_ARCH_HAS_SET_DIRECT_MAP
static inline int set_direct_map_invalid_noflush(struct page *page)
{
--
2.34.1
^ permalink raw reply related [flat|nested] 23+ messages in thread* [RFC PATCH 07/18] arm64: add support of memory protection for NUMA replicas
2026-08-27 16:11 [RFC PATCH 00/18] mm: arm64: Add kernel replication feature Nikita Panov
` (5 preceding siblings ...)
2026-08-27 16:11 ` [RFC PATCH 06/18] mm: add support of memory protection for NUMA replicas Nikita Panov
@ 2026-08-27 16:11 ` Nikita Panov
2026-08-27 16:11 ` [RFC PATCH 08/18] mm: set memory permissions for BPF handlers replicas Nikita Panov
` (12 subsequent siblings)
19 siblings, 0 replies; 23+ messages in thread
From: Nikita Panov @ 2026-08-27 16:11 UTC (permalink / raw)
To: catalin.marinas, akpm, david, ljs, vbabka, cl, linux, will,
mark.rutland, liam, rppt, surenb, mhocko
Cc: linux-mm, linux-kernel, linux-arm-kernel, wangkefeng.wang,
artem.kuzin, panov.nikita
Acked-by: Artem Kuzin <artem.kuzin@huawei.com>
Acked-by: Alexander Grubnikov <alexander.grubnikov@huawei.com>
Acked-by: Ilya Hanov <ilya.hanov@huawei-partners.com>
Acked-by: Denis Darvish <darvish.denis@huawei.com>
Signed-off-by: Nikita Panov <panov.nikita@huawei.com>
---
arch/arm64/mm/pageattr.c | 72 ++++++++++++++++++++++++++++++++++++++--
1 file changed, 70 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c
index bbe98ac9ad8c..50a76385b184 100644
--- a/arch/arm64/mm/pageattr.c
+++ b/arch/arm64/mm/pageattr.c
@@ -9,6 +9,7 @@
#include <linux/sched.h>
#include <linux/vmalloc.h>
#include <linux/pagewalk.h>
+#include <linux/numa_kernel_replication.h>
#include <asm/cacheflush.h>
#include <asm/pgtable-prot.h>
@@ -102,7 +103,7 @@ bool can_set_direct_map(void)
arm64_kfence_can_set_direct_map() || is_realm_world();
}
-static int update_range_prot(unsigned long start, unsigned long size,
+static int update_range_prot_pgd(pgd_t *pgtable, unsigned long start, unsigned long size,
pgprot_t set_mask, pgprot_t clear_mask)
{
struct page_change_data data;
@@ -123,12 +124,19 @@ static int update_range_prot(unsigned long start, unsigned long size,
* must be eliminated by splitting the mapping.
*/
ret = walk_kernel_page_table_range_lockless(start, start + size,
- &pageattr_ops, NULL, &data);
+ &pageattr_ops, pgtable, &data);
lazy_mmu_mode_disable();
return ret;
}
+static int update_range_prot(unsigned long start, unsigned long size,
+ pgprot_t set_mask, pgprot_t clear_mask)
+{
+ return update_range_prot_pgd(NULL, start, size, set_mask, clear_mask);
+}
+
+
static int __change_memory_common(unsigned long start, unsigned long size,
pgprot_t set_mask, pgprot_t clear_mask)
{
@@ -147,6 +155,22 @@ static int __change_memory_common(unsigned long start, unsigned long size,
return ret;
}
+#ifdef CONFIG_KERNEL_REPLICATION
+static int __change_memory_common_replicas(unsigned long start, unsigned long size,
+ pgprot_t set_mask, pgprot_t clear_mask)
+{
+ int nid;
+ int ret;
+
+ for_each_memory_node(nid) {
+ ret |= update_range_prot_pgd(per_node_pgd(&init_mm, nid), start, size, set_mask, clear_mask);
+ }
+
+ flush_tlb_kernel_range(start, start + size);
+ return ret;
+}
+#endif /* CONFIG_KERNEL_REPLICATION */
+
static int change_memory_common(unsigned long addr, int numpages,
pgprot_t set_mask, pgprot_t clear_mask)
{
@@ -211,6 +235,20 @@ static int change_memory_common(unsigned long addr, int numpages,
return __change_memory_common(start, size, set_mask, clear_mask);
}
+#ifdef CONFIG_KERNEL_REPLICATION
+static int numa_change_memory_common(unsigned long addr, int numpages,
+ pgprot_t set_mask, pgprot_t clear_mask)
+{
+ int ret;
+
+ ret = change_memory_common(addr, numpages, set_mask, clear_mask);
+ if (ret)
+ return ret;
+
+ return __change_memory_common_replicas(addr, numpages * PAGE_SIZE, set_mask, clear_mask);
+}
+#endif /* CONFIG_KERNEL_REPLICATION */
+
int set_memory_ro(unsigned long addr, int numpages)
{
return change_memory_common(addr, numpages,
@@ -239,6 +277,36 @@ int set_memory_x(unsigned long addr, int numpages)
__pgprot(PTE_PXN));
}
+#ifdef CONFIG_KERNEL_REPLICATION
+int numa_set_memory_x(unsigned long addr, int numpages)
+{
+ return numa_change_memory_common(addr, numpages,
+ __pgprot(PTE_MAYBE_GP),
+ __pgprot(PTE_PXN));
+}
+
+int numa_set_memory_nx(unsigned long addr, int numpages)
+{
+ return numa_change_memory_common(addr, numpages,
+ __pgprot(PTE_PXN),
+ __pgprot(PTE_MAYBE_GP));
+}
+
+int numa_set_memory_ro(unsigned long addr, int numpages)
+{
+ return numa_change_memory_common(addr, numpages,
+ __pgprot(PTE_RDONLY),
+ __pgprot(PTE_WRITE));
+}
+
+int numa_set_memory_rw(unsigned long addr, int numpages)
+{
+ return numa_change_memory_common(addr, numpages,
+ __pgprot(PTE_WRITE),
+ __pgprot(PTE_RDONLY));
+}
+#endif /*CONFIG_KERNEL_REPLICATION*/
+
int set_memory_valid(unsigned long addr, int numpages, int enable)
{
if (enable)
--
2.34.1
^ permalink raw reply related [flat|nested] 23+ messages in thread* [RFC PATCH 08/18] mm: set memory permissions for BPF handlers replicas
2026-08-27 16:11 [RFC PATCH 00/18] mm: arm64: Add kernel replication feature Nikita Panov
` (6 preceding siblings ...)
2026-08-27 16:11 ` [RFC PATCH 07/18] arm64: " Nikita Panov
@ 2026-08-27 16:11 ` Nikita Panov
2026-08-27 16:11 ` [RFC PATCH 09/18] mm: add replicas allocation support for vmalloc Nikita Panov
` (11 subsequent siblings)
19 siblings, 0 replies; 23+ messages in thread
From: Nikita Panov @ 2026-08-27 16:11 UTC (permalink / raw)
To: catalin.marinas, akpm, david, ljs, vbabka, cl, linux, will,
mark.rutland, liam, rppt, surenb, mhocko
Cc: linux-mm, linux-kernel, linux-arm-kernel, wangkefeng.wang,
artem.kuzin, panov.nikita
Acked-by: Artem Kuzin <artem.kuzin@huawei.com>
Acked-by: Alexander Grubnikov <alexander.grubnikov@huawei.com>
Acked-by: Ilya Hanov <ilya.hanov@huawei-partners.com>
Acked-by: Denis Darvish <darvish.denis@huawei.com>
Signed-off-by: Nikita Panov <panov.nikita@huawei.com>
---
arch/arm64/net/bpf_jit_comp.c | 4 ++--
include/asm-generic/set_memory.h | 14 ++++++++++++++
include/linux/set_memory.h | 14 --------------
kernel/bpf/core.c | 4 ++--
kernel/bpf/trampoline.c | 2 +-
5 files changed, 19 insertions(+), 19 deletions(-)
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index c18e005a41db..588d1b2a2753 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -3236,10 +3236,10 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t,
/* non-zero plt_target indicates we're patching a bpf prog,
* which is read only.
*/
- if (set_memory_rw(PAGE_MASK & ((uintptr_t)&plt->target), 1))
+ if (numa_set_memory_rw(PAGE_MASK & ((uintptr_t)&plt->target), 1))
return -EFAULT;
WRITE_ONCE(plt->target, plt_target);
- set_memory_ro(PAGE_MASK & ((uintptr_t)&plt->target), 1);
+ numa_set_memory_ro(PAGE_MASK & ((uintptr_t)&plt->target), 1);
/* since plt target points to either the new trampoline
* or dummy_tramp, even if another CPU reads the old plt
* target value before fetching the bl instruction to plt,
diff --git a/include/asm-generic/set_memory.h b/include/asm-generic/set_memory.h
index c86abf6bc7ba..4c97978d2bfb 100644
--- a/include/asm-generic/set_memory.h
+++ b/include/asm-generic/set_memory.h
@@ -10,4 +10,18 @@ int set_memory_rw(unsigned long addr, int numpages);
int set_memory_x(unsigned long addr, int numpages);
int set_memory_nx(unsigned long addr, int numpages);
+#ifdef CONFIG_KERNEL_REPLICATION
+int numa_set_memory_ro(unsigned long addr, int numpages);
+int numa_set_memory_rw(unsigned long addr, int numpages);
+int numa_set_memory_x(unsigned long addr, int numpages);
+int numa_set_memory_nx(unsigned long addr, int numpages);
+#else
+
+#define numa_set_memory_ro set_memory_ro
+#define numa_set_memory_rw set_memory_rw
+#define numa_set_memory_x set_memory_x
+#define numa_set_memory_nx set_memory_nx
+
+#endif /* CONFIG_KERNEL_REPLICATION */
+
#endif
diff --git a/include/linux/set_memory.h b/include/linux/set_memory.h
index 03fccacef037..00b21d1c4331 100644
--- a/include/linux/set_memory.h
+++ b/include/linux/set_memory.h
@@ -8,20 +8,6 @@
#ifdef CONFIG_ARCH_HAS_SET_MEMORY
#include <asm/set_memory.h>
-#ifdef CONFIG_KERNEL_REPLICATION
-int numa_set_memory_ro(unsigned long addr, int numpages);
-int numa_set_memory_rw(unsigned long addr, int numpages);
-int numa_set_memory_x(unsigned long addr, int numpages);
-int numa_set_memory_nx(unsigned long addr, int numpages);
-#else
-
-#define numa_set_memory_ro set_memory_ro
-#define numa_set_memory_rw set_memory_rw
-#define numa_set_memory_x set_memory_x
-#define numa_set_memory_nx set_memory_nx
-
-#endif /* CONFIG_KERNEL_REPLICATION */
-
#else
static inline int __must_check set_memory_ro(unsigned long addr, int numpages) { return 0; }
static inline int __must_check set_memory_rw(unsigned long addr, int numpages) { return 0; }
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 8b294dfc1ad4..854db8e9dcf7 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -939,7 +939,7 @@ static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_ins
if (!bpf_jit_mem_is_rox()) {
bpf_fill_ill_insns(pack->ptr, BPF_PROG_PACK_SIZE);
set_vm_flush_reset_perms(pack->ptr);
- err = set_memory_rox((unsigned long)pack->ptr,
+ err = numa_set_memory_rox((unsigned long)pack->ptr,
BPF_PROG_PACK_SIZE / PAGE_SIZE);
if (err)
goto out;
@@ -977,7 +977,7 @@ void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns, bool
bpf_fill_ill_insns(ptr, size);
set_vm_flush_reset_perms(ptr);
- err = set_memory_rox((unsigned long)ptr,
+ err = numa_set_memory_rox((unsigned long)ptr,
size / PAGE_SIZE);
if (err) {
bpf_jit_free_exec(ptr);
diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
index 90b70ea0d370..66656eb4714a 100644
--- a/kernel/bpf/trampoline.c
+++ b/kernel/bpf/trampoline.c
@@ -1508,7 +1508,7 @@ void __weak arch_free_bpf_trampoline(void *image, unsigned int size)
int __weak arch_protect_bpf_trampoline(void *image, unsigned int size)
{
WARN_ON_ONCE(size > PAGE_SIZE);
- return set_memory_rox((long)image, 1);
+ return numa_set_memory_rox((long)image, 1);
}
int __weak arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
--
2.34.1
^ permalink raw reply related [flat|nested] 23+ messages in thread* [RFC PATCH 09/18] mm: add replicas allocation support for vmalloc
2026-08-27 16:11 [RFC PATCH 00/18] mm: arm64: Add kernel replication feature Nikita Panov
` (7 preceding siblings ...)
2026-08-27 16:11 ` [RFC PATCH 08/18] mm: set memory permissions for BPF handlers replicas Nikita Panov
@ 2026-08-27 16:11 ` Nikita Panov
2026-08-27 16:11 ` [RFC PATCH 10/18] arm64: enable per-NUMA node kernel text and rodata replication Nikita Panov
` (10 subsequent siblings)
19 siblings, 0 replies; 23+ messages in thread
From: Nikita Panov @ 2026-08-27 16:11 UTC (permalink / raw)
To: catalin.marinas, akpm, david, ljs, vbabka, cl, linux, will,
mark.rutland, liam, rppt, surenb, mhocko
Cc: linux-mm, linux-kernel, linux-arm-kernel, wangkefeng.wang,
artem.kuzin, panov.nikita
In order to support generic vmalloc, several modifications are required.
If the allocated region belongs to the non-replicated part of the table,
then the normal vmalloc mechanism is suitable.
If the allocated region belongs to the replicated part of the table,
a replicated table for each replica node must be created for this memory.
This region might be replicated after its initialization,
for example, to support replication of text and ro-data
for loadable kernel modules.
Acked-by: Alexander Grubnikov <alexander.grubnikov@huawei.com>
Acked-by: Ilya Hanov <ilya.hanov@huawei-partners.com>
Acked-by: Denis Darvish <darvish.denis@huawei.com>
Co-developed-by: Artem Kuzin <artem.kuzin@huawei.com>
Signed-off-by: Artem Kuzin <artem.kuzin@huawei.com>
Co-developed-by: Nikita Panov <panov.nikita@huawei.com>
Signed-off-by: Nikita Panov <panov.nikita@huawei.com>
---
include/linux/mm.h | 2 +
include/linux/vmalloc.h | 18 ++
mm/memory.c | 61 ++++++
mm/vmalloc.c | 454 +++++++++++++++++++++++++++++++++-------
4 files changed, 461 insertions(+), 74 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 95213a81907c..8a7083c02203 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1885,6 +1885,8 @@ int region_intersects(resource_size_t offset, size_t size, unsigned long flags,
struct page *vmalloc_to_page(const void *addr);
unsigned long vmalloc_to_pfn(const void *addr);
+struct page *walk_to_page_node(int nid, const void *addr);
+
/*
* Determine if an address is within the vmalloc range
*
diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h
index aed121d729b0..44eda2175a28 100644
--- a/include/linux/vmalloc.h
+++ b/include/linux/vmalloc.h
@@ -31,6 +31,10 @@ struct iov_iter; /* in uio.h */
#define VM_MAP_PUT_PAGES 0x00000200 /* put pages and free array in vfree */
#define VM_ALLOW_HUGE_VMAP 0x00000400 /* Allow for huge pages on archs with HAVE_ARCH_HUGE_VMALLOC */
+#ifdef CONFIG_KERNEL_REPLICATION
+#define VM_NUMA_SHARED 0x00002000 /* Pages shared between per-NUMA node TT*/
+#endif
+
#if (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)) && \
!defined(CONFIG_KASAN_VMALLOC)
#define VM_DEFER_KMEMLEAK 0x00000800 /* defer kmemleak object creation */
@@ -66,6 +70,10 @@ struct vm_struct {
phys_addr_t phys_addr;
const void *caller;
unsigned long requested_size;
+#ifdef CONFIG_KERNEL_REPLICATION
+ int node;
+ bool replicated;
+#endif
};
struct vmap_area {
@@ -177,6 +185,16 @@ extern void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align
const void *caller) __alloc_size(1);
#define __vmalloc_node_range(...) alloc_hooks(__vmalloc_node_range_noprof(__VA_ARGS__))
+#ifdef CONFIG_KERNEL_REPLICATION
+ /*
+ * DO NOT USE this function if you don't understand what it is doing
+ * Use only in pair with vmalloc(vm_flags|=VM_NUMA_SHARED)
+ */
+int __vmalloc_node_replicate_range(const void *addr, gfp_t gfp_mask,
+ pgprot_t prot, unsigned long vm_flags);
+void vunmap_range_replicas(unsigned long addr, unsigned long end);
+#endif
+
void *__vmalloc_node_noprof(unsigned long size, unsigned long align, gfp_t gfp_mask,
int node, const void *caller) __alloc_size(1);
#define __vmalloc_node(...) alloc_hooks(__vmalloc_node_noprof(__VA_ARGS__))
diff --git a/mm/memory.c b/mm/memory.c
index 05853e4fb266..49e5a502abd8 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -78,6 +78,7 @@
#include <linux/sched/sysctl.h>
#include <linux/pgalloc.h>
#include <linux/uaccess.h>
+#include <linux/numa_kernel_replication.h>
#include <trace/events/kmem.h>
@@ -7810,4 +7811,64 @@ void vma_pgtable_walk_end(struct vm_area_struct *vma)
{
if (is_vm_hugetlb_page(vma))
hugetlb_vma_unlock_read(vma);
+
+}
+/**
+ * Walk in replicated tranlation table specified by nid.
+ * If kernel replication is disabled or text is not replicated yet,
+ * value of nid is not used
+ */
+struct page *walk_to_page_node(int nid, const void *vmalloc_addr)
+{
+ unsigned long addr = (unsigned long)vmalloc_addr;
+ struct page *page = NULL;
+ pgd_t *pgd;
+ p4d_t *p4d;
+ pud_t *pud;
+ pmd_t *pmd;
+ pte_t *ptep, pte;
+
+ if (is_text_replicated())
+ pgd = pgd_offset_pgd(per_node_pgd(&init_mm, nid), addr);
+ else
+ pgd = pgd_offset_pgd(init_mm.pgd, addr);
+
+ if (pgd_none(*pgd))
+ return NULL;
+ if (WARN_ON_ONCE(pgd_leaf(*pgd)))
+ return NULL; /* XXX: no allowance for huge pgd */
+ if (WARN_ON_ONCE(pgd_bad(*pgd)))
+ return NULL;
+
+ p4d = p4d_offset(pgd, addr);
+ if (p4d_none(*p4d))
+ return NULL;
+ if (p4d_leaf(*p4d))
+ return p4d_page(*p4d) + ((addr & ~P4D_MASK) >> PAGE_SHIFT);
+ if (WARN_ON_ONCE(p4d_bad(*p4d)))
+ return NULL;
+
+ pud = pud_offset(p4d, addr);
+ if (pud_none(*pud))
+ return NULL;
+ if (pud_leaf(*pud))
+ return pud_page(*pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT);
+ if (WARN_ON_ONCE(pud_bad(*pud)))
+ return NULL;
+
+ pmd = pmd_offset(pud, addr);
+ if (pmd_none(*pmd))
+ return NULL;
+ if (pmd_leaf(*pmd))
+ return pmd_page(*pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
+ if (WARN_ON_ONCE(pmd_bad(*pmd)))
+ return NULL;
+
+ ptep = pte_offset_map(pmd, addr);
+ pte = *ptep;
+ if (pte_present(pte))
+ page = pte_page(pte);
+ pte_unmap(ptep);
+
+ return page;
}
diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index bea9f76ed7e7..da6facf64db8 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -10,6 +10,7 @@
#include <linux/vmalloc.h>
#include <linux/mm.h>
+#include <linux/numa_kernel_replication.h>
#include <linux/module.h>
#include <linux/highmem.h>
#include <linux/sched/signal.h>
@@ -318,7 +319,7 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end,
return err;
}
-static int vmap_range_noflush(unsigned long addr, unsigned long end,
+static int vmap_range_noflush_pgd(pgd_t *pgtable, unsigned long addr, unsigned long end,
phys_addr_t phys_addr, pgprot_t prot,
unsigned int max_page_shift)
{
@@ -337,7 +338,7 @@ static int vmap_range_noflush(unsigned long addr, unsigned long end,
BUG_ON(addr >= end);
start = addr;
- pgd = pgd_offset_k(addr);
+ pgd = pgd_offset_pgd(pgtable, addr);
do {
next = pgd_addr_end(addr, end);
err = vmap_p4d_range(pgd, addr, next, phys_addr, prot,
@@ -352,6 +353,13 @@ static int vmap_range_noflush(unsigned long addr, unsigned long end,
return err;
}
+static int vmap_range_noflush(unsigned long addr, unsigned long end,
+ phys_addr_t phys_addr, pgprot_t prot,
+ unsigned int max_page_shift)
+{
+ return vmap_range_noflush_pgd(init_mm.pgd, addr, end, phys_addr, prot, max_page_shift);
+}
+
int vmap_page_range(unsigned long addr, unsigned long end,
phys_addr_t phys_addr, pgprot_t prot)
{
@@ -490,18 +498,17 @@ static void vunmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end,
}
/*
- * vunmap_range_noflush is similar to vunmap_range, but does not
- * flush caches or TLBs.
+ * __vunmap_range_noflush_pgd is similar to vunmap_range, but does not
+ * flush caches or TLBs, and able to work with pgd granularity.
*
* The caller is responsible for calling flush_cache_vmap() before calling
* this function, and flush_tlb_kernel_range after it has returned
* successfully (and before the addresses are expected to cause a page fault
* or be re-mapped for something else, if TLB flushes are being delayed or
* coalesced).
- *
- * This is an internal function only. Do not use outside mm/.
*/
-void __vunmap_range_noflush(unsigned long start, unsigned long end)
+static void __vunmap_range_noflush_pgd(pgd_t *pgtable,
+ unsigned long start, unsigned long end)
{
unsigned long next;
pgd_t *pgd;
@@ -509,7 +516,7 @@ void __vunmap_range_noflush(unsigned long start, unsigned long end)
pgtbl_mod_mask mask = 0;
BUG_ON(addr >= end);
- pgd = pgd_offset_k(addr);
+ pgd = pgd_offset_pgd(pgtable, addr);
do {
next = pgd_addr_end(addr, end);
if (pgd_bad(*pgd))
@@ -523,6 +530,17 @@ void __vunmap_range_noflush(unsigned long start, unsigned long end)
arch_sync_kernel_mappings(start, end);
}
+/*
+ * vunmap_range_noflush is similar to __vunmap_range_noflush_pgd, but works
+ * only with init_mm->pgd.
+ *
+ * This is an internal function only. Do not use outside mm/.
+ */
+void __vunmap_range_noflush(unsigned long start, unsigned long end)
+{
+ __vunmap_range_noflush_pgd(init_mm.pgd, start, end);
+}
+
void vunmap_range_noflush(unsigned long start, unsigned long end)
{
kmsan_vunmap_range_noflush(start, end);
@@ -545,6 +563,18 @@ void vunmap_range(unsigned long addr, unsigned long end)
flush_tlb_kernel_range(addr, end);
}
+#ifdef CONFIG_KERNEL_REPLICATION
+void vunmap_range_replicas(unsigned long addr, unsigned long end)
+{
+ int nid;
+
+ flush_cache_vunmap(addr, end);
+ for_each_memory_node(nid)
+ __vunmap_range_noflush_pgd(init_mm.pgd_numa[nid], addr, end);
+ flush_tlb_kernel_range(addr, end);
+}
+#endif
+
static int vmap_pages_pte_range(pmd_t *pmd, unsigned long addr,
unsigned long end, pgprot_t prot, struct page **pages, int *nr,
pgtbl_mod_mask *mask)
@@ -643,7 +673,8 @@ static int vmap_pages_p4d_range(pgd_t *pgd, unsigned long addr,
return 0;
}
-static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end,
+static int vmap_small_pages_range_noflush_pgd(pgd_t *pgtable,
+ unsigned long addr, unsigned long end,
pgprot_t prot, struct page **pages)
{
unsigned long start = addr;
@@ -654,7 +685,7 @@ static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end,
pgtbl_mod_mask mask = 0;
BUG_ON(addr >= end);
- pgd = pgd_offset_k(addr);
+ pgd = pgd_offset_pgd(pgtable, addr);
do {
next = pgd_addr_end(addr, end);
if (pgd_bad(*pgd))
@@ -671,7 +702,7 @@ static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end,
}
/*
- * vmap_pages_range_noflush is similar to vmap_pages_range, but does not
+ * __vmap_pages_range_noflush_pgd is similar to vmap_pages_range, but does not
* flush caches.
*
* The caller is responsible for calling flush_cache_vmap() after this
@@ -679,7 +710,7 @@ static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end,
*
* This is an internal function only. Do not use outside mm/.
*/
-int __vmap_pages_range_noflush(unsigned long addr, unsigned long end,
+static int __vmap_pages_range_noflush_pgd(pgd_t *pgtable, unsigned long addr, unsigned long end,
pgprot_t prot, struct page **pages, unsigned int page_shift)
{
unsigned int i, nr = (end - addr) >> PAGE_SHIFT;
@@ -688,12 +719,13 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end,
if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) ||
page_shift == PAGE_SHIFT)
- return vmap_small_pages_range_noflush(addr, end, prot, pages);
+ return vmap_small_pages_range_noflush_pgd(pgtable, addr, end,
+ prot, pages);
for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) {
int err;
- err = vmap_range_noflush(addr, addr + (1UL << page_shift),
+ err = vmap_range_noflush_pgd(pgtable, addr, addr + (1UL << page_shift),
page_to_phys(pages[i]), prot,
page_shift);
if (err)
@@ -705,7 +737,7 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end,
return 0;
}
-int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
+static int vmap_pages_range_noflush_pgd(pgd_t *pgtable, unsigned long addr, unsigned long end,
pgprot_t prot, struct page **pages, unsigned int page_shift,
gfp_t gfp_mask)
{
@@ -714,20 +746,35 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
if (ret)
return ret;
- return __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+ return __vmap_pages_range_noflush_pgd(pgtable, addr, end, prot, pages, page_shift);
}
-static int __vmap_pages_range(unsigned long addr, unsigned long end,
+int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
+ pgprot_t prot, struct page **pages, unsigned int page_shift,
+ gfp_t gfp_mask)
+{
+ return vmap_pages_range_noflush_pgd(init_mm.pgd, addr, end, prot, pages,
+ page_shift, gfp_mask);
+}
+
+static int __vmap_pages_range_pgd(pgd_t *pgtable, unsigned long addr, unsigned long end,
pgprot_t prot, struct page **pages, unsigned int page_shift,
gfp_t gfp_mask)
{
int err;
- err = vmap_pages_range_noflush(addr, end, prot, pages, page_shift, gfp_mask);
+ err = vmap_pages_range_noflush_pgd(pgtable, addr, end, prot, pages, page_shift, gfp_mask);
flush_cache_vmap(addr, end);
return err;
}
+static int __vmap_pages_range(unsigned long addr, unsigned long end,
+ pgprot_t prot, struct page **pages, unsigned int page_shift,
+ gfp_t gfp_mask)
+{
+ return __vmap_pages_range_pgd(init_mm.pgd, addr, end, prot, pages, page_shift, gfp_mask);
+}
+
/**
* vmap_pages_range - map pages to a kernel virtual address
* @addr: start of the VM area to map
@@ -821,57 +868,12 @@ EXPORT_SYMBOL_GPL(is_vmalloc_or_module_addr);
*/
struct page *vmalloc_to_page(const void *vmalloc_addr)
{
- unsigned long addr = (unsigned long) vmalloc_addr;
- struct page *page = NULL;
- pgd_t *pgd = pgd_offset_k(addr);
- p4d_t *p4d;
- pud_t *pud;
- pmd_t *pmd;
- pte_t *ptep, pte;
-
/*
* XXX we might need to change this if we add VIRTUAL_BUG_ON for
* architectures that do not vmalloc module space
*/
VIRTUAL_BUG_ON(!is_vmalloc_or_module_addr(vmalloc_addr));
-
- if (pgd_none(*pgd))
- return NULL;
- if (WARN_ON_ONCE(pgd_leaf(*pgd)))
- return NULL; /* XXX: no allowance for huge pgd */
- if (WARN_ON_ONCE(pgd_bad(*pgd)))
- return NULL;
-
- p4d = p4d_offset(pgd, addr);
- if (p4d_none(*p4d))
- return NULL;
- if (p4d_leaf(*p4d))
- return p4d_page(*p4d) + ((addr & ~P4D_MASK) >> PAGE_SHIFT);
- if (WARN_ON_ONCE(p4d_bad(*p4d)))
- return NULL;
-
- pud = pud_offset(p4d, addr);
- if (pud_none(*pud))
- return NULL;
- if (pud_leaf(*pud))
- return pud_page(*pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT);
- if (WARN_ON_ONCE(pud_bad(*pud)))
- return NULL;
-
- pmd = pmd_offset(pud, addr);
- if (pmd_none(*pmd))
- return NULL;
- if (pmd_leaf(*pmd))
- return pmd_page(*pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
- if (WARN_ON_ONCE(pmd_bad(*pmd)))
- return NULL;
-
- ptep = pte_offset_kernel(pmd, addr);
- pte = ptep_get(ptep);
- if (pte_present(pte))
- page = pte_page(pte);
-
- return page;
+ return walk_to_page_node(first_memory_node, vmalloc_addr);
}
EXPORT_SYMBOL(vmalloc_to_page);
@@ -2490,13 +2492,39 @@ static void free_vmap_area_noflush(struct vmap_area *va)
schedule_work(&drain_vmap_work);
}
+#ifdef CONFIG_KERNEL_REPLICATION
+static void vunmap_range_noflush_replicas(unsigned long start, unsigned long end)
+{
+ if (numa_addr_has_replica((void *)start)) {
+ int node;
+ kmsan_vunmap_range_noflush(start, end);
+
+ /**
+ * In some scenarios we might clear
+ * empty entries here, which is totally fine
+ */
+ for_each_memory_node(node)
+ __vunmap_range_noflush_pgd(per_node_pgd(&init_mm, node), start, end);
+ } else {
+ vunmap_range_noflush(start, end);
+ }
+}
+#else
+static void vunmap_range_noflush_replicas(unsigned long start, unsigned long end)
+{
+ vunmap_range_noflush(start, end);
+}
+#endif
+
/*
* Free and unmap a vmap area
*/
static void free_unmap_vmap_area(struct vmap_area *va)
{
flush_cache_vunmap(va->va_start, va->va_end);
- vunmap_range_noflush(va->va_start, va->va_end);
+
+ vunmap_range_noflush_replicas(va->va_start, va->va_end);
+
if (debug_pagealloc_enabled_static())
flush_tlb_kernel_range(va->va_start, va->va_end);
@@ -3193,7 +3221,11 @@ void __init vm_area_register_early(struct vm_struct *vm, size_t align)
BUG_ON(vmap_initialized);
for (p = &vmlist; (cur = *p) != NULL; p = &cur->next) {
- if ((unsigned long)cur->addr - addr >= vm->size)
+ /* Why change this? vm_structs for kerenl text
+ * might end up outside of vmalloc region. (kernel replcation + nokaslr case)
+ * In that case, underflow will happen and normal vm will be added in the beginning
+ */
+ if ((unsigned long)cur->addr >= addr + vm->size)
break;
addr = ALIGN((unsigned long)cur->addr + cur->size, align);
}
@@ -3357,17 +3389,74 @@ struct vm_struct *remove_vm_area(const void *addr)
return vm;
}
+#ifdef CONFIG_KERNEL_REPLICATION
+static inline void set_direct_map_page_replicas(const struct vm_struct *area,
+ struct page *page,
+ int (*set_direct_map)(struct page *page))
+{
+ if (area->replicated) {
+ struct page *cursor;
+
+ list_for_each_entry(cursor, &page->lru, lru) {
+ if (page_address(cursor))
+ set_direct_map(cursor);
+ }
+ }
+}
+#endif /* CONFIG_KERNEL_REPLICATION */
+
static inline void set_area_direct_map(const struct vm_struct *area,
int (*set_direct_map)(struct page *page))
{
unsigned long i;
/* HUGE_VMALLOC passes small pages to set_direct_map */
- for (i = 0; i < area->nr_pages; i++)
+ for (i = 0; i < area->nr_pages; i++) {
if (page_address(area->pages[i]))
set_direct_map(area->pages[i]);
+#ifdef CONFIG_KERNEL_REPLICATION
+ set_direct_map_page_replicas(area,
+ area->pages[i], set_direct_map);
+#endif /* CONFIG_KERNEL_REPLICATION */
+ }
}
+#ifdef CONFIG_KERNEL_REPLICATION
+static void vm_account_replicated_range(struct vm_struct *area,
+ struct page *page,
+ unsigned long *s,
+ unsigned long *e,
+ int *flush)
+{
+ int flush_dmap = 0;
+ unsigned long start = ULONG_MAX, end = 0;
+ unsigned int page_order = vm_area_page_order(area);
+
+ if (area->replicated) {
+ struct page *cursor;
+
+ list_for_each_entry(cursor, &page->lru, lru) {
+ unsigned long addr = (unsigned long)page_address(cursor);
+
+ if (addr) {
+ unsigned long page_size;
+
+ page_size = PAGE_SIZE << page_order;
+ start = min(addr, start);
+ end = max(addr + page_size, end);
+ flush_dmap = 1;
+ }
+ }
+ }
+
+ if (flush_dmap)
+ *flush = flush_dmap;
+
+ *s = start;
+ *e = end;
+}
+#endif /* CONFIG_KERNEL_REPLICATION */
+
/*
* Flush the vm mapping and reset the direct map.
*/
@@ -3393,6 +3482,10 @@ static void vm_reset_perms(struct vm_struct *area)
end = max(addr + page_size, end);
flush_dmap = 1;
}
+#ifdef CONFIG_KERNEL_REPLICATION
+ vm_account_replicated_range(area, area->pages[i],
+ &start, &end, &flush_dmap);
+#endif /* CONFIG_KERNEL_REPLICATION */
}
/*
@@ -3438,6 +3531,30 @@ void vfree_atomic(const void *addr)
schedule_work(&p->wq);
}
+#ifdef CONFIG_KERNEL_REPLICATION
+static void vfree_page_replicas(struct vm_struct *area, struct page *page)
+{
+ if (area->replicated) {
+ struct page *cursor, *tmp;
+
+ list_for_each_entry_safe(cursor, tmp, &page->lru, lru) {
+ BUG_ON(!cursor);
+
+ list_del(&cursor->lru);
+ mod_lruvec_page_state(cursor, NR_VMALLOC, -1);
+ /*
+ * High-order allocs for huge vmallocs are split, so
+ * can be freed as an array of order-0 allocations
+ */
+ __free_pages(cursor, 0);
+ cond_resched();
+ }
+ }
+}
+#else
+static void vfree_page_replicas(struct vm_struct *area, struct page *page) { }
+#endif /* CONFIG_KERNEL_REPLICATION */
+
/*
* vm_area_free_pages - free a range of pages from a vmalloc allocation
* @vm: the vm_struct containing the pages
@@ -3455,8 +3572,10 @@ static void vm_area_free_pages(struct vm_struct *vm, unsigned long start_idx,
unsigned long i;
if (!(vm->flags & VM_MAP_PUT_PAGES)) {
- for (i = start_idx; i < end_idx; i++)
+ for (i = start_idx; i < end_idx; i++) {
+ vfree_page_replicas(vm, vm->pages[i]);
mod_lruvec_page_state(vm->pages[i], NR_VMALLOC, -1);
+ }
}
free_pages_bulk(vm->pages + start_idx, end_idx - start_idx);
@@ -3862,6 +3981,58 @@ memalloc_restore_scope(unsigned int flags)
memalloc_flags_restore(flags);
}
+static int __vmap_pages_range_retry(unsigned long addr, unsigned long end,
+ pgprot_t prot, struct page **pages, unsigned int page_shift,
+ gfp_t gfp_mask, bool nofail)
+{
+ int ret;
+
+ do {
+ ret = __vmap_pages_range(addr, end, prot, pages,
+ page_shift, gfp_mask);
+ if (nofail && (ret < 0))
+ schedule_timeout_uninterruptible(1);
+ } while (nofail && (ret < 0));
+
+ return ret;
+}
+
+#ifdef CONFIG_KERNEL_REPLICATION
+static int __vmap_pages_range_retry_replica_aware(unsigned long addr, unsigned long end,
+ pgprot_t prot, struct page **pages, unsigned int page_shift,
+ gfp_t gfp_mask, bool nofail, unsigned long flags)
+{
+ int ret;
+
+ if (flags & VM_NUMA_SHARED) {
+ int nid;
+
+ ret = kmsan_vmap_pages_range_noflush(addr, end, prot, pages,
+ page_shift, gfp_mask);
+ for_each_memory_node(nid) {
+ pgd_t *pgd = per_node_pgd(&init_mm, nid);
+
+ do {
+ ret = __vmap_pages_range_noflush_pgd(pgd, addr, end, prot, pages,
+ page_shift);
+ if (nofail && (ret < 0))
+ schedule_timeout_uninterruptible(1);
+ } while (nofail && (ret < 0));
+
+ if (ret < 0)
+ goto out;
+ }
+
+ flush_cache_vmap(addr, end);
+ } else {
+ ret = __vmap_pages_range_retry(addr, end, prot, pages,
+ page_shift, gfp_mask, nofail);
+ }
+out:
+ return ret;
+}
+#endif
+
static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,
pgprot_t prot, unsigned int page_shift,
int node)
@@ -3942,12 +4113,14 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,
* by the scope API
*/
flags = memalloc_apply_gfp_scope(gfp_mask);
- do {
- ret = __vmap_pages_range(addr, addr + size, prot, area->pages,
- page_shift, nested_gfp);
- if (nofail && (ret < 0))
- schedule_timeout_uninterruptible(1);
- } while (nofail && (ret < 0));
+
+#ifdef CONFIG_KERNEL_REPLICATION
+ ret = __vmap_pages_range_retry_replica_aware(addr, addr + size, prot, area->pages,
+ page_shift, nested_gfp, nofail, area->flags);
+#else
+ ret = __vmap_pages_range_retry(addr, addr + size, prot, area->pages,
+ page_shift, nested_gfp, nofail);
+#endif
memalloc_restore_scope(flags);
if (ret < 0) {
@@ -4076,6 +4249,16 @@ void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align,
goto fail;
}
+#ifdef CONFIG_KERNEL_REPLICATION
+ /* Check, that for requested region correct replication flag was passed */
+ if (WARN_ON_ONCE(numa_addr_has_replica(area->addr) && !(vm_flags & VM_NUMA_SHARED)))
+ vm_flags |= VM_NUMA_SHARED;
+
+ if (WARN_ON_ONCE(!numa_addr_has_replica(area->addr) && (vm_flags & VM_NUMA_SHARED)))
+ vm_flags &= ~VM_NUMA_SHARED;
+
+ area->node = node;
+#endif
/*
* Prepare arguments for __vmalloc_area_node() and
* kasan_unpoison_vmalloc().
@@ -4165,6 +4348,129 @@ void *__vmalloc_node_noprof(unsigned long size, unsigned long align,
return __vmalloc_node_range_noprof(size, align, VMALLOC_START, VMALLOC_END,
gfp_mask, PAGE_KERNEL, 0, node, caller);
}
+
+#ifdef CONFIG_KERNEL_REPLICATION
+static void numa_replicate_page_range(struct page **src, struct page **dst, int nr_pages)
+{
+ int i;
+ void *from, *to;
+
+ for (i = 0; i < nr_pages; i++) {
+ from = kmap_local_page(src[i]);
+ to = kmap_local_page(dst[i]);
+
+ copy_page(to, from);
+
+ kunmap_local(to);
+ kunmap_local(from);
+ }
+}
+
+int __vmalloc_node_replicate_range(const void *addr, gfp_t gfp_mask,
+ pgprot_t prot, unsigned long vm_flags)
+{
+ int i, ret, node = 0;
+ struct vm_struct *area;
+ unsigned int page_order;
+ unsigned int nr_allocated;
+ struct page **pages;
+ unsigned long area_start, area_end;
+ const gfp_t nested_gfp = (gfp_mask & GFP_RECLAIM_MASK) | __GFP_ZERO;
+ unsigned long array_size;
+ unsigned int flags;
+
+
+ gfp_mask |= __GFP_NOWARN;
+ if (!(gfp_mask & (GFP_DMA | GFP_DMA32)))
+ gfp_mask |= __GFP_HIGHMEM;
+
+ if (unlikely(!numa_addr_has_replica(addr)))
+ return -EINVAL;
+
+ area = find_vm_area(addr);
+ if (unlikely(!area))
+ return -ENOENT;
+
+ if (area->node == NUMA_NO_NODE)
+ return -EINVAL;
+
+ array_size = sizeof(struct page *) * area->nr_pages;
+ if (array_size > PAGE_SIZE)
+ pages = __vmalloc(array_size, nested_gfp);
+ else
+ pages = kmalloc(array_size, nested_gfp);
+
+ if (!pages)
+ return -ENOMEM;
+
+ page_order = vm_area_page_order(area);
+ for (i = 0; i < area->nr_pages; i++)
+ INIT_LIST_HEAD(&area->pages[i]->lru);
+
+ area_start = (unsigned long)area->addr;
+ area_end = (unsigned long)(area->addr + area->nr_pages * PAGE_SIZE);
+
+ for_each_memory_node(node) {
+ if (area->node == node)
+ continue;
+
+ nr_allocated = vm_area_alloc_pages(gfp_mask | __GFP_NOWARN,
+ node, page_order, area->nr_pages, pages);
+ if (nr_allocated != area->nr_pages)
+ goto fail_alloc_pages;
+
+ for (i = 0; i < area->nr_pages; i++)
+ list_add(&pages[i]->lru, &area->pages[i]->lru);
+
+ __vunmap_range_noflush_pgd(init_mm.pgd_numa[node],
+ area_start, area_end);
+
+ /*
+ * We can't fail here (hopefully)
+ * Possible errors: not enough memory for tables and not empty entries.
+ * Both unrealistic because we just cleared entries in existed tables.
+ */
+
+ numa_replicate_page_range(area->pages, pages, area->nr_pages);
+
+ flags = memalloc_apply_gfp_scope(gfp_mask);
+
+
+ ret = __vmap_pages_range_noflush_pgd(per_node_pgd(&init_mm, node), area_start,
+ area_end, prot, area->pages, page_order + PAGE_SHIFT);
+
+ memalloc_restore_scope(flags);
+ if (ret != 0)
+ goto fail_map_pages;
+
+ for (i = 0; i < area->nr_pages; i++)
+ pages[i] = NULL;
+ }
+ kvfree(pages);
+ flush_cache_vmap(area_start, area_end);
+ flush_tlb_kernel_range(area_start, area_end);
+ area->replicated = true;
+
+ return 0;
+fail_alloc_pages:
+ for (i = 0; i < nr_allocated; i++)
+ __free_pages(pages[i], 0);
+
+fail_map_pages:
+ kfree(pages);
+ for (i = 0; i < area->nr_pages; i++) {
+ struct page *page, *tmp;
+
+ list_for_each_entry_safe(page, tmp, &area->pages[i]->lru, lru) {
+ list_del(&page->lru);
+ __free_pages(page, 0);
+ }
+ }
+
+ return ret;
+}
+#endif /* CONFIG_KERNEL_REPLICATION */
+
/*
* This is only for performance analysis of vmalloc and stress purpose.
* It is required by vmalloc test module, therefore do not use it other
--
2.34.1
^ permalink raw reply related [flat|nested] 23+ messages in thread* [RFC PATCH 10/18] arm64: enable per-NUMA node kernel text and rodata replication
2026-08-27 16:11 [RFC PATCH 00/18] mm: arm64: Add kernel replication feature Nikita Panov
` (8 preceding siblings ...)
2026-08-27 16:11 ` [RFC PATCH 09/18] mm: add replicas allocation support for vmalloc Nikita Panov
@ 2026-08-27 16:11 ` Nikita Panov
2026-08-27 16:11 ` [RFC PATCH 11/18] mm: " Nikita Panov
` (9 subsequent siblings)
19 siblings, 0 replies; 23+ messages in thread
From: Nikita Panov @ 2026-08-27 16:11 UTC (permalink / raw)
To: catalin.marinas, akpm, david, ljs, vbabka, cl, linux, will,
mark.rutland, liam, rppt, surenb, mhocko
Cc: linux-mm, linux-kernel, linux-arm-kernel, wangkefeng.wang,
artem.kuzin, panov.nikita
During boot memory for replicas is allocated,
local translation tables are created,
original text and rodata are copied to replicas,
and replicas are mapped to local tables.
On startup of the secondary CPUs, after minimal initialization,
the local pgtable is loaded to ttbr1.
Acked-by: Artem Kuzin <artem.kuzin@huawei.com>
Acked-by: Alexander Grubnikov <alexander.grubnikov@huawei.com>
Acked-by: Ilya Hanov <ilya.hanov@huawei-partners.com>
Acked-by: Denis Darvish <darvish.denis@huawei.com>
Signed-off-by: Nikita Panov <panov.nikita@huawei.com>
---
arch/arm64/include/asm/pgtable.h | 4 +++
arch/arm64/kernel/smp.c | 8 ++++++
arch/arm64/mm/context.c | 1 +
arch/arm64/mm/init.c | 49 ++++++++++++++++++++++++++++++++
arch/arm64/mm/kasan_init.c | 2 ++
arch/arm64/mm/mmu.c | 38 ++++++++++++++++++++++++-
6 files changed, 101 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 6000905a2e86..f11d77829890 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -20,7 +20,11 @@
* VMALLOC_START: beginning of the kernel vmalloc space
* VMALLOC_END: extends to the available space below vmemmap
*/
+#ifdef CONFIG_KERNEL_REPLICATION
+#define VMALLOC_START ((MODULES_END & PGDIR_MASK) + PGDIR_SIZE)
+#else /* !CONFIG_KERNEL_REPLICATION */
#define VMALLOC_START (MODULES_END)
+#endif /* CONFIG_KERNEL_REPLICATION */
#if VA_BITS == VA_BITS_MIN
#define VMALLOC_END (VMEMMAP_START - SZ_8M)
#else
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index a61dc3016a11..4832a89ba992 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -36,6 +36,7 @@
#include <linux/kprobes.h>
#include <linux/kvm_host.h>
#include <linux/nmi.h>
+#include <linux/numa_kernel_replication.h>
#include <asm/alternative.h>
#include <asm/atomic.h>
@@ -208,6 +209,13 @@ asmlinkage notrace void secondary_start_kernel(void)
mmgrab(mm);
current->active_mm = mm;
+ /*
+ * Setup per-NUMA node page table if kernel
+ * replication is enabled. Option supported
+ * only for 64-bit mode.
+ */
+ numa_setup_pgd();
+
/*
* TTBR0 is only used for the identity mapping at this stage. Make it
* point to zero page to avoid speculatively fetching new entries.
diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c
index 0f4a28b87469..3afdae62784e 100644
--- a/arch/arm64/mm/context.c
+++ b/arch/arm64/mm/context.c
@@ -11,6 +11,7 @@
#include <linux/sched.h>
#include <linux/slab.h>
#include <linux/mm.h>
+#include <linux/numa_kernel_replication.h>
#include <asm/cpufeature.h>
#include <asm/mmu_context.h>
diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index fbf215ecc7d0..3c2394b20c03 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -331,6 +331,47 @@ void __init bootmem_init(void)
memblock_dump_all();
}
+#ifdef CONFIG_KERNEL_REPLICATION
+/*
+ * It is necessary to preallocate vmalloc pages in advance,
+ * otherwise the replicated page-tables can be incomplete.
+ */
+void __init preallocate_vmalloc_pages(void)
+{
+ unsigned long addr;
+
+ for (addr = MODULES_VADDR; addr <= VMALLOC_END && addr != 0UL;
+ addr = ALIGN(addr + 1, PGDIR_SIZE)) {
+ pgd_t *pgd = pgd_offset_k(addr);
+ p4d_t *p4d;
+ pud_t *pud;
+ pmd_t *pmd;
+ int pte;
+
+ p4d = p4d_alloc(&init_mm, pgd, addr);
+ /*
+ * No need to check p4d here due to
+ * only 4-stage page table is possible
+ */
+ pud = pud_alloc(&init_mm, p4d, addr);
+ if (!pud)
+ panic("Failed to pre-allocate pud pages for vmalloc area\n");
+ if (!mm_pud_folded(&init_mm))
+ continue;
+
+ pmd = pmd_alloc(&init_mm, pud, addr);
+ if (!pmd)
+ panic("Failed to pre-allocate pmd pages for vmalloc area\n");
+ if (!mm_pmd_folded(&init_mm))
+ continue;
+
+ pte = pte_alloc(&init_mm, pmd);
+ if (pte)
+ panic("Failed to pre-allocate pte pages for vmalloc area\n");
+ }
+}
+#endif /* CONFIG_KERNEL_REPLICATION */
+
void __init arch_setup_zero_pages(void)
{
__zero_page = phys_to_page(__pa_symbol(empty_zero_page));
@@ -401,7 +442,15 @@ void free_initmem(void)
* prevents the region from being reused for kernel modules, which
* is not supported by kallsyms.
*/
+#ifdef CONFIG_KERNEL_REPLICATION
+ /*
+ * In case of replicated kernel the per-NUMA node vmalloc
+ * memory should be released.
+ */
+ vunmap_range_replicas((u64)__init_begin, (u64)__init_end);
+#else
vunmap_range((u64)__init_begin, (u64)__init_end);
+#endif /* CONFIG_KERNEL_REPLICATION */
}
void dump_mem_limit(void)
diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c
index 45fbdce684c8..37eb408830f0 100644
--- a/arch/arm64/mm/kasan_init.c
+++ b/arch/arm64/mm/kasan_init.c
@@ -345,7 +345,9 @@ static void __init kasan_init_shadow(void)
kasan_populate_early_shadow(kasan_mem_to_shadow((void *)PAGE_END),
(void *)mod_shadow_start);
+#ifndef CONFIG_KERNEL_REPLICATION
BUILD_BUG_ON(VMALLOC_START != MODULES_END);
+#endif
kasan_populate_early_shadow((void *)vmalloc_shadow_end,
(void *)KASAN_SHADOW_END);
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 79d90226fd5d..fa12461c0d8e 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -30,6 +30,7 @@
#include <linux/mm_inline.h>
#include <linux/pagewalk.h>
#include <linux/stop_machine.h>
+#include <linux/numa_kernel_replication.h>
#include <asm/barrier.h>
#include <asm/cputype.h>
@@ -1039,6 +1040,22 @@ void __init create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys,
pgd_pgtable_alloc_special_mm, flags);
}
+static void populate_mappings_prot(phys_addr_t phys, unsigned long virt,
+ phys_addr_t size, pgprot_t prot)
+{
+#ifdef CONFIG_KERNEL_REPLICATION
+ int nid;
+
+ for_each_memory_node(nid) {
+ early_create_pgd_mapping(per_node_pgd(&init_mm, nid),
+ page_to_phys(walk_to_page_node(nid, (void *)virt)),
+ virt, size, prot, NULL, 0);
+ }
+#else
+ early_create_pgd_mapping(init_mm.pgd, phys, virt, size, prot, NULL, 0);
+#endif /* CONFIG_KERNEL_REPLICATION */
+}
+
static void update_mapping_prot(phys_addr_t phys, unsigned long virt,
phys_addr_t size, pgprot_t prot)
{
@@ -1048,7 +1065,7 @@ static void update_mapping_prot(phys_addr_t phys, unsigned long virt,
return;
}
- early_create_pgd_mapping(init_mm.pgd, phys, virt, size, prot, NULL, 0);
+ populate_mappings_prot(phys, virt, size, prot);
/* flush the TLBs after updating live kernel mappings */
flush_tlb_kernel_range(virt, virt + size);
@@ -1390,6 +1407,21 @@ static pgprot_t __init kernel_exec_prot(void)
return rodata_enabled ? PAGE_KERNEL_ROX : PAGE_KERNEL_EXEC;
}
+#ifdef CONFIG_KERNEL_REPLICATION
+static void __init populate_trampoline_mappings(void)
+{
+ int nid;
+
+ /* Copy trampoline mappings in replicated tables */
+ for_each_memory_node(nid) {
+ memcpy(per_node_pgd(&init_mm, nid) - (PAGE_SIZE * 2 / sizeof(pgd_t)),
+ tramp_pg_dir, PGD_SIZE);
+ }
+ /* Be sure that replicated page table can be observed properly */
+ dsb(ishst);
+}
+#endif /* CONFIG_KERNEL_REPLICATION */
+
static int __init map_entry_trampoline(void)
{
int i;
@@ -1418,6 +1450,10 @@ static int __init map_entry_trampoline(void)
__set_fixmap(FIX_ENTRY_TRAMP_TEXT1 - i,
pa_start + i * PAGE_SIZE, PAGE_KERNEL_RO);
+#ifdef CONFIG_KERNEL_REPLICATION
+ populate_trampoline_mappings();
+#endif /* CONFIG_KERNEL_REPLICATION */
+
return 0;
}
core_initcall(map_entry_trampoline);
--
2.34.1
^ permalink raw reply related [flat|nested] 23+ messages in thread* [RFC PATCH 11/18] mm: enable per-NUMA node kernel text and rodata replication
2026-08-27 16:11 [RFC PATCH 00/18] mm: arm64: Add kernel replication feature Nikita Panov
` (9 preceding siblings ...)
2026-08-27 16:11 ` [RFC PATCH 10/18] arm64: enable per-NUMA node kernel text and rodata replication Nikita Panov
@ 2026-08-27 16:11 ` Nikita Panov
2026-08-27 16:11 ` [RFC PATCH 12/18] arm64: make power management aware about kernel replication Nikita Panov
` (8 subsequent siblings)
19 siblings, 0 replies; 23+ messages in thread
From: Nikita Panov @ 2026-08-27 16:11 UTC (permalink / raw)
To: catalin.marinas, akpm, david, ljs, vbabka, cl, linux, will,
mark.rutland, liam, rppt, surenb, mhocko
Cc: linux-mm, linux-kernel, linux-arm-kernel, wangkefeng.wang,
artem.kuzin, panov.nikita
Acked-by: Artem Kuzin <artem.kuzin@huawei.com>
Acked-by: Alexander Grubnikov <alexander.grubnikov@huawei.com>
Acked-by: Ilya Hanov <ilya.hanov@huawei-partners.com>
Acked-by: Denis Darvish <darvish.denis@huawei.com>
Signed-off-by: Nikita Panov <panov.nikita@huawei.com>
---
arch/arm64/mm/context.c | 1 -
arch/arm64/mm/mmu.c | 8 ++++++--
include/linux/mm.h | 3 +++
init/main.c | 17 +++++++++++++++++
mm/mm_init.c | 3 +++
5 files changed, 29 insertions(+), 3 deletions(-)
diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c
index 3afdae62784e..0f4a28b87469 100644
--- a/arch/arm64/mm/context.c
+++ b/arch/arm64/mm/context.c
@@ -11,7 +11,6 @@
#include <linux/sched.h>
#include <linux/slab.h>
#include <linux/mm.h>
-#include <linux/numa_kernel_replication.h>
#include <asm/cpufeature.h>
#include <asm/mmu_context.h>
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index fa12461c0d8e..4bc8b748e303 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -1040,10 +1040,10 @@ void __init create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys,
pgd_pgtable_alloc_special_mm, flags);
}
+#ifdef CONFIG_KERNEL_REPLICATION
static void populate_mappings_prot(phys_addr_t phys, unsigned long virt,
phys_addr_t size, pgprot_t prot)
{
-#ifdef CONFIG_KERNEL_REPLICATION
int nid;
for_each_memory_node(nid) {
@@ -1051,10 +1051,14 @@ static void populate_mappings_prot(phys_addr_t phys, unsigned long virt,
page_to_phys(walk_to_page_node(nid, (void *)virt)),
virt, size, prot, NULL, 0);
}
+}
#else
+static void populate_mappings_prot(phys_addr_t phys, unsigned long virt,
+ phys_addr_t size, pgprot_t prot)
+{
early_create_pgd_mapping(init_mm.pgd, phys, virt, size, prot, NULL, 0);
-#endif /* CONFIG_KERNEL_REPLICATION */
}
+#endif /* CONFIG_KERNEL_REPLICATION */
static void update_mapping_prot(phys_addr_t phys, unsigned long virt,
phys_addr_t size, pgprot_t prot)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 8a7083c02203..0f3592038823 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -4240,6 +4240,9 @@ extern int __meminit early_pfn_to_nid(unsigned long pfn);
#endif
extern void mem_init(void);
+#ifdef CONFIG_KERNEL_REPLICATION
+extern void preallocate_vmalloc_pages(void);
+#endif
extern void __init mmap_init(void);
extern void __show_mem(unsigned int flags, const nodemask_t *nodemask, int max_zone_idx);
diff --git a/init/main.c b/init/main.c
index d9d936707c19..249307e46818 100644
--- a/init/main.c
+++ b/init/main.c
@@ -109,6 +109,7 @@
#include <linux/unaligned.h>
#include <linux/vdso_datastore.h>
#include <linux/hazptr.h>
+#include <linux/numa_kernel_replication.h>
#include <net/net_namespace.h>
#include <asm/io.h>
@@ -1038,12 +1039,20 @@ void start_kernel(void)
* These use large bootmem allocations and must precede
* initalization of page allocator
*/
+ numa_replication_init();
setup_log_buf(0);
vfs_caches_init_early();
sort_main_extable();
trap_init();
mm_core_init();
maple_tree_init();
+ /*
+ * Kernel text replication should be done before
+ * alloc/init first mm struct, due to it is necessary
+ * to setup per-NUMA node translation tables and kernel
+ * instances properly.
+ */
+ numa_replicate_kernel_text();
poking_init();
ftrace_init();
@@ -1571,6 +1580,14 @@ static int __ref kernel_init(void *unused)
free_initmem();
mark_readonly();
+ /*
+ * RODATA replication is done here due to
+ * it is necessary to finalize the kernel
+ * and modules initialization before
+ */
+ numa_replicate_kernel_rodata();
+ numa_replication_fini();
+
/*
* Kernel mappings are now finalized - update the userspace page-table
* to finalize PTI.
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 1533aebafb68..d2b13d8887c8 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -2679,6 +2679,9 @@ void __init mm_core_init(void)
ptlock_cache_init();
pgtable_cache_init();
debug_objects_mem_init();
+#ifdef CONFIG_KERNEL_REPLICATION
+ preallocate_vmalloc_pages();
+#endif
vmalloc_init();
/* If no deferred init page_ext now, as vmap is fully initialized */
if (!deferred_struct_pages)
--
2.34.1
^ permalink raw reply related [flat|nested] 23+ messages in thread* [RFC PATCH 12/18] arm64: make power management aware about kernel replication
2026-08-27 16:11 [RFC PATCH 00/18] mm: arm64: Add kernel replication feature Nikita Panov
` (10 preceding siblings ...)
2026-08-27 16:11 ` [RFC PATCH 11/18] mm: " Nikita Panov
@ 2026-08-27 16:11 ` Nikita Panov
2026-08-27 16:11 ` [RFC PATCH 13/18] arm64: make kernel text patching aware about replicas Nikita Panov
` (7 subsequent siblings)
19 siblings, 0 replies; 23+ messages in thread
From: Nikita Panov @ 2026-08-27 16:11 UTC (permalink / raw)
To: catalin.marinas, akpm, david, ljs, vbabka, cl, linux, will,
mark.rutland, liam, rppt, surenb, mhocko
Cc: linux-mm, linux-kernel, linux-arm-kernel, wangkefeng.wang,
artem.kuzin, panov.nikita
Acked-by: Artem Kuzin <artem.kuzin@huawei.com>
Acked-by: Alexander Grubnikov <alexander.grubnikov@huawei.com>
Acked-by: Ilya Hanov <ilya.hanov@huawei-partners.com>
Acked-by: Denis Darvish <darvish.denis@huawei.com>
Signed-off-by: Nikita Panov <panov.nikita@huawei.com>
---
arch/arm64/include/asm/mmu_context.h | 4 ++++
arch/arm64/kernel/hibernate.c | 5 +++++
arch/arm64/kernel/sleep.S | 8 ++++++++
mm/numa_kernel_replication.c | 5 +++++
4 files changed, 22 insertions(+)
diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h
index 803b68758152..f364281d3442 100644
--- a/arch/arm64/include/asm/mmu_context.h
+++ b/arch/arm64/include/asm/mmu_context.h
@@ -136,10 +136,14 @@ static inline void cpu_install_ttbr0(phys_addr_t ttbr0, unsigned long t0sz)
void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp);
+#ifndef CONFIG_KERNEL_REPLICATION
static inline void cpu_enable_swapper_cnp(void)
{
__cpu_replace_ttbr1(lm_alias(swapper_pg_dir), true);
}
+#else
+void cpu_enable_swapper_cnp(void);
+#endif
static inline void cpu_replace_ttbr1(pgd_t *pgdp)
{
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 7bf117427777..e9f5ddf37c4a 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -16,6 +16,7 @@
#include <linux/sched.h>
#include <linux/suspend.h>
#include <linux/utsname.h>
+#include <linux/numa_kernel_replication.h>
#include <asm/barrier.h>
#include <asm/cacheflush.h>
@@ -113,7 +114,11 @@ int arch_hibernation_header_save(void *addr, unsigned int max_size)
return -EOVERFLOW;
arch_hdr_invariants(&hdr->invariants);
+#ifdef CONFIG_KERNEL_REPLICATION
+ hdr->ttbr1_el1 = virt_to_phys(this_node_pgd(&init_mm));
+#else
hdr->ttbr1_el1 = __pa_symbol(swapper_pg_dir);
+#endif /* CONFIG_KERNEL_REPLICATION */
hdr->reenter_kernel = _cpu_resume;
/* We can't use __hyp_get_vectors() because kvm may still be loaded */
diff --git a/arch/arm64/kernel/sleep.S b/arch/arm64/kernel/sleep.S
index f093cdf71be1..8ad66ed70122 100644
--- a/arch/arm64/kernel/sleep.S
+++ b/arch/arm64/kernel/sleep.S
@@ -5,6 +5,10 @@
#include <asm/assembler.h>
#include <asm/smp.h>
+#ifdef CONFIG_KERNEL_REPLICATION
+.extern numa_setup_pgd
+#endif
+
.text
/*
* Implementation of MPIDR_EL1 hash algorithm through shifting
@@ -144,6 +148,10 @@ SYM_FUNC_START(_cpu_resume)
bl kasan_unpoison_task_stack_below
#endif
+#ifdef CONFIG_KERNEL_REPLICATION
+ bl numa_setup_pgd
+#endif
+
ldp x19, x20, [x29, #16]
ldp x21, x22, [x29, #32]
ldp x23, x24, [x29, #48]
diff --git a/mm/numa_kernel_replication.c b/mm/numa_kernel_replication.c
index 082aabc6b8bc..b1c5cd2a1fa1 100644
--- a/mm/numa_kernel_replication.c
+++ b/mm/numa_kernel_replication.c
@@ -67,6 +67,11 @@ static int node_to_memory_node[MAX_NUMNODES];
static bool pgtables_extra;
static DEFINE_SPINLOCK(debugfs_lock);
+void cpu_enable_swapper_cnp(void)
+{
+ __cpu_replace_ttbr1(this_node_pgd(&init_mm), true);
+}
+
bool is_text_replicated(void)
{
return text_replicated;
--
2.34.1
^ permalink raw reply related [flat|nested] 23+ messages in thread* [RFC PATCH 13/18] arm64: make kernel text patching aware about replicas
2026-08-27 16:11 [RFC PATCH 00/18] mm: arm64: Add kernel replication feature Nikita Panov
` (11 preceding siblings ...)
2026-08-27 16:11 ` [RFC PATCH 12/18] arm64: make power management aware about kernel replication Nikita Panov
@ 2026-08-27 16:11 ` Nikita Panov
2026-08-27 16:11 ` [RFC PATCH 14/18] arm64: add correct alignment to kimage in efi code Nikita Panov
` (6 subsequent siblings)
19 siblings, 0 replies; 23+ messages in thread
From: Nikita Panov @ 2026-08-27 16:11 UTC (permalink / raw)
To: catalin.marinas, akpm, david, ljs, vbabka, cl, linux, will,
mark.rutland, liam, rppt, surenb, mhocko
Cc: linux-mm, linux-kernel, linux-arm-kernel, wangkefeng.wang,
artem.kuzin, panov.nikita
Acked-by: Artem Kuzin <artem.kuzin@huawei.com>
Acked-by: Alexander Grubnikov <alexander.grubnikov@huawei.com>
Acked-by: Ilya Hanov <ilya.hanov@huawei-partners.com>
Acked-by: Denis Darvish <darvish.denis@huawei.com>
Signed-off-by: Nikita Panov <panov.nikita@huawei.com>
---
arch/arm64/kernel/alternative.c | 33 +++++++++++-
arch/arm64/kernel/patching.c | 96 +++++++++++++++++++++++++++++++++
2 files changed, 128 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/kernel/alternative.c b/arch/arm64/kernel/alternative.c
index f5ec7e7c1d3f..e85644d64bbd 100644
--- a/arch/arm64/kernel/alternative.c
+++ b/arch/arm64/kernel/alternative.c
@@ -10,6 +10,7 @@
#include <linux/init.h>
#include <linux/cpu.h>
+#include <linux/numa_kernel_replication.h>
#include <linux/elf.h>
#include <asm/cacheflush.h>
#include <asm/alternative.h>
@@ -139,6 +140,36 @@ static noinstr void clean_dcache_range_nopatch(u64 start, u64 end)
} while (cur += d_size, cur < end);
}
+#ifdef CONFIG_KERNEL_REPLICATION
+static void __write_alternatives(struct alt_instr *alt,
+ alternative_cb_t alt_cb,
+ __le32 *origptr, __le32 *updptr,
+ int nr_inst)
+{
+ if (is_text_replicated() && is_kernel_text((unsigned long)origptr)) {
+ int nid;
+
+ for_each_memory_node(nid) {
+ __le32 *ptr = numa_get_replica(origptr, nid);
+
+ alt_cb(alt, origptr, ptr, nr_inst);
+ clean_dcache_range_nopatch((u64)ptr,
+ (u64)(ptr + nr_inst));
+ }
+ } else {
+ alt_cb(alt, origptr, updptr, nr_inst);
+ }
+}
+#else
+static void __write_alternatives(struct alt_instr *alt,
+ alternative_cb_t alt_cb,
+ __le32 *origptr, __le32 *updptr,
+ int nr_inst)
+{
+ alt_cb(alt, origptr, updptr, nr_inst);
+}
+#endif /* CONFIG_KERNEL_REPLICATION */
+
static int __apply_alternatives(const struct alt_region *region,
bool is_module,
unsigned long *cpucap_mask)
@@ -174,7 +205,7 @@ static int __apply_alternatives(const struct alt_region *region,
alt_cb = patch_alternative;
}
- alt_cb(alt, origptr, updptr, nr_inst);
+ __write_alternatives(alt, alt_cb, origptr, updptr, nr_inst);
if (!is_module) {
clean_dcache_range_nopatch((u64)origptr,
diff --git a/arch/arm64/kernel/patching.c b/arch/arm64/kernel/patching.c
index 09f019c6547a..8bac14198459 100644
--- a/arch/arm64/kernel/patching.c
+++ b/arch/arm64/kernel/patching.c
@@ -5,6 +5,7 @@
#include <linux/spinlock.h>
#include <linux/stop_machine.h>
#include <linux/uaccess.h>
+#include <linux/numa_kernel_replication.h>
#include <asm/cacheflush.h>
#include <asm/fixmap.h>
@@ -15,6 +16,7 @@
static DEFINE_RAW_SPINLOCK(patch_lock);
+#ifndef CONFIG_KERNEL_REPLICATION
static bool is_exit_text(unsigned long addr)
{
/* discarded with init text/data */
@@ -42,6 +44,17 @@ static void __kprobes *patch_map(void *addr, int fixmap)
return (void *)set_fixmap_offset(fixmap, phys);
}
+#else
+static void __kprobes *patch_map(void *addr, int fixmap, int nid)
+{
+ struct page *page;
+
+ page = walk_to_page_node(nid, addr);
+ BUG_ON(!page);
+
+ return (void *)set_fixmap_offset(fixmap, page_to_phys(page) + offset_in_page(addr));
+}
+#endif /* CONFIG_KERNEL_REPLICATION */
static void __kprobes patch_unmap(int fixmap)
{
@@ -63,6 +76,28 @@ int __kprobes aarch64_insn_read(void *addr, u32 *insnp)
return ret;
}
+#ifdef CONFIG_KERNEL_REPLICATION
+static int __kprobes __aarch64_insn_write(void *addr, __le32 insn)
+{
+ int nid;
+ void *waddr = addr;
+ unsigned long flags = 0;
+ int ret;
+
+ raw_spin_lock_irqsave(&patch_lock, flags);
+ for_each_memory_node(nid) {
+ waddr = patch_map(addr, FIX_TEXT_POKE0, nid);
+ ret = copy_to_kernel_nofault(waddr, &insn, AARCH64_INSN_SIZE);
+ patch_unmap(FIX_TEXT_POKE0);
+
+ if (ret || !is_text_replicated())
+ break;
+ }
+ raw_spin_unlock_irqrestore(&patch_lock, flags);
+
+ return ret;
+}
+#else
static int __kprobes __aarch64_insn_write(void *addr, __le32 insn)
{
void *waddr = addr;
@@ -79,12 +114,37 @@ static int __kprobes __aarch64_insn_write(void *addr, __le32 insn)
return ret;
}
+#endif /* CONFIG_KERNEL_REPLICATION */
int __kprobes aarch64_insn_write(void *addr, u32 insn)
{
return __aarch64_insn_write(addr, cpu_to_le32(insn));
}
+#ifdef CONFIG_KERNEL_REPLICATION
+noinstr int aarch64_insn_write_literal_u64(void *addr, u64 val)
+{
+ int nid;
+ u64 *waddr;
+ unsigned long flags;
+ int ret;
+
+ raw_spin_lock_irqsave(&patch_lock, flags);
+ for_each_memory_node(nid) {
+ waddr = patch_map(addr, FIX_TEXT_POKE0, nid);
+
+ ret = copy_to_kernel_nofault(waddr, &val, sizeof(val));
+
+ patch_unmap(FIX_TEXT_POKE0);
+
+ if (ret || !is_text_replicated())
+ break;
+ }
+ raw_spin_unlock_irqrestore(&patch_lock, flags);
+
+ return ret;
+}
+#else
noinstr int aarch64_insn_write_literal_u64(void *addr, u64 val)
{
u64 *waddr;
@@ -101,9 +161,44 @@ noinstr int aarch64_insn_write_literal_u64(void *addr, u64 val)
return ret;
}
+#endif /* CONFIG_KERNEL_REPLICATION */
typedef void text_poke_f(void *dst, void *src, size_t patched, size_t len);
+#ifdef CONFIG_KERNEL_REPLICATION
+static void *__text_poke(text_poke_f func, void *addr, void *src, size_t len)
+{
+ unsigned long flags;
+ size_t patched = 0;
+ size_t size;
+ void *waddr;
+ void *ptr;
+
+ raw_spin_lock_irqsave(&patch_lock, flags);
+
+ while (patched < len) {
+ int nid;
+
+ ptr = addr + patched;
+ size = min(PAGE_SIZE - offset_in_page(ptr), len - patched);
+
+ for_each_memory_node(nid) {
+ waddr = patch_map(ptr, FIX_TEXT_POKE0, nid);
+ func(waddr, src, patched, size);
+ patch_unmap(FIX_TEXT_POKE0);
+
+ if (!is_text_replicated())
+ break;
+ }
+ patched += size;
+ }
+ raw_spin_unlock_irqrestore(&patch_lock, flags);
+
+ flush_icache_range((uintptr_t)addr, (uintptr_t)addr + len);
+
+ return addr;
+}
+#else
static void *__text_poke(text_poke_f func, void *addr, void *src, size_t len)
{
unsigned long flags;
@@ -130,6 +225,7 @@ static void *__text_poke(text_poke_f func, void *addr, void *src, size_t len)
return addr;
}
+#endif
static void text_poke_memcpy(void *dst, void *src, size_t patched, size_t len)
{
--
2.34.1
^ permalink raw reply related [flat|nested] 23+ messages in thread* [RFC PATCH 14/18] arm64: add correct alignment to kimage in efi code
2026-08-27 16:11 [RFC PATCH 00/18] mm: arm64: Add kernel replication feature Nikita Panov
` (12 preceding siblings ...)
2026-08-27 16:11 ` [RFC PATCH 13/18] arm64: make kernel text patching aware about replicas Nikita Panov
@ 2026-08-27 16:11 ` Nikita Panov
2026-08-27 16:11 ` [RFC PATCH 15/18] arm64: add support of NUMA replication for ptdump Nikita Panov
` (5 subsequent siblings)
19 siblings, 0 replies; 23+ messages in thread
From: Nikita Panov @ 2026-08-27 16:11 UTC (permalink / raw)
To: catalin.marinas, akpm, david, ljs, vbabka, cl, linux, will,
mark.rutland, liam, rppt, surenb, mhocko
Cc: linux-mm, linux-kernel, linux-arm-kernel, wangkefeng.wang,
artem.kuzin, panov.nikita
Acked-by: Artem Kuzin <artem.kuzin@huawei.com>
Acked-by: Alexander Grubnikov <alexander.grubnikov@huawei.com>
Acked-by: Ilya Hanov <ilya.hanov@huawei-partners.com>
Acked-by: Denis Darvish <darvish.denis@huawei.com>
Signed-off-by: Nikita Panov <panov.nikita@huawei.com>
---
arch/arm64/include/asm/efi.h | 18 ++++++++++++++++--
1 file changed, 16 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h
index e8a9783235cb..08ec4d8b48cd 100644
--- a/arch/arm64/include/asm/efi.h
+++ b/arch/arm64/include/asm/efi.h
@@ -87,8 +87,6 @@ static inline unsigned long efi_get_max_initrd_addr(unsigned long image_addr)
static inline unsigned long efi_get_kimg_min_align(void)
{
- extern bool efi_nokaslr;
-
/*
* Although relocatable kernels can fix up the misalignment with
* respect to MIN_KIMG_ALIGN, the resulting virtual text addresses are
@@ -97,7 +95,23 @@ static inline unsigned long efi_get_kimg_min_align(void)
* 2M alignment if KASLR was explicitly disabled, even if it was not
* going to be activated to begin with.
*/
+
+#ifdef CONFIG_KERNEL_REPLICATION
+ /* If kernel replication is enabled, the special alignment is necessary.
+ * Due to this fact for now we map kernel by huge pages even
+ * in case of KASLR enabled. Ugly but works.
+ */
+#ifdef CONFIG_ARM64_4K_PAGES
+ return HPAGE_SIZE;
+#else
+ return CONT_PTE_SIZE;
+#endif
+
+#else
+ extern bool efi_nokaslr;
+
return efi_nokaslr ? MIN_KIMG_ALIGN : EFI_KIMG_ALIGN;
+#endif
}
#define EFI_ALLOC_ALIGN SZ_64K
--
2.34.1
^ permalink raw reply related [flat|nested] 23+ messages in thread* [RFC PATCH 15/18] arm64: add support of NUMA replication for ptdump
2026-08-27 16:11 [RFC PATCH 00/18] mm: arm64: Add kernel replication feature Nikita Panov
` (13 preceding siblings ...)
2026-08-27 16:11 ` [RFC PATCH 14/18] arm64: add correct alignment to kimage in efi code Nikita Panov
@ 2026-08-27 16:11 ` Nikita Panov
2026-08-27 16:11 ` [RFC PATCH 16/18] arm64: add kernel modules text and rodata replication support Nikita Panov
` (4 subsequent siblings)
19 siblings, 0 replies; 23+ messages in thread
From: Nikita Panov @ 2026-08-27 16:11 UTC (permalink / raw)
To: catalin.marinas, akpm, david, ljs, vbabka, cl, linux, will,
mark.rutland, liam, rppt, surenb, mhocko
Cc: linux-mm, linux-kernel, linux-arm-kernel, wangkefeng.wang,
artem.kuzin, panov.nikita
Acked-by: Artem Kuzin <artem.kuzin@huawei.com>
Acked-by: Alexander Grubnikov <alexander.grubnikov@huawei.com>
Acked-by: Ilya Hanov <ilya.hanov@huawei-partners.com>
Acked-by: Denis Darvish <darvish.denis@huawei.com>
Signed-off-by: Nikita Panov <panov.nikita@huawei.com>
---
arch/arm64/mm/ptdump.c | 24 ++++++++++++++++++++++--
1 file changed, 22 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c
index 5a76c59b5ada..17c4de4b9b3e 100644
--- a/arch/arm64/mm/ptdump.c
+++ b/arch/arm64/mm/ptdump.c
@@ -18,6 +18,7 @@
#include <linux/ptdump.h>
#include <linux/sched.h>
#include <linux/seq_file.h>
+#include <linux/numa_kernel_replication.h>
#include <asm/fixmap.h>
#include <asm/kasan.h>
@@ -328,7 +329,7 @@ static struct ptdump_info kernel_ptdump_info __ro_after_init = {
.mm = &init_mm,
};
-bool ptdump_check_wx(void)
+static bool ptdump_check_wx_pgd(struct mm_struct *mm, pgd_t *pgd)
{
struct ptdump_pg_state st = {
.seq = NULL,
@@ -353,7 +354,7 @@ bool ptdump_check_wx(void)
}
};
- ptdump_walk_pgd(&st.ptdump, &init_mm, NULL);
+ ptdump_walk_pgd(&st.ptdump, mm, pgd);
if (st.wx_pages || st.uxn_pages) {
pr_warn("Checked W+X mappings: FAILED, %lu W+X pages found, %lu non-UXN pages found\n",
@@ -367,6 +368,25 @@ bool ptdump_check_wx(void)
}
}
+
+#ifdef CONFIG_KERNEL_REPLICATION
+bool ptdump_check_wx(void)
+{
+ bool res = false;
+ int nid;
+
+ for_each_memory_node(nid)
+ res |= ptdump_check_wx_pgd(&init_mm, per_node_pgd(&init_mm, nid));
+
+ return res;
+}
+#else
+bool ptdump_check_wx(void)
+{
+ return ptdump_check_wx_pgd(&init_mm, init_mm->pgd);
+}
+#endif
+
static int __init ptdump_init(void)
{
u64 page_offset = _PAGE_OFFSET(vabits_actual);
--
2.34.1
^ permalink raw reply related [flat|nested] 23+ messages in thread* [RFC PATCH 16/18] arm64: add kernel modules text and rodata replication support
2026-08-27 16:11 [RFC PATCH 00/18] mm: arm64: Add kernel replication feature Nikita Panov
` (14 preceding siblings ...)
2026-08-27 16:11 ` [RFC PATCH 15/18] arm64: add support of NUMA replication for ptdump Nikita Panov
@ 2026-08-27 16:11 ` Nikita Panov
2026-08-27 16:11 ` [RFC PATCH 17/18] mm: init kernel modules with " Nikita Panov
` (3 subsequent siblings)
19 siblings, 0 replies; 23+ messages in thread
From: Nikita Panov @ 2026-08-27 16:11 UTC (permalink / raw)
To: catalin.marinas, akpm, david, ljs, vbabka, cl, linux, will,
mark.rutland, liam, rppt, surenb, mhocko
Cc: linux-mm, linux-kernel, linux-arm-kernel, wangkefeng.wang,
artem.kuzin, panov.nikita
Acked-by: Artem Kuzin <artem.kuzin@huawei.com>
Acked-by: Alexander Grubnikov <alexander.grubnikov@huawei.com>
Acked-by: Ilya Hanov <ilya.hanov@huawei-partners.com>
Acked-by: Denis Darvish <darvish.denis@huawei.com>
Signed-off-by: Nikita Panov <panov.nikita@huawei.com>
---
arch/arm64/kernel/module.c | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c
index 24adb581af0e..61f605f74d86 100644
--- a/arch/arm64/kernel/module.c
+++ b/arch/arm64/kernel/module.c
@@ -18,6 +18,7 @@
#include <linux/moduleloader.h>
#include <linux/random.h>
#include <linux/scs.h>
+#include <linux/vmalloc.h>
#include <asm/alternative.h>
#include <asm/insn.h>
@@ -25,6 +26,16 @@
#include <asm/sections.h>
#include <asm/text-patching.h>
+#ifdef CONFIG_KERNEL_REPLICATION
+void module_replicate(void *ptr)
+{
+ gfp_t gfp_mask = GFP_KERNEL;
+
+ __vmalloc_node_replicate_range(ptr, gfp_mask,
+ PAGE_KERNEL, 0);
+}
+#endif /*CONFIG_KERNEL_REPLICATION*/
+
enum aarch64_reloc_op {
RELOC_OP_NONE,
RELOC_OP_ABS,
--
2.34.1
^ permalink raw reply related [flat|nested] 23+ messages in thread* [RFC PATCH 17/18] mm: init kernel modules with replication support
2026-08-27 16:11 [RFC PATCH 00/18] mm: arm64: Add kernel replication feature Nikita Panov
` (15 preceding siblings ...)
2026-08-27 16:11 ` [RFC PATCH 16/18] arm64: add kernel modules text and rodata replication support Nikita Panov
@ 2026-08-27 16:11 ` Nikita Panov
2026-08-27 16:11 ` [RFC PATCH 18/18] mm: introduce kernel cmdline option "kernel_replication=" Nikita Panov
` (2 subsequent siblings)
19 siblings, 0 replies; 23+ messages in thread
From: Nikita Panov @ 2026-08-27 16:11 UTC (permalink / raw)
To: catalin.marinas, akpm, david, ljs, vbabka, cl, linux, will,
mark.rutland, liam, rppt, surenb, mhocko
Cc: linux-mm, linux-kernel, linux-arm-kernel, wangkefeng.wang,
artem.kuzin, panov.nikita
Acked-by: Artem Kuzin <artem.kuzin@huawei.com>
Acked-by: Alexander Grubnikov <alexander.grubnikov@huawei.com>
Acked-by: Ilya Hanov <ilya.hanov@huawei-partners.com>
Acked-by: Denis Darvish <darvish.denis@huawei.com>
Signed-off-by: Nikita Panov <panov.nikita@huawei.com>
---
include/linux/moduleloader.h | 4 ++++
kernel/module/main.c | 17 ++++++++++++++++
kernel/module/strict_rwx.c | 12 +++++------
mm/execmem.c | 39 +++++++++++++++++++++++++++++++-----
4 files changed, 61 insertions(+), 11 deletions(-)
diff --git a/include/linux/moduleloader.h b/include/linux/moduleloader.h
index e395461d59e5..1d100dfc8251 100644
--- a/include/linux/moduleloader.h
+++ b/include/linux/moduleloader.h
@@ -25,6 +25,10 @@ int module_frob_arch_sections(Elf_Ehdr *hdr,
/* Additional bytes needed by arch in front of individual sections */
unsigned int arch_mod_section_prepend(struct module *mod, unsigned int section);
+#ifdef CONFIG_KERNEL_REPLICATION
+void module_replicate(void *ptr);
+#endif /* CONFIG_KERNEL_REPLICATION */
+
/* Determines if the section name is an init section (that is only used during
* module loading).
*/
diff --git a/kernel/module/main.c b/kernel/module/main.c
index d0e1e0bd2ad0..d5c8f041be48 100644
--- a/kernel/module/main.c
+++ b/kernel/module/main.c
@@ -60,6 +60,7 @@
#include <linux/codetag.h>
#include <linux/debugfs.h>
#include <linux/execmem.h>
+#include <linux/numa_kernel_replication.h>
#include <uapi/linux/module.h>
#include "internal.h"
@@ -1338,6 +1339,18 @@ void __weak module_arch_freeing_init(struct module *mod)
{
}
+#ifdef CONFIG_KERNEL_REPLICATION
+static int sections_to_replicate[] = {MOD_TEXT, MOD_RODATA};
+
+static void module_replicate_sections(struct module *mod)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(sections_to_replicate); i++)
+ module_replicate(mod->mem[sections_to_replicate[i]].base);
+}
+#endif /* CONFIG_KERNEL_REPLICATION */
+
static int module_memory_alloc(struct module *mod, enum mod_mem_type type)
{
unsigned int size = PAGE_ALIGN(mod->mem[type].size);
@@ -3330,6 +3343,10 @@ static int complete_formation(struct module *mod, struct load_info *info)
module_bug_finalize(info->hdr, info->sechdrs, mod);
module_cfi_finalize(info->hdr, info->sechdrs, mod);
+#ifdef CONFIG_KERNEL_REPLICATION
+ module_replicate_sections(mod);
+#endif
+
err = module_enable_rodata_ro(mod);
if (err)
goto out_strict_rwx;
diff --git a/kernel/module/strict_rwx.c b/kernel/module/strict_rwx.c
index 8fd438529fbc..07ad1af69eaa 100644
--- a/kernel/module/strict_rwx.c
+++ b/kernel/module/strict_rwx.c
@@ -39,9 +39,9 @@ int module_enable_text_rox(const struct module *mod)
if (mem->is_rox)
ret = execmem_restore_rox(mem->base, mem->size);
else if (IS_ENABLED(CONFIG_STRICT_MODULE_RWX))
- ret = module_set_memory(mod, type, set_memory_rox);
+ ret = module_set_memory(mod, type, numa_set_memory_rox);
else
- ret = module_set_memory(mod, type, set_memory_x);
+ ret = module_set_memory(mod, type, numa_set_memory_x);
if (ret)
return ret;
}
@@ -55,10 +55,10 @@ int module_enable_rodata_ro(const struct module *mod)
if (!IS_ENABLED(CONFIG_STRICT_MODULE_RWX) || !rodata_enabled)
return 0;
- ret = module_set_memory(mod, MOD_RODATA, set_memory_ro);
+ ret = module_set_memory(mod, MOD_RODATA, numa_set_memory_ro);
if (ret)
return ret;
- ret = module_set_memory(mod, MOD_INIT_RODATA, set_memory_ro);
+ ret = module_set_memory(mod, MOD_INIT_RODATA, numa_set_memory_ro);
if (ret)
return ret;
@@ -70,7 +70,7 @@ int module_enable_rodata_ro_after_init(const struct module *mod)
if (!IS_ENABLED(CONFIG_STRICT_MODULE_RWX) || !rodata_enabled)
return 0;
- return module_set_memory(mod, MOD_RO_AFTER_INIT, set_memory_ro);
+ return module_set_memory(mod, MOD_RO_AFTER_INIT, numa_set_memory_ro);
}
int module_enable_data_nx(const struct module *mod)
@@ -79,7 +79,7 @@ int module_enable_data_nx(const struct module *mod)
return 0;
for_class_mod_mem_type(type, data) {
- int ret = module_set_memory(mod, type, set_memory_nx);
+ int ret = module_set_memory(mod, type, numa_set_memory_nx);
if (ret)
return ret;
diff --git a/mm/execmem.c b/mm/execmem.c
index 74a178a87e75..ea4b15c8a788 100644
--- a/mm/execmem.c
+++ b/mm/execmem.c
@@ -16,6 +16,7 @@
#include <linux/set_memory.h>
#include <linux/moduleloader.h>
#include <linux/text-patching.h>
+#include <linux/numa_kernel_replication.h>
#include <asm/tlbflush.h>
@@ -26,8 +27,9 @@ static struct execmem_info *execmem_info __ro_after_init;
static struct execmem_info default_execmem_info __ro_after_init;
#ifdef CONFIG_MMU
-static void *execmem_vmalloc(struct execmem_range *range, size_t size,
- pgprot_t pgprot, unsigned long vm_flags)
+
+static void *execmem_vmalloc_node(struct execmem_range *range, size_t size,
+ pgprot_t pgprot, unsigned long vm_flags, int node)
{
bool kasan = range->flags & EXECMEM_KASAN_SHADOW;
gfp_t gfp_flags = GFP_KERNEL | __GFP_NOWARN;
@@ -40,13 +42,13 @@ static void *execmem_vmalloc(struct execmem_range *range, size_t size,
vm_flags |= VM_DEFER_KMEMLEAK;
p = __vmalloc_node_range(size, align, start, end, gfp_flags,
- pgprot, vm_flags, NUMA_NO_NODE,
+ pgprot, vm_flags, node,
__builtin_return_address(0));
if (!p && range->fallback_start) {
start = range->fallback_start;
end = range->fallback_end;
p = __vmalloc_node_range(size, align, start, end, gfp_flags,
- pgprot, vm_flags, NUMA_NO_NODE,
+ pgprot, vm_flags, node,
__builtin_return_address(0));
}
@@ -63,6 +65,26 @@ static void *execmem_vmalloc(struct execmem_range *range, size_t size,
return p;
}
+#ifdef CONFIG_KERNEL_REPLICATION
+static void *execmem_vmalloc_type(struct execmem_range *range, size_t size,
+ pgprot_t pgprot, unsigned long vm_flags, enum execmem_type type)
+{
+ if (is_text_replicated() && (type == EXECMEM_MODULE_TEXT || type == EXECMEM_MODULE_DATA))
+ /* Need to specify some numa node id for correct allocation and further replication */
+ return execmem_vmalloc_node(range, size, pgprot,
+ vm_flags | VM_NUMA_SHARED, numa_node_id());
+ else
+ return execmem_vmalloc_node(range, size, pgprot,
+ vm_flags, NUMA_NO_NODE);
+}
+#else
+static void *execmem_vmalloc_type(struct execmem_range *range, size_t size,
+ pgprot_t pgprot, unsigned long vm_flags, enum execmem_type type)
+{
+ return execmem_vmalloc_node(range, size, pgprot, vm_flags, NUMA_NO_NODE);
+}
+#endif
+
struct vm_struct *execmem_vmap(size_t size)
{
struct execmem_range *range = &execmem_info->ranges[EXECMEM_MODULE_DATA];
@@ -87,6 +109,13 @@ static void *execmem_vmalloc(struct execmem_range *range, size_t size,
#endif /* CONFIG_MMU */
#ifdef CONFIG_ARCH_HAS_EXECMEM_ROX
+
+static void *execmem_vmalloc(struct execmem_range *range, size_t size,
+ pgprot_t pgprot, unsigned long vm_flags)
+{
+ return execmem_vmalloc_node(range, size, pgprot, vm_flags, NUMA_NO_NODE);
+}
+
struct execmem_cache {
struct mutex mutex;
struct maple_tree busy_areas;
@@ -475,7 +504,7 @@ void *execmem_alloc(enum execmem_type type, size_t size)
if (use_cache)
p = execmem_cache_alloc(range, size);
else
- p = execmem_vmalloc(range, size, pgprot, vm_flags);
+ p = execmem_vmalloc_type(range, size, pgprot, vm_flags, type);
return kasan_reset_tag(p);
}
--
2.34.1
^ permalink raw reply related [flat|nested] 23+ messages in thread* [RFC PATCH 18/18] mm: introduce kernel cmdline option "kernel_replication="
2026-08-27 16:11 [RFC PATCH 00/18] mm: arm64: Add kernel replication feature Nikita Panov
` (16 preceding siblings ...)
2026-08-27 16:11 ` [RFC PATCH 17/18] mm: init kernel modules with " Nikita Panov
@ 2026-08-27 16:11 ` Nikita Panov
2026-08-27 17:25 ` [RFC PATCH 00/18] mm: arm64: Add kernel replication feature Lorenzo Stoakes (ARM)
2026-08-27 19:11 ` Matthew Wilcox
19 siblings, 0 replies; 23+ messages in thread
From: Nikita Panov @ 2026-08-27 16:11 UTC (permalink / raw)
To: catalin.marinas, akpm, david, ljs, vbabka, cl, linux, will,
mark.rutland, liam, rppt, surenb, mhocko
Cc: linux-mm, linux-kernel, linux-arm-kernel, wangkefeng.wang,
artem.kuzin, panov.nikita
The new cmdline option allows enabling/disabling
kernel replication feature.
By default, the replication feature is disabled.
This allows to set CONFIG_KERNEL_REPLICATION=y by default,
but enabling this feature is done via kernel cmdline.
Signed-off-by: Nikita Panov <panov.nikita@huawei.com>
---
.../admin-guide/kernel-parameters.txt | 7 ++
include/linux/numa_kernel_replication.h | 27 ++++++-
kernel/module/main.c | 3 +
mm/numa_kernel_replication.c | 78 ++++++++++++++++++-
mm/vmalloc.c | 2 +-
5 files changed, 113 insertions(+), 4 deletions(-)
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index d55524e3b724..133afdd7dab5 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -3039,6 +3039,13 @@ Kernel parameters
for Movable pages. "nn[KMGTPE]", "nn%", and "mirror"
are exclusive, so you cannot specify multiple forms.
+ kernel_replication=
+ [ARM64]
+ Format: [on|off]
+ If CONFIG_KERNEL_REPLICATION is set, it allows
+ enabling/disabling the kernel replication feature
+ via cmdline. Default value is off.
+
kfence.burst= [MM,KFENCE] The number of additional successive
allocations to be attempted through KFENCE for each
sample interval.
diff --git a/include/linux/numa_kernel_replication.h b/include/linux/numa_kernel_replication.h
index 34aa063d42e6..23069ca82fa8 100644
--- a/include/linux/numa_kernel_replication.h
+++ b/include/linux/numa_kernel_replication.h
@@ -18,8 +18,31 @@ extern nodemask_t replica_nodes;
nid != MAX_NUMNODES; \
nid = next_node(nid, replica_nodes))
-#define this_node_pgd(mm) ((mm)->pgd_numa[numa_node_id()])
-#define per_node_pgd(mm, nid) ((mm)->pgd_numa[nid])
+bool is_text_replicated(void);
+
+static inline pgd_t *this_node_pgd(struct mm_struct *mm)
+{
+ if (is_text_replicated())
+ return mm->pgd_numa[numa_node_id()];
+ else
+ return mm->pgd;
+}
+
+static inline pgd_t *per_node_pgd(struct mm_struct *mm, int nid)
+{
+ if (is_text_replicated())
+ return mm->pgd_numa[nid];
+ else
+ return mm->pgd;
+}
+
+static inline pgd_t **per_node_pgd_ptr(struct mm_struct *mm, int nid)
+{
+ if (is_text_replicated())
+ return &mm->pgd_numa[nid];
+ else
+ return &mm->pgd;
+}
static inline bool numa_addr_has_replica(const void *addr)
{
diff --git a/kernel/module/main.c b/kernel/module/main.c
index d5c8f041be48..25c8816588ce 100644
--- a/kernel/module/main.c
+++ b/kernel/module/main.c
@@ -1346,6 +1346,9 @@ static void module_replicate_sections(struct module *mod)
{
int i;
+ if (!is_text_replicated())
+ return;
+
for (i = 0; i < ARRAY_SIZE(sections_to_replicate); i++)
module_replicate(mod->mem[sections_to_replicate[i]].base);
}
diff --git a/mm/numa_kernel_replication.c b/mm/numa_kernel_replication.c
index b1c5cd2a1fa1..c8c3424fa20c 100644
--- a/mm/numa_kernel_replication.c
+++ b/mm/numa_kernel_replication.c
@@ -64,6 +64,8 @@ static unsigned int master_node = INT_MAX;
*/
static int node_to_memory_node[MAX_NUMNODES];
+static bool kernel_replication_enabled;
+
static bool pgtables_extra;
static DEFINE_SPINLOCK(debugfs_lock);
@@ -253,6 +255,9 @@ static void dump_pgtables(struct mm_struct *mm,
start = start & PAGE_MASK;
end = (end & PAGE_MASK) - 1 + PAGE_SIZE;
+ if (!mm->pgd_numa)
+ return;
+
replication_log(data,
"----PER-NUMA NODE KERNEL REPLICATION ENABLED----\n");
@@ -619,6 +624,18 @@ static void replicate_pgtables(void)
}
}
+static void __init numa_replicate_kernel_text_disabled(void)
+{
+ int nid;
+
+ init_mm.pgd_numa = (pgd_t **)kmalloc(sizeof(pgd_t *) * MAX_NUMNODES, GFP_PGTABLE_KERNEL);
+ BUG_ON(!init_mm.pgd_numa);
+ for_each_online_node(nid) {
+ init_mm.pgd_numa[nid] = init_mm.pgd;
+ }
+}
+
+
/*
* Kernel text replication includes two steps:
* 1. page tables replication for init_mm
@@ -636,6 +653,11 @@ void __init numa_replicate_kernel_text(void)
{
int nid;
+ if (!kernel_replication_enabled) {
+ numa_replicate_kernel_text_disabled();
+ return;
+ }
+
replicate_pgtables();
for_each_memory_node(nid) {
@@ -653,6 +675,10 @@ void numa_replicate_kernel_rodata(void)
{
int nid;
+ if (!kernel_replication_enabled) {
+ return;
+ }
+
for_each_memory_node(nid) {
if (nid == master_node)
continue;
@@ -664,7 +690,7 @@ void numa_replicate_kernel_rodata(void)
void numa_setup_pgd(void)
{
- numa_load_replicated_pgd(init_mm.pgd_numa[numa_node_id()]);
+ numa_load_replicated_pgd(this_node_pgd(&init_mm));
}
void __init_or_module *numa_get_replica(void *vaddr, int nid)
@@ -679,8 +705,48 @@ void __init_or_module *numa_get_replica(void *vaddr, int nid)
return node_desc[nid].text_vaddr + offset;
}
+static int __init setup_kernel_replication(char *str)
+{
+ int ret = 0;
+
+ if (!str)
+ goto out;
+ if (!strcmp(str, "on")) {
+ kernel_replication_enabled = true;
+ pr_info("Kernel replication enabled via cmdline\n");
+ ret = 1;
+ } else if (!strcmp(str, "off")) {
+ kernel_replication_enabled = false;
+ pr_info("Kernel replication disabled via cmdline\n");
+ ret = 1;
+ }
+out:
+ if (!ret)
+ pr_warn("kernel_replication= cannot parse, ignored\n");
+ return ret;
+}
+__setup("kernel_replication=", setup_kernel_replication);
+
+
nodemask_t __ro_after_init replica_nodes = { { [0] = 1UL } };
+/*
+ * Let us pretend, that we have only single node fore replicas.
+ * Do not replicate anything.
+ */
+static void __init numa_replication_init_disabled(void)
+{
+ int nid;
+
+ __node_set(0, &replica_nodes);
+ for_each_online_node(nid) {
+ node_to_memory_node[nid] = 0;
+ }
+
+ node_desc[0].text_vaddr = lm_alias((void *)KERNEL_TEXT_START);
+ node_desc[0].rodata_vaddr = lm_alias((void *)KERNEL_RODATA_START);
+}
+
void __init numa_replication_init(void)
{
int nid;
@@ -693,6 +759,16 @@ void __init numa_replication_init(void)
#endif
nodes_clear(replica_nodes);
+ if (kernel_replication_enabled)
+ pr_info("WARNING! WARNING! WARNING! Kernel replication enabled WARNING! WARNING! WARNING!\n");
+ else
+ pr_info("Kernel replication disabled\n");
+
+ if (!kernel_replication_enabled) {
+ numa_replication_init_disabled();
+ return;
+ }
+
for_each_node_state(nid, N_MEMORY) {
__node_set(nid, &replica_nodes);
}
diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index da6facf64db8..532298ab4263 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -4422,7 +4422,7 @@ int __vmalloc_node_replicate_range(const void *addr, gfp_t gfp_mask,
for (i = 0; i < area->nr_pages; i++)
list_add(&pages[i]->lru, &area->pages[i]->lru);
- __vunmap_range_noflush_pgd(init_mm.pgd_numa[node],
+ __vunmap_range_noflush_pgd(per_node_pgd(&init_mm, node),
area_start, area_end);
/*
--
2.34.1
^ permalink raw reply related [flat|nested] 23+ messages in thread* Re: [RFC PATCH 00/18] mm: arm64: Add kernel replication feature
2026-08-27 16:11 [RFC PATCH 00/18] mm: arm64: Add kernel replication feature Nikita Panov
` (17 preceding siblings ...)
2026-08-27 16:11 ` [RFC PATCH 18/18] mm: introduce kernel cmdline option "kernel_replication=" Nikita Panov
@ 2026-08-27 17:25 ` Lorenzo Stoakes (ARM)
2026-08-27 19:04 ` David Hildenbrand (Arm)
2026-08-27 19:11 ` Matthew Wilcox
19 siblings, 1 reply; 23+ messages in thread
From: Lorenzo Stoakes (ARM) @ 2026-08-27 17:25 UTC (permalink / raw)
To: artem.kuzin
Cc: catalin.marinas, akpm, david, vbabka, cl, linux, will,
mark.rutland, liam, rppt, surenb, mhocko, linux-mm, linux-kernel,
linux-arm-kernel, wangkefeng.wang, panov.nikita
Hi,
On Fri, Aug 28, 2026 at 12:11:40AM +0800, Nikita Panov wrote:
> Current status:
>
> There were several prior submissions with some sort of replication
> for NUMA systems, including one from our side for the x86_64 platform.
> In the last couple years, several research articles related to solving
> locality issues on NUMA machines through replication emerged as well.
>
> [1] - arm64 kernel text replication
> [2] - x86 NUMA-aware kernel replication
> [3] - x86 kernel text replication
> [4] - NUMA replication of user data
> [5] - Mitosis: Transparently Self-Replicating Page-Tables for Large-Memory Machines
> [6] - WASP: Workload-Aware Self-Replicating Page-Tables for NUMA Servers
> [7] - PaCaR: Improved Buffered I/O Locality on NUMA Systems with Page Cache Replication
> [8] - Memory page replication for Linux on X86 processors
(What happened to 9-11? :P)
> [12] - Optimize this_cpu_*() ops for non-x86 (ARM64 for this series)
>
> As of today, none of it was merged into mainline.
I mean :) maybe take that as a hint? If a number of series trying to do X get
rejected by upstream, that is maybe suggestive of barking up the wrong tree?
> However, after thorough re-evaluation, we were not able to observe
> performance improvement for the x86 platform, so we have decided to stop this
> direction and switch on arm64.
I mean, why? You should provide details here, this is quite hand-wavey. You are
also proposing core mm changes for something that seems specific to unique
hardware as far as I can tell, which is a big ask.
<snip>
> Known problems:
>
> 1. Other combinations of base page size and va size (especially with 16K pages)
> should be adapted and verified.
Umm, yeah this is basic stuff for upstreamability :)
> 2. Replicated translation tables for the vmalloc region are not local right now.
> Allocation performed with default memory policy, so translation tables
> for kernel modules will not be local. However,
> replicated text and rodata of the modules are local.
> In general, vmalloc patch should be cleaned up.
Again, this is really more of an alpha pre-RFC I'd say.
> 3. Any modifications of kernel PGD level. These modifications
> should be synchronized across all replicated tables.
> Right now, for example, memory hotplug/hotunplug
> lacks this support, vmemmap and kasan regions for
> added memory might not be observed correctly. This could be fixed
> by patching all places in the kernel where swapper_pg_dir
> is modified, or by "lazy" propagation on kernel faults in the pgd-level.
> Propagation approach will not help in the case of pgd_clear()
> on swapper_pg_dir though.
Yeah OK this suggests to me you've got the locking and synchronisation all wrong
and it's worrying :)
In general I really oppose anything that adds additional kernel page tables or
complicates kernel page table handling.
We already have singificant complexity and bugs/races emerging from people doing
odd things with kernel page tables on assumption that it's 'safe'.
You'll need very compelling evidence to justify anything that touches such
sensitive stuff.
The code is also fiddling with PGD assignment in a way that could interact badly
with how these PGDs are synchronised. These things are very subtle, and even if
it's limited to one arch the core mm code is not.
>
> Overall, this patch set in an early PoC stage and require some improvements.
>
> Overhead:
>
> Memory overhead for the kernel itself is about 30MB per NUMA node
> on our deployment. For kernel modules - depends on their sizes, but text
> and ro-data are not that big.
> CPU overhead - replication performed on the boot stage. After boot
> only "rare" operations are slowed down -
> module loading, text patching, kernel table pgd-level modifications.
Hmm. I wonder if they're as rare as you think though? It all depends also on how
slowed down they are, how that manifests, etc.
>
> Performance evaluation:
>
> Our local testing was performed on
> Kunpeng 920, 128 CPU, 4 nodes, 100Gb for each node.
Thanks for providing details of the hardware used!
>
> Microbenchmark:
> Kernel module with a huge text section (~50MB) filled with CPU-bound
> instructions. For each NUMA node thread is spawned, each thread in a loop
> executes isntructions. Total execution time of each thread is measured.
> The insmod call bound to node 0 through numactl (less time is better).
So wait, you bound it to node 0, then rely on kernel text replication to improve
performance due to a bad hint?
That seems like you could fix the issue by binding correctly? :)
>
> node 0 1 2 3
> Before time, s 5.567 7.598 13.294 18.905
> After time, s 5.469 6.960 6.777 5.531
>
> Diff ~0% -8.5% -49% -70%
> In this benchmark, interconnect was not used by any other actors,
> so microbenchmark numbers might be significantly improved.
This benchmark seems entirely synthetic and it seems odd to me, prima facie, to
implement a feature to correct for incorrect NUMA binding?
Maybe I'm missing something though.
>
> Customer's evaluation:
> We were provided with the following feedback on this patch set
> directly from our customers. Unfortunately, we do not have details
> regarding how these measurements were done other than it was
> a production setup.
> Evaulation was performed on Kunpeng 920 and 920B platforms:
> CEPH distributed storage +5%
> StarRocksDB +5%
This isn't hugely encouraging.
>
> Couple more words about patch set and technology:
>
> This patchset was merged into the innovative branch of
> the openEuler distributive 1.5 year ago (openEuler-25.03)
> and was actively tested in production environment [9], [10].
> In addition, besides the kernel part, we have published
> user space replication (for translation tables and rodata) as well,
> but it is very complex and experimental
> even compared to this patch set [11]. With replication in user
> space, we were able to achieve the following numbers in
> performance improvement:
> MySQL + sysbench 1-6%
> Spark TPC-H 4-20%
> Phoronix test-suite 0-25%
These seem very vague and wide-ranged, I'm not sure they're really saying much
at all?
>
> Discussion:
>
> The main question we'd like to discuss is the following:
> Should the kernel replication feature be merged into the Linux
> somewhere in the future? In any form, not specifically this patch set,
> but the core concept itself.
I will leave the broader topic to the NUMA experts.
>
> If the answer is yes, please share your thoughts on this patch set. What else
> should be fixed (or reimplemented and redsigned completly) in this patch
> for mainline in your opinion? We'd be glad to do it, and in that case
> I'll send an updated version in the near future.
Glancing thorugh, The patch set seems very far from being upstreamable:
if (kernel_replication_enabled)
pr_info("WARNING! WARNING! WARNING! Kernel replication enabled WARNING! WARNING! WARNING!\n");
For instance... this really shouts some alpha effort.
Annnd :) this:
if (rwsem_is_locked(&mm->mmap_lock))
locked = true;
else
mmap_read_lock(mm);
is just utterly, utterly broken.
rwsem_is_locked() can be raced at any time, you don't own the lock so it can
just be unlocked underneath you.
That you do that suggests to me you've not thought about locks correctly _at
all_ here.
And as I said above, locking issues around page table walking and manipulation
is very subtle and difficult to manage correctly.
Then there's stuff like this:
/* TODO: remove last condition and do something better
* In the case of a folded P4D level, pgd_none and pgd_huge
* always return 0, so we might start to replicate empty entries.
* We obviously want to avoid this, so the last check is performed here.
*/
if (pgd_none(*orig_pgd) || pgd_val(*orig_pgd) == 0)
goto skip;
This whole block seems confused, and it's nitty but you're using completely
incorrect comment style for the linux kernel which again doesn't fill me with
confidence that you've really thought things through or understand mm code
correctly.
Your replicate_memory() function seems to not synchronise _at all_, but you do
for some reason synchronise on dumping memory, bizarely.
You write a ton of duplicative page table code that doesn't seem to handle huge
pages at all, again doesn't seem to be performing any locking correctly at all,
and yeah the list goes on.
In general the code looks like an alpha experimental thing and a million miles
away from anything even vaguely upstreamable.
And in general for this kind of thing - the devil is in the detail.
So if you want to assert that something is viable, you need code that at
least looks _vaguely_ upstreamable and demonstates understanding of the
issues at play here, and you are not doing so.
In conclusion:
- 12 (or is it 9? :) attempts have been made at this kind of thing and all
were rejected - this isn't an encouraging sign that the approach is
viable.
- You've already found it has limited use, and your benchmark numbers seem
either entirely synthetic or bordeline not statistically significant.
- The code is, as discussed, not even vaguely close to being upstreamable.
So overall it seems to me that perhaps better NUMA policy decisions could
solve your problems.
But yeah, what's presented in this series doesn't seem like a worthwhile
road to travel down to me.
--
Cheers, Lorenzo
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [RFC PATCH 00/18] mm: arm64: Add kernel replication feature
2026-08-27 17:25 ` [RFC PATCH 00/18] mm: arm64: Add kernel replication feature Lorenzo Stoakes (ARM)
@ 2026-08-27 19:04 ` David Hildenbrand (Arm)
0 siblings, 0 replies; 23+ messages in thread
From: David Hildenbrand (Arm) @ 2026-08-27 19:04 UTC (permalink / raw)
To: Lorenzo Stoakes (ARM), artem.kuzin
Cc: catalin.marinas, akpm, vbabka, cl, linux, will, mark.rutland,
liam, rppt, surenb, mhocko, linux-mm, linux-kernel,
linux-arm-kernel, wangkefeng.wang, panov.nikita
[...]
About to go on PTO, some random points :)
>
>> [12] - Optimize this_cpu_*() ops for non-x86 (ARM64 for this series)
>>
>> As of today, none of it was merged into mainline.
>
> I mean :) maybe take that as a hint? If a number of series trying to do X get
> rejected by upstream, that is maybe suggestive of barking up the wrong tree?
I'd have expected an explanation at that point why previous approaches were not
merged and how this patch set is sufficiently different that it is worth
maintainer's time.
>
>> However, after thorough re-evaluation, we were not able to observe
>> performance improvement for the x86 platform, so we have decided to stop this
>> direction and switch on arm64.
>
> I mean, why? You should provide details here, this is quite hand-wavey. You are
> also proposing core mm changes for something that seems specific to unique
> hardware as far as I can tell, which is a big ask.
What I consider interesting is "after thorough re-evaluation, we were not able
to observe performance improvement for the x86 platform".
How confident are we that a thorough re-evaluation on arm64 will not similarly
have the same outcome? IOW, what was the problematic part when doing the x86
evaluation, and how was that avoided when evaluating the arm64 implementation?
--
Cheers,
David
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [RFC PATCH 00/18] mm: arm64: Add kernel replication feature
2026-08-27 16:11 [RFC PATCH 00/18] mm: arm64: Add kernel replication feature Nikita Panov
` (18 preceding siblings ...)
2026-08-27 17:25 ` [RFC PATCH 00/18] mm: arm64: Add kernel replication feature Lorenzo Stoakes (ARM)
@ 2026-08-27 19:11 ` Matthew Wilcox
19 siblings, 0 replies; 23+ messages in thread
From: Matthew Wilcox @ 2026-08-27 19:11 UTC (permalink / raw)
To: artem.kuzin
Cc: catalin.marinas, akpm, david, ljs, vbabka, cl, linux, will,
mark.rutland, liam, rppt, surenb, mhocko, linux-mm, linux-kernel,
linux-arm-kernel, wangkefeng.wang, panov.nikita
On Fri, Aug 28, 2026 at 12:11:40AM +0800, Nikita Panov wrote:
> There were several prior submissions with some sort of replication
> for NUMA systems, including one from our side for the x86_64 platform.
> In the last couple years, several research articles related to solving
> locality issues on NUMA machines through replication emerged as well.
You're lumping a lot of different NUMA replication things together in
your list. That doesn't give me confidence you know what you're doing.
I *think* what you're doing here is kernel text replication. Other
architectures (mips, arm) have managed to do this without core MM
changes. Why can't you?
^ permalink raw reply [flat|nested] 23+ messages in thread