From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A433FC61DC6 for ; Thu, 27 Aug 2026 16:25:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Reply-To:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:CC:To:From:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=gzG5ol8Yr6RKYwTAU+BAIanpIkNMA3tETIj6mvgS0RQ=; b=hWmZEoaba47TzinHqZ1NlD23uD myz0z8eIRSMTh0nOGpH+2REpHezRpy/jz2ctxXi2fHwlJMF24ZqBy02waL3wjo+KQaa9Ptmy0ZBje Bmggn6QOxlWQKutLJemRQIHIX2HFjZEPlN++YoJYQac50ESRZT0ZOExafbKkoXLBHpiMy1OtOEbrA cF8sfwLMRU6YgxhAb2eJoRdIFWRKt9G21oSMK/ai5LERZFGgvfYvemJiBGw2yRFoSNhPnNMUIhutk geWyAdI3P9ipsJNy97jMnPO6jLww7/PzItgETazcUO+hYH/PkSFNYNMA+GY/JQQgiaF6LG9fSKgs1 7hr/lKDQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.99.1 #2 (Red Hat Linux)) id 1wzcuP-00000004JvO-3RfF; Thu, 27 Aug 2026 16:24:57 +0000 Received: from frasgout.his.huawei.com ([185.176.79.56]) by bombadil.infradead.org with esmtps (Exim 4.99.1 #2 (Red Hat Linux)) id 1wzcuG-00000004JkE-3sUu for linux-arm-kernel@lists.infradead.org; Thu, 27 Aug 2026 16:24:51 +0000 dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=gzG5ol8Yr6RKYwTAU+BAIanpIkNMA3tETIj6mvgS0RQ=; b=F63kYZZRaWckjPUhtZOR2u+9xVDb4ayB+XJ7djeLCD8+HZj9Ituii5PFkTq09FW8YQWa/kROT 427t61dwdHdiaK2yyzTeCLrtvj4uBWR5I3vJ1UmSPJ72M17crIgWvQ4lDFSBleLV0vCWWAvY/PI IwFkSVRGhZ2SPjPU+HD5Af8= Received: from mail.maildlp.com (unknown [172.18.224.107]) by frasgout.his.huawei.com (SkyGuard) with ESMTPS id 4hW6KW4D2GzJ46Yq; Fri, 28 Aug 2026 00:24:15 +0800 (CST) Received: from mscpeml500004.china.huawei.com (unknown [7.188.26.250]) by mail.maildlp.com (Postfix) with ESMTPS id 25DEC4058D; Fri, 28 Aug 2026 00:24:37 +0800 (CST) Received: from mscphis00949.huawei.com (10.123.66.11) by mscpeml500004.china.huawei.com (7.188.26.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Thu, 27 Aug 2026 19:24:36 +0300 From: Nikita Panov To: , , , , , , , , , , , , CC: , , , , , Subject: [RFC PATCH 10/18] arm64: enable per-NUMA node kernel text and rodata replication Date: Fri, 28 Aug 2026 00:11:50 +0800 Message-ID: <20260827161158.3618409-11-panov.nikita@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260827161158.3618409-1-panov.nikita@huawei.com> References: <20260827161158.3618409-1-panov.nikita@huawei.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.123.66.11] X-ClientProxiedBy: mscpeml100003.china.huawei.com (10.199.174.67) To mscpeml500004.china.huawei.com (7.188.26.250) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.9.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20260827_092449_301151_BD64D765 X-CRM114-Status: GOOD ( 19.88 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: artem.kuzin@huawei.com Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org During boot memory for replicas is allocated, local translation tables are created, original text and rodata are copied to replicas, and replicas are mapped to local tables. On startup of the secondary CPUs, after minimal initialization, the local pgtable is loaded to ttbr1. Acked-by: Artem Kuzin Acked-by: Alexander Grubnikov Acked-by: Ilya Hanov Acked-by: Denis Darvish Signed-off-by: Nikita Panov --- arch/arm64/include/asm/pgtable.h | 4 +++ arch/arm64/kernel/smp.c | 8 ++++++ arch/arm64/mm/context.c | 1 + arch/arm64/mm/init.c | 49 ++++++++++++++++++++++++++++++++ arch/arm64/mm/kasan_init.c | 2 ++ arch/arm64/mm/mmu.c | 38 ++++++++++++++++++++++++- 6 files changed, 101 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 6000905a2e86..f11d77829890 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -20,7 +20,11 @@ * VMALLOC_START: beginning of the kernel vmalloc space * VMALLOC_END: extends to the available space below vmemmap */ +#ifdef CONFIG_KERNEL_REPLICATION +#define VMALLOC_START ((MODULES_END & PGDIR_MASK) + PGDIR_SIZE) +#else /* !CONFIG_KERNEL_REPLICATION */ #define VMALLOC_START (MODULES_END) +#endif /* CONFIG_KERNEL_REPLICATION */ #if VA_BITS == VA_BITS_MIN #define VMALLOC_END (VMEMMAP_START - SZ_8M) #else diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index a61dc3016a11..4832a89ba992 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -36,6 +36,7 @@ #include #include #include +#include #include #include @@ -208,6 +209,13 @@ asmlinkage notrace void secondary_start_kernel(void) mmgrab(mm); current->active_mm = mm; + /* + * Setup per-NUMA node page table if kernel + * replication is enabled. Option supported + * only for 64-bit mode. + */ + numa_setup_pgd(); + /* * TTBR0 is only used for the identity mapping at this stage. Make it * point to zero page to avoid speculatively fetching new entries. diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c index 0f4a28b87469..3afdae62784e 100644 --- a/arch/arm64/mm/context.c +++ b/arch/arm64/mm/context.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index fbf215ecc7d0..3c2394b20c03 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -331,6 +331,47 @@ void __init bootmem_init(void) memblock_dump_all(); } +#ifdef CONFIG_KERNEL_REPLICATION +/* + * It is necessary to preallocate vmalloc pages in advance, + * otherwise the replicated page-tables can be incomplete. + */ +void __init preallocate_vmalloc_pages(void) +{ + unsigned long addr; + + for (addr = MODULES_VADDR; addr <= VMALLOC_END && addr != 0UL; + addr = ALIGN(addr + 1, PGDIR_SIZE)) { + pgd_t *pgd = pgd_offset_k(addr); + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + int pte; + + p4d = p4d_alloc(&init_mm, pgd, addr); + /* + * No need to check p4d here due to + * only 4-stage page table is possible + */ + pud = pud_alloc(&init_mm, p4d, addr); + if (!pud) + panic("Failed to pre-allocate pud pages for vmalloc area\n"); + if (!mm_pud_folded(&init_mm)) + continue; + + pmd = pmd_alloc(&init_mm, pud, addr); + if (!pmd) + panic("Failed to pre-allocate pmd pages for vmalloc area\n"); + if (!mm_pmd_folded(&init_mm)) + continue; + + pte = pte_alloc(&init_mm, pmd); + if (pte) + panic("Failed to pre-allocate pte pages for vmalloc area\n"); + } +} +#endif /* CONFIG_KERNEL_REPLICATION */ + void __init arch_setup_zero_pages(void) { __zero_page = phys_to_page(__pa_symbol(empty_zero_page)); @@ -401,7 +442,15 @@ void free_initmem(void) * prevents the region from being reused for kernel modules, which * is not supported by kallsyms. */ +#ifdef CONFIG_KERNEL_REPLICATION + /* + * In case of replicated kernel the per-NUMA node vmalloc + * memory should be released. + */ + vunmap_range_replicas((u64)__init_begin, (u64)__init_end); +#else vunmap_range((u64)__init_begin, (u64)__init_end); +#endif /* CONFIG_KERNEL_REPLICATION */ } void dump_mem_limit(void) diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index 45fbdce684c8..37eb408830f0 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -345,7 +345,9 @@ static void __init kasan_init_shadow(void) kasan_populate_early_shadow(kasan_mem_to_shadow((void *)PAGE_END), (void *)mod_shadow_start); +#ifndef CONFIG_KERNEL_REPLICATION BUILD_BUG_ON(VMALLOC_START != MODULES_END); +#endif kasan_populate_early_shadow((void *)vmalloc_shadow_end, (void *)KASAN_SHADOW_END); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 79d90226fd5d..fa12461c0d8e 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -30,6 +30,7 @@ #include #include #include +#include #include #include @@ -1039,6 +1040,22 @@ void __init create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, pgd_pgtable_alloc_special_mm, flags); } +static void populate_mappings_prot(phys_addr_t phys, unsigned long virt, + phys_addr_t size, pgprot_t prot) +{ +#ifdef CONFIG_KERNEL_REPLICATION + int nid; + + for_each_memory_node(nid) { + early_create_pgd_mapping(per_node_pgd(&init_mm, nid), + page_to_phys(walk_to_page_node(nid, (void *)virt)), + virt, size, prot, NULL, 0); + } +#else + early_create_pgd_mapping(init_mm.pgd, phys, virt, size, prot, NULL, 0); +#endif /* CONFIG_KERNEL_REPLICATION */ +} + static void update_mapping_prot(phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot) { @@ -1048,7 +1065,7 @@ static void update_mapping_prot(phys_addr_t phys, unsigned long virt, return; } - early_create_pgd_mapping(init_mm.pgd, phys, virt, size, prot, NULL, 0); + populate_mappings_prot(phys, virt, size, prot); /* flush the TLBs after updating live kernel mappings */ flush_tlb_kernel_range(virt, virt + size); @@ -1390,6 +1407,21 @@ static pgprot_t __init kernel_exec_prot(void) return rodata_enabled ? PAGE_KERNEL_ROX : PAGE_KERNEL_EXEC; } +#ifdef CONFIG_KERNEL_REPLICATION +static void __init populate_trampoline_mappings(void) +{ + int nid; + + /* Copy trampoline mappings in replicated tables */ + for_each_memory_node(nid) { + memcpy(per_node_pgd(&init_mm, nid) - (PAGE_SIZE * 2 / sizeof(pgd_t)), + tramp_pg_dir, PGD_SIZE); + } + /* Be sure that replicated page table can be observed properly */ + dsb(ishst); +} +#endif /* CONFIG_KERNEL_REPLICATION */ + static int __init map_entry_trampoline(void) { int i; @@ -1418,6 +1450,10 @@ static int __init map_entry_trampoline(void) __set_fixmap(FIX_ENTRY_TRAMP_TEXT1 - i, pa_start + i * PAGE_SIZE, PAGE_KERNEL_RO); +#ifdef CONFIG_KERNEL_REPLICATION + populate_trampoline_mappings(); +#endif /* CONFIG_KERNEL_REPLICATION */ + return 0; } core_initcall(map_entry_trampoline); -- 2.34.1