From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8DA63C61DF4 for ; Fri, 24 Nov 2023 10:23:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=1Goug3pQKwUUc9DN7+5ISjWsBsIe2zJNSWQMLZ9wujw=; b=mVhiKerFUbbaZXpTA/hTEsJfMD 0EAqDOu5wEkPB/29ppc4DHrpLobpweT/nY1lezWq/gqS1XovL1xH6oaPQ5Rin78QuGApHB6glLu31 PgUOLIil6r+MGNnrH9uIdbcqLbydOnqWAzPj7jGPhG8CZ6Qrtk2P3oRT9yKy/88sh5OrXT9cR85TH 4uZKopIL0CYw8Ia+NdbN0d0zcFELgFwLedukPMskockuaMFDfln+43MjSmP3CBwYRPLrWgvZ92V4d Dlcq4rshKMewZncWId1QhrUMOVd9SP9IZ4QdVMKz0rS1S/SDlvWqjd04y1X6ImaVWlUT9o4BPv2xf 9T2Ix2Xg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TKz-006psf-0N; Fri, 24 Nov 2023 10:23:05 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJi-006pNs-0o for linux-arm-kernel@bombadil.infradead.org; Fri, 24 Nov 2023 10:21:46 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=AVRRN4DuFbRb3wrfT0sNswvm342zPXmkt8+v7cdul60=; b=HZyc/PRqROWYgfP5My5HTW/VVn smvFCP1q+gAFiLhTF73yOJITwaMkjAGWOGEpnq4oXwdh/3PxmbIGtwdrq4tdq2/fhCIsedzrMKYy8 znWVT95hG0UzmqCrELCS3gzI0ygHYnAu89OsnAfM+TbCqeEGA6D4G6nNWYsUhDL25U0+KnDWJTFCp yp4vvRsEewNCMr1ftNp3oMVSV9n/se0KRRYh3IKHbfF0HWiSLpDXDgtubaPz9092zF9qtLC6GqEbr xO1qX9bsBb/5z+QGzkHNOUjQZU7iHi2NXyeFy0SSUNF2FJzqBHh4oVib2zFt8S1g57/YvM3E/SGRY XRlg/bXw==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1r6TJe-008SuG-Gp for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:21:45 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5ae5b12227fso22947017b3.0 for ; Fri, 24 Nov 2023 02:21:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821300; x=1701426100; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=AVRRN4DuFbRb3wrfT0sNswvm342zPXmkt8+v7cdul60=; b=te8t9m6Gk+4ZKgGpV/TguoGSGwyjiSDv2nlZapKfWE8P9DIIr0+W8rNL+yOJYQ5j6l dZbrZqMre3+NX8V6te7LC5rA5o7czF20DP0MU7mbiV7e/dGp8Mk78gBoS+tY3Bm50o2e vYKj9iZcEE1Ha1pYygtJ1GxsotPd7qW9Wio/BWhuM8oxJi+IybYWv4Xp6Kq4y6R9BEuK JuYzhH/iXG+GMgj31IeHbZv87AXKg0Xoa+F1ZT8lNw7DKrasObBKVb0VL0Oe4dkghlEV NecVqOYh6jhwcIOuZSejz3un+rac/VMJOwuUVlQFF7s3Qjqnw3R1gIQJHlouqI2Vh3iT 7G6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821300; x=1701426100; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AVRRN4DuFbRb3wrfT0sNswvm342zPXmkt8+v7cdul60=; b=plrhBD3W0zPNF/qbhN1fT8hVpK83pq5ZxcHf900LPCS5Bbyqqj/jQ4p6RCa2iZaqLU eFFSHYqjlT8q4jHctNf7uZouqnz4l7DRKUaENDHSRhPtL2kGaiTD5PGyKRHo+Rov70aw OefpDwItVyF5QkpOjEucz5/Lsiqg+0jzyDr66NCvePAqEzh0gWHEqfzSkygQZQeQlwa5 CZT03GAbj+MPqNP3wQlcQblPeIN+T/d5y+xc2Xn5HLLUYVMTw9mqx37LSFdUPySRWS7R Jd9/UbPhw9E7EN7c2Kgmvp7O7LqYJwmqnswm81wVU1C2YSg1fazveUo4R0ry0mBLe16R 3RSA== X-Gm-Message-State: AOJu0YyEsX/P222WPhyrfY4o5lNOFiP5GrSBV/XIVO9+OoZoBhp2fCym be7DJTIrVP53LCbGWraZxhuNnpzYqB0WrHr/0z9syszhNZsQSw0jeBs9D53Tgq9NoKSyEut7UvQ mfLGb1upTn6I/QELhqs9ExlUP288uBxvLB4tK3C+oFN3nGVct7azcF4LHWoljqV81kA0M9b4fCt o= X-Google-Smtp-Source: AGHT+IF8qokzwi8ONAOq1Gj44a7z5Urdn56cOdHGdCb/yja+TbjFXM72WNAyb32v6KcAw6EwIW00SOXv X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:ab8e:0:b0:db4:5db7:9519 with SMTP id v14-20020a25ab8e000000b00db45db79519mr56770ybi.3.1700821300151; Fri, 24 Nov 2023 02:21:40 -0800 (PST) Date: Fri, 24 Nov 2023 11:19:18 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=10407; i=ardb@kernel.org; h=from:subject; bh=Yx1U2ztevh8h0bIECIvTnyCuqgsmKWI951lILqnMOJs=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYrP15WsZYTaX/8oxVWrJ21sv+qvrcOCMyA+r9pTXC aJPl3l2lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgInsOcTIcNAj/TT3v5oTNpF9 FczLzihsOOGkN+vv9ZbuE0cCGJ17HjP801XO/HbSokX0ZrToouPTfEtmuITMSk/XCRA8qHjLRGA zOwA= X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-79-ardb@google.com> Subject: [PATCH v5 38/39] arm64: mm: omit redundant remap of kernel image From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_102142_626466_7626D6C2 X-CRM114-Status: GOOD ( 20.70 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Now that the early kernel mapping is created with all the right attributes and segment boundaries, there is no longer a need to recreate it and switch to it. This also means we no longer have to copy the kasan shadow or some parts of the fixmap from one set of page tables to the other. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/fixmap.h | 1 - arch/arm64/include/asm/kasan.h | 2 - arch/arm64/include/asm/mmu.h | 2 +- arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kernel/pi/map_kernel.c | 6 +- arch/arm64/mm/fixmap.c | 34 -------- arch/arm64/mm/kasan_init.c | 15 ---- arch/arm64/mm/mmu.c | 85 ++++---------------- 8 files changed, 21 insertions(+), 125 deletions(-) diff --git a/arch/arm64/include/asm/fixmap.h b/arch/arm64/include/asm/fixmap.h index 58c294a96676..8aabd45e9a13 100644 --- a/arch/arm64/include/asm/fixmap.h +++ b/arch/arm64/include/asm/fixmap.h @@ -100,7 +100,6 @@ enum fixed_addresses { #define FIXMAP_PAGE_IO __pgprot(PROT_DEVICE_nGnRE) void __init early_fixmap_init(void); -void __init fixmap_copy(pgd_t *pgdir); #define __early_set_fixmap __set_fixmap diff --git a/arch/arm64/include/asm/kasan.h b/arch/arm64/include/asm/kasan.h index 12d5f47f7dbe..ab52688ac4bd 100644 --- a/arch/arm64/include/asm/kasan.h +++ b/arch/arm64/include/asm/kasan.h @@ -36,12 +36,10 @@ void kasan_init(void); #define _KASAN_SHADOW_START(va) (KASAN_SHADOW_END - (1UL << ((va) - KASAN_SHADOW_SCALE_SHIFT))) #define KASAN_SHADOW_START _KASAN_SHADOW_START(vabits_actual) -void kasan_copy_shadow(pgd_t *pgdir); asmlinkage void kasan_early_init(void); #else static inline void kasan_init(void) { } -static inline void kasan_copy_shadow(pgd_t *pgdir) { } #endif #endif diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index d0b8b4b413b6..65977c7783c5 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -110,7 +110,7 @@ static inline bool kaslr_requires_kpti(void) } #define INIT_MM_CONTEXT(name) \ - .pgd = init_pg_dir, + .pgd = swapper_pg_dir, #endif /* !__ASSEMBLY__ */ #endif diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 941a14c05184..e140c5bda90b 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -55,6 +55,7 @@ PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed); PROVIDE(__pi_init_idmap_pg_dir = init_idmap_pg_dir); PROVIDE(__pi_init_pg_dir = init_pg_dir); PROVIDE(__pi_init_pg_end = init_pg_end); +PROVIDE(__pi_swapper_pg_dir = swapper_pg_dir); PROVIDE(__pi__text = _text); PROVIDE(__pi__stext = _stext); diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index a9aad3a8b575..d5b345d68f07 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -125,8 +125,12 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) text_prot, true, root_level); map_segment(init_pg_dir, NULL, va_offset, __inittext_begin, __inittext_end, text_prot, false, root_level); - dsb(ishst); } + + /* Copy the root page table to its final location */ + memcpy((void *)swapper_pg_dir + va_offset, init_pg_dir, PGD_SIZE); + dsb(ishst); + idmap_cpu_replace_ttbr1(swapper_pg_dir); } static void __init map_fdt(u64 fdt) diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index c0a3301203bd..9436a12e1882 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -167,37 +167,3 @@ void *__init fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot) return dt_virt; } - -/* - * Copy the fixmap region into a new pgdir. - */ -void __init fixmap_copy(pgd_t *pgdir) -{ - if (!READ_ONCE(pgd_val(*pgd_offset_pgd(pgdir, FIXADDR_TOT_START)))) { - /* - * The fixmap falls in a separate pgd to the kernel, and doesn't - * live in the carveout for the swapper_pg_dir. We can simply - * re-use the existing dir for the fixmap. - */ - set_pgd(pgd_offset_pgd(pgdir, FIXADDR_TOT_START), - READ_ONCE(*pgd_offset_k(FIXADDR_TOT_START))); - } else if (CONFIG_PGTABLE_LEVELS > 3) { - pgd_t *bm_pgdp; - p4d_t *bm_p4dp; - pud_t *bm_pudp; - /* - * The fixmap shares its top level pgd entry with the kernel - * mapping. This can really only occur when we are running - * with 16k/4 levels, so we can simply reuse the pud level - * entry instead. - */ - BUG_ON(!IS_ENABLED(CONFIG_ARM64_16K_PAGES)); - bm_pgdp = pgd_offset_pgd(pgdir, FIXADDR_TOT_START); - bm_p4dp = p4d_offset(bm_pgdp, FIXADDR_TOT_START); - bm_pudp = pud_set_fixmap_offset(bm_p4dp, FIXADDR_TOT_START); - pud_populate(&init_mm, bm_pudp, lm_alias(bm_pmd)); - pud_clear_fixmap(); - } else { - BUG(); - } -} diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index 555285ebd5af..dd91f5942fd2 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -184,21 +184,6 @@ static void __init kasan_map_populate(unsigned long start, unsigned long end, kasan_pgd_populate(start & PAGE_MASK, PAGE_ALIGN(end), node, false); } -/* - * Copy the current shadow region into a new pgdir. - */ -void __init kasan_copy_shadow(pgd_t *pgdir) -{ - pgd_t *pgdp, *pgdp_new, *pgdp_end; - - pgdp = pgd_offset_k(KASAN_SHADOW_START); - pgdp_end = pgd_offset_k(KASAN_SHADOW_END); - pgdp_new = pgd_offset_pgd(pgdir, KASAN_SHADOW_START); - do { - set_pgd(pgdp_new, READ_ONCE(*pgdp)); - } while (pgdp++, pgdp_new++, pgdp != pgdp_end); -} - static void __init clear_pgds(unsigned long start, unsigned long end) { diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index d36ba0811f5e..bb9e13d54a5d 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -648,9 +648,9 @@ void mark_rodata_ro(void) debug_checkwx(); } -static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, - pgprot_t prot, struct vm_struct *vma, - int flags, unsigned long vm_flags) +static void __init declare_vma(struct vm_struct *vma, + void *va_start, void *va_end, + unsigned long vm_flags) { phys_addr_t pa_start = __pa_symbol(va_start); unsigned long size = va_end - va_start; @@ -658,9 +658,6 @@ static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, BUG_ON(!PAGE_ALIGNED(pa_start)); BUG_ON(!PAGE_ALIGNED(size)); - __create_pgd_mapping(pgdp, pa_start, (unsigned long)va_start, size, prot, - early_pgtable_alloc, flags); - if (!(vm_flags & VM_NO_GUARD)) size += PAGE_SIZE; @@ -673,12 +670,12 @@ static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, vm_area_add_early(vma); } +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 static pgprot_t kernel_exec_prot(void) { return rodata_enabled ? PAGE_KERNEL_ROX : PAGE_KERNEL_EXEC; } -#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 static int __init map_entry_trampoline(void) { int i; @@ -710,60 +707,17 @@ core_initcall(map_entry_trampoline); #endif /* - * Open coded check for BTI, only for use to determine configuration - * for early mappings for before the cpufeature code has run. - */ -static bool arm64_early_this_cpu_has_bti(void) -{ - u64 pfr1; - - if (!IS_ENABLED(CONFIG_ARM64_BTI_KERNEL)) - return false; - - pfr1 = __read_sysreg_by_encoding(SYS_ID_AA64PFR1_EL1); - return cpuid_feature_extract_unsigned_field(pfr1, - ID_AA64PFR1_EL1_BT_SHIFT); -} - -/* - * Create fine-grained mappings for the kernel. + * Declare the VMA areas for the kernel */ -static void __init map_kernel(pgd_t *pgdp) +static void __init declare_kernel_vmas(void) { - static struct vm_struct vmlinux_text, vmlinux_rodata, vmlinux_inittext, - vmlinux_initdata, vmlinux_data; - - /* - * External debuggers may need to write directly to the text - * mapping to install SW breakpoints. Allow this (only) when - * explicitly requested with rodata=off. - */ - pgprot_t text_prot = kernel_exec_prot(); - - /* - * If we have a CPU that supports BTI and a kernel built for - * BTI then mark the kernel executable text as guarded pages - * now so we don't have to rewrite the page tables later. - */ - if (arm64_early_this_cpu_has_bti()) - text_prot = __pgprot_modify(text_prot, PTE_GP, PTE_GP); + static struct vm_struct vmlinux_seg[KERNEL_SEGMENT_COUNT]; - /* - * Only rodata will be remapped with different permissions later on, - * all other segments are allowed to use contiguous mappings. - */ - map_kernel_segment(pgdp, _stext, _etext, text_prot, &vmlinux_text, 0, - VM_NO_GUARD); - map_kernel_segment(pgdp, __start_rodata, __inittext_begin, PAGE_KERNEL, - &vmlinux_rodata, NO_CONT_MAPPINGS, VM_NO_GUARD); - map_kernel_segment(pgdp, __inittext_begin, __inittext_end, text_prot, - &vmlinux_inittext, 0, VM_NO_GUARD); - map_kernel_segment(pgdp, __initdata_begin, __initdata_end, PAGE_KERNEL, - &vmlinux_initdata, 0, VM_NO_GUARD); - map_kernel_segment(pgdp, _data, _end, PAGE_KERNEL, &vmlinux_data, 0, 0); - - fixmap_copy(pgdp); - kasan_copy_shadow(pgdp); + declare_vma(&vmlinux_seg[0], _stext, _etext, VM_NO_GUARD); + declare_vma(&vmlinux_seg[1], __start_rodata, __inittext_begin, VM_NO_GUARD); + declare_vma(&vmlinux_seg[2], __inittext_begin, __inittext_end, VM_NO_GUARD); + declare_vma(&vmlinux_seg[3], __initdata_begin, __initdata_end, VM_NO_GUARD); + declare_vma(&vmlinux_seg[4], _data, _end, 0); } void __pi_map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, @@ -799,23 +753,12 @@ static void __init create_idmap(void) void __init paging_init(void) { - pgd_t *pgdp = pgd_set_fixmap(__pa_symbol(swapper_pg_dir)); - extern pgd_t init_idmap_pg_dir[]; - - map_kernel(pgdp); - map_mem(pgdp); - - pgd_clear_fixmap(); - - cpu_replace_ttbr1(lm_alias(swapper_pg_dir), init_idmap_pg_dir); - init_mm.pgd = swapper_pg_dir; - - memblock_phys_free(__pa_symbol(init_pg_dir), - __pa_symbol(init_pg_end) - __pa_symbol(init_pg_dir)); + map_mem(swapper_pg_dir); memblock_allow_resize(); create_idmap(); + declare_kernel_vmas(); } #ifdef CONFIG_MEMORY_HOTPLUG -- 2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel