From: Ard Biesheuvel <ardb@kernel.org>
To: linux-arm-kernel@lists.infradead.org
Cc: linux-efi@vger.kernel.org, Ard Biesheuvel <ardb@kernel.org>,
Marc Zyngier <maz@kernel.org>, Will Deacon <will@kernel.org>,
Mark Rutland <mark.rutland@arm.com>,
Kees Cook <keescook@chromium.org>,
Catalin Marinas <catalin.marinas@arm.com>,
Mark Brown <broonie@kernel.org>
Subject: [PATCH v3 26/30] arm64: head: avoid cache invalidation when entering with the MMU on
Date: Mon, 11 Apr 2022 11:48:20 +0200 [thread overview]
Message-ID: <20220411094824.4176877-27-ardb@kernel.org> (raw)
In-Reply-To: <20220411094824.4176877-1-ardb@kernel.org>
If we enter with the MMU on, there is no need for explicit cache
invalidation for stores to memory, as they will be coherent with the
caches.
Let's take advantage of this, and create the ID map with the MMU still
enabled if that is how we entered, and avoid any cache invalidation
calls in that case.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
arch/arm64/kernel/head.S | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 1ff474701e99..4a05f4480207 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -92,9 +92,9 @@
SYM_CODE_START(primary_entry)
bl record_mmu_state
bl preserve_boot_args
+ bl create_idmap
bl init_kernel_el // w0=cpu_boot_mode
mov x20, x0
- bl create_idmap
/*
* The following calls CPU setup code, see arch/arm64/mm/proc.S for
@@ -127,11 +127,13 @@ SYM_CODE_START_LOCAL(preserve_boot_args)
stp x21, x1, [x0] // x0 .. x3 at kernel entry
stp x2, x3, [x0, #16]
+ cbnz x22, 0f // skip cache invalidation if MMU is on
dmb sy // needed before dc ivac with
// MMU off
add x1, x0, #0x20 // 4 x 8 bytes
b dcache_inval_poc // tail call
+0: ret
SYM_CODE_END(preserve_boot_args)
SYM_FUNC_START_LOCAL(clear_page_tables)
@@ -366,12 +368,13 @@ SYM_FUNC_START_LOCAL(create_idmap)
* accesses (MMU disabled), invalidate those tables again to
* remove any speculatively loaded cache lines.
*/
+ cbnz x22, 0f // skip cache invalidation if MMU is on
dmb sy
adrp x0, init_idmap_pg_dir
adrp x1, init_idmap_pg_end
bl dcache_inval_poc
- ret x28
+0: ret x28
SYM_FUNC_END(create_idmap)
SYM_FUNC_START_LOCAL(create_kernel_mapping)
--
2.30.2
next prev parent reply other threads:[~2022-04-11 9:51 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-04-11 9:47 [PATCH v3 00/30] arm64: support WXN and entry with MMU enabled Ard Biesheuvel
2022-04-11 9:47 ` [PATCH v3 01/30] arm64: head: move kimage_vaddr variable into C file Ard Biesheuvel
2022-04-11 9:47 ` [PATCH v3 02/30] arm64: mm: make vabits_actual a build time constant if possible Ard Biesheuvel
2022-04-11 9:47 ` [PATCH v3 03/30] arm64: head: move assignment of idmap_t0sz to C code Ard Biesheuvel
2022-04-11 9:47 ` [PATCH v3 04/30] arm64: head: drop idmap_ptrs_per_pgd Ard Biesheuvel
2022-04-11 9:47 ` [PATCH v3 05/30] arm64: head: simplify page table mapping macros (slightly) Ard Biesheuvel
2022-04-11 9:48 ` [PATCH v3 06/30] arm64: head: switch to map_memory macro for the extended ID map Ard Biesheuvel
2022-04-11 9:48 ` [PATCH v3 07/30] arm64: head: split off idmap creation code Ard Biesheuvel
2022-04-11 9:48 ` [PATCH v3 08/30] arm64: kernel: drop unnecessary PoC cache clean+invalidate Ard Biesheuvel
2022-04-11 9:48 ` [PATCH v3 09/30] arm64: head: pass ID map root table address to __enable_mmu() Ard Biesheuvel
2022-04-11 9:48 ` [PATCH v3 10/30] arm64: mm: provide idmap pointer to cpu_replace_ttbr1() Ard Biesheuvel
2022-04-11 9:48 ` [PATCH v3 11/30] arm64: head: add helper function to remap regions in early page tables Ard Biesheuvel
2022-04-11 9:48 ` [PATCH v3 12/30] arm64: head: cover entire kernel image in initial ID map Ard Biesheuvel
2022-04-11 9:48 ` [PATCH v3 13/30] arm64: head: use relative references to the RELA and RELR tables Ard Biesheuvel
2022-04-11 9:48 ` [PATCH v3 14/30] arm64: head: create a temporary FDT mapping in the initial ID map Ard Biesheuvel
2022-04-11 9:48 ` [PATCH v3 15/30] arm64: idreg-override: use early FDT mapping in " Ard Biesheuvel
2022-04-11 9:48 ` [PATCH v3 16/30] arm64: head: factor out TTBR1 assignment into a macro Ard Biesheuvel
2022-04-11 9:48 ` [PATCH v3 17/30] arm64: head: populate kernel page tables with MMU and caches on Ard Biesheuvel
2022-04-11 9:48 ` [PATCH v3 18/30] arm64: head: record CPU boot mode after enabling the MMU Ard Biesheuvel
2022-04-11 9:48 ` [PATCH v3 19/30] arm64: kaslr: deal with init called with VA randomization enabled Ard Biesheuvel
2022-04-11 9:48 ` [PATCH v3 20/30] arm64: head: relocate kernel only a single time if KASLR is enabled Ard Biesheuvel
2022-04-11 9:48 ` [PATCH v3 21/30] arm64: head: remap the kernel text/inittext region read-only Ard Biesheuvel
2022-04-11 9:48 ` [PATCH v3 22/30] arm64: setup: drop early FDT pointer helpers Ard Biesheuvel
2022-04-11 9:48 ` [PATCH v3 23/30] arm64: mm: move ro_after_init section into the data segment Ard Biesheuvel
2022-04-11 9:48 ` [PATCH v3 24/30] arm64: mm: add support for WXN memory translation attribute Ard Biesheuvel
2022-04-11 9:48 ` [PATCH v3 25/30] arm64: head: record the MMU state at primary entry Ard Biesheuvel
2022-04-11 9:48 ` Ard Biesheuvel [this message]
2022-04-11 9:48 ` [PATCH v3 27/30] arm64: head: clean the ID map page to the PoC Ard Biesheuvel
2022-04-11 9:48 ` [PATCH v3 28/30] efi: libstub: pass image handle to handle_kernel_image() Ard Biesheuvel
2022-04-11 9:48 ` [PATCH v3 29/30] efi/arm64: libstub: run image in place if randomized by the loader Ard Biesheuvel
2022-04-11 9:48 ` [PATCH v3 30/30] arm64: efi/libstub: enter with the MMU on if executing in place Ard Biesheuvel
2022-04-12 16:59 ` [PATCH v3 00/30] arm64: support WXN and entry with MMU enabled Kees Cook
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220411094824.4176877-27-ardb@kernel.org \
--to=ardb@kernel.org \
--cc=broonie@kernel.org \
--cc=catalin.marinas@arm.com \
--cc=keescook@chromium.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-efi@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=maz@kernel.org \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).