* [PATCH v3 1/2] arm64: add an MTE support check to the top of mte_thread_switch() @ 2021-09-15 19:03 Peter Collingbourne 2021-09-15 19:03 ` [PATCH v3 2/2] arm64: kasan: mte: move GCR_EL1 switch to task switch when KASAN disabled Peter Collingbourne 2021-09-21 12:45 ` (subset) [PATCH v3 1/2] arm64: add an MTE support check to the top of mte_thread_switch() Catalin Marinas 0 siblings, 2 replies; 5+ messages in thread From: Peter Collingbourne @ 2021-09-15 19:03 UTC (permalink / raw) To: Catalin Marinas, Vincenzo Frascino, Will Deacon, Andrey Konovalov Cc: Peter Collingbourne, Evgenii Stepanov, linux-arm-kernel This lets us avoid doing unnecessary work on hardware that does not support MTE, and will allow us to freely use MTE instructions in the code called by mte_thread_switch(). Since this would mean that we do a redundant check in mte_check_tfsr_el1(), remove it and add two checks now required in its callers. Signed-off-by: Peter Collingbourne <pcc@google.com> Link: https://linux-review.googlesource.com/id/I02fd000d1ef2c86c7d2952a7f099b254ec227a5d --- v3: - remove check from mte_check_tfsr_el1() arch/arm64/include/asm/mte.h | 6 ++++++ arch/arm64/kernel/mte.c | 10 ++++------ 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h index 3f93b9e0b339..02511650cffe 100644 --- a/arch/arm64/include/asm/mte.h +++ b/arch/arm64/include/asm/mte.h @@ -99,11 +99,17 @@ void mte_check_tfsr_el1(void); static inline void mte_check_tfsr_entry(void) { + if (!system_supports_mte()) + return; + mte_check_tfsr_el1(); } static inline void mte_check_tfsr_exit(void) { + if (!system_supports_mte()) + return; + /* * The asynchronous faults are sync'ed automatically with * TFSR_EL1 on kernel entry but for exit an explicit dsb() diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index 9d314a3bad3b..e5e801bc5312 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -142,12 +142,7 @@ void mte_enable_kernel_async(void) #ifdef CONFIG_KASAN_HW_TAGS void mte_check_tfsr_el1(void) { - u64 tfsr_el1; - - if (!system_supports_mte()) - return; - - tfsr_el1 = read_sysreg_s(SYS_TFSR_EL1); + u64 tfsr_el1 = read_sysreg_s(SYS_TFSR_EL1); if (unlikely(tfsr_el1 & SYS_TFSR_EL1_TF1)) { /* @@ -199,6 +194,9 @@ void mte_thread_init_user(void) void mte_thread_switch(struct task_struct *next) { + if (!system_supports_mte()) + return; + mte_update_sctlr_user(next); /* -- 2.33.0.309.g3052b89438-goog _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH v3 2/2] arm64: kasan: mte: move GCR_EL1 switch to task switch when KASAN disabled 2021-09-15 19:03 [PATCH v3 1/2] arm64: add an MTE support check to the top of mte_thread_switch() Peter Collingbourne @ 2021-09-15 19:03 ` Peter Collingbourne 2021-09-21 12:39 ` Catalin Marinas 2021-09-21 12:45 ` (subset) [PATCH v3 1/2] arm64: add an MTE support check to the top of mte_thread_switch() Catalin Marinas 1 sibling, 1 reply; 5+ messages in thread From: Peter Collingbourne @ 2021-09-15 19:03 UTC (permalink / raw) To: Catalin Marinas, Vincenzo Frascino, Will Deacon, Andrey Konovalov Cc: Peter Collingbourne, Evgenii Stepanov, linux-arm-kernel It is not necessary to write to GCR_EL1 on every kernel entry and exit when HW tag-based KASAN is disabled because the kernel will not execute any IRG instructions in that mode. Since accessing GCR_EL1 can be expensive on some microarchitectures, avoid doing so by moving the access to task switch when HW tag-based KASAN is disabled. Signed-off-by: Peter Collingbourne <pcc@google.com> Acked-by: Andrey Konovalov <andreyknvl@gmail.com> Link: https://linux-review.googlesource.com/id/I78e90d60612a94c24344526f476ac4ff216e10d2 --- v2: - add prerequisite patch to fix booting on non-MTE hardware arch/arm64/kernel/entry.S | 10 +++++----- arch/arm64/kernel/mte.c | 24 +++++++++++++++++++++--- include/linux/kasan.h | 9 +++++++-- 3 files changed, 33 insertions(+), 10 deletions(-) diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index bc6d5a970a13..2f69ae43941d 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -168,9 +168,9 @@ alternative_else_nop_endif .macro mte_set_kernel_gcr, tmp, tmp2 #ifdef CONFIG_KASAN_HW_TAGS -alternative_if_not ARM64_MTE +alternative_cb kasan_hw_tags_enable b 1f -alternative_else_nop_endif +alternative_cb_end mov \tmp, KERNEL_GCR_EL1 msr_s SYS_GCR_EL1, \tmp 1: @@ -178,10 +178,10 @@ alternative_else_nop_endif .endm .macro mte_set_user_gcr, tsk, tmp, tmp2 -#ifdef CONFIG_ARM64_MTE -alternative_if_not ARM64_MTE +#ifdef CONFIG_KASAN_HW_TAGS +alternative_cb kasan_hw_tags_enable b 1f -alternative_else_nop_endif +alternative_cb_end ldr \tmp, [\tsk, #THREAD_MTE_CTRL] mte_set_gcr \tmp, \tmp2 diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index e5e801bc5312..bb8204d4c5d3 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -157,7 +157,7 @@ void mte_check_tfsr_el1(void) } #endif -static void mte_update_sctlr_user(struct task_struct *task) +static void mte_update_sctlr_user_and_gcr_excl(struct task_struct *task) { /* * This must be called with preemption disabled and can only be called @@ -177,6 +177,24 @@ static void mte_update_sctlr_user(struct task_struct *task) else if (resolved_mte_tcf & MTE_CTRL_TCF_SYNC) sctlr |= SCTLR_EL1_TCF0_SYNC; task->thread.sctlr_user = sctlr; + + /* + * SYS_GCR_EL1 will be set to current->thread.mte_ctrl value by + * mte_set_user_gcr() in kernel_exit, but only if KASAN is enabled. + */ + if (!kasan_hw_tags_enabled()) + write_sysreg_s(((mte_ctrl >> MTE_CTRL_GCR_USER_EXCL_SHIFT) & + SYS_GCR_EL1_EXCL_MASK) | SYS_GCR_EL1_RRND, + SYS_GCR_EL1); +} + +void __init kasan_hw_tags_enable(struct alt_instr *alt, __le32 *origptr, + __le32 *updptr, int nr_inst) +{ + BUG_ON(nr_inst != 1); /* Branch -> NOP */ + + if (kasan_hw_tags_enabled()) + *updptr = cpu_to_le32(aarch64_insn_gen_nop()); } void mte_thread_init_user(void) @@ -197,7 +215,7 @@ void mte_thread_switch(struct task_struct *next) if (!system_supports_mte()) return; - mte_update_sctlr_user(next); + mte_update_sctlr_user_and_gcr_excl(next); /* * Check if an async tag exception occurred at EL1. @@ -242,7 +260,7 @@ long set_mte_ctrl(struct task_struct *task, unsigned long arg) task->thread.mte_ctrl = mte_ctrl; if (task == current) { preempt_disable(); - mte_update_sctlr_user(task); + mte_update_sctlr_user_and_gcr_excl(task); update_sctlr_el1(task->thread.sctlr_user); preempt_enable(); } diff --git a/include/linux/kasan.h b/include/linux/kasan.h index dd874a1ee862..de5f5913374d 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -89,7 +89,7 @@ static __always_inline bool kasan_enabled(void) return static_branch_likely(&kasan_flag_enabled); } -static inline bool kasan_has_integrated_init(void) +static inline bool kasan_hw_tags_enabled(void) { return kasan_enabled(); } @@ -104,7 +104,7 @@ static inline bool kasan_enabled(void) return IS_ENABLED(CONFIG_KASAN); } -static inline bool kasan_has_integrated_init(void) +static inline bool kasan_hw_tags_enabled(void) { return false; } @@ -125,6 +125,11 @@ static __always_inline void kasan_free_pages(struct page *page, #endif /* CONFIG_KASAN_HW_TAGS */ +static inline bool kasan_has_integrated_init(void) +{ + return kasan_hw_tags_enabled(); +} + #ifdef CONFIG_KASAN struct kasan_cache { -- 2.33.0.309.g3052b89438-goog _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH v3 2/2] arm64: kasan: mte: move GCR_EL1 switch to task switch when KASAN disabled 2021-09-15 19:03 ` [PATCH v3 2/2] arm64: kasan: mte: move GCR_EL1 switch to task switch when KASAN disabled Peter Collingbourne @ 2021-09-21 12:39 ` Catalin Marinas 2021-09-24 1:08 ` Peter Collingbourne 0 siblings, 1 reply; 5+ messages in thread From: Catalin Marinas @ 2021-09-21 12:39 UTC (permalink / raw) To: Peter Collingbourne Cc: Vincenzo Frascino, Will Deacon, Andrey Konovalov, Evgenii Stepanov, linux-arm-kernel On Wed, Sep 15, 2021 at 12:03:36PM -0700, Peter Collingbourne wrote: > -static void mte_update_sctlr_user(struct task_struct *task) > +static void mte_update_sctlr_user_and_gcr_excl(struct task_struct *task) > { > /* > * This must be called with preemption disabled and can only be called > @@ -177,6 +177,24 @@ static void mte_update_sctlr_user(struct task_struct *task) > else if (resolved_mte_tcf & MTE_CTRL_TCF_SYNC) > sctlr |= SCTLR_EL1_TCF0_SYNC; > task->thread.sctlr_user = sctlr; > + > + /* > + * SYS_GCR_EL1 will be set to current->thread.mte_ctrl value by > + * mte_set_user_gcr() in kernel_exit, but only if KASAN is enabled. > + */ > + if (!kasan_hw_tags_enabled()) > + write_sysreg_s(((mte_ctrl >> MTE_CTRL_GCR_USER_EXCL_SHIFT) & > + SYS_GCR_EL1_EXCL_MASK) | SYS_GCR_EL1_RRND, > + SYS_GCR_EL1); > +} The only nitpick I have is that I'd like to keep the GCR_EL1 updating in a separate function, mte_update_gcr_user() or something (it can be static inline) that you'd explicitly call from mte_thread_switch() and set_mte_ctrl(). The original mte_update_sctlr_user() was not writing any system registers. Maybe a better alternative would be to move the update_sctlr_el1() call in mte_update_sctlr_user() as well (and rename it more generically to mte_update_user_ctrl()) but with a new bool update_sctlr argument that's false on the thread switch path. Either way: Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v3 2/2] arm64: kasan: mte: move GCR_EL1 switch to task switch when KASAN disabled 2021-09-21 12:39 ` Catalin Marinas @ 2021-09-24 1:08 ` Peter Collingbourne 0 siblings, 0 replies; 5+ messages in thread From: Peter Collingbourne @ 2021-09-24 1:08 UTC (permalink / raw) To: Catalin Marinas Cc: Vincenzo Frascino, Will Deacon, Andrey Konovalov, Evgenii Stepanov, Linux ARM On Tue, Sep 21, 2021 at 5:40 AM Catalin Marinas <catalin.marinas@arm.com> wrote: > > On Wed, Sep 15, 2021 at 12:03:36PM -0700, Peter Collingbourne wrote: > > -static void mte_update_sctlr_user(struct task_struct *task) > > +static void mte_update_sctlr_user_and_gcr_excl(struct task_struct *task) > > { > > /* > > * This must be called with preemption disabled and can only be called > > @@ -177,6 +177,24 @@ static void mte_update_sctlr_user(struct task_struct *task) > > else if (resolved_mte_tcf & MTE_CTRL_TCF_SYNC) > > sctlr |= SCTLR_EL1_TCF0_SYNC; > > task->thread.sctlr_user = sctlr; > > + > > + /* > > + * SYS_GCR_EL1 will be set to current->thread.mte_ctrl value by > > + * mte_set_user_gcr() in kernel_exit, but only if KASAN is enabled. > > + */ > > + if (!kasan_hw_tags_enabled()) > > + write_sysreg_s(((mte_ctrl >> MTE_CTRL_GCR_USER_EXCL_SHIFT) & > > + SYS_GCR_EL1_EXCL_MASK) | SYS_GCR_EL1_RRND, > > + SYS_GCR_EL1); > > +} > > The only nitpick I have is that I'd like to keep the GCR_EL1 updating in > a separate function, mte_update_gcr_user() or something (it can be > static inline) that you'd explicitly call from mte_thread_switch() and > set_mte_ctrl(). The original mte_update_sctlr_user() was not writing any > system registers. Done in v4. (I mildly prefer the way I originally wrote it as I think it's less error prone, but I don't mind that much.) Sorry, I forgot to add your Reviewed-by. Peter _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: (subset) [PATCH v3 1/2] arm64: add an MTE support check to the top of mte_thread_switch() 2021-09-15 19:03 [PATCH v3 1/2] arm64: add an MTE support check to the top of mte_thread_switch() Peter Collingbourne 2021-09-15 19:03 ` [PATCH v3 2/2] arm64: kasan: mte: move GCR_EL1 switch to task switch when KASAN disabled Peter Collingbourne @ 2021-09-21 12:45 ` Catalin Marinas 1 sibling, 0 replies; 5+ messages in thread From: Catalin Marinas @ 2021-09-21 12:45 UTC (permalink / raw) To: Andrey Konovalov, Will Deacon, Peter Collingbourne, Vincenzo Frascino Cc: linux-arm-kernel, Evgenii Stepanov On Wed, 15 Sep 2021 12:03:35 -0700, Peter Collingbourne wrote: > This lets us avoid doing unnecessary work on hardware that does > not support MTE, and will allow us to freely use MTE instructions > in the code called by mte_thread_switch(). > > Since this would mean that we do a redundant check in > mte_check_tfsr_el1(), remove it and add two checks now required in > its callers. I applied the first patch to arm64 (for-next/fixes), it should appear in 5.15-rc3. It looks to me more like a performance regression with the additional dsb+isb on the syscall path for hw not supporting MTE (and I added a Fixes tag). I'll leave the second patch to Will for 5.16. Thanks! [1/2] arm64: add an MTE support check to the top of mte_thread_switch() https://git.kernel.org/arm64/c/8c8a3b5bd960 -- Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-09-24 1:10 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2021-09-15 19:03 [PATCH v3 1/2] arm64: add an MTE support check to the top of mte_thread_switch() Peter Collingbourne 2021-09-15 19:03 ` [PATCH v3 2/2] arm64: kasan: mte: move GCR_EL1 switch to task switch when KASAN disabled Peter Collingbourne 2021-09-21 12:39 ` Catalin Marinas 2021-09-24 1:08 ` Peter Collingbourne 2021-09-21 12:45 ` (subset) [PATCH v3 1/2] arm64: add an MTE support check to the top of mte_thread_switch() Catalin Marinas
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).