From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E121FC46CA3 for ; Wed, 29 Nov 2023 11:19:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=wDqd/gGDvMGtMO1/KD7n1W8okVzZ73kGrZ0PzlAcs8Q=; b=41PlrGOszRAOmMqyfpwLQk80nI 62YLogZnvvnrtT5QDLtoFq/2tHMFAcZh6g+SD4hHFZO+tUw6C6cQ4AQE5JfL9+1lQ70wPHvlLEd8k xoSR1H1tBNENMpQmImzuM0cryYf+rXyz0yDXtZd8HcuKsXtUMuUr9oAMK18j16onHf0VHoc48kANy u/Up+0RQ/bE0UauF2wo2SEi5s0I7UiLU42b1gu0aM5H101bkdcuydVEhpxdIlOIAQJe8pR7hFW5Hm TrxmVK9qyY/dvU+063pK35e/3Ru2CTAEhGbXOscATDNAk8mqpx3vb9MBVWE0LJVswSJgqbLUCdI8z /rj1Zl1g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r8Iaw-0080Sm-0S; Wed, 29 Nov 2023 11:19:06 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r8Ia9-007zjH-2D for linux-arm-kernel@lists.infradead.org; Wed, 29 Nov 2023 11:18:28 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5d1b431fa7bso26288597b3.1 for ; Wed, 29 Nov 2023 03:18:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701256695; x=1701861495; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=oTb5YT02RwqmCRs+XMdOFbLPPfCLhuRkcodO2emiEKQ=; b=pp6EuxzK8q18OOSQfJBYJxm2K4c3eL5LoAmHfNoL8xsQD5QNWXTCRh3dAcDpHvR+UE z20dD0t0zke/oqSavzscGKsSBznipy4MS2pjMWVz4mAJsKWnfw4V6lMOFEPExEgJW4Dv kDp6PXFDvquh7pdmvFQfZjffou4uJ40VbbYt5JYxtwVYA+R1cbqte+nO+m7jVrwiwxlM t2KpteEE2FLtxWTZsKlxmGxpWoD/pfyZiBu6MCdqSRf6AsUdOsMv3G//ANJHPDY7DieN l5mca9PmelOsuRDiGj9Q0pDMXfw+W0JmPKNOSqkIC7jC6f5WJuN1oHZ3nj5cFjI17BRw jdjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701256695; x=1701861495; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=oTb5YT02RwqmCRs+XMdOFbLPPfCLhuRkcodO2emiEKQ=; b=pDTQNPsDDLCt2C0BjzpsIE8EgGcaoN+Ncz58RqnZjwb06LcN77ZcIFWW4XsJga/NuR yzPxg1ZTiPB0F+X0pUZhMA3jnJtr0FsR+7nrJzS9IbOd8KpW+7ucM9fz115CBplq944I 9d95KKsmV79sxtoWqp6ErTlz0wsXFT3ClDGJO5IF9N/Z/bv+b1L+8I8NS15INZtCdwK+ uUZJW1IUIUAJAiTYUc4zVDF7PhqFnQs3hzNSZv7BVbmEQ8qCVfv6gJjwmZunLJ6+INaq 4bDvDVFBdkUPGEmOEkJ+Yr47+J7OrELeO3tpTbQfUvtZnogV4O5KPZ5njLllIHbEglm+ ij/A== X-Gm-Message-State: AOJu0Yx6qaYK7AaECUasfm2vjaCPgmGwToX6bSmoIiHPo75BxAFGCKCR tgWjE9XTFjZ1otKsQMFgs5ErgpwlklPW0pVk+8zOU+unsSuoKDmzdtELkH/M+0GVjOaY637iYIr rPod+LzEFkeuTLQ4SffOLmJvVjYvBVcjtf4d734QmUgx3FlIDQv0hauXd/idV+CSzo4aGxm6QNa A= X-Google-Smtp-Source: AGHT+IG1HpvWIOFipWZhTxibzDdxI/GIjxF90B2Pn9Nx1jtdm+aiHqxdL1e81bWBmgQw/UuBWwZBK8hH X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:690c:3183:b0:5cc:412c:27c7 with SMTP id fd3-20020a05690c318300b005cc412c27c7mr567350ywb.5.1701256695316; Wed, 29 Nov 2023 03:18:15 -0800 (PST) Date: Wed, 29 Nov 2023 12:16:22 +0100 In-Reply-To: <20231129111555.3594833-43-ardb@google.com> Mime-Version: 1.0 References: <20231129111555.3594833-43-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6307; i=ardb@kernel.org; h=from:subject; bh=+CAandW2REwobvpWTcepqeb8aChD0Vxr0cE2KhPU7EE=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITVdtlP2cW/X88niv1weWuY86llb9Sd4BnPe/jOCKpNye PdrXxXqKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABM5fYuRYf6sBlXv3hbFV1MV 7ve8vPLGVXNFGaNaNFMBx/cd7wWvf2T4H/bBM7E/3XEL3+9lk7fV31Xo0vEQbJe861jyRDd+eyU zNwA= X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231129111555.3594833-69-ardb@google.com> Subject: [PATCH v6 26/41] arm64: head: move dynamic shadow call stack patching into early C runtime From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231129_031817_783595_8B45B378 X-CRM114-Status: GOOD ( 18.69 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Once we update the early kernel mapping code to only map the kernel once with the right permissions, we can no longer perform code patching via this mapping. So move this code to an earlier stage of the boot, right after applying the relocations. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/scs.h | 4 +-- arch/arm64/kernel/Makefile | 2 -- arch/arm64/kernel/head.S | 8 +++--- arch/arm64/kernel/module.c | 2 +- arch/arm64/kernel/pi/Makefile | 10 +++++--- arch/arm64/kernel/{ => pi}/patch-scs.c | 26 ++++++++++---------- 6 files changed, 27 insertions(+), 25 deletions(-) diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h index 3fdae5fe3142..eca2ba5a6276 100644 --- a/arch/arm64/include/asm/scs.h +++ b/arch/arm64/include/asm/scs.h @@ -72,8 +72,8 @@ static inline void dynamic_scs_init(void) static inline void dynamic_scs_init(void) {} #endif -int scs_patch(const u8 eh_frame[], int size); -asmlinkage void scs_patch_vmlinux(void); +int __pi_scs_patch(const u8 eh_frame[], int size); +asmlinkage void __pi_scs_patch_vmlinux(void); #endif /* __ASSEMBLY __ */ diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index dc85dc2ee4ed..14b4a179bad3 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -71,8 +71,6 @@ obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o obj-$(CONFIG_ARM64_MTE) += mte.o obj-y += vdso-wrap.o obj-$(CONFIG_COMPAT_VDSO) += vdso32-wrap.o -obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.o -CFLAGS_patch-scs.o += -mbranch-protection=none # Force dependency (vdso*-wrap.S includes vdso.so through incbin) $(obj)/vdso-wrap.o: $(obj)/vdso/vdso.so diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 865ecc1f8255..b320702032a7 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -490,9 +490,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) #endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init -#endif -#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS - bl scs_patch_vmlinux #endif mov x0, x20 bl finalise_el2 // Prefer VHE if possible @@ -794,6 +791,11 @@ SYM_FUNC_START_LOCAL(__primary_switch) #ifdef CONFIG_RELOCATABLE mov x0, x23 bl __pi_relocate_kernel +#endif +#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS + ldr x0, =__eh_frame_start + ldr x1, =__eh_frame_end + bl __pi_scs_patch_vmlinux #endif ldr x8, =__primary_switched adrp x0, KERNEL_START // __pa(KERNEL_START) diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c index dd851297596e..47e0be610bb6 100644 --- a/arch/arm64/kernel/module.c +++ b/arch/arm64/kernel/module.c @@ -595,7 +595,7 @@ int module_finalize(const Elf_Ehdr *hdr, if (scs_is_dynamic()) { s = find_section(hdr, sechdrs, ".init.eh_frame"); if (s) - scs_patch((void *)s->sh_addr, s->sh_size); + __pi_scs_patch((void *)s->sh_addr, s->sh_size); } return module_init_ftrace_plt(hdr, sechdrs, me); diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 7f6dfce893c3..a8b302245f15 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -38,7 +38,9 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := idreg-override.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o -obj-$(CONFIG_RELOCATABLE) += relocate.pi.o -obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o -extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) +obj-y := idreg-override.pi.o \ + lib-fdt.pi.o lib-fdt_ro.pi.o +obj-$(CONFIG_RELOCATABLE) += relocate.pi.o +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o +obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.pi.o +extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) diff --git a/arch/arm64/kernel/patch-scs.c b/arch/arm64/kernel/pi/patch-scs.c similarity index 91% rename from arch/arm64/kernel/patch-scs.c rename to arch/arm64/kernel/pi/patch-scs.c index a1fe4b4ff591..c65ef40d1e6b 100644 --- a/arch/arm64/kernel/patch-scs.c +++ b/arch/arm64/kernel/pi/patch-scs.c @@ -4,14 +4,11 @@ * Author: Ard Biesheuvel */ -#include #include #include #include -#include #include -#include #include // @@ -81,7 +78,11 @@ static void __always_inline scs_patch_loc(u64 loc) */ return; } - dcache_clean_pou(loc, loc + sizeof(u32)); + if (IS_ENABLED(CONFIG_ARM64_WORKAROUND_CLEAN_CACHE)) + asm("dc civac, %0" :: "r"(loc)); + else + asm(ALTERNATIVE("dc cvau, %0", "nop", ARM64_HAS_CACHE_IDC) + :: "r"(loc)); } /* @@ -128,10 +129,10 @@ struct eh_frame { }; }; -static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, - bool fde_has_augmentation_data, - int code_alignment_factor, - bool dry_run) +static int scs_handle_fde_frame(const struct eh_frame *frame, + bool fde_has_augmentation_data, + int code_alignment_factor, + bool dry_run) { int size = frame->size - offsetof(struct eh_frame, opcodes) + 4; u64 loc = (u64)offset_to_ptr(&frame->initial_loc); @@ -198,14 +199,13 @@ static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, break; default: - pr_err("unhandled opcode: %02x in FDE frame %lx\n", opcode[-1], (uintptr_t)frame); return -ENOEXEC; } } return 0; } -int noinstr scs_patch(const u8 eh_frame[], int size) +int scs_patch(const u8 eh_frame[], int size) { const u8 *p = eh_frame; @@ -251,12 +251,12 @@ int noinstr scs_patch(const u8 eh_frame[], int size) return 0; } -asmlinkage void __init scs_patch_vmlinux(void) +asmlinkage void __init scs_patch_vmlinux(const u8 start[], const u8 end[]) { if (!should_patch_pac_into_scs()) return; - WARN_ON(scs_patch(__eh_frame_start, __eh_frame_end - __eh_frame_start)); - icache_inval_all_pou(); + scs_patch(start, end - start); + asm("ic ialluis"); isb(); } -- 2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel