From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 26A42C433EF for ; Thu, 28 Apr 2022 06:59:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=s52PhEA3bAWvtzkJc34N7HfVcB2yOlEkhFjiGCu8F6M=; b=LRWIEsJw0JVf2Q9t7CIyFBQrFp Q+8EFScHK7wyNNLKs09hDoWBspaFVX0/RV3p+MK37rQW99IVNFyLgd4rp7ovOy7DevFPHkhwV5RGl 6Xt+uJ2sdKBoHbn2BGTnnRXGAGL1PYBSKq/111LVVaCkQpMZux84ojaSL0zkaO6YvJW9Cuk68hvc/ SRp6bJ1mCbEmu51wIBYGmGYtOgMZTKDBsjSY4I+kSmU5Wcjf2bAmwfkguShuuraALFEe8bdMHJgtQ zsS0Z8A+KxyGtIYZStMlY6OmG9zsPFdYQye9OsV/rBdZrvspt6g0vaqSf3BMpw6Y79ScCVN/rlRAo GF6D9Fgg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1njy6A-004zmb-Fx; Thu, 28 Apr 2022 06:57:58 +0000 Received: from mail-pj1-x102c.google.com ([2607:f8b0:4864:20::102c]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1njy66-004ziD-P5 for linux-arm-kernel@lists.infradead.org; Thu, 28 Apr 2022 06:57:57 +0000 Received: by mail-pj1-x102c.google.com with SMTP id cx11-20020a17090afd8b00b001d9fe5965b3so4967063pjb.3 for ; Wed, 27 Apr 2022 23:57:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=applOq61A1kPgrAqYMQ6DdRNoLWzc8K0I42SOGrOzNQ=; b=KYkXTkpfJAPLmrAAc5TUDIy1iPx5b0HjNNj6grxJJojjKpOoC0xMnmG+AmAQ4gn9qF qpA/RWUYizGzCegha1WUMJqKq/xLXe+Ez9RRl65SpLo8BR+plPOsNs4LKrKg5VVWAI8j NmKmR6dsC6n41e/Lx1PeSGhbe3EBo7cPjQzqExeJy9iBas7byoZO9SFYdW6LL/ZneJaj wVTjm+KuaxYdtCmAYHkSQyOppGL3IhKk8HuzPjJ+qG5WyTofC9TFmFv/0xLcQgQn7hBB o4bBRUybyFpZy9NprDDg+RVbq0Wi20Jm/56Iql/bK17gJB5tqlQD2uGT4nkHKQFIYAKT Oa7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=applOq61A1kPgrAqYMQ6DdRNoLWzc8K0I42SOGrOzNQ=; b=nYKrqKlZnqQsv78RjVZIjSvShzCmjoqvlwnHKf5PHZ+S0o2pP+N/rVuB9A9lIXtadL 7qau2qISMVVbe04kAtxq2vK0JrjN9hT4Vbh8D3oJBabrKYPPvL9Nsrry7LxSSCk/2pop PIFQNvrZ+YH5Tzr7qxEbU71+4+pUQSE5ZCdBq5SjgQeGwRw6w9Z/1OEt636DK3JUirrO JTrbffyhmP/0oNakTO569kDiKhUmzTBSrgxL4C/eraBULVRN///zsdAr/SyqfBX2HYm0 Ix5D43q6cKD/OAS24zcESe6FLHEolqYAfzZYuOPcemA8Tx4ld9TX7B44J5ZkMA9icwu3 sB4A== X-Gm-Message-State: AOAM533gt80H3ZghvuOSQDj4c+P3hOLoPiLiJ58r7QEieF5VlCbIwE0A ysaO1KBq4WIqgOUV0/YZEX7cTA== X-Google-Smtp-Source: ABdhPJzItn9leydOjmQ2j9Ls9SvEhxwEv9OM0VWq5QTZcPsoM+NBFIALB7ioOE+57y7/cHKgVDaZ5g== X-Received: by 2002:a17:90b:606:b0:1d9:5dd5:1489 with SMTP id gb6-20020a17090b060600b001d95dd51489mr24210556pjb.171.1651129066875; Wed, 27 Apr 2022 23:57:46 -0700 (PDT) Received: from google.com ([2620:15c:2ce:200:97d8:6a3e:53c0:230d]) by smtp.gmail.com with ESMTPSA id v189-20020a622fc6000000b004fb72e95806sm20868781pfv.48.2022.04.27.23.57.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Apr 2022 23:57:46 -0700 (PDT) Date: Wed, 27 Apr 2022 23:57:42 -0700 From: Fangrui Song To: Ard Biesheuvel Cc: Linux ARM , clang-built-linux , Will Deacon , Catalin Marinas , Kees Cook , Mark Rutland , Nathan Chancellor , Sami Tolvanen , Nick Desaulniers Subject: Re: [RFC PATCH 2/2] arm64: kernel: switch to PIE code generation for relocatable kernels Message-ID: <20220428065742.rl3w5rz2ni2fhngl@google.com> References: <20220427171241.2426592-1-ardb@kernel.org> <20220427171241.2426592-3-ardb@kernel.org> <20220428024030.gwxb746c5gwvcnw6@google.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220427_235754_894436_0E8F14A7 X-CRM114-Status: GOOD ( 38.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 2022-04-28, Ard Biesheuvel wrote: >On Thu, 28 Apr 2022 at 04:40, Fangrui Song wrote: >> >> On 2022-04-27, Ard Biesheuvel wrote: >> >We currently use ordinary, position dependent code generation for the >> >core kernel, which happens to default to the 'small' code model on both >> >GCC and Clang. This is the code model that relies on ADRP/ADD or >> >ADRP/LDR pairs for symbol references, which are PC-relative with a range >> >of -/+ 4 GiB, and therefore happen to be position independent in >> >practice. >> > >> >This means that the fact that we can link the relocatable KASLR kernel >> >using the -pie linker flag (which generates the runtime relocations and >> >inserts them into the binary) is somewhat of a coincidence, and not >> >something which is explicitly supported by the toolchains. >> >> Agree. The current -fno-PIE + -shared -Bsymbolic combo works as a >> conincidence, not guaranteed by the toolchain. >> >> -shared needs -fpic object files. -shared -Bsymbolic is very similar to >> -pie and therefore works with -fpie object files, but the usage is not >> recommended from the toolchain perspective. >> > >So are you suggesting we should also switch from -shared to -Bsymbol >to -pie if we can? I don't remember the details, but IIRC ld.bfd >didn't set the ELF binary type correctly, but perhaps this has now >been fixed. Yes, -shared -Bsymbolic => -pie, but that can be done later. For e_type: ET_DYN, I think unlikely there was a bug. -pie was added by binutils in 2003: it's close to -shared but doesn't allow its definitions to be preempted/interposed. Code earlier than that might use -shared -Bsymbolic before -pie was available. >> >The reason we have not used -fpie for code generation so far (which is >> >the compiler flag that should be used to generate code that is to be >> >linked with -pie) is that by default, it generates code based on >> >assumptions that only hold for shared libraries and PIE executables, >> >i.e., that gathering all relocatable quantities into a Global Offset >> >Table (GOT) is desirable because it reduces the CoW footprint, and >> >because it permits ELF symbol preemption (which lets an executable >> >override symbols defined in a shared library, in a way that forces the >> >shared library to update all of its internal references as well). >> >Ironically, this means we end up with many more absolute references that >> >all need to be fixed up at boot. >> >> This is not about symbol preemption (when the executable and a shared >> objectdefine the same symbol, which one wins). An executable using a GOT >> which will be resolved to a shared object => this is regular relocation >> resolving and there is no preemption. >> >> It is that the compiler prefers code generation which can avoid text >> relocations / copy relocations / canonical PLT entries >> (https://maskray.me/blog/2021-01-09-copy-relocations-canonical-plt-entries-and-protected#summary). >> > >Fair enough. So the compiler cannot generate relative references to >undefined external symbols since it doesn't know at codegen time >whether the symbol reference will be satisfied by the executable >itself or by a shared library, and in the latter case, the relative >distance to the symbol is not known at build time, and so a runtime >relocation is required. Right. >But how about references to symbols with >external visibility that are defined in the same compilation unit? I >don't quite understand why those references need to go via the GOT as >well. If you mean references to a non-local STV_DEFAULT (default visibility) definition => * -fpic: use GOT because the definition may be replaced by another at run time. Conservatively use a GOT-generating code sequence to allow potential symbol preemption(interposition). The linker may optimize out the GOT (x86-64 GOTPCRELX, recent ld.lld for aarch64, powerpc64 TOC-indirect to TOC-relative optimization). * -fpie or -fno-pie: the definition cannot be replaced. GOT is unneeded. -fpie is an optimization on top of -fpic: (a) non-local STV_DEFAULT definitions can be assumed non-interposable (b) (irrelevant to the kernel) TLS can use more optimized models. >> >Fortunately, we can convince the compiler to handle this in a way that >> >is a bit more suitable for freestanding binaries such as the kernel, by >> >setting the 'hidden' visibility #pragma, which informs the compiler that >> >symbol preemption or CoW footprint are of no concern to us, and so >> >PC-relative references that are resolved at link time are perfectly >> >fine. >> >> Agree >> > >The only unfortunate thing is that -fvisibility=hidden does not give >us the behavior we want, and we are forced to use the #pragma instead. Right. For a very long time there had been no option controlling the access mode for undefined symbols (-fvisibility= is for defined symbols). I added -fdirect-access-external-data to Clang which supports many architectures (x86, aarch64, arm, riscv, ...). GCC's x86 port added -mdirect-extern-access in 2022-02 (not available on aarch64). The use of `#pragma GCC visibility push(hidden)` looks good as a portable solution. >> >So let's enable this #pragma and build with -fpie when building a >> >relocatable kernel. This also means that all constant data items that >> >carry statically initialized pointer variables are now emitted into the >> >.data.rel.ro* sections, so move these into .rodata where they belong. >> >> LGTM, except: is ".rodata" a typo? The patch doesn't reference .rodata >> > >I am referring to the .rodata pseudo-segment that we have in the >kernel, which runs from _etext to __inittext_begin. OK >> >Code size impact (GCC): >> > >> >Before: >> > >> > text data bss total filename >> > 16712396 18659064 534556 35906016 vmlinux >> > >> >After: >> > >> > text data bss total filename >> > 16804400 18612876 534556 35951832 vmlinux >> > >> >Code size impact (Clang): >> > >> >Before: >> > >> > text data bss total filename >> > 17194584 13335060 535268 31064912 vmlinux >> > >> >After: >> > >> > text data bss total filename >> > 17194536 13310032 535268 31039836 vmlinux The size difference for Clang matches my expecation:) I am somewhat surprised that data is smaller, though... I wonder how GCC makes code bloated so much... >> >Signed-off-by: Ard Biesheuvel >> >--- >> > arch/arm64/Makefile | 4 ++++ >> > arch/arm64/kernel/vmlinux.lds.S | 9 ++++----- >> > 2 files changed, 8 insertions(+), 5 deletions(-) >> > >> >diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile >> >index 2f1de88651e6..94b6c51f5de6 100644 >> >--- a/arch/arm64/Makefile >> >+++ b/arch/arm64/Makefile >> >@@ -18,6 +18,10 @@ ifeq ($(CONFIG_RELOCATABLE), y) >> > # with the relocation offsets always being zero. >> > LDFLAGS_vmlinux += -shared -Bsymbolic -z notext \ >> > $(call ld-option, --no-apply-dynamic-relocs) >> >+ >> >+# Generate position independent code without relying on a Global Offset Table >> >+KBUILD_CFLAGS_KERNEL += -fpie -include $(srctree)/include/linux/hidden.h >> >+ >> > endif >> > >> > ifeq ($(CONFIG_ARM64_ERRATUM_843419),y) >> >diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S >> >index edaf0faf766f..b1e071ac1acf 100644 >> >--- a/arch/arm64/kernel/vmlinux.lds.S >> >+++ b/arch/arm64/kernel/vmlinux.lds.S >> >@@ -174,8 +174,6 @@ SECTIONS >> > KEXEC_TEXT >> > TRAMP_TEXT >> > *(.gnu.warning) >> >- . = ALIGN(16); >> >- *(.got) /* Global offset table */ >> > } >> > >> > /* >> >@@ -192,6 +190,8 @@ SECTIONS >> > /* everything from this point to __init_begin will be marked RO NX */ >> > RO_DATA(PAGE_SIZE) >> > >> >+ .data.rel.ro : ALIGN(8) { *(.got) *(.data.rel.ro*) } >> >+ >> > HYPERVISOR_DATA_SECTIONS >> > >> > idmap_pg_dir = .; >> >@@ -273,6 +273,8 @@ SECTIONS >> > _sdata = .; >> > RW_DATA(L1_CACHE_BYTES, PAGE_SIZE, THREAD_ALIGN) >> > >> >+ .data.rel : ALIGN(8) { *(.data.rel*) } >> >+ >> > /* >> > * Data written with the MMU off but read with the MMU on requires >> > * cache lines to be invalidated, discarding up to a Cache Writeback >> >@@ -320,9 +322,6 @@ SECTIONS >> > *(.plt) *(.plt.*) *(.iplt) *(.igot .igot.plt) >> > } >> > ASSERT(SIZEOF(.plt) == 0, "Unexpected run-time procedure linkages detected!") >> >- >> >- .data.rel.ro : { *(.data.rel.ro) } >> >- ASSERT(SIZEOF(.data.rel.ro) == 0, "Unexpected RELRO detected!") >> > } >> > >> > #include "image-vars.h" >> >-- >> >2.30.2 >> > >> >-- >> >You received this message because you are subscribed to the Google Groups "Clang Built Linux" group. >> >To unsubscribe from this group and stop receiving emails from it, send an email to clang-built-linux+unsubscribe@googlegroups.com. >> >To view this discussion on the web visit https://groups.google.com/d/msgid/clang-built-linux/20220427171241.2426592-3-ardb%40kernel.org. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel