[RFT PATCH] arm64: relocatable: build the kernel as a proper shared library

[RFT PATCH] arm64: relocatable: build the kernel as a proper shared library

[Ard Biesheuvel]

readelf complains about the section layout of vmlinux when building
with CONFIG_RELOCATABLE=y (for KASLR):

  readelf: Warning: [21]: Link field (0) should index a symtab section.
  readelf: Warning: [21]: Info field (0) should index a relocatable section.

Also, it seems that our use of '-pie -shared' is contradictory, and
thus ambiguous. In general, the way KASLR is wired up at the moment
is highly tailored to how ld.bfd happens to implement (and conflate)
PIE executables and shared libraries, so given the current effort to
support other toolchains, let's fix some of these issues as well.

- Drop the -pie linker argument and just leave -shared. In ld.bfd,
  the differences between them are unclear (except for the ELF type
  of the produced image [0]) but lld chokes on seeing both at the
  same time.

- Rename the .rela output section to .rela.dyn, as is customary for
  shared libraries and PIE executables.

- Don't discard the .dynamic, .dynsym, .dynstr and .hash sections.
  Instead, make sure that they are [mostly] empty by marking all
  symbols as local, and emit them into the .init segment.

These changes only affect the ELF image, and produce the same binary
image, with the exception of a couple of bytes of .init data for the
empty .dynsym and .dynstr sections.

[0] b9dce7f1ba01 ("arm64: kernel: force ET_DYN ELF type for ...")

Cc: Nick Desaulniers <ndesaulniers@google.com>
Cc: Peter Smith <peter.smith@linaro.org>
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
 arch/arm64/Makefile             |  2 +-
 arch/arm64/kernel/vmlinux.lds.S | 30 ++++++++++++++++++++++++------
 2 files changed, 25 insertions(+), 7 deletions(-)

diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
index 6cb9fc7e9382..7221494bcf60 100644
--- a/arch/arm64/Makefile
+++ b/arch/arm64/Makefile
@@ -18,7 +18,7 @@ ifeq ($(CONFIG_RELOCATABLE), y)
 # Pass --no-apply-dynamic-relocs to restore pre-binutils-2.27 behaviour
 # for relative relocs, since this leads to better Image compression
 # with the relocation offsets always being zero.
-LDFLAGS_vmlinux		+= -pie -shared -Bsymbolic \
+LDFLAGS_vmlinux		+= -shared -Bsymbolic \
 			$(call ld-option, --no-apply-dynamic-relocs)
 endif
 
diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
index 03b00007553d..9ba4016090b1 100644
--- a/arch/arm64/kernel/vmlinux.lds.S
+++ b/arch/arm64/kernel/vmlinux.lds.S
@@ -98,8 +98,6 @@ SECTIONS
 		EXIT_CALL
 		*(.discard)
 		*(.discard.*)
-		*(.interp .dynamic)
-		*(.dynsym .dynstr .hash)
 	}
 
 	. = KIMAGE_VADDR + TEXT_OFFSET;
@@ -192,12 +190,25 @@ SECTIONS
 
 	PERCPU_SECTION(L1_CACHE_BYTES)
 
-	.rela : ALIGN(8) {
+#ifdef CONFIG_RELOCATABLE
+	.rela.dyn : ALIGN(8) {
 		*(.rela .rela*)
 	}
-
-	__rela_offset	= ABSOLUTE(ADDR(.rela) - KIMAGE_VADDR);
-	__rela_size	= SIZEOF(.rela);
+	.dynamic : {
+		*(.dynamic)
+	}
+	.dynsym : ALIGN(8) {
+		*(.dynsym)
+	}
+	.dynstr : {
+		*(.dynstr)
+	}
+	.hash : {
+		*(.hash)
+	}
+	__rela_offset	= ABSOLUTE(ADDR(.rela.dyn) - KIMAGE_VADDR);
+	__rela_size	= SIZEOF(.rela.dyn);
+#endif
 
 	. = ALIGN(SEGMENT_ALIGN);
 	__initdata_end = .;
@@ -244,6 +255,13 @@ SECTIONS
 	HEAD_SYMBOLS
 }
 
+#ifdef CONFIG_RELOCATABLE
+VERSION {
+	/* mark all symbols as local so they are not listed in .dynsym */
+	{ local: *; };
+}
+#endif
+
 /*
  * The HYP init code and ID map text can't be longer than a page each,
  * and should not cross a page boundary.
-- 
2.19.2


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFT PATCH] arm64: relocatable: build the kernel as a proper shared library

[Nick Desaulniers]

On Sat, Dec 1, 2018 at 3:54 AM Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
>
> readelf complains about the section layout of vmlinux when building
> with CONFIG_RELOCATABLE=y (for KASLR):
>
>   readelf: Warning: [21]: Link field (0) should index a symtab section.
>   readelf: Warning: [21]: Info field (0) should index a relocatable section.
>
> Also, it seems that our use of '-pie -shared' is contradictory, and
> thus ambiguous. In general, the way KASLR is wired up at the moment
> is highly tailored to how ld.bfd happens to implement (and conflate)
> PIE executables and shared libraries, so given the current effort to
> support other toolchains, let's fix some of these issues as well.
>
> - Drop the -pie linker argument and just leave -shared. In ld.bfd,
>   the differences between them are unclear (except for the ELF type
>   of the produced image [0]) but lld chokes on seeing both at the
>   same time.

This lines up with Peter's suggestion in
https://bugs.llvm.org/show_bug.cgi?id=39810#c15

>
> - Rename the .rela output section to .rela.dyn, as is customary for
>   shared libraries and PIE executables.
>
> - Don't discard the .dynamic, .dynsym, .dynstr and .hash sections.
>   Instead, make sure that they are [mostly] empty by marking all
>   symbols as local, and emit them into the .init segment.
>
> These changes only affect the ELF image, and produce the same binary
> image, with the exception of a couple of bytes of .init data for the
> empty .dynsym and .dynstr sections.
>
> [0] b9dce7f1ba01 ("arm64: kernel: force ET_DYN ELF type for ...")
>
> Cc: Nick Desaulniers <ndesaulniers@google.com>
> Cc: Peter Smith <peter.smith@linaro.org>
> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>

Thanks for this patch Ard!
Boot tested in QEMU w/ and w/o CONFIG_RELOCATABLE=y.
Tested-by: Nick Desaulniers <ndesaulniers@google.com>

> ---
>  arch/arm64/Makefile             |  2 +-
>  arch/arm64/kernel/vmlinux.lds.S | 30 ++++++++++++++++++++++++------
>  2 files changed, 25 insertions(+), 7 deletions(-)
>
> diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
> index 6cb9fc7e9382..7221494bcf60 100644
> --- a/arch/arm64/Makefile
> +++ b/arch/arm64/Makefile
> @@ -18,7 +18,7 @@ ifeq ($(CONFIG_RELOCATABLE), y)
>  # Pass --no-apply-dynamic-relocs to restore pre-binutils-2.27 behaviour
>  # for relative relocs, since this leads to better Image compression
>  # with the relocation offsets always being zero.
> -LDFLAGS_vmlinux                += -pie -shared -Bsymbolic \
> +LDFLAGS_vmlinux                += -shared -Bsymbolic \
>                         $(call ld-option, --no-apply-dynamic-relocs)
>  endif
>
> diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
> index 03b00007553d..9ba4016090b1 100644
> --- a/arch/arm64/kernel/vmlinux.lds.S
> +++ b/arch/arm64/kernel/vmlinux.lds.S
> @@ -98,8 +98,6 @@ SECTIONS
>                 EXIT_CALL
>                 *(.discard)
>                 *(.discard.*)
> -               *(.interp .dynamic)
> -               *(.dynsym .dynstr .hash)
>         }
>
>         . = KIMAGE_VADDR + TEXT_OFFSET;
> @@ -192,12 +190,25 @@ SECTIONS
>
>         PERCPU_SECTION(L1_CACHE_BYTES)
>
> -       .rela : ALIGN(8) {
> +#ifdef CONFIG_RELOCATABLE
> +       .rela.dyn : ALIGN(8) {
>                 *(.rela .rela*)
>         }
> -
> -       __rela_offset   = ABSOLUTE(ADDR(.rela) - KIMAGE_VADDR);
> -       __rela_size     = SIZEOF(.rela);
> +       .dynamic : {
> +               *(.dynamic)
> +       }
> +       .dynsym : ALIGN(8) {
> +               *(.dynsym)
> +       }
> +       .dynstr : {
> +               *(.dynstr)
> +       }
> +       .hash : {
> +               *(.hash)
> +       }
> +       __rela_offset   = ABSOLUTE(ADDR(.rela.dyn) - KIMAGE_VADDR);
> +       __rela_size     = SIZEOF(.rela.dyn);
> +#endif
>
>         . = ALIGN(SEGMENT_ALIGN);
>         __initdata_end = .;
> @@ -244,6 +255,13 @@ SECTIONS
>         HEAD_SYMBOLS
>  }
>
> +#ifdef CONFIG_RELOCATABLE
> +VERSION {
> +       /* mark all symbols as local so they are not listed in .dynsym */
> +       { local: *; };
> +}
> +#endif
> +
>  /*
>   * The HYP init code and ID map text can't be longer than a page each,
>   * and should not cross a page boundary.
> --
> 2.19.2
>


-- 
Thanks,
~Nick Desaulniers

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFT PATCH] arm64: relocatable: build the kernel as a proper shared library

[Will Deacon]

Hi Ard,

On Sat, Dec 01, 2018 at 12:53:24PM +0100, Ard Biesheuvel wrote:
> readelf complains about the section layout of vmlinux when building
> with CONFIG_RELOCATABLE=y (for KASLR):
> 
>   readelf: Warning: [21]: Link field (0) should index a symtab section.
>   readelf: Warning: [21]: Info field (0) should index a relocatable section.
> 
> Also, it seems that our use of '-pie -shared' is contradictory, and
> thus ambiguous. In general, the way KASLR is wired up at the moment
> is highly tailored to how ld.bfd happens to implement (and conflate)
> PIE executables and shared libraries, so given the current effort to
> support other toolchains, let's fix some of these issues as well.
> 
> - Drop the -pie linker argument and just leave -shared. In ld.bfd,
>   the differences between them are unclear (except for the ELF type
>   of the produced image [0]) but lld chokes on seeing both at the
>   same time.
> 
> - Rename the .rela output section to .rela.dyn, as is customary for
>   shared libraries and PIE executables.
> 
> - Don't discard the .dynamic, .dynsym, .dynstr and .hash sections.
>   Instead, make sure that they are [mostly] empty by marking all
>   symbols as local, and emit them into the .init segment.

These second two changes seem fairly arbitrary to me: is lld unhappy without
them? My concern is that we'll make some other change in the future and run
into more arbitrary breakage. Is there any build bots hooked up with lld so
help us catch regressions quickly? (i.e nightly builds of mainline and next)

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFT PATCH] arm64: relocatable: build the kernel as a proper shared library

[Ard Biesheuvel]

On Mon, 3 Dec 2018 at 18:08, Will Deacon <will.deacon@arm.com> wrote:
>
> Hi Ard,
>
> On Sat, Dec 01, 2018 at 12:53:24PM +0100, Ard Biesheuvel wrote:
> > readelf complains about the section layout of vmlinux when building
> > with CONFIG_RELOCATABLE=y (for KASLR):
> >
> >   readelf: Warning: [21]: Link field (0) should index a symtab section.
> >   readelf: Warning: [21]: Info field (0) should index a relocatable section.
> >
> > Also, it seems that our use of '-pie -shared' is contradictory, and
> > thus ambiguous. In general, the way KASLR is wired up at the moment
> > is highly tailored to how ld.bfd happens to implement (and conflate)
> > PIE executables and shared libraries, so given the current effort to
> > support other toolchains, let's fix some of these issues as well.
> >
> > - Drop the -pie linker argument and just leave -shared. In ld.bfd,
> >   the differences between them are unclear (except for the ELF type
> >   of the produced image [0]) but lld chokes on seeing both at the
> >   same time.
> >
> > - Rename the .rela output section to .rela.dyn, as is customary for
> >   shared libraries and PIE executables.
> >
> > - Don't discard the .dynamic, .dynsym, .dynstr and .hash sections.
> >   Instead, make sure that they are [mostly] empty by marking all
> >   symbols as local, and emit them into the .init segment.
>
> These second two changes seem fairly arbitrary to me: is lld unhappy without
> them? My concern is that we'll make some other change in the future and run
> into more arbitrary breakage. Is there any build bots hooked up with lld so
> help us catch regressions quickly? (i.e nightly builds of mainline and next)
>

The .rela.dyn change is for readelf actually, not for lld. Discarding
the .dyn* sections is indeed something lld chokes on.

It seems reasonable to make the vmlinux ELF image look as much as an
ordinary shared library as we can, to avoid issues with tooling in
general, not just lld. This is why I separated this patch from the one
that caters for lld in particular.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFT PATCH] arm64: relocatable: build the kernel as a proper shared library

[Peter Smith]

On Mon, 3 Dec 2018 at 17:08, Will Deacon <will.deacon@arm.com> wrote:
>
> Hi Ard,
>
> On Sat, Dec 01, 2018 at 12:53:24PM +0100, Ard Biesheuvel wrote:
> > readelf complains about the section layout of vmlinux when building
> > with CONFIG_RELOCATABLE=y (for KASLR):
> >
> >   readelf: Warning: [21]: Link field (0) should index a symtab section.
> >   readelf: Warning: [21]: Info field (0) should index a relocatable section.
> >
> > Also, it seems that our use of '-pie -shared' is contradictory, and
> > thus ambiguous. In general, the way KASLR is wired up at the moment
> > is highly tailored to how ld.bfd happens to implement (and conflate)
> > PIE executables and shared libraries, so given the current effort to
> > support other toolchains, let's fix some of these issues as well.
> >
> > - Drop the -pie linker argument and just leave -shared. In ld.bfd,
> >   the differences between them are unclear (except for the ELF type
> >   of the produced image [0]) but lld chokes on seeing both at the
> >   same time.
> >
> > - Rename the .rela output section to .rela.dyn, as is customary for
> >   shared libraries and PIE executables.
> >
> > - Don't discard the .dynamic, .dynsym, .dynstr and .hash sections.
> >   Instead, make sure that they are [mostly] empty by marking all
> >   symbols as local, and emit them into the .init segment.
>
> These second two changes seem fairly arbitrary to me: is lld unhappy without
> them? My concern is that we'll make some other change in the future and run
> into more arbitrary breakage. Is there any build bots hooked up with lld so
> help us catch regressions quickly? (i.e nightly builds of mainline and next)
>
> Will

Hello Will,

LLD currently doesn't permit the .dynamic, .dynstr and .hash sections
from being removed. At the moment the Linaro kernel CI job doesn't
have a LLD configuration. We are planning to rectify this shortly,
hopefully next week and if not then straight after Christmas.

If you do need support for removal of .dynamic etc. to happen it would
be great if you could comment on the LLD PR
https://bugs.llvm.org/show_bug.cgi?id=39810 . There is some pushback
over whether LLD needs to do this if it has already been worked
around.

Peter

p.s. apologies to the people that got this twice, the message bounced
first time.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFT PATCH] arm64: relocatable: build the kernel as a proper shared library

[Nick Desaulniers]

On Mon, Dec 3, 2018 at 9:08 AM Will Deacon <will.deacon@arm.com> wrote:
>
> Hi Ard,
>
> On Sat, Dec 01, 2018 at 12:53:24PM +0100, Ard Biesheuvel wrote:
> > readelf complains about the section layout of vmlinux when building
> > with CONFIG_RELOCATABLE=y (for KASLR):
> >
> >   readelf: Warning: [21]: Link field (0) should index a symtab section.
> >   readelf: Warning: [21]: Info field (0) should index a relocatable section.
> >
> > Also, it seems that our use of '-pie -shared' is contradictory, and
> > thus ambiguous. In general, the way KASLR is wired up at the moment
> > is highly tailored to how ld.bfd happens to implement (and conflate)
> > PIE executables and shared libraries, so given the current effort to
> > support other toolchains, let's fix some of these issues as well.
> >
> > - Drop the -pie linker argument and just leave -shared. In ld.bfd,
> >   the differences between them are unclear (except for the ELF type
> >   of the produced image [0]) but lld chokes on seeing both at the
> >   same time.
> >
> > - Rename the .rela output section to .rela.dyn, as is customary for
> >   shared libraries and PIE executables.
> >
> > - Don't discard the .dynamic, .dynsym, .dynstr and .hash sections.
> >   Instead, make sure that they are [mostly] empty by marking all
> >   symbols as local, and emit them into the .init segment.
>
> These second two changes seem fairly arbitrary to me: is lld unhappy without
> them?

I don't know about .rela -> .rela.dyn, or .hash, but LLD developers
just issued patches allowing the following sections to be discarded:

.dynamic:
https://reviews.llvm.org/D55211
.dynsym:
https://reviews.llvm.org/D55218
.dynstr:
https://reviews.llvm.org/D55215

> My concern is that we'll make some other change in the future and run
> into more arbitrary breakage. Is there any build bots hooked up with lld so
> help us catch regressions quickly? (i.e nightly builds of mainline and next)

Yes.  We're running continuous integration at:
https://github.com/ClangBuiltLinux/continuous-integration

We have nightly cron jobs building arm64 defconfigs with clang for
mainline, -next, and some LTS branches of the stable tree. ex.
https://travis-ci.com/ClangBuiltLinux/continuous-integration/builds/93312295

Some arm64 targets are additionally being linked with LLD.
CONFIG_RANDOMIZE_BASE is not on in the defconfig (see:
http://lists.infradead.org/pipermail/linux-arm-kernel/2018-December/616877.html)

We're working with kernel CI to get Clang builds (and LLD) setup ASAP,
as they cover more trees, bisect failures and notify developers
automatically.
-- 
Thanks,
~Nick Desaulniers

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFT PATCH] arm64: relocatable: build the kernel as a proper shared library

[Will Deacon]

On Mon, Dec 03, 2018 at 06:11:47PM +0100, Ard Biesheuvel wrote:
> On Mon, 3 Dec 2018 at 18:08, Will Deacon <will.deacon@arm.com> wrote:
> >
> > Hi Ard,
> >
> > On Sat, Dec 01, 2018 at 12:53:24PM +0100, Ard Biesheuvel wrote:
> > > readelf complains about the section layout of vmlinux when building
> > > with CONFIG_RELOCATABLE=y (for KASLR):
> > >
> > >   readelf: Warning: [21]: Link field (0) should index a symtab section.
> > >   readelf: Warning: [21]: Info field (0) should index a relocatable section.
> > >
> > > Also, it seems that our use of '-pie -shared' is contradictory, and
> > > thus ambiguous. In general, the way KASLR is wired up at the moment
> > > is highly tailored to how ld.bfd happens to implement (and conflate)
> > > PIE executables and shared libraries, so given the current effort to
> > > support other toolchains, let's fix some of these issues as well.
> > >
> > > - Drop the -pie linker argument and just leave -shared. In ld.bfd,
> > >   the differences between them are unclear (except for the ELF type
> > >   of the produced image [0]) but lld chokes on seeing both at the
> > >   same time.
> > >
> > > - Rename the .rela output section to .rela.dyn, as is customary for
> > >   shared libraries and PIE executables.
> > >
> > > - Don't discard the .dynamic, .dynsym, .dynstr and .hash sections.
> > >   Instead, make sure that they are [mostly] empty by marking all
> > >   symbols as local, and emit them into the .init segment.
> >
> > These second two changes seem fairly arbitrary to me: is lld unhappy without
> > them? My concern is that we'll make some other change in the future and run
> > into more arbitrary breakage. Is there any build bots hooked up with lld so
> > help us catch regressions quickly? (i.e nightly builds of mainline and next)
> >
> 
> The .rela.dyn change is for readelf actually, not for lld. Discarding
> the .dyn* sections is indeed something lld chokes on.
> 
> It seems reasonable to make the vmlinux ELF image look as much as an
> ordinary shared library as we can, to avoid issues with tooling in
> general, not just lld. This is why I separated this patch from the one
> that caters for lld in particular.

Given that the LLD developers are looking at addressing the section
discards, can you repost without that part please?

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFT PATCH] arm64: relocatable: build the kernel as a proper shared library

[Will Deacon]

On Mon, Dec 03, 2018 at 10:29:00AM -0800, Nick Desaulniers wrote:
> On Mon, Dec 3, 2018 at 9:08 AM Will Deacon <will.deacon@arm.com> wrote:
> >
> > Hi Ard,
> >
> > On Sat, Dec 01, 2018 at 12:53:24PM +0100, Ard Biesheuvel wrote:
> > > readelf complains about the section layout of vmlinux when building
> > > with CONFIG_RELOCATABLE=y (for KASLR):
> > >
> > >   readelf: Warning: [21]: Link field (0) should index a symtab section.
> > >   readelf: Warning: [21]: Info field (0) should index a relocatable section.
> > >
> > > Also, it seems that our use of '-pie -shared' is contradictory, and
> > > thus ambiguous. In general, the way KASLR is wired up at the moment
> > > is highly tailored to how ld.bfd happens to implement (and conflate)
> > > PIE executables and shared libraries, so given the current effort to
> > > support other toolchains, let's fix some of these issues as well.
> > >
> > > - Drop the -pie linker argument and just leave -shared. In ld.bfd,
> > >   the differences between them are unclear (except for the ELF type
> > >   of the produced image [0]) but lld chokes on seeing both at the
> > >   same time.
> > >
> > > - Rename the .rela output section to .rela.dyn, as is customary for
> > >   shared libraries and PIE executables.
> > >
> > > - Don't discard the .dynamic, .dynsym, .dynstr and .hash sections.
> > >   Instead, make sure that they are [mostly] empty by marking all
> > >   symbols as local, and emit them into the .init segment.
> >
> > These second two changes seem fairly arbitrary to me: is lld unhappy without
> > them?
> 
> I don't know about .rela -> .rela.dyn, or .hash, but LLD developers
> just issued patches allowing the following sections to be discarded:
> 
> .dynamic:
> https://reviews.llvm.org/D55211
> .dynsym:
> https://reviews.llvm.org/D55218
> .dynstr:
> https://reviews.llvm.org/D55215

Thanks again.

> > My concern is that we'll make some other change in the future and run
> > into more arbitrary breakage. Is there any build bots hooked up with lld so
> > help us catch regressions quickly? (i.e nightly builds of mainline and next)
> 
> Yes.  We're running continuous integration at:
> https://github.com/ClangBuiltLinux/continuous-integration
> 
> We have nightly cron jobs building arm64 defconfigs with clang for
> mainline, -next, and some LTS branches of the stable tree. ex.
> https://travis-ci.com/ClangBuiltLinux/continuous-integration/builds/93312295

Great, that looks very promising! Hopefully that means we can keep things
building reliably once we get to that point.

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel