[PATCH 5.15 5.10 5.4 0/1] arm64: fix Build ID if CONFIG_MODVERSIONS

[PATCH 5.15 5.10 5.4 0/1] arm64: fix Build ID if CONFIG_MODVERSIONS

[Tom Saeger]

Hi Greg,
    Our internal rpm build for aarch64 started failing after merging
v5.15.60.  The build actually succeeds, but packaging failed due to tools
which extract the Build ID from vmlinux.


Expected:
readelf -n vmlinux

Displaying notes found in: .notes
  Owner                Data size  Description
  GNU                  0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
    Build ID: ae5803ded69a15fb0a75905ec226a76cc1823d22
  Linux                0x00000004 func
   description data: 00 00 00 00
  Linux                0x00000001 OPEN
   description data: 00

Actual (post v5.15.60 merge):
readelf -n vmlinux
<no output>


Digging deeper...

Expected:
readelf --headers vmlinux | sed -ne '/Program Header/,$p'
Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000010000 0xffff800008000000 0xffff800008000000
                 0x0000000002012a00 0x000000000208f724  RWE    0x10000
  NOTE           0x0000000001705948 0xffff8000096f5948 0xffff8000096f5948
                 0x0000000000000054 0x0000000000000054  R      0x4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x10

 Section to Segment mapping:
  Segment Sections...
   00     .head.text .text .got.plt .rodata .pci_fixup __ksymtab __ksymtab_gpl __kcrctab __kcrctab_gpl __ksymtab_strings __param __modver __ex_table .notes .hyp.rodata .init.text .exit.text .altinstructions .init.data .data..percpu .hyp.data..percpu .hyp.reloc .rela.dyn .data __bug_table .mmuoff.data.write .mmuoff.data.read .pecoff_edata_padding .bss
   01     .notes
   02

Actual:
readelf --headers vmlinux | sed -ne '/Program Header/,$p'
Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000010000 0xffff800008000000 0xffff800008000000
                 0x0000000002012a00 0x000000000208f724  RWE    0x10000
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x10

 Section to Segment mapping:
  Segment Sections...
   00     .head.text .text .got.plt .rodata .pci_fixup __ksymtab __ksymtab_gpl __kcrctab __kcrctab_gpl __ksymtab_strings __param __modver __ex_table .notes .hyp.rodata .init.text .exit.text .altinstructions .init.data .data..percpu .hyp.data..percpu .hyp.reloc .rela.dyn .data __bug_table .mmuoff.data.write .mmuoff.data.read .pecoff_edata_padding .bss
   01

readelf -n fails since NOTE segment is missing from vmlinux.

Why?

5.15 Commit: 4c7ee827da2c ("Makefile: link with -z noexecstack --no-warn-rwx-segments")
surfaced this issue.

1. Reverting this patch fixed this issue.

2. Removing all .note.GNU-stack sections in cmd_modversions_S also worked:

diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index 17aa8ef2d52a..b95cfbb43cee 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -379,6 +379,8 @@ cmd_modversions_S =								\
 	if $(OBJDUMP) -h $@ | grep -q __ksymtab; then				\
 		$(call cmd_gensymtypes_S,$(KBUILD_SYMTYPES),$(@:.o=.symtypes))	\
 		    > $(@D)/.tmp_$(@F:.o=.ver);					\
+		echo "SECTIONS { /DISCARD/ : { *(.note.GNU-stack) }}"		\
+		>> $(@D)/.tmp_$(@F:.o=.ver); 					\
 										\
 		$(LD) $(KBUILD_LDFLAGS) -r -o $(@D)/.tmp_$(@F) $@ 		\
 			-T $(@D)/.tmp_$(@F:.o=.ver);				\

3. But the simplest fix I found was to remove -z noexecstack just for
head.S (this patch).


This issue exists, in my testing of arm64 builds, for stable versions
5.15.30+, 5.10.136+, and 5.4.210+
and reproduces with every attempted combination of gcc{11,12}
ld{2.36,2.37,2.38,2.39}.

It can also be reproduced upstream by cherry-picking 0d362be5b142 on-top until
7b4537199a4a ("kbuild: link symbol CRCs at final link, removing CONFIG_MODULE_REL_CRCS")
which was part of merge
df202b452fe6 ("Merge tag 'kbuild-v5.19' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild")

It seems kernels prior to v5.4 and 5.19+ use differing KBUILD rules
(with CONFIG_MODVERSIONS=y) for linking components of vmlinux, and in my
testing these other versions did not encounter this issue.

Other architectures besides arm64 might also have similar bug/feature of ld.
Though, x86_64 v5.15.60 worked as expected, so no further experiments
were performed for x86_64.

arm64 repro test:

make ARCH=arm64 mrproper
cp arch/arm64/configs/defconfig ".config"
scripts/config -e MODVERSIONS
make ARCH=arm64 olddefconfig
make ARCH=arm64 -j16 vmlinux
readelf -n vmlinux | grep -F "Build ID:"

Please consider applying for 5.15, 5.10, and 5.4.


Regards,

--Tom

Tom Saeger (1):
  arm64: fix Build ID if CONFIG_MODVERSIONS

 arch/arm64/kernel/Makefile | 5 +++++
 1 file changed, 5 insertions(+)


base-commit: e4a7232c917cd1b56d5b4fa9d7a23e3eabfecba0
-- 
2.38.1

[PATCH 5.15 5.10 5.4 1/1] arm64: fix Build ID if CONFIG_MODVERSIONS

[Tom Saeger]

Backport of: 0d362be5b142 ("Makefile: link with -z noexecstack --no-warn-rwx-segments")
breaks arm64 Build ID when CONFIG_MODVERSIONS=y.

CONFIG_MODVERSIONS adds extra tooling to calculate symbol versions.
This kernel's KBUILD tooling uses both
relocatable (-r) and (-z noexecstack) to link head.o
which results in ld adding a .note.GNU-stack section.
Final linking of vmlinux should add a .NOTES segment containing the
Build ID, but does NOT if head.o has a .note.GNU-stack section.

Selectively remove -z noexecstack from head.o's KBUILD_LDFLAGS to
prevent .note.GNU-stack from being added to head.o.  Final link of
vmlinux then properly adds .NOTES segment containing Build ID that can
be read using tools like 'readelf -n'.

Cc: <stable@vger.kernel.org> # 5.15, 5.10, 5.4
Signed-off-by: Tom Saeger <tom.saeger@oracle.com>
---
 arch/arm64/kernel/Makefile | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile
index 749e31475e41..8bdee0f655e2 100644
--- a/arch/arm64/kernel/Makefile
+++ b/arch/arm64/kernel/Makefile
@@ -85,3 +85,8 @@ extra-y					+= $(head-y) vmlinux.lds
 ifeq ($(CONFIG_DEBUG_EFI),y)
 AFLAGS_head.o += -DVMLINUX_PATH="\"$(realpath $(objtree)/vmlinux)\""
 endif
+
+ifeq ($(CONFIG_MODVERSIONS),y)
+# filter-out -z noexecstack for head.S
+$(obj)/head.o: KBUILD_LDFLAGS := $(shell echo "$(KBUILD_LDFLAGS)" | sed -e 's/-z\s\+noexecstack//')
+endif
-- 
2.38.1

Re: [PATCH 5.15 5.10 5.4 1/1] arm64: fix Build ID if CONFIG_MODVERSIONS

[Greg Kroah-Hartman]

On Tue, Dec 06, 2022 at 01:43:08PM -0700, Tom Saeger wrote:
> Backport of: 0d362be5b142 ("Makefile: link with -z noexecstack --no-warn-rwx-segments")
> breaks arm64 Build ID when CONFIG_MODVERSIONS=y.
> 
> CONFIG_MODVERSIONS adds extra tooling to calculate symbol versions.
> This kernel's KBUILD tooling uses both
> relocatable (-r) and (-z noexecstack) to link head.o
> which results in ld adding a .note.GNU-stack section.
> Final linking of vmlinux should add a .NOTES segment containing the
> Build ID, but does NOT if head.o has a .note.GNU-stack section.
> 
> Selectively remove -z noexecstack from head.o's KBUILD_LDFLAGS to
> prevent .note.GNU-stack from being added to head.o.  Final link of
> vmlinux then properly adds .NOTES segment containing Build ID that can
> be read using tools like 'readelf -n'.
> 
> Cc: <stable@vger.kernel.org> # 5.15, 5.10, 5.4
> Signed-off-by: Tom Saeger <tom.saeger@oracle.com>
> ---
>  arch/arm64/kernel/Makefile | 5 +++++
>  1 file changed, 5 insertions(+)

Why isn't this needed in Linus's tree?

And why not cc: everyone involved in this, I would need acks from
maintainers to be able to accept this...

thanks,

greg k-h

Re: [PATCH 5.15 5.10 5.4 1/1] arm64: fix Build ID if CONFIG_MODVERSIONS

[Tom Saeger]

On Thu, Dec 08, 2022 at 09:34:40PM +0100, Greg Kroah-Hartman wrote:
> On Tue, Dec 06, 2022 at 01:43:08PM -0700, Tom Saeger wrote:
> > Backport of: 0d362be5b142 ("Makefile: link with -z noexecstack --no-warn-rwx-segments")
> > breaks arm64 Build ID when CONFIG_MODVERSIONS=y.
> > 
> > CONFIG_MODVERSIONS adds extra tooling to calculate symbol versions.
> > This kernel's KBUILD tooling uses both
> > relocatable (-r) and (-z noexecstack) to link head.o
> > which results in ld adding a .note.GNU-stack section.
> > Final linking of vmlinux should add a .NOTES segment containing the
> > Build ID, but does NOT if head.o has a .note.GNU-stack section.
> > 
> > Selectively remove -z noexecstack from head.o's KBUILD_LDFLAGS to
> > prevent .note.GNU-stack from being added to head.o.  Final link of
> > vmlinux then properly adds .NOTES segment containing Build ID that can
> > be read using tools like 'readelf -n'.
> > 
> > Cc: <stable@vger.kernel.org> # 5.15, 5.10, 5.4
> > Signed-off-by: Tom Saeger <tom.saeger@oracle.com>
> > ---
> >  arch/arm64/kernel/Makefile | 5 +++++
> >  1 file changed, 5 insertions(+)
> 
> Why isn't this needed in Linus's tree?

0d362be5b142 ("Makefile: link with -z noexecstack --no-warn-rwx-segments")

was merged after
7b4537199a4a ("kbuild: link symbol CRCs at final link, removing CONFIG_MODULE_REL_CRCS")

Linus's tree never had -z noexecstack with these same KBUILD rules.

> 
> And why not cc: everyone involved in this, I would need acks from
> maintainers to be able to accept this...

Fair request.

Between ~5.3 and 5.19-rc1 cherry-picking 
0d362be5b142 ("Makefile: link with -z noexecstack --no-warn-rwx-segments")
and building arm64 with CONFIG_MODVERSIONS=y
results in vmlinux missing Build ID

head.S is compiled to head.o
head.o is linked (ld) with -r and -z noexecstack  which adds .note.GNU-stack section in head.o
head.o is then linked again with vmlinux (resulting vmlinux is missing .NOTE segment)


Can folks confirm/deny ld behavior is expected (arm64)?
And whether the above patch would be an acceptable fix for these kernel
versions?

repro test in cover letter: https://lore.kernel.org/all/cover.1670358255.git.tom.saeger@oracle.com/#r

Regards,
--Tom


> 
> thanks,
> 
> greg k-h

Re: [PATCH 5.15 5.10 5.4 1/1] arm64: fix Build ID if CONFIG_MODVERSIONS

[Greg Kroah-Hartman]

On Thu, Dec 08, 2022 at 05:31:06PM -0600, Tom Saeger wrote:
> On Thu, Dec 08, 2022 at 09:34:40PM +0100, Greg Kroah-Hartman wrote:
> > On Tue, Dec 06, 2022 at 01:43:08PM -0700, Tom Saeger wrote:
> > > Backport of: 0d362be5b142 ("Makefile: link with -z noexecstack --no-warn-rwx-segments")
> > > breaks arm64 Build ID when CONFIG_MODVERSIONS=y.
> > > 
> > > CONFIG_MODVERSIONS adds extra tooling to calculate symbol versions.
> > > This kernel's KBUILD tooling uses both
> > > relocatable (-r) and (-z noexecstack) to link head.o
> > > which results in ld adding a .note.GNU-stack section.
> > > Final linking of vmlinux should add a .NOTES segment containing the
> > > Build ID, but does NOT if head.o has a .note.GNU-stack section.
> > > 
> > > Selectively remove -z noexecstack from head.o's KBUILD_LDFLAGS to
> > > prevent .note.GNU-stack from being added to head.o.  Final link of
> > > vmlinux then properly adds .NOTES segment containing Build ID that can
> > > be read using tools like 'readelf -n'.
> > > 
> > > Cc: <stable@vger.kernel.org> # 5.15, 5.10, 5.4
> > > Signed-off-by: Tom Saeger <tom.saeger@oracle.com>
> > > ---
> > >  arch/arm64/kernel/Makefile | 5 +++++
> > >  1 file changed, 5 insertions(+)
> > 
> > Why isn't this needed in Linus's tree?
> 
> 0d362be5b142 ("Makefile: link with -z noexecstack --no-warn-rwx-segments")
> 
> was merged after
> 7b4537199a4a ("kbuild: link symbol CRCs at final link, removing CONFIG_MODULE_REL_CRCS")
> 
> Linus's tree never had -z noexecstack with these same KBUILD rules.

Then it needs to say that, in detail, in this changelog please.

> > And why not cc: everyone involved in this, I would need acks from
> > maintainers to be able to accept this...
> 
> Fair request.
> 
> Between ~5.3 and 5.19-rc1 cherry-picking 
> 0d362be5b142 ("Makefile: link with -z noexecstack --no-warn-rwx-segments")
> and building arm64 with CONFIG_MODVERSIONS=y
> results in vmlinux missing Build ID
> 
> head.S is compiled to head.o
> head.o is linked (ld) with -r and -z noexecstack  which adds .note.GNU-stack section in head.o
> head.o is then linked again with vmlinux (resulting vmlinux is missing .NOTE segment)
> 
> 
> Can folks confirm/deny ld behavior is expected (arm64)?
> And whether the above patch would be an acceptable fix for these kernel
> versions?
> 
> repro test in cover letter: https://lore.kernel.org/all/cover.1670358255.git.tom.saeger@oracle.com/#r

Please resend, no need for a cover letter for just one patch, and
include all of the needed info in the changelog itself.

thanks,

greg k-h

Re: [PATCH 5.15 5.10 5.4 1/1] arm64: fix Build ID if CONFIG_MODVERSIONS

[Nick Desaulniers]

On Thu, Dec 8, 2022 at 3:31 PM Tom Saeger <tom.saeger@oracle.com> wrote:
>
> Can folks confirm/deny ld behavior is expected (arm64)?
> And whether the above patch would be an acceptable fix for these kernel
> versions?

If you remove `-z noexecstack`, aren't you just going to trigger
warnings from BFD again?

At the least consider adding a fixes tag for 0d362be5b142, and note
that stable doesn't have
7b4537199a4a, in the commit message.
-- 
Thanks,
~Nick Desaulniers

Re: [PATCH 5.15 5.10 5.4 1/1] arm64: fix Build ID if CONFIG_MODVERSIONS

[Tom Saeger]

On Fri, Dec 09, 2022 at 09:58:42AM -0800, Nick Desaulniers wrote:
> On Thu, Dec 8, 2022 at 3:31 PM Tom Saeger <tom.saeger@oracle.com> wrote:
> >
> > Can folks confirm/deny ld behavior is expected (arm64)?
> > And whether the above patch would be an acceptable fix for these kernel
> > versions?
> 
> If you remove `-z noexecstack`, aren't you just going to trigger
> warnings from BFD again?

hmm, probably so, let me check

> 
> At the least consider adding a fixes tag for 0d362be5b142, and note
> that stable doesn't have
> 7b4537199a4a, in the commit message.
> -- 
> Thanks,
> ~Nick Desaulniers