Broken 32-bit riscv debug build with ZSTD and FTRACE

Broken 32-bit riscv debug build with ZSTD and FTRACE

[Marco Bonelli]

I am building v6.15 for 32-bit riscv and I have noticed that modpost fails with
a ton of errors and warnings (tens of thousands). I am on a Debian 12 x86-64
host using the riscv32-linux-gcc from mirrors.edge.kernel.org [1]. I can
reproduce with different GCC versions (I tested with 10.1.0, 14.2.0, 15.1.0).
For whatever reason though, only when targeting riscv 32-bit.

Steps to reproduce:

    export ARCH=riscv CROSS_COMPILE=riscv32-linux-
    make distclean
    make defconfig
    make 32-bit.config
    ./scripts/config \
        -e DEBUG_KERNEL \
        -e DEBUG_INFO \
        -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
        -d DEBUG_INFO_NONE \
        -d DEBUG_INFO_REDUCED
    make olddefconfig
    make -j vmlinux

Build output:

    ...
    MODPOST vmlinux.symvers
    WARNING: modpost: vmlinux: section mismatch in reference: tcp_write_timer_handler+0xa4 (section: .text) -> register_kernel_offset_dumper (section: .init.text)
    ...
    ERROR: modpost: vmlinux: local symbol 'find_get_pid' was exported
    ...
    ERROR: modpost: __ex_table+0x17e8 references non-executable section '.debug_str'
    ...
    make[2]: *** [scripts/Makefile.modpost:147: vmlinux.symvers] Error 1
    make[1]: *** [/mnt/m2data/linux/Makefile:1958: modpost] Error 2
    make: *** [Makefile:248: __sub-make] Error 2

There are thousands of warnings/errors like the three above, complaining
about "section mismatch in reference", "local symbol xxx was exported" and
"references non-executable section".

I did a bisect run on torvalds/linux and narrowed it down to:

    e61f33273ca755b3e2ebee4520a76097199dc7a8 Merge tag 'zstd-linus-v6.15-rc1' of https://github.com/terrelln/linux

I was also able to reproduce on next/linux-next tag next-20250530. 
Looking at the merge from github.com/terrelln/linux it seems to be a single
commit [2].

Through trial and error I noticed that the build only seems to fail when
*all* the following conditions are met:

1. Building for 32-bit riscv (ARCH=riscv + defconfig + 32-bit.config)
2. FTRACE=y
3. ZSTD_COMPRESS=y
4. Debug info enabled

With ZSTD_COMPRESS=m (selecting SECURITY_APPARMOR_EXPORT_BINARY=n) or
ZSTD_COMPRESS=n (selecting also BTRFS_FS=n), modpost goes through fine.
With FTRACE=n or disabling debug info modpost also goes through fine.

So it seems to be a combination of factors, but I am unsure how exactly
these things can interact/interfere with each other. Any clue?

Full bisect run log:

    # bad: [0ff41df1cb268fc69e703a08a57ee14ae967d0ca] Linux 6.15
    # good: [38fec10eb60d687e30c8c6b5420d86e8149f7557] Linux 6.14
    git bisect start 'v6.15' 'v6.14'
    # bad: [390513642ee6763c7ada07f0a1470474986e6c1c] io_uring: always do atomic put from iowq
    git bisect bad 390513642ee6763c7ada07f0a1470474986e6c1c
    # good: [9b960d8cd6f712cb2c03e2bdd4d5ca058238037f] Merge tag 'for-6.15/block-20250322' of git://git.kernel.dk/linux
    git bisect good 9b960d8cd6f712cb2c03e2bdd4d5ca058238037f
    # good: [023b1e9d265ca0662111a9df23d22b4632717a8a] Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net
    git bisect good 023b1e9d265ca0662111a9df23d22b4632717a8a
    # bad: [3a90a72aca0a98125f0c7350ffb7cc63665f8047] Merge tag 'asm-generic-6.15-2' of git://git.kernel.org/pub/scm/linux/kernel/git/arnd/asm-generic
    git bisect bad 3a90a72aca0a98125f0c7350ffb7cc63665f8047
    # good: [3ef47a0436219359a0838d74bb353caa6aad3fc0] Merge tag 'at91-dt-6.15' of https://git.kernel.org/pub/scm/linux/kernel/git/at91/linux into soc/dt
    git bisect good 3ef47a0436219359a0838d74bb353caa6aad3fc0
    # bad: [1a9239bb4253f9076b5b4b2a1a4e8d7defd77a95] Merge tag 'net-next-6.15' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next
    git bisect bad 1a9239bb4253f9076b5b4b2a1a4e8d7defd77a95
    # good: [ffd6c179286daf31ea0d6413228bbc2a3c7ee1de] Merge patch series "scsi: ufs: renesas: Add support for R-Car S4-8 ES1.2"
    git bisect good ffd6c179286daf31ea0d6413228bbc2a3c7ee1de
    # good: [22df63a23a9e53d06ff2c67f863e9ce1640b73cb] Merge branches 'apple/dart', 'arm/smmu/updates', 'arm/smmu/bindings', 'rockchip', 's390', 'core', 'intel/vt-d' and 'amd/amd-vi' into next
    git bisect good 22df63a23a9e53d06ff2c67f863e9ce1640b73cb
    # good: [29fa7d7934216e0a93102a930ef28e2a6ae852b1] selftests/sysctl: fix wording of help messages
    git bisect good 29fa7d7934216e0a93102a930ef28e2a6ae852b1
    # good: [22093997ac9220d3c606313efbf4ce564962d095] Merge tag 'ata-6.15-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/libata/linux
    git bisect good 22093997ac9220d3c606313efbf4ce564962d095
    # good: [a018d1cf990d0c339fe0e29b762ea5dc10567d67] scsi: st: Fix array overflow in st_setup()
    git bisect good a018d1cf990d0c339fe0e29b762ea5dc10567d67
    # good: [336b4dae6dfecc9aa53a3a68c71b9c1c1d466388] Merge tag 'iommu-updates-v6.15' of git://git.kernel.org/pub/scm/linux/kernel/git/iommu/linux
    git bisect good 336b4dae6dfecc9aa53a3a68c71b9c1c1d466388
    # good: [65d1f5507ed2c78c64fce40e44e5574a9419eb09] zstd: Import upstream v1.5.7
    git bisect good 65d1f5507ed2c78c64fce40e44e5574a9419eb09
    # bad: [e61f33273ca755b3e2ebee4520a76097199dc7a8] Merge tag 'zstd-linus-v6.15-rc1' of https://github.com/terrelln/linux
    git bisect bad e61f33273ca755b3e2ebee4520a76097199dc7a8
    # good: [592329e5e94e26080f4815c6cc6cd0f487a91064] Merge tag 'sysctl-6.15-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/sysctl/sysctl
    git bisect good 592329e5e94e26080f4815c6cc6cd0f487a91064
    # first bad commit: [e61f33273ca755b3e2ebee4520a76097199dc7a8] Merge tag 'zstd-linus-v6.15-rc1' of https://github.com/terrelln/linux

[1]: https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/14.2.0/
[2]: https://github.com/terrelln/linux/commit/65d1f5507ed2c78c64fce40e44e5574a9419eb09

--
Marco Bonelli

Re: Broken 32-bit riscv debug build with ZSTD and FTRACE

[Marco Bonelli]

> Steps to reproduce:
> 
>     export ARCH=riscv CROSS_COMPILE=riscv32-linux-
>     make distclean
>     make defconfig
>     make 32-bit.config
>     ./scripts/config \
>         -e DEBUG_KERNEL \
>         -e DEBUG_INFO \
>         -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
>         -d DEBUG_INFO_NONE \
>         -d DEBUG_INFO_REDUCED
>     make olddefconfig
>     make -j vmlinux

Sorry, forgot to add "-e FTRACE" to the steps above. Here it is:

	export ARCH=riscv CROSS_COMPILE=riscv32-linux-
	make distclean
	make defconfig
	make 32-bit.config
	./scripts/config \
		-e DEBUG_KERNEL \
		-e DEBUG_INFO \
		-e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
		-d DEBUG_INFO_NONE \
		-d DEBUG_INFO_REDUCED \
		-e FTRACE
	make olddefconfig
	make -j vmlinux

Everything else still applies.

--
Marco Bonelli

Re: Broken 32-bit riscv debug build with ZSTD and FTRACE

[Alexandre Ghiti]

Hi Marco,

On 5/31/25 13:52, Marco Bonelli wrote:
>> Steps to reproduce:
>>
>>      export ARCH=riscv CROSS_COMPILE=riscv32-linux-
>>      make distclean
>>      make defconfig
>>      make 32-bit.config
>>      ./scripts/config \
>>          -e DEBUG_KERNEL \
>>          -e DEBUG_INFO \
>>          -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
>>          -d DEBUG_INFO_NONE \
>>          -d DEBUG_INFO_REDUCED
>>      make olddefconfig
>>      make -j vmlinux
> Sorry, forgot to add "-e FTRACE" to the steps above. Here it is:
>
> 	export ARCH=riscv CROSS_COMPILE=riscv32-linux-
> 	make distclean
> 	make defconfig
> 	make 32-bit.config
> 	./scripts/config \
> 		-e DEBUG_KERNEL \
> 		-e DEBUG_INFO \
> 		-e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
> 		-d DEBUG_INFO_NONE \
> 		-d DEBUG_INFO_REDUCED \
> 		-e FTRACE
> 	make olddefconfig
> 	make -j vmlinux
>
> Everything else still applies.
>
> --
> Marco Bonelli
>
> _______________________________________________
> linux-riscv mailing list
> linux-riscv@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-riscv


First, thanks for the report!

I unfortunately cannot reproduce this issue locally, I tried on both 
v6.15 and latest linus master, with gcc 13.1.0 and gcc 14.2.0 . I made 
sure that I have FTRACE, ZSTD_COMPRESS and DEBUG_INFO enabled.

Can you attach your full config?

Thanks,

Alex

Re: Broken 32-bit riscv debug build with ZSTD and FTRACE

[Marco Bonelli]

> Can you attach your full config?

Here it is: https://pastebin.com/raw/wL2Q38g0 - uploaded to pastebin to
avoid clogging everyone's inbox.

> I unfortunately cannot reproduce this issue locally, I tried on both 
> v6.15 and latest linus master, with gcc 13.1.0 and gcc 14.2.0 . I made 
> sure that I have FTRACE, ZSTD_COMPRESS and DEBUG_INFO enabled.

Are you building on a x86-64 host? Maybe something in my build environment
is relevant (not sure what it could be). I am building on x86-64 Debian 12
with host GCC 12.2.0, target cross GCC 14.2.0 [1]. 

Here is a minimal Dockerfile that reproduces the issue (docker build should
fail on the last command when it gets to MODPOST vmlinux.symvers):

	# docker build --progress=plain .
	FROM debian:12

	ENV DEBIAN_FRONTEND=noninteractive
	RUN apt-get update && \
		apt-get install -y \
			build-essential git make gcc-12 binutils util-linux bc gawk flex \
			bison dwarves tar wget

	WORKDIR /work
	RUN git clone --depth 1 --single-branch --branch v6.15 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
	RUN wget 'https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/14.2.0/x86_64-gcc-14.2.0-nolibc-riscv32-linux.tar.xz'
	RUN tar xf x86_64-gcc-14.2.0-nolibc-riscv32-linux.tar.xz

	WORKDIR /work/linux
	ENV PATH=/work/gcc-14.2.0-nolibc/riscv32-linux/bin:$PATH
	ENV CROSS_COMPILE=riscv32-linux-
	ENV ARCH=riscv
	RUN make distclean && \
		make defconfig && \
		make 32-bit.config && \
		./scripts/config \
			-e DEBUG_KERNEL \
			-e DEBUG_INFO \
			-e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
			-d DEBUG_INFO_NONE \
			-d DEBUG_INFO_REDUCED \
			-e FTRACE && \
		make olddefconfig

	RUN make -j19 vmlinux

Hope this helps. Let me know if you need any additional info or test.

[1]: https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/14.2.0/x86_64-gcc-14.2.0-nolibc-riscv32-linux.tar.xz

--
Marco Bonelli

Re: Broken 32-bit riscv debug build with ZSTD and FTRACE

[Marco Bonelli]

I was able to also reproduce without ZSTD (i.e. both ZSTD_COMPRESS=n
and ZSTD_DECOMPRESS=n) like this:

	export ARCH=riscv CROSS_COMPILE=riscv32-linux-
	make distclean
	make defconfig
	make 32-bit.config
	./scripts/config \
		-e FTRACE \
		-e CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
		-d RD_ZSTD \
		-d SECURITY_APPARMOR_INTROSPECT_POLICY \
		-d BTRFS_FS
	make olddefconfig
	make -j vmlinux

Did another bisect run between v6.14 and v6.15 with the above commands
in a bash script and got:

	494e7fe591bf834d57c6607cdc26ab8873708aa7 Merge tag 'bpf_res_spin_lock' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next

This leaves me more puzzled than before honestly. Not sure whether it is
a real bug or a problem on my end at this point? The fact that I can repro
in a Docker makes me think of the former, but the fact that I was able to
bisect it down to two different commits depending on ZSTD vs no ZSTD is
weird.

Alex (or anyone else really): are you able to reproduce with my Dockerfile
or config I provided in my last mail by any chance?

--
Marco Bonelli

Re: Broken 32-bit riscv debug build with ZSTD and FTRACE

[Alexandre Ghiti]

Hi Marco,

On 6/6/25 21:42, Marco Bonelli wrote:
> I was able to also reproduce without ZSTD (i.e. both ZSTD_COMPRESS=n
> and ZSTD_DECOMPRESS=n) like this:
>
> 	export ARCH=riscv CROSS_COMPILE=riscv32-linux-
> 	make distclean
> 	make defconfig
> 	make 32-bit.config
> 	./scripts/config \
> 		-e FTRACE \
> 		-e CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
> 		-d RD_ZSTD \
> 		-d SECURITY_APPARMOR_INTROSPECT_POLICY \
> 		-d BTRFS_FS
> 	make olddefconfig
> 	make -j vmlinux
>
> Did another bisect run between v6.14 and v6.15 with the above commands
> in a bash script and got:
>
> 	494e7fe591bf834d57c6607cdc26ab8873708aa7 Merge tag 'bpf_res_spin_lock' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next
>
> This leaves me more puzzled than before honestly. Not sure whether it is
> a real bug or a problem on my end at this point? The fact that I can repro
> in a Docker makes me think of the former, but the fact that I was able to
> bisect it down to two different commits depending on ZSTD vs no ZSTD is
> weird.
>
> Alex (or anyone else really): are you able to reproduce with my Dockerfile
> or config I provided in my last mail by any chance?

I'm able to reproduce your issue with your Dockerfile and locally too 
now using your last instructions, with the rv32 toolchain you provided 
and my regular rv64 toolchain (13.1.0).

So, that's weird, let's take the following failure:

ERROR: modpost: vmlinux: local symbol 'riscv_cached_mvendorid' was exported

When attaching with gdb, I get this:

Breakpoint 2, check_export_symbol (sym=0x7fffcfc5aca8, 
secname=0x7ffff7b7e3aa ".debug_str", faddr=72, elf=0x7fffffffdec0, 
mod=0x55555556b2a0) at ../scripts/mod/modpost.c:1087
1087            error("%s: local symbol '%s' was exported\n", mod->name,
1: mod->name = 0x55555556b334 "vmlinux"
2: label_name = 0x7fffdfcd965f "__export_symbol_riscv_cached_mvendorid"
(gdb) p/x sym->st_info
$1 = 0x0
(gdb) p/x *sym
$2 = {st_name = 0x321ded, st_value = 0x3d9111, st_size = 0x0, st_info = 
0x0,
   st_other = 0x0, st_shndx = 0xb6}

where st_info == 0 == STB_LOCAL.

On a working rv64 build with the same configs, I get:

Breakpoint 1, check_export_symbol (sym=0x7fffc56d94a8, 
secname=0x7ffff7ae9e54 ".text", faddr=120, elf=0x7fffffffded0, 
mod=0x55555556b2a0) at ../scripts/mod/modpost.c:1085
1085        if (ELF_ST_BIND(sym->st_info) != STB_GLOBAL &&
(gdb) p/x *sym
$1 = {st_name = 0xe6aa, st_info = 0x12, st_other = 0x0, st_shndx = 0x2,
   st_value = 0x11ba, st_size = 0x2c}

The difference is that the symbol is not in the same section.

I'm still looking into it, just wanted to let you know that I can 
reproduce and hoping someone who knows could jump in :)

Thanks,

Alex


>
> --
> Marco Bonelli

Re: Broken 32-bit riscv debug build with ZSTD and FTRACE

[Marco Bonelli]

I did some digging adding prints to modpost.c, trying to understand the errors
complaining about "local symbol 'xxx' was exported" for now. For reference, I am
using commit 494e7fe591bf834d57c6607cdc26ab8873708aa7 for the broken build and
tag v6.14 for the sane build. Config as per my previous mail:

	export ARCH=riscv CROSS_COMPILE=riscv32-linux-
	make distclean
	make defconfig
	make 32-bit.config
	./scripts/config \
		-e FTRACE \
		-e CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
		-d RD_ZSTD \
		-d SECURITY_APPARMOR_INTROSPECT_POLICY \
		-d BTRFS_FS
	make olddefconfig
	make -j vmlinux

It seems like modpost is right: the sanity check for exported symbols fails
because somehow some of the entries of .rela.export_symbol reference wrong
LOCAL symbols. 

Inspecting the sane vmlinux.o built on v6.14 I see:

	$ readelf -W -r vmlinux.o | grep -A10 -F .rela.export_symbol
	Relocation section '.rela.export_symbol' at offset 0x255c28c contains 11089 entries:
	 Offset     Info    Type                Sym. Value  Symbol's Name + Addend
	00000004  0862be01 R_RISCV_32             00000028   system_state + 0
	00000010  0892f901 R_RISCV_32             00000018   static_key_initialized + 0
	00000018  084f0e01 R_RISCV_32             00000014   reset_devices + 0
	00000020  08933601 R_RISCV_32             00000008   loops_per_jiffy + 0
	0000002c  083fcf01 R_RISCV_32             00000110   init_uts_ns + 0
	00000038  08761f01 R_RISCV_32             00000354   wait_for_initramfs + 0
	00000040  084c4c01 R_RISCV_32             00000340   init_task + 0
	00000048  0863e901 R_RISCV_32             00000aca   riscv_cached_mvendorid + 0
	00000050  08423d01 R_RISCV_32             00000af4   riscv_cached_marchid + 0

While for the broken vmlinux.o on 494e7fe5 I see:

	$ readelf -W -r vmlinux.o | grep -A10 -F .rela.export_symbol
	Relocation section '.rela.export_symbol' at offset 0x27aa24cc contains 11425 entries:
	 Offset     Info    Type                Sym. Value  Symbol's Name + Addend
	00000004  fff15b01 R_RISCV_32             00000028   system_state + 0
	00000010  ffeed801 R_RISCV_32             00000018   static_key_initialized + 0
	00000018  fff07d01 R_RISCV_32             00000014   reset_devices + 0
	00000020  00149201 R_RISCV_32             000036ed   .LASF3838 + 0
	0000002c  ffde7c01 R_RISCV_32             00000324   init_uts_ns + 0
	00000038  ffc4c801 R_RISCV_32             00000b3a   wait_for_initramfs + 0
	00000040  000d6801 R_RISCV_32             00011e04   .LASF2005 + 0
	00000048  ffb72201 R_RISCV_32             000012b0   riscv_cached_mvendorid + 0
	00000050  ffd14901 R_RISCV_32             000012da   riscv_cached_marchid + 0

Notice how the 4th and 7th relocation entries reference some .LASFxxx label in
the broken build, but reference loops_per_jiffy and init_task in the sane build.

Those relocation entries seem bogus and point to LOCAL symbols. For example,
for .LASF3838 we have ELF32_R_SYM(rela->r_info) = 0x1492 = 5266, which is:

	$ readelf -s vmlinux.o | grep ' 5266:'
	5266: 000036ed     0 NOTYPE  LOCAL  DEFAULT  178 .LASF3838

So rela->r_info is bad, and when modpost looks at it, it rightfully complains
about the symbol being local. However, rela->r_offset (0x20) seems correct as it
points to __export_symbol_loops_per_jiffy (taken as offset in .export_symbols).
Therefore, we get the weird error message:

	ERROR: modpost: vmlinux: local symbol 'loops_per_jiffy' was exported

Even though loops_per_jiffy is *GLOBAL* and *should* be exported.

FWIW, the call chain is:

	section_rela() -> check_section_mismatch() -> check_export_symbol().

Anyhow, that's all I have for now. Seems like vmlinux.o is being built with
RELA relocations that have bogus r_info. Not sure how, will have to do more
digging if/when I have more time.

--
Marco Bonelli