* RISC-V 32-bit debug builds reaching breaking point: too many symbols
@ 2025-06-18 10:55 ` Marco Bonelli
0 siblings, 0 replies; 6+ messages in thread
From: Marco Bonelli @ 2025-06-18 10:55 UTC (permalink / raw)
To: linux-riscv@lists.infradead.org
Cc: linux-kernel@vger.kernel.org, paul.walmsley@sifive.com,
aou@eecs.berkeley.edu, palmer@dabbelt.com, alex@ghiti.fr
RISC-V debug builds generate *millions* of symbols in vmlinux.o, and this number
is now getting beyond the limit of the maximum symbol table index that can be
represented by 32-bit Rela relocations. The ELF32_R_SYM portion of Elf32_Rela
r_info is only 24 bits, therefore symtab indexes larger than 16777215 overflow
and cause bogus Rela entries pointing to wrong symbols, ultimately breaking the
build.
I recently noticed this [1] when "MODPOST vmlinux.symvers" failed with thousands
of errors and warnings on a particular build of mine for v6.15 RISC-V 32-bit.
The majority (99%) of the symbols are for local temporary labels (.Lxxx) that
would normally be stripped by default at link time, but seems they cannot be
stripped as they are referenced by Rela relocations. The number of such symbols
has always been huge from the very first RISC-V Linux version (v4.15) and has
been steadily growing since (I plotted a bar chart for v6.5-v6.15 here [3]).
For reference, on v4.15 a simple defconfig + debug build with GCC 11.1 produces
a vmlinux.o with around 7 million such symbols. On v6.15 with GCC 14.2 it gets
to around 15 million. We are close to the 16.8M limit, and already exceeding it
on some configurations (the discussion in [1] is an example).
What can be done to reduce them down to an acceptable number, or even better get
rid of them entirely? These local temporary symbols referenced by relocations
seem to be ubiquitous, so I suppose this is some RISC-V ELF ABI design choice.
Could those Rela relocs just avoid referencing any symbol? Or could Rel/Relr be
used instead?
Otherwise, if those really need to be kept as is, perhaps splitting debug info
out of vmlinux.o before linking into final vmlinux (when those are finally
stripped) could also be a viable solution, though the debug info would have to
be split into multiple files or the same problem would arise.
Thoughts?
Here are some stats from a custom script I hacked together (can provide source
if needed).
# Clean v6.15 tree
export PATH=/path/to/gcc-14.2.0-nolibc/riscv32-linux/bin:$PATH
export ARCH=riscv CROSS_COMPILE=riscv32-linux-
make defconfig
make 32-bit.config
./scripts/config -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT
make olddefconfig
make -j vmlinux
Stats:
Total symbols: 15138556
Temporary local symbols: 14972044
Referenced by relocations: 14971527
By section:
.rela.debug_info 8783573
.rela.debug_line 3649412
.rela.debug_ranges 1110915
.rela.debug_loc 972349
.rela.text 398789
.rela.debug_frame 283422
.rela.rodata 17182
.rela.init.text 15732
.rela__bug_table 11163
.rela__jump_table 10830
.rela.debug_aranges 10253
.rela.alternative 4510
.rela.data 3504
.rela.text.unlikely 2650
.rela__ex_table 2044
.rela.sched.text 1405
.rela.init.rodata 541
.rela.exit.text 538
.rela.ref.text 344
.rela.init.data 321
.rela.noinstr.text 234
.rela.spinlock.text 194
.rela.data..ro_after_init 98
.rela.cpuidle.text 36
.rela.srodata 30
.rela__modver 27
.rela.head.text 21
.rela.sdata 15
.rela.irqentry.text 12
.relaruntime_shift_d_hash_shift 10
.relaruntime_ptr_dentry_hashtable 10
.rela.lsm_info.init 3
.rela.data..percpu 1
By relocation kind:
R_RISCV_32 9665623
R_RISCV_SUB16 3652043
R_RISCV_ADD16 3647245
R_RISCV_ADD32 1174187
R_RISCV_SUB32 345881
R_RISCV_BRANCH 168074
R_RISCV_RVC_BRANCH 104031
R_RISCV_SET6 87790
R_RISCV_SUB6 87790
R_RISCV_PCREL_LO12_I 81819
R_RISCV_RVC_JUMP 76539
R_RISCV_SET8 29260
R_RISCV_SUB8 29260
R_RISCV_PCREL_HI20 23416
R_RISCV_SET16 4798
R_RISCV_JAL 4676
R_RISCV_PCREL_LO12_S 1410
R_RISCV_CALL_PLT 1
[1]: https://lore.kernel.org/lkml/960240908.630790.1748641210849@privateemail.com/
[2]: https://x.com/mebeim/status/1934950596693635410
--
Marco Bonelli
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 6+ messages in thread
* RISC-V 32-bit debug builds reaching breaking point: too many symbols
@ 2025-06-18 10:55 ` Marco Bonelli
0 siblings, 0 replies; 6+ messages in thread
From: Marco Bonelli @ 2025-06-18 10:55 UTC (permalink / raw)
To: linux-riscv@lists.infradead.org
Cc: linux-kernel@vger.kernel.org, paul.walmsley@sifive.com,
aou@eecs.berkeley.edu, palmer@dabbelt.com, alex@ghiti.fr
RISC-V debug builds generate *millions* of symbols in vmlinux.o, and this number
is now getting beyond the limit of the maximum symbol table index that can be
represented by 32-bit Rela relocations. The ELF32_R_SYM portion of Elf32_Rela
r_info is only 24 bits, therefore symtab indexes larger than 16777215 overflow
and cause bogus Rela entries pointing to wrong symbols, ultimately breaking the
build.
I recently noticed this [1] when "MODPOST vmlinux.symvers" failed with thousands
of errors and warnings on a particular build of mine for v6.15 RISC-V 32-bit.
The majority (99%) of the symbols are for local temporary labels (.Lxxx) that
would normally be stripped by default at link time, but seems they cannot be
stripped as they are referenced by Rela relocations. The number of such symbols
has always been huge from the very first RISC-V Linux version (v4.15) and has
been steadily growing since (I plotted a bar chart for v6.5-v6.15 here [3]).
For reference, on v4.15 a simple defconfig + debug build with GCC 11.1 produces
a vmlinux.o with around 7 million such symbols. On v6.15 with GCC 14.2 it gets
to around 15 million. We are close to the 16.8M limit, and already exceeding it
on some configurations (the discussion in [1] is an example).
What can be done to reduce them down to an acceptable number, or even better get
rid of them entirely? These local temporary symbols referenced by relocations
seem to be ubiquitous, so I suppose this is some RISC-V ELF ABI design choice.
Could those Rela relocs just avoid referencing any symbol? Or could Rel/Relr be
used instead?
Otherwise, if those really need to be kept as is, perhaps splitting debug info
out of vmlinux.o before linking into final vmlinux (when those are finally
stripped) could also be a viable solution, though the debug info would have to
be split into multiple files or the same problem would arise.
Thoughts?
Here are some stats from a custom script I hacked together (can provide source
if needed).
# Clean v6.15 tree
export PATH=/path/to/gcc-14.2.0-nolibc/riscv32-linux/bin:$PATH
export ARCH=riscv CROSS_COMPILE=riscv32-linux-
make defconfig
make 32-bit.config
./scripts/config -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT
make olddefconfig
make -j vmlinux
Stats:
Total symbols: 15138556
Temporary local symbols: 14972044
Referenced by relocations: 14971527
By section:
.rela.debug_info 8783573
.rela.debug_line 3649412
.rela.debug_ranges 1110915
.rela.debug_loc 972349
.rela.text 398789
.rela.debug_frame 283422
.rela.rodata 17182
.rela.init.text 15732
.rela__bug_table 11163
.rela__jump_table 10830
.rela.debug_aranges 10253
.rela.alternative 4510
.rela.data 3504
.rela.text.unlikely 2650
.rela__ex_table 2044
.rela.sched.text 1405
.rela.init.rodata 541
.rela.exit.text 538
.rela.ref.text 344
.rela.init.data 321
.rela.noinstr.text 234
.rela.spinlock.text 194
.rela.data..ro_after_init 98
.rela.cpuidle.text 36
.rela.srodata 30
.rela__modver 27
.rela.head.text 21
.rela.sdata 15
.rela.irqentry.text 12
.relaruntime_shift_d_hash_shift 10
.relaruntime_ptr_dentry_hashtable 10
.rela.lsm_info.init 3
.rela.data..percpu 1
By relocation kind:
R_RISCV_32 9665623
R_RISCV_SUB16 3652043
R_RISCV_ADD16 3647245
R_RISCV_ADD32 1174187
R_RISCV_SUB32 345881
R_RISCV_BRANCH 168074
R_RISCV_RVC_BRANCH 104031
R_RISCV_SET6 87790
R_RISCV_SUB6 87790
R_RISCV_PCREL_LO12_I 81819
R_RISCV_RVC_JUMP 76539
R_RISCV_SET8 29260
R_RISCV_SUB8 29260
R_RISCV_PCREL_HI20 23416
R_RISCV_SET16 4798
R_RISCV_JAL 4676
R_RISCV_PCREL_LO12_S 1410
R_RISCV_CALL_PLT 1
[1]: https://lore.kernel.org/lkml/960240908.630790.1748641210849@privateemail.com/
[2]: https://x.com/mebeim/status/1934950596693635410
--
Marco Bonelli
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RISC-V 32-bit debug builds reaching breaking point: too many symbols
2025-06-18 10:55 ` Marco Bonelli
(?)
@ 2025-06-18 11:02 ` Abhigyan ghosh
-1 siblings, 0 replies; 6+ messages in thread
From: Abhigyan ghosh @ 2025-06-18 11:02 UTC (permalink / raw)
To: Marco Bonelli; +Cc: linux-kernel
Hi Marco,
Thanks for this incredibly detailed analysis — those stats are eye-opening.
I had a quick question as I'm exploring ELF internals and kernel compilation workflows, especially in the RISC-V context:
Would using split debug info earlier in the build (e.g. extracting `.debug_*` before `vmlinux.o` grows too large) help mitigate the symbol table overflow — or is the linker still forced to carry them through regardless due to reloc dependencies?
Also, does this behavior differ between GNU ld and LLD in your tests?
Appreciate the time you've taken to compile all this — the plots and breakdown by section/reloc type really help visualize the scale.
Best regards,
Abhigyan Ghosh
@zscript.team.zs (zsml.zscript.org)
On 18 June 2025 4:25:08 pm IST, Marco Bonelli <marco@mebeim.net> wrote:
>RISC-V debug builds generate *millions* of symbols in vmlinux.o, and this number
>is now getting beyond the limit of the maximum symbol table index that can be
>represented by 32-bit Rela relocations. The ELF32_R_SYM portion of Elf32_Rela
>r_info is only 24 bits, therefore symtab indexes larger than 16777215 overflow
>and cause bogus Rela entries pointing to wrong symbols, ultimately breaking the
>build.
>
>I recently noticed this [1] when "MODPOST vmlinux.symvers" failed with thousands
>of errors and warnings on a particular build of mine for v6.15 RISC-V 32-bit.
>
>The majority (99%) of the symbols are for local temporary labels (.Lxxx) that
>would normally be stripped by default at link time, but seems they cannot be
>stripped as they are referenced by Rela relocations. The number of such symbols
>has always been huge from the very first RISC-V Linux version (v4.15) and has
>been steadily growing since (I plotted a bar chart for v6.5-v6.15 here [3]).
>
>For reference, on v4.15 a simple defconfig + debug build with GCC 11.1 produces
>a vmlinux.o with around 7 million such symbols. On v6.15 with GCC 14.2 it gets
>to around 15 million. We are close to the 16.8M limit, and already exceeding it
>on some configurations (the discussion in [1] is an example).
>
>What can be done to reduce them down to an acceptable number, or even better get
>rid of them entirely? These local temporary symbols referenced by relocations
>seem to be ubiquitous, so I suppose this is some RISC-V ELF ABI design choice.
>Could those Rela relocs just avoid referencing any symbol? Or could Rel/Relr be
>used instead?
>
>Otherwise, if those really need to be kept as is, perhaps splitting debug info
>out of vmlinux.o before linking into final vmlinux (when those are finally
>stripped) could also be a viable solution, though the debug info would have to
>be split into multiple files or the same problem would arise.
>
>Thoughts?
>
>Here are some stats from a custom script I hacked together (can provide source
>if needed).
>
> # Clean v6.15 tree
> export PATH=/path/to/gcc-14.2.0-nolibc/riscv32-linux/bin:$PATH
> export ARCH=riscv CROSS_COMPILE=riscv32-linux-
>
> make defconfig
> make 32-bit.config
> ./scripts/config -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT
> make olddefconfig
> make -j vmlinux
>
>Stats:
>
> Total symbols: 15138556
> Temporary local symbols: 14972044
> Referenced by relocations: 14971527
> By section:
> .rela.debug_info 8783573
> .rela.debug_line 3649412
> .rela.debug_ranges 1110915
> .rela.debug_loc 972349
> .rela.text 398789
> .rela.debug_frame 283422
> .rela.rodata 17182
> .rela.init.text 15732
> .rela__bug_table 11163
> .rela__jump_table 10830
> .rela.debug_aranges 10253
> .rela.alternative 4510
> .rela.data 3504
> .rela.text.unlikely 2650
> .rela__ex_table 2044
> .rela.sched.text 1405
> .rela.init.rodata 541
> .rela.exit.text 538
> .rela.ref.text 344
> .rela.init.data 321
> .rela.noinstr.text 234
> .rela.spinlock.text 194
> .rela.data..ro_after_init 98
> .rela.cpuidle.text 36
> .rela.srodata 30
> .rela__modver 27
> .rela.head.text 21
> .rela.sdata 15
> .rela.irqentry.text 12
> .relaruntime_shift_d_hash_shift 10
> .relaruntime_ptr_dentry_hashtable 10
> .rela.lsm_info.init 3
> .rela.data..percpu 1
> By relocation kind:
> R_RISCV_32 9665623
> R_RISCV_SUB16 3652043
> R_RISCV_ADD16 3647245
> R_RISCV_ADD32 1174187
> R_RISCV_SUB32 345881
> R_RISCV_BRANCH 168074
> R_RISCV_RVC_BRANCH 104031
> R_RISCV_SET6 87790
> R_RISCV_SUB6 87790
> R_RISCV_PCREL_LO12_I 81819
> R_RISCV_RVC_JUMP 76539
> R_RISCV_SET8 29260
> R_RISCV_SUB8 29260
> R_RISCV_PCREL_HI20 23416
> R_RISCV_SET16 4798
> R_RISCV_JAL 4676
> R_RISCV_PCREL_LO12_S 1410
> R_RISCV_CALL_PLT 1
>
>[1]: https://lore.kernel.org/lkml/960240908.630790.1748641210849@privateemail.com/
>[2]: https://x.com/mebeim/status/1934950596693635410
>
>--
>Marco Bonelli
>
aghosh
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RISC-V 32-bit debug builds reaching breaking point: too many symbols
2025-06-18 10:55 ` Marco Bonelli
(?)
(?)
@ 2025-06-18 11:04 ` Abhigyan ghosh
2025-06-18 19:21 ` Marco Bonelli
-1 siblings, 1 reply; 6+ messages in thread
From: Abhigyan ghosh @ 2025-06-18 11:04 UTC (permalink / raw)
To: Marco Bonelli; +Cc: linux-kernel
Hi Marco,
Thanks for this incredibly detailed analysis — those stats are eye-opening.
I had a quick question as I'm exploring ELF internals and kernel compilation workflows, especially in the RISC-V context:
Would using split debug info earlier in the build (e.g. extracting `.debug_*` before `vmlinux.o` grows too large) help mitigate the symbol table overflow — or is the linker still forced to carry them through regardless due to reloc dependencies?
Also, does this behavior differ between GNU ld and LLD in your tests?
Appreciate the time you've taken to compile all this — the plots and breakdown by section/reloc type really help visualize the scale.
Best regards,
Abhigyan Ghosh
On 18 June 2025 4:25:08 pm IST, Marco Bonelli <marco@mebeim.net> wrote:
>RISC-V debug builds generate *millions* of symbols in vmlinux.o, and this number
>is now getting beyond the limit of the maximum symbol table index that can be
>represented by 32-bit Rela relocations. The ELF32_R_SYM portion of Elf32_Rela
>r_info is only 24 bits, therefore symtab indexes larger than 16777215 overflow
>and cause bogus Rela entries pointing to wrong symbols, ultimately breaking the
>build.
>
>I recently noticed this [1] when "MODPOST vmlinux.symvers" failed with thousands
>of errors and warnings on a particular build of mine for v6.15 RISC-V 32-bit.
>
>The majority (99%) of the symbols are for local temporary labels (.Lxxx) that
>would normally be stripped by default at link time, but seems they cannot be
>stripped as they are referenced by Rela relocations. The number of such symbols
>has always been huge from the very first RISC-V Linux version (v4.15) and has
>been steadily growing since (I plotted a bar chart for v6.5-v6.15 here [3]).
>
>For reference, on v4.15 a simple defconfig + debug build with GCC 11.1 produces
>a vmlinux.o with around 7 million such symbols. On v6.15 with GCC 14.2 it gets
>to around 15 million. We are close to the 16.8M limit, and already exceeding it
>on some configurations (the discussion in [1] is an example).
>
>What can be done to reduce them down to an acceptable number, or even better get
>rid of them entirely? These local temporary symbols referenced by relocations
>seem to be ubiquitous, so I suppose this is some RISC-V ELF ABI design choice.
>Could those Rela relocs just avoid referencing any symbol? Or could Rel/Relr be
>used instead?
>
>Otherwise, if those really need to be kept as is, perhaps splitting debug info
>out of vmlinux.o before linking into final vmlinux (when those are finally
>stripped) could also be a viable solution, though the debug info would have to
>be split into multiple files or the same problem would arise.
>
>Thoughts?
>
>Here are some stats from a custom script I hacked together (can provide source
>if needed).
>
> # Clean v6.15 tree
> export PATH=/path/to/gcc-14.2.0-nolibc/riscv32-linux/bin:$PATH
> export ARCH=riscv CROSS_COMPILE=riscv32-linux-
>
> make defconfig
> make 32-bit.config
> ./scripts/config -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT
> make olddefconfig
> make -j vmlinux
>
>Stats:
>
> Total symbols: 15138556
> Temporary local symbols: 14972044
> Referenced by relocations: 14971527
> By section:
> .rela.debug_info 8783573
> .rela.debug_line 3649412
> .rela.debug_ranges 1110915
> .rela.debug_loc 972349
> .rela.text 398789
> .rela.debug_frame 283422
> .rela.rodata 17182
> .rela.init.text 15732
> .rela__bug_table 11163
> .rela__jump_table 10830
> .rela.debug_aranges 10253
> .rela.alternative 4510
> .rela.data 3504
> .rela.text.unlikely 2650
> .rela__ex_table 2044
> .rela.sched.text 1405
> .rela.init.rodata 541
> .rela.exit.text 538
> .rela.ref.text 344
> .rela.init.data 321
> .rela.noinstr.text 234
> .rela.spinlock.text 194
> .rela.data..ro_after_init 98
> .rela.cpuidle.text 36
> .rela.srodata 30
> .rela__modver 27
> .rela.head.text 21
> .rela.sdata 15
> .rela.irqentry.text 12
> .relaruntime_shift_d_hash_shift 10
> .relaruntime_ptr_dentry_hashtable 10
> .rela.lsm_info.init 3
> .rela.data..percpu 1
> By relocation kind:
> R_RISCV_32 9665623
> R_RISCV_SUB16 3652043
> R_RISCV_ADD16 3647245
> R_RISCV_ADD32 1174187
> R_RISCV_SUB32 345881
> R_RISCV_BRANCH 168074
> R_RISCV_RVC_BRANCH 104031
> R_RISCV_SET6 87790
> R_RISCV_SUB6 87790
> R_RISCV_PCREL_LO12_I 81819
> R_RISCV_RVC_JUMP 76539
> R_RISCV_SET8 29260
> R_RISCV_SUB8 29260
> R_RISCV_PCREL_HI20 23416
> R_RISCV_SET16 4798
> R_RISCV_JAL 4676
> R_RISCV_PCREL_LO12_S 1410
> R_RISCV_CALL_PLT 1
>
>[1]: https://lore.kernel.org/lkml/960240908.630790.1748641210849@privateemail.com/
>[2]: https://x.com/mebeim/status/1934950596693635410
>
>--
>Marco Bonelli
>
aghosh
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RISC-V 32-bit debug builds reaching breaking point: too many symbols
2025-06-18 11:04 ` Abhigyan ghosh
@ 2025-06-18 19:21 ` Marco Bonelli
0 siblings, 0 replies; 6+ messages in thread
From: Marco Bonelli @ 2025-06-18 19:21 UTC (permalink / raw)
To: Abhigyan ghosh; +Cc: linux-kernel
(sorry for the dupe mail, forgot to hit "reply all")
> Would using split debug info earlier in the build (e.g. extracting `.debug_*` before `vmlinux.o` grows too large) help mitigate the symbol table overflow — or is the linker still forced to carry them through regardless due to reloc dependencies?
Wish I knew honestly, not sure. That is essentially what I was asking in my mail.
If I had to take a guess, I think it wouldn't matter and just move the problem
because you would still have a huge split .dwo file, but I am not 100% sure. More
smaller files could be generated (e.g. one per compilation unit) but those would
be painful to carry around.
> Also, does this behavior differ between GNU ld and LLD in your tests?
I have tested a LLVM build and while it does produce less of those annoying
symbols, the same problem persists.
--
Marco Bonelli
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RISC-V 32-bit debug builds reaching breaking point: too many symbols
@ 2025-06-19 6:11 Abhigyan ghosh
0 siblings, 0 replies; 6+ messages in thread
From: Abhigyan ghosh @ 2025-06-19 6:11 UTC (permalink / raw)
To: marco; +Cc: linux-kernel
Hi Marco,
Thanks for the quick reply — and no worries about the dupe!
Got it. That makes sense. I was wondering if splitting earlier would help reduce linker overhead, but I see now that the .dwo files (or .dwp) would likely still carry a heavy load and relocations would remain a bottleneck.
Interesting to hear that the same behavior persists with LLVM — I guess that confirms it's more of a fundamental tooling limit than a front-end quirk.
Appreciate your input! If I end up testing with smaller compilation units or tweaking DWARF emission flags, I’ll report back.
Best regards,
Abhigyan Ghosh
zsml.zscript.org
@zscript.team.zs
aghosh
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2025-06-19 6:11 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-06-19 6:11 RISC-V 32-bit debug builds reaching breaking point: too many symbols Abhigyan ghosh
-- strict thread matches above, loose matches on Subject: below --
2025-06-18 10:55 Marco Bonelli
2025-06-18 10:55 ` Marco Bonelli
2025-06-18 11:02 ` Abhigyan ghosh
2025-06-18 11:04 ` Abhigyan ghosh
2025-06-18 19:21 ` Marco Bonelli
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.