Re: [PATCH v2 0/4] Enable measuring the kernel's Source-based Code Coverage and MC/DC with Clang

Nathan Chancellor
2025-08-29T18:10:18Z

Hi Jinghao and Wentao,

On Thu, Nov 21, 2024 at 11:05:14PM -0600, Jinghao Jia wrote:
...
> On 10/3/24 6:29 PM, Nathan Chancellor wrote:
> > I seem to have narrowed down it to a few different configurations on top
> > of x86_64_defconfig but I will include the full bad configuration as an
> > attachment just in case anything else is relevant.
...
> > $ qemu-system-x86_64 \
> > -display none \
> > -nodefaults \
> > -M q35 \
> > -d unimp,guest_errors \
> > -append 'console=ttyS0 earlycon=uart8250,io,0x3f8' \
> > -kernel arch/x86/boot/bzImage
> > -initrd rootfs.cpio \
> > -cpu host \
> > -enable-kvm \
> > -m 8G \
> > -smp 8 \
> > -serial mon:stdio
> > <hangs with no output>
>
> This hang is caused by an early boot exception -- gdb shows the execution
> reaches the halt loop in early_fixup_exception(). Dumping regs->ip associated
> with this exception points us to the following instruction:
>
> ffffffff89b58074: 48 ff 05 85 7f 4a 76 incq 0x764a7f85(%rip) # 0 <fixed_percpu_data>
>
> This is apparently an incorrect access to the per-cpu variable (the cpu offset
> in %gs is needed) and triggers a null-ptr-deref. Without CONFIG_AMD_MEM_ENCRYPT
> (one of the bad configs), it turns out the instruction is actually accessing
> the llvm prof-counter of strscpy():
>
> ffffffff89b85a04: 48 ff 05 6d 94 7d fa incq -0x5826b93(%rip) # ffffffff8435ee78 <__profc__Z13sized_strscpyPcU25pass_dynamic_object_size1PKcU25pass_dynamic_object_size1m>
>
> This symbol is left undefined in the bad vmlinux, which explains why the
> faulting instruction is accessing address 0. Tracing through the kernel
> linking process shows that the symbol is still defined (as a weak symbol) in
> vmlinux.a and vmlinux.o, but becomes undefined after the first round of linking
> of the kernel image (.tmp_vmlinux1).
>
> After playing with it a little bit, we found the creation of vmlinux.o to be
> the problem. Specifically, if we use mold[1] instead of lld to create the
> object and pass it to the later stages of kernel linking, the symbol will be
> properly defined as a data symbol (and the kernel can boot).
>
> It seems that the issue does not reproduce with LLVM-20. Nevertheless we have
> reported[2] this to upstream llvm.
>
> [1]: https://github.com/rui314/mold
> [2]: https://github.com/llvm/llvm-project/issues/116575

Sasha pinged me on IRC earlier this week about this series and this
issue, noting that he was unable to reproduce it with a similar
toolchain version and the instructions above. I was able to confirm that
at 6.17-rc1 with this patch set applied (after fixing a couple of minor
conflicts), I no longer see this boot issue but it is still reproducible
on 6.12.

In attempting to narrow my bisect window to find what patch fixes this
issue, I noticed that this configuration actually fails to build with

Absolute reference to symbol '__llvm_prf_cnts' not permitted in .head.text

in 6.15 and 6.14 as a result of Ard's commit faf0ed487415 ("x86/boot:
Reject absolute references in .head.text"). Bisecting between 6.15 and
6.16 reveals Ard's commit a3cbbb4717e1 ("x86/boot: Move SEV startup code
into startup/") resolves the build error and that kernel boots, which
seems to make sense to me given what code was involved here. It is
possible that arch/x86/boot/startup will want 'LLVM_COV_PROFILE := n'
since all other instrumentation is disabled.

I built v6.17-rc1 + this series with a fuller distribution configuration
and CONFIG_LLVM_COV_PROFILE_ALL=y. That kernel boots fine in QEMU but I
have done no further evaluation.

Cheers,
Nathan