Linux Trace Kernel
 help / color / mirror / Atom feed
* [PATCH] tracing: Disable KCOV instrumentation for trace_irqsoff.o
@ 2026-05-25 17:04 Karl Mehltretter
  2026-05-26  6:07 ` Masami Hiramatsu
  0 siblings, 1 reply; 3+ messages in thread
From: Karl Mehltretter @ 2026-05-25 17:04 UTC (permalink / raw)
  To: Steven Rostedt, Masami Hiramatsu
  Cc: Mathieu Desnoyers, Dmitry Vyukov, Andrey Konovalov, Marco Elver,
	kasan-dev, linux-trace-kernel, linux-kernel, Karl Mehltretter

When KCOV runs its boot selftest with whole-kernel instrumentation
enabled, it sets current->kcov_mode to KCOV_MODE_TRACE_PC without
installing a coverage area. Any instrumented code accepted as task-context
coverage in that window dereferences current->kcov_area and crashes.

On ARMv5 Versatile PB with CONFIG_KCOV_SELFTEST=y,
CONFIG_KCOV_INSTRUMENT_ALL=y and CONFIG_IRQSOFF_TRACER=y, boot hits a
NULL pointer fault during the selftest:

  kcov: running self test
  Internal error: Oops: 5 [#1] ARM
  PC is at __sanitizer_cov_trace_pc+0x4c/0x90
  Kernel panic - not syncing: Fatal exception

A diagnostic run showed the unwanted coverage comes from the IRQs-off
tracer callbacks reached from ARM IRQ entry before hardirq context is
visible to KCOV:

  __sanitizer_cov_trace_pc from tracer_hardirqs_off+0x18/0x1cc
  tracer_hardirqs_off from trace_hardirqs_off+0x34/0x54
  trace_hardirqs_off from __irq_svc+0x58/0xb0
  __irq_svc from kcov_init+0x7c/0xdc

and similarly through tracer_hardirqs_on().

trace_preemptirq.o is already excluded because this tracing path can run
from early interrupt code and produce coverage unrelated to syscall
inputs. Exclude trace_irqsoff.o as well, instead of requiring users to
turn off CONFIG_KCOV_INSTRUMENT_ALL=y, which is the default whole-kernel
KCOV mode.

With the exclusion in place, the same ARMv5 Versatile PB QEMU test boots
through the KCOV selftest and reaches userspace.

Tested on ARMv5 Versatile PB QEMU with CONFIG_KCOV_SELFTEST=y,
CONFIG_KCOV_INSTRUMENT_ALL=y and CONFIG_IRQSOFF_TRACER=y.

Assisted-by: Codex:gpt-5
Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>
---
 kernel/trace/Makefile | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/kernel/trace/Makefile b/kernel/trace/Makefile
index 8d3d96e847d8..f934ff586bd4 100644
--- a/kernel/trace/Makefile
+++ b/kernel/trace/Makefile
@@ -48,9 +48,10 @@ ifdef CONFIG_GCOV_PROFILE_FTRACE
 GCOV_PROFILE := y
 endif
 
-# Functions in this file could be invoked from early interrupt
-# code and produce random code coverage.
+# Functions in these files can run from IRQ entry before hardirq context
+# is visible to KCOV, and produce coverage unrelated to syscall inputs.
 KCOV_INSTRUMENT_trace_preemptirq.o := n
+KCOV_INSTRUMENT_trace_irqsoff.o := n
 
 CFLAGS_bpf_trace.o := -I$(src)
 
-- 
2.39.5 (Apple Git-154)


^ permalink raw reply related	[flat|nested] 3+ messages in thread

* Re: [PATCH] tracing: Disable KCOV instrumentation for trace_irqsoff.o
  2026-05-25 17:04 [PATCH] tracing: Disable KCOV instrumentation for trace_irqsoff.o Karl Mehltretter
@ 2026-05-26  6:07 ` Masami Hiramatsu
  2026-05-26 10:22   ` Karl Mehltretter
  0 siblings, 1 reply; 3+ messages in thread
From: Masami Hiramatsu @ 2026-05-26  6:07 UTC (permalink / raw)
  To: Karl Mehltretter
  Cc: Steven Rostedt, Mathieu Desnoyers, Dmitry Vyukov,
	Andrey Konovalov, Marco Elver, kasan-dev, linux-trace-kernel,
	linux-kernel

On Mon, 25 May 2026 19:04:28 +0200
Karl Mehltretter <kmehltretter@gmail.com> wrote:

> When KCOV runs its boot selftest with whole-kernel instrumentation
> enabled, it sets current->kcov_mode to KCOV_MODE_TRACE_PC without
> installing a coverage area. Any instrumented code accepted as task-context
> coverage in that window dereferences current->kcov_area and crashes.
> 
> On ARMv5 Versatile PB with CONFIG_KCOV_SELFTEST=y,
> CONFIG_KCOV_INSTRUMENT_ALL=y and CONFIG_IRQSOFF_TRACER=y, boot hits a
> NULL pointer fault during the selftest:
> 
>   kcov: running self test
>   Internal error: Oops: 5 [#1] ARM
>   PC is at __sanitizer_cov_trace_pc+0x4c/0x90
>   Kernel panic - not syncing: Fatal exception
> 
> A diagnostic run showed the unwanted coverage comes from the IRQs-off
> tracer callbacks reached from ARM IRQ entry before hardirq context is
> visible to KCOV:
> 
>   __sanitizer_cov_trace_pc from tracer_hardirqs_off+0x18/0x1cc
>   tracer_hardirqs_off from trace_hardirqs_off+0x34/0x54
>   trace_hardirqs_off from __irq_svc+0x58/0xb0
>   __irq_svc from kcov_init+0x7c/0xdc
> 
> and similarly through tracer_hardirqs_on().
> 
> trace_preemptirq.o is already excluded because this tracing path can run
> from early interrupt code and produce coverage unrelated to syscall
> inputs. Exclude trace_irqsoff.o as well, instead of requiring users to
> turn off CONFIG_KCOV_INSTRUMENT_ALL=y, which is the default whole-kernel
> KCOV mode.
> 
> With the exclusion in place, the same ARMv5 Versatile PB QEMU test boots
> through the KCOV selftest and reaches userspace.
> 
> Tested on ARMv5 Versatile PB QEMU with CONFIG_KCOV_SELFTEST=y,
> CONFIG_KCOV_INSTRUMENT_ALL=y and CONFIG_IRQSOFF_TRACER=y.


Thanks for reporting. This looks good to me for a mitigation.
BTW, I could not reproduce the bug with above configs.
Is this only for arm32?


> 
> Assisted-by: Codex:gpt-5
> Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>
> ---
>  kernel/trace/Makefile | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/trace/Makefile b/kernel/trace/Makefile
> index 8d3d96e847d8..f934ff586bd4 100644
> --- a/kernel/trace/Makefile
> +++ b/kernel/trace/Makefile
> @@ -48,9 +48,10 @@ ifdef CONFIG_GCOV_PROFILE_FTRACE
>  GCOV_PROFILE := y
>  endif
>  
> -# Functions in this file could be invoked from early interrupt
> -# code and produce random code coverage.
> +# Functions in these files can run from IRQ entry before hardirq context
> +# is visible to KCOV, and produce coverage unrelated to syscall inputs.
>  KCOV_INSTRUMENT_trace_preemptirq.o := n
> +KCOV_INSTRUMENT_trace_irqsoff.o := n
>  
>  CFLAGS_bpf_trace.o := -I$(src)
>  
> -- 
> 2.39.5 (Apple Git-154)
> 


-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [PATCH] tracing: Disable KCOV instrumentation for trace_irqsoff.o
  2026-05-26  6:07 ` Masami Hiramatsu
@ 2026-05-26 10:22   ` Karl Mehltretter
  0 siblings, 0 replies; 3+ messages in thread
From: Karl Mehltretter @ 2026-05-26 10:22 UTC (permalink / raw)
  To: Masami Hiramatsu
  Cc: Steven Rostedt, Mathieu Desnoyers, Dmitry Vyukov,
	Andrey Konovalov, Marco Elver, kasan-dev, linux-trace-kernel,
	linux-kernel

[-- Attachment #1: Type: text/plain, Size: 3082 bytes --]

On Tue, May 26, 2026 at 03:07:58PM +0100, Masami Hiramatsu wrote:
> Thanks for reporting. This looks good to me for a mitigation.
> BTW, I could not reproduce the bug with above configs.
> Is this only for arm32?

I was able to reproduce this on arm64 QEMU virt with the attached
config and log.

Test base:
  4cbfe4502e3d ("Merge tag 'v7.1-rc5-ksmbd-server-fixes' ...")

QEMU command:
  qemu-system-aarch64 \
    -machine virt,gic-version=2 -cpu cortex-a57 -m 512M -smp 1 \
    -kernel arch/arm64/boot/Image \
    -append "console=ttyAMA0,115200 earlycon=pl011,0x9000000 rdinit=/init panic_on_warn=0 oops=panic loglevel=8 printk.time=1" \
    -nographic -no-reboot

Relevant config options:
  CONFIG_TRACE_IRQFLAGS=y
  CONFIG_IRQSOFF_TRACER=y
  CONFIG_KCOV=y
  CONFIG_KCOV_INSTRUMENT_ALL=y
  CONFIG_KCOV_SELFTEST=y

The raw arm64 crash first runs into other KCOV-instrumented early
IRQ/stack helpers. To isolate the trace_irqsoff.o part, I used the
following additional changes. This is not intended for merge:

diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile
index 74b76bb70452..d69eb3fd0577 100644
--- a/arch/arm64/kernel/Makefile
+++ b/arch/arm64/kernel/Makefile
@@ -24,6 +24,9 @@ KASAN_SANITIZE_stacktrace.o := n
 # inhibit KCOV instrumentation, disable it for the entire compilation unit.
 KCOV_INSTRUMENT_entry-common.o := n
 KCOV_INSTRUMENT_idle.o := n
+KCOV_INSTRUMENT_irq.o := n
+KCOV_INSTRUMENT_return_address.o := n
+KCOV_INSTRUMENT_stacktrace.o := n
 
 # Object file lists.
 obj-y			:= debug-monitors.o entry.o irq.o fpsimd.o		\
diff --git a/kernel/time/Makefile b/kernel/time/Makefile
index eaf290c972f9..2641a44f6339 100644
--- a/kernel/time/Makefile
+++ b/kernel/time/Makefile
@@ -21,6 +21,7 @@ ifeq ($(CONFIG_GENERIC_CLOCKEVENTS_BROADCAST),y)
  obj-$(CONFIG_TICK_ONESHOT)			+= tick-broadcast-hrtimer.o
 endif
 obj-$(CONFIG_GENERIC_SCHED_CLOCK)		+= sched_clock.o
+KCOV_INSTRUMENT_sched_clock.o := n
 obj-$(CONFIG_TICK_ONESHOT)			+= tick-oneshot.o tick-sched.o
 obj-$(CONFIG_LEGACY_TIMER_TICK)			+= tick-legacy.o
 ifeq ($(CONFIG_SMP),y)

With these changes, but with trace_irqsoff.o still instrumented,
the kernel still crashes during the KCOV selftest:

  kcov: running self test
  pc : __sanitizer_cov_trace_pc+0x64/0x84
  Kernel panic - not syncing: kernel stack overflow
  ...
  tracer_hardirqs_off+0x1c/0x78
  trace_hardirqs_off.part.0+0x70/0x1a0
  trace_hardirqs_off_finish+0x60/0x6c
  arm64_enter_from_kernel_mode.isra.0+0x18/0x38
  el1_interrupt+0x24/0x58
  el1h_64_irq+0x6c/0x70
  kcov_init+0xc8/0x118

Then adding the line from my original ARMv5
mitigation makes the arm64 kernel boot through the KCOV selftest:

  KCOV_INSTRUMENT_trace_irqsoff.o := n

The boot log then shows:

  kcov: running self test
  kcov: done running self test
  tiny-init: reached userspace

So arm64 also confirms that trace_irqsoff.o is reachable from this early
IRQ entry path while KCOV selftest mode is active.

Arm64 appears to have additional KCOV/early-entry paths with this config,
which probably need to be investigated independently.

Regards,
Karl

[-- Attachment #2: arm64-kcov.config.gz --]
[-- Type: application/x-gunzip, Size: 11591 bytes --]

[-- Attachment #3: arm64-kcov-trace-irqsoff-crash.log.gz --]
[-- Type: application/x-gunzip, Size: 4458 bytes --]

^ permalink raw reply related	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2026-05-26 10:22 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-05-25 17:04 [PATCH] tracing: Disable KCOV instrumentation for trace_irqsoff.o Karl Mehltretter
2026-05-26  6:07 ` Masami Hiramatsu
2026-05-26 10:22   ` Karl Mehltretter

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox