* [PATCH RFC v3 01/12] kcov: wire up compiler instrumentation for CONFIG_KCOV_EXT_RECORDS
2026-09-08 16:54 [PATCH RFC v3 00/12] KCOV: entry/exit records, memory access records, and delay injection Jann Horn
@ 2026-09-08 16:54 ` Jann Horn
2026-09-08 16:54 ` [PATCH RFC v3 02/12] kcov: refactor mode check out of check_kcov_mode() Jann Horn
` (10 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Jann Horn @ 2026-09-08 16:54 UTC (permalink / raw)
To: Dmitry Vyukov, Andrey Konovalov, Alexander Potapenko
Cc: Nathan Chancellor, Nick Desaulniers, Bill Wendling, Justin Stitt,
linux-kernel, kasan-dev, llvm, Jann Horn
This is the first half of CONFIG_KCOV_EXT_RECORDS.
Set the appropriate compiler flags to call separate hooks for function
entry/exit, and provide these hooks, but don't make it visible in the KCOV
UAPI yet.
With -fsanitize-coverage=trace-pc-entry-exit, the compiler behavior changes
as follows:
- The __sanitizer_cov_trace_pc() call on function entry is replaced with a
call to __sanitizer_cov_trace_pc_entry(); so for now,
__sanitizer_cov_trace_pc_entry() must be treated the same way as
__sanitizer_cov_trace_pc().
- On function exit, an extra call to __sanitizer_cov_trace_pc_exit()
happens; since function exit produced no coverage in the old UAPI,
__sanitizer_cov_trace_pc_exit() should do nothing for now.
This feature was added to LLVM in commit:
https://github.com/llvm/llvm-project/commit/dc5c6d008f487eea8f5d646011f9b3dca6caebd7
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
Signed-off-by: Jann Horn <jannh@google.com>
---
include/linux/kcov.h | 2 ++
kernel/kcov.c | 34 +++++++++++++++++++++++++++-------
lib/Kconfig.debug | 12 ++++++++++++
scripts/Makefile.kcov | 2 ++
tools/objtool/check.c | 2 ++
5 files changed, 45 insertions(+), 7 deletions(-)
diff --git a/include/linux/kcov.h b/include/linux/kcov.h
index 895b761b2db1..cd79715db241 100644
--- a/include/linux/kcov.h
+++ b/include/linux/kcov.h
@@ -79,6 +79,8 @@ typedef unsigned long long kcov_u64;
#endif
void __sanitizer_cov_trace_pc(void);
+void __sanitizer_cov_trace_pc_entry(void);
+void __sanitizer_cov_trace_pc_exit(void);
void __sanitizer_cov_trace_cmp1(u8 arg1, u8 arg2);
void __sanitizer_cov_trace_cmp2(u16 arg1, u16 arg2);
void __sanitizer_cov_trace_cmp4(u32 arg1, u32 arg2);
diff --git a/kernel/kcov.c b/kernel/kcov.c
index 35420f0ac524..5d9686c8b3ec 100644
--- a/kernel/kcov.c
+++ b/kernel/kcov.c
@@ -198,15 +198,10 @@ static notrace unsigned long canonicalize_ip(unsigned long ip)
return ip;
}
-/*
- * Entry point from instrumented code.
- * This is called once per basic-block/edge.
- */
-void notrace __sanitizer_cov_trace_pc(void)
+static __always_inline void notrace kcov_add_pc_record(unsigned long record)
{
struct task_struct *t;
unsigned long *area;
- unsigned long ip = canonicalize_ip(_RET_IP_);
unsigned long pos;
t = current;
@@ -226,11 +221,36 @@ void notrace __sanitizer_cov_trace_pc(void)
*/
WRITE_ONCE(area[0], pos);
barrier();
- area[pos] = ip;
+ area[pos] = record;
}
}
+
+/*
+ * Entry point from instrumented code.
+ * This is called once per basic-block/edge.
+ */
+void notrace __sanitizer_cov_trace_pc(void)
+{
+ kcov_add_pc_record(canonicalize_ip(_RET_IP_));
+}
EXPORT_SYMBOL(__sanitizer_cov_trace_pc);
+#ifdef CONFIG_KCOV_EXT_RECORDS
+void notrace __sanitizer_cov_trace_pc_entry(void)
+{
+ unsigned long record = canonicalize_ip(_RET_IP_);
+
+ /*
+ * This hook replaces __sanitizer_cov_trace_pc() for the function entry
+ * basic block; it should still emit a record even in classic kcov mode.
+ */
+ kcov_add_pc_record(record);
+}
+void notrace __sanitizer_cov_trace_pc_exit(void)
+{
+}
+#endif
+
#ifdef CONFIG_KCOV_ENABLE_COMPARISONS
static void notrace write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip)
{
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index a2f0d3e97889..6ddf58692b09 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -2195,6 +2195,18 @@ config KCOV
For more details, see Documentation/dev-tools/kcov.rst.
+config KCOV_EXT_RECORDS
+ bool "Support extended KCOV records with function entry/exit records"
+ depends on KCOV
+ depends on 64BIT
+ depends on $(cc-option,-fsanitize-coverage=trace-pc-entry-exit)
+ help
+ Extended KCOV records allow distinguishing between multiple types of
+ records: Normal edge coverage, function entry, and function exit.
+
+ This will likely cause a small additional slowdown compared to normal
+ KCOV.
+
config KCOV_ENABLE_COMPARISONS
bool "Enable comparison operands collection by KCOV"
depends on KCOV
diff --git a/scripts/Makefile.kcov b/scripts/Makefile.kcov
index 78305a84ba9d..aa0be904268f 100644
--- a/scripts/Makefile.kcov
+++ b/scripts/Makefile.kcov
@@ -1,10 +1,12 @@
# SPDX-License-Identifier: GPL-2.0-only
kcov-flags-y += -fsanitize-coverage=trace-pc
+kcov-flags-$(CONFIG_KCOV_EXT_RECORDS) += -fsanitize-coverage=trace-pc-entry-exit
kcov-flags-$(CONFIG_KCOV_ENABLE_COMPARISONS) += -fsanitize-coverage=trace-cmp
kcov-rflags-y += -Cpasses=sancov-module
kcov-rflags-y += -Cllvm-args=-sanitizer-coverage-level=3
kcov-rflags-y += -Cllvm-args=-sanitizer-coverage-trace-pc
+kcov-rflags-$(CONFIG_KCOV_EXT_RECORDS) += -Cllvm-args=-sanitizer-coverage-trace-pc-entry-exit
kcov-rflags-$(CONFIG_KCOV_ENABLE_COMPARISONS) += -Cllvm-args=-sanitizer-coverage-trace-compares
export CFLAGS_KCOV := $(kcov-flags-y)
diff --git a/tools/objtool/check.c b/tools/objtool/check.c
index df04e6be2f66..d70cb640e2ec 100644
--- a/tools/objtool/check.c
+++ b/tools/objtool/check.c
@@ -1220,6 +1220,8 @@ static const char *uaccess_safe_builtin[] = {
"write_comp_data",
"check_kcov_mode",
"__sanitizer_cov_trace_pc",
+ "__sanitizer_cov_trace_pc_entry",
+ "__sanitizer_cov_trace_pc_exit",
"__sanitizer_cov_trace_const_cmp1",
"__sanitizer_cov_trace_const_cmp2",
"__sanitizer_cov_trace_const_cmp4",
--
2.55.0.979.g7e5102b832-goog
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH RFC v3 02/12] kcov: refactor mode check out of check_kcov_mode()
2026-09-08 16:54 [PATCH RFC v3 00/12] KCOV: entry/exit records, memory access records, and delay injection Jann Horn
2026-09-08 16:54 ` [PATCH RFC v3 01/12] kcov: wire up compiler instrumentation for CONFIG_KCOV_EXT_RECORDS Jann Horn
@ 2026-09-08 16:54 ` Jann Horn
2026-09-08 16:54 ` [PATCH RFC v3 03/12] kcov: introduce extended PC coverage collection mode Jann Horn
` (9 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Jann Horn @ 2026-09-08 16:54 UTC (permalink / raw)
To: Dmitry Vyukov, Andrey Konovalov, Alexander Potapenko
Cc: Nathan Chancellor, Nick Desaulniers, Bill Wendling, Justin Stitt,
linux-kernel, kasan-dev, llvm, Jann Horn
The following patch will need to check t->kcov_mode in different ways at
different check_kcov_mode() call sites. In preparation for that, move the
mode check up the call hierarchy.
Signed-off-by: Jann Horn <jannh@google.com>
---
kernel/kcov.c | 31 +++++++++++++++++--------------
1 file changed, 17 insertions(+), 14 deletions(-)
diff --git a/kernel/kcov.c b/kernel/kcov.c
index 5d9686c8b3ec..26baccaaefa9 100644
--- a/kernel/kcov.c
+++ b/kernel/kcov.c
@@ -167,10 +167,8 @@ static __always_inline bool in_softirq_really(void)
return in_serving_softirq() && !in_hardirq() && !in_nmi();
}
-static notrace bool check_kcov_mode(enum kcov_mode needed_mode, struct task_struct *t)
+static notrace bool check_kcov_context(struct task_struct *t)
{
- unsigned int mode;
-
/*
* We are interested in code coverage as a function of a syscall inputs,
* so we ignore code executed in interrupts, unless we are in a remote
@@ -178,7 +176,6 @@ static notrace bool check_kcov_mode(enum kcov_mode needed_mode, struct task_stru
*/
if (!in_task() && !(in_softirq_really() && t->kcov_softirq))
return false;
- mode = READ_ONCE(t->kcov_mode);
/*
* There is some code that runs in interrupts but for which
* in_interrupt() returns false (e.g. preempt_schedule_irq()).
@@ -187,7 +184,7 @@ static notrace bool check_kcov_mode(enum kcov_mode needed_mode, struct task_stru
* kcov_start().
*/
barrier();
- return mode == needed_mode;
+ return true;
}
static notrace unsigned long canonicalize_ip(unsigned long ip)
@@ -198,14 +195,12 @@ static notrace unsigned long canonicalize_ip(unsigned long ip)
return ip;
}
-static __always_inline void notrace kcov_add_pc_record(unsigned long record)
+static __always_inline void notrace kcov_add_pc_record(struct task_struct *t, unsigned long record)
{
- struct task_struct *t;
unsigned long *area;
unsigned long pos;
- t = current;
- if (!check_kcov_mode(KCOV_MODE_TRACE_PC, t))
+ if (!check_kcov_context(t))
return;
area = t->kcov_area;
@@ -213,7 +208,7 @@ static __always_inline void notrace kcov_add_pc_record(unsigned long record)
pos = READ_ONCE(area[0]) + 1;
if (likely(pos < t->kcov_size)) {
/* Previously we write pc before updating pos. However, some
- * early interrupt code could bypass check_kcov_mode() check
+ * early interrupt code could bypass check_kcov_context() check
* and invoke __sanitizer_cov_trace_pc(). If such interrupt is
* raised between writing pc and updating pos, the pc could be
* overitten by the recursive __sanitizer_cov_trace_pc().
@@ -231,20 +226,28 @@ static __always_inline void notrace kcov_add_pc_record(unsigned long record)
*/
void notrace __sanitizer_cov_trace_pc(void)
{
- kcov_add_pc_record(canonicalize_ip(_RET_IP_));
+ struct task_struct *cur = current;
+
+ if (READ_ONCE(cur->kcov_mode) != KCOV_MODE_TRACE_PC)
+ return;
+ kcov_add_pc_record(cur, canonicalize_ip(_RET_IP_));
}
EXPORT_SYMBOL(__sanitizer_cov_trace_pc);
#ifdef CONFIG_KCOV_EXT_RECORDS
void notrace __sanitizer_cov_trace_pc_entry(void)
{
+ struct task_struct *cur = current;
unsigned long record = canonicalize_ip(_RET_IP_);
+ unsigned int kcov_mode = READ_ONCE(cur->kcov_mode);
/*
* This hook replaces __sanitizer_cov_trace_pc() for the function entry
* basic block; it should still emit a record even in classic kcov mode.
*/
- kcov_add_pc_record(record);
+ if (kcov_mode != KCOV_MODE_TRACE_PC)
+ return;
+ kcov_add_pc_record(cur, record);
}
void notrace __sanitizer_cov_trace_pc_exit(void)
{
@@ -259,7 +262,7 @@ static void notrace write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip)
u64 count, start_index, end_pos, max_pos;
t = current;
- if (!check_kcov_mode(KCOV_MODE_TRACE_CMP, t))
+ if (READ_ONCE(t->kcov_mode) != KCOV_MODE_TRACE_CMP || !check_kcov_context(t))
return;
ip = canonicalize_ip(ip);
@@ -379,7 +382,7 @@ static void kcov_start(struct task_struct *t, struct kcov *kcov,
t->kcov_size = size;
t->kcov_area = area;
t->kcov_sequence = sequence;
- /* See comment in check_kcov_mode(). */
+ /* See comment in check_kcov_context(). */
barrier();
WRITE_ONCE(t->kcov_mode, mode);
}
--
2.55.0.979.g7e5102b832-goog
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH RFC v3 03/12] kcov: introduce extended PC coverage collection mode
2026-09-08 16:54 [PATCH RFC v3 00/12] KCOV: entry/exit records, memory access records, and delay injection Jann Horn
2026-09-08 16:54 ` [PATCH RFC v3 01/12] kcov: wire up compiler instrumentation for CONFIG_KCOV_EXT_RECORDS Jann Horn
2026-09-08 16:54 ` [PATCH RFC v3 02/12] kcov: refactor mode check out of check_kcov_mode() Jann Horn
@ 2026-09-08 16:54 ` Jann Horn
2026-09-08 16:54 ` [PATCH RFC v3 04/12] kcov: summarize entry/exit while disabled Jann Horn
` (8 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Jann Horn @ 2026-09-08 16:54 UTC (permalink / raw)
To: Dmitry Vyukov, Andrey Konovalov, Alexander Potapenko
Cc: Nathan Chancellor, Nick Desaulniers, Bill Wendling, Justin Stitt,
linux-kernel, kasan-dev, llvm, Jann Horn
This is the second half of CONFIG_KCOV_EXT_RECORDS.
Introduce a new KCOV mode KCOV_TRACE_PC_EXT which replaces the upper 8 bits
of recorded instruction pointers with metadata. For now, userspace can use
this metadata to distinguish three types of records:
- function entry
- function exit
- normal basic block inside the function
Internally, this new mode is represented as a variant of
KCOV_MODE_TRACE_PC, distinguished with the flag KCOV_EXT_FORMAT.
Store this flag as part of the mode in task_struct::kcov_mode and in
kcov::mode to avoid having to pass it around separately everywhere.
Signed-off-by: Jann Horn <jannh@google.com>
---
include/linux/kcov.h | 7 +++++++
include/uapi/linux/kcov.h | 12 ++++++++++++
kernel/kcov.c | 39 ++++++++++++++++++++++++++++++++++-----
3 files changed, 53 insertions(+), 5 deletions(-)
diff --git a/include/linux/kcov.h b/include/linux/kcov.h
index cd79715db241..6c9f0373022f 100644
--- a/include/linux/kcov.h
+++ b/include/linux/kcov.h
@@ -23,8 +23,15 @@ enum kcov_mode {
KCOV_MODE_TRACE_CMP = 3,
};
+/*
+ * Modifier for KCOV_MODE_TRACE_PC to record function entry/exit marked with
+ * metadata bits.
+ */
+#define KCOV_EXT_FORMAT (1 << 29)
#define KCOV_IN_CTXSW (1 << 30)
+#define KCOV_MODE_TRACE_PC_EXT (KCOV_MODE_TRACE_PC | KCOV_EXT_FORMAT)
+
void kcov_task_init(struct task_struct *t);
void kcov_task_exit(struct task_struct *t);
diff --git a/include/uapi/linux/kcov.h b/include/uapi/linux/kcov.h
index ed95dba9fa37..8d8a233bd61f 100644
--- a/include/uapi/linux/kcov.h
+++ b/include/uapi/linux/kcov.h
@@ -35,8 +35,20 @@ enum {
KCOV_TRACE_PC = 0,
/* Collecting comparison operands mode. */
KCOV_TRACE_CMP = 1,
+ /*
+ * Extended PC coverage collection mode.
+ * In this mode, the top byte of the PC is replaced with flag bits
+ * (KCOV_RECORDFLAG_*).
+ */
+ KCOV_TRACE_PC_EXT = 2,
};
+#define KCOV_RECORD_IP_MASK 0x00ffffffffffffff
+#define KCOV_RECORDFLAG_TYPEMASK 0xf000000000000000
+#define KCOV_RECORDFLAG_TYPE_NORMAL 0xf000000000000000
+#define KCOV_RECORDFLAG_TYPE_ENTRY 0x0000000000000000
+#define KCOV_RECORDFLAG_TYPE_EXIT 0x1000000000000000
+
/*
* The format for the types of collected comparisons.
*
diff --git a/kernel/kcov.c b/kernel/kcov.c
index 26baccaaefa9..701ad69493bf 100644
--- a/kernel/kcov.c
+++ b/kernel/kcov.c
@@ -55,7 +55,12 @@ struct kcov {
refcount_t refcount;
/* The lock protects mode, size, area and t. */
spinlock_t lock;
- enum kcov_mode mode __guarded_by(&lock);
+ /*
+ * Mode, consists of:
+ * - enum kcov_mode
+ * - flag KCOV_EXT_FORMAT
+ */
+ unsigned int mode __guarded_by(&lock);
/* Size of arena (in long's). */
unsigned int size __guarded_by(&lock);
/* Coverage buffer shared with user space. */
@@ -228,8 +233,14 @@ void notrace __sanitizer_cov_trace_pc(void)
{
struct task_struct *cur = current;
- if (READ_ONCE(cur->kcov_mode) != KCOV_MODE_TRACE_PC)
+ if ((READ_ONCE(cur->kcov_mode) & ~KCOV_EXT_FORMAT) != KCOV_MODE_TRACE_PC)
return;
+ /*
+ * No bitops are needed here for setting the record type because
+ * KCOV_RECORDFLAG_TYPE_NORMAL has the high bits set.
+ * This relies on userspace not caring about the rest of the top byte
+ * for KCOV_RECORDFLAG_TYPE_NORMAL records.
+ */
kcov_add_pc_record(cur, canonicalize_ip(_RET_IP_));
}
EXPORT_SYMBOL(__sanitizer_cov_trace_pc);
@@ -245,12 +256,28 @@ void notrace __sanitizer_cov_trace_pc_entry(void)
* This hook replaces __sanitizer_cov_trace_pc() for the function entry
* basic block; it should still emit a record even in classic kcov mode.
*/
- if (kcov_mode != KCOV_MODE_TRACE_PC)
+ if ((kcov_mode & ~KCOV_EXT_FORMAT) != KCOV_MODE_TRACE_PC)
return;
+ if ((kcov_mode & KCOV_EXT_FORMAT) != 0)
+ record = (record & KCOV_RECORD_IP_MASK) | KCOV_RECORDFLAG_TYPE_ENTRY;
kcov_add_pc_record(cur, record);
}
void notrace __sanitizer_cov_trace_pc_exit(void)
{
+ struct task_struct *cur = current;
+ unsigned long record;
+
+ /*
+ * This hook is not called at the beginning of a basic block; the basic
+ * block from which the hook was invoked is already covered by a
+ * preceding hook call.
+ * So unlike __sanitizer_cov_trace_pc_entry(), this PC should only be
+ * reported in extended mode, where function exit events are recorded.
+ */
+ if (READ_ONCE(cur->kcov_mode) != KCOV_MODE_TRACE_PC_EXT)
+ return;
+ record = (canonicalize_ip(_RET_IP_) & KCOV_RECORD_IP_MASK) | KCOV_RECORDFLAG_TYPE_EXIT;
+ kcov_add_pc_record(cur, record);
}
#endif
@@ -373,7 +400,7 @@ EXPORT_SYMBOL(__sanitizer_cov_trace_switch);
#endif /* ifdef CONFIG_KCOV_ENABLE_COMPARISONS */
static void kcov_start(struct task_struct *t, struct kcov *kcov,
- unsigned int size, void *area, enum kcov_mode mode,
+ unsigned int size, void *area, unsigned int mode,
int sequence)
{
kcov_debug("t = %px, size = %u, area = %px\n", t, size, area);
@@ -590,6 +617,8 @@ static int kcov_get_mode(unsigned long arg)
#else
return -ENOTSUPP;
#endif
+ else if (arg == KCOV_TRACE_PC_EXT)
+ return IS_ENABLED(CONFIG_KCOV_EXT_RECORDS) ? KCOV_MODE_TRACE_PC_EXT : -ENOTSUPP;
else
return -EINVAL;
}
@@ -1098,7 +1127,7 @@ void kcov_remote_stop(void)
* and kcov_remote_stop(), hence the sequence check.
*/
if (sequence == kcov->sequence && kcov->remote)
- kcov_move_area(kcov->mode, kcov->area, kcov->size, area);
+ kcov_move_area(kcov->mode & ~KCOV_EXT_FORMAT, kcov->area, kcov->size, area);
spin_unlock(&kcov->lock);
spin_lock(&kcov_remote_lock);
--
2.55.0.979.g7e5102b832-goog
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH RFC v3 04/12] kcov: summarize entry/exit while disabled
2026-09-08 16:54 [PATCH RFC v3 00/12] KCOV: entry/exit records, memory access records, and delay injection Jann Horn
` (2 preceding siblings ...)
2026-09-08 16:54 ` [PATCH RFC v3 03/12] kcov: introduce extended PC coverage collection mode Jann Horn
@ 2026-09-08 16:54 ` Jann Horn
2026-09-08 16:54 ` [PATCH RFC v3 05/12] kasan: refactor write/is_write arguments to flags Jann Horn
` (7 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Jann Horn @ 2026-09-08 16:54 UTC (permalink / raw)
To: Dmitry Vyukov, Andrey Konovalov, Alexander Potapenko
Cc: Nathan Chancellor, Nick Desaulniers, Bill Wendling, Justin Stitt,
linux-kernel, kasan-dev, llvm, Jann Horn
In case kcov is re-enabled with a different call stack than the one it
was disabled with, emit events that summarize changes to the call stack
so that userspace can continue tracking the call stack across context
switches.
Signed-off-by: Jann Horn <jannh@google.com>
---
include/linux/kcov.h | 11 ++---------
include/uapi/linux/kcov.h | 2 ++
kernel/kcov.c | 48 +++++++++++++++++++++++++++++++++++++++++++++--
3 files changed, 50 insertions(+), 11 deletions(-)
diff --git a/include/linux/kcov.h b/include/linux/kcov.h
index 6c9f0373022f..357f4de8790a 100644
--- a/include/linux/kcov.h
+++ b/include/linux/kcov.h
@@ -35,15 +35,8 @@ enum kcov_mode {
void kcov_task_init(struct task_struct *t);
void kcov_task_exit(struct task_struct *t);
-#define kcov_prepare_switch(t) \
-do { \
- (t)->kcov_mode |= KCOV_IN_CTXSW; \
-} while (0)
-
-#define kcov_finish_switch(t) \
-do { \
- (t)->kcov_mode &= ~KCOV_IN_CTXSW; \
-} while (0)
+void kcov_prepare_switch(struct task_struct *cur);
+void kcov_finish_switch(struct task_struct *cur);
/* See Documentation/dev-tools/kcov.rst for usage details. */
void kcov_remote_start(u64 handle);
diff --git a/include/uapi/linux/kcov.h b/include/uapi/linux/kcov.h
index 8d8a233bd61f..75c582784055 100644
--- a/include/uapi/linux/kcov.h
+++ b/include/uapi/linux/kcov.h
@@ -48,6 +48,8 @@ enum {
#define KCOV_RECORDFLAG_TYPE_NORMAL 0xf000000000000000
#define KCOV_RECORDFLAG_TYPE_ENTRY 0x0000000000000000
#define KCOV_RECORDFLAG_TYPE_EXIT 0x1000000000000000
+/* Summarized entry/exit events that occurred in an untraced region. */
+#define KCOV_RECORDFLAG_TYPE_EESUM 0x2000000000000000
/*
* The format for the types of collected comparisons.
diff --git a/kernel/kcov.c b/kernel/kcov.c
index 701ad69493bf..712f0f744ec5 100644
--- a/kernel/kcov.c
+++ b/kernel/kcov.c
@@ -76,6 +76,8 @@ struct kcov {
* kcov_remote_stop(), see the comment there.
*/
int sequence;
+ int suppressed_stack_delta;
+ int suppressed_stack_mindelta;
};
struct kcov_remote_area {
@@ -256,8 +258,12 @@ void notrace __sanitizer_cov_trace_pc_entry(void)
* This hook replaces __sanitizer_cov_trace_pc() for the function entry
* basic block; it should still emit a record even in classic kcov mode.
*/
- if ((kcov_mode & ~KCOV_EXT_FORMAT) != KCOV_MODE_TRACE_PC)
+ if ((kcov_mode & ~(KCOV_EXT_FORMAT|KCOV_IN_CTXSW)) != KCOV_MODE_TRACE_PC)
return;
+ if (kcov_mode & KCOV_IN_CTXSW) {
+ cur->kcov->suppressed_stack_delta++;
+ return;
+ }
if ((kcov_mode & KCOV_EXT_FORMAT) != 0)
record = (record & KCOV_RECORD_IP_MASK) | KCOV_RECORDFLAG_TYPE_ENTRY;
kcov_add_pc_record(cur, record);
@@ -266,6 +272,7 @@ void notrace __sanitizer_cov_trace_pc_exit(void)
{
struct task_struct *cur = current;
unsigned long record;
+ unsigned int kcov_mode = READ_ONCE(cur->kcov_mode);
/*
* This hook is not called at the beginning of a basic block; the basic
@@ -274,8 +281,16 @@ void notrace __sanitizer_cov_trace_pc_exit(void)
* So unlike __sanitizer_cov_trace_pc_entry(), this PC should only be
* reported in extended mode, where function exit events are recorded.
*/
- if (READ_ONCE(cur->kcov_mode) != KCOV_MODE_TRACE_PC_EXT)
+ if ((kcov_mode & ~KCOV_IN_CTXSW) != KCOV_MODE_TRACE_PC_EXT)
return;
+ if (kcov_mode & KCOV_IN_CTXSW) {
+ struct kcov *kcov = cur->kcov;
+
+ if (kcov->suppressed_stack_mindelta == kcov->suppressed_stack_delta)
+ kcov->suppressed_stack_mindelta--;
+ kcov->suppressed_stack_delta--;
+ return;
+ }
record = (canonicalize_ip(_RET_IP_) & KCOV_RECORD_IP_MASK) | KCOV_RECORDFLAG_TYPE_EXIT;
kcov_add_pc_record(cur, record);
}
@@ -399,6 +414,35 @@ void notrace __sanitizer_cov_trace_switch(kcov_u64 val, void *arg)
EXPORT_SYMBOL(__sanitizer_cov_trace_switch);
#endif /* ifdef CONFIG_KCOV_ENABLE_COMPARISONS */
+void kcov_prepare_switch(struct task_struct *cur)
+{
+#ifdef CONFIG_KCOV_EXT_RECORDS
+ struct kcov *kcov = cur->kcov;
+
+ if (kcov) {
+ kcov->suppressed_stack_mindelta = 0;
+ kcov->suppressed_stack_delta = 0;
+ }
+#endif
+ cur->kcov_mode |= KCOV_IN_CTXSW;
+}
+
+void kcov_finish_switch(struct task_struct *cur)
+{
+ struct kcov *kcov = cur->kcov;
+ unsigned long record;
+
+ cur->kcov_mode &= ~KCOV_IN_CTXSW;
+ if (!IS_ENABLED(CONFIG_KCOV_EXT_RECORDS))
+ return;
+ if ((cur->kcov_mode & KCOV_EXT_FORMAT) == 0)
+ return;
+ record = KCOV_RECORDFLAG_TYPE_EESUM |
+ (((u16)(s16)kcov->suppressed_stack_mindelta)<<16) |
+ (((u16)(s16)kcov->suppressed_stack_delta)<<16);
+ kcov_add_pc_record(cur, record);
+}
+
static void kcov_start(struct task_struct *t, struct kcov *kcov,
unsigned int size, void *area, unsigned int mode,
int sequence)
--
2.55.0.979.g7e5102b832-goog
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH RFC v3 05/12] kasan: refactor write/is_write arguments to flags
2026-09-08 16:54 [PATCH RFC v3 00/12] KCOV: entry/exit records, memory access records, and delay injection Jann Horn
` (3 preceding siblings ...)
2026-09-08 16:54 ` [PATCH RFC v3 04/12] kcov: summarize entry/exit while disabled Jann Horn
@ 2026-09-08 16:54 ` Jann Horn
2026-09-08 16:54 ` [PATCH RFC v3 06/12] kcov: introduce memory access tracing Jann Horn
` (6 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Jann Horn @ 2026-09-08 16:54 UTC (permalink / raw)
To: Dmitry Vyukov, Andrey Konovalov, Alexander Potapenko
Cc: Nathan Chancellor, Nick Desaulniers, Bill Wendling, Justin Stitt,
linux-kernel, kasan-dev, llvm, Jann Horn
Refactor the "write"/"is_write" arguments of kasan_check_range() and
kasan_report() into "flags" arguments that can contain the flag
KASAN_TYPE_WRITE.
This prepares for a following patch that introduces a second flag.
This should hopefully not change the machine code in hotpaths - both before
and after this change, the argument is either 0 or 1 at the assembly level
depending on whether the access is a write.
Only kasan_report() should have to do a tiny bit of extra work to do a
bittest.
Signed-off-by: Jann Horn <jannh@google.com>
---
arch/arm64/kernel/traps.c | 2 +-
arch/arm64/mm/fault.c | 2 +-
include/linux/kasan.h | 6 ++++--
mm/kasan/common.c | 2 +-
mm/kasan/generic.c | 20 ++++++++++----------
mm/kasan/kasan.h | 6 +++---
mm/kasan/report.c | 3 ++-
mm/kasan/report_generic.c | 8 ++++----
mm/kasan/shadow.c | 24 ++++++++++++------------
mm/kasan/sw_tags.c | 20 ++++++++++----------
10 files changed, 48 insertions(+), 45 deletions(-)
diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
index 914282016069..9f31fe3f1660 100644
--- a/arch/arm64/kernel/traps.c
+++ b/arch/arm64/kernel/traps.c
@@ -1069,7 +1069,7 @@ int kasan_brk_handler(struct pt_regs *regs, unsigned long esr)
void *addr = (void *)regs->regs[0];
u64 pc = regs->pc;
- kasan_report(addr, size, write, pc);
+ kasan_report(addr, size, write ? KASAN_TYPE_WRITE : 0, pc);
/*
* The instrumentation allows to control whether we can proceed after
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 0b52557652be..221e39869ae6 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -365,7 +365,7 @@ static void report_tag_fault(unsigned long addr, unsigned long esr,
* find out access size.
*/
bool is_write = !!(esr & ESR_ELx_WNR);
- kasan_report((void *)addr, 0, is_write, regs->pc);
+ kasan_report((void *)addr, 0, is_write ? KASAN_TYPE_WRITE : 0, regs->pc);
}
#else
/* Tag faults aren't enabled without CONFIG_KASAN_HW_TAGS. */
diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index bf233bde68c7..03c7ac79345d 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -33,6 +33,8 @@ typedef unsigned int __bitwise kasan_vmalloc_flags_t;
#define KASAN_VMALLOC_PAGE_RANGE 0x1 /* Apply exsiting page range */
#define KASAN_VMALLOC_TLB_FLUSH 0x2 /* TLB flush */
+#define KASAN_TYPE_WRITE 0x1
+
#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
#include <linux/pgtable.h>
@@ -526,11 +528,11 @@ static inline void *kasan_reset_tag(const void *addr)
* kasan_report - print a report about a bad memory access detected by KASAN
* @addr: address of the bad access
* @size: size of the bad access
- * @is_write: whether the bad access is a write or a read
+ * @flags: bitmask, can contain KASAN_TYPE_* flags
* @ip: instruction pointer for the accessibility check or the bad access itself
*/
bool kasan_report(const void *addr, size_t size,
- bool is_write, unsigned long ip);
+ unsigned int flags, unsigned long ip);
#else /* CONFIG_KASAN_SW_TAGS || CONFIG_KASAN_HW_TAGS */
diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index b7d05c2a6d93..1ab77ac9719c 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -571,7 +571,7 @@ void __kasan_mempool_unpoison_object(void *ptr, size_t size, unsigned long ip)
bool __kasan_check_byte(const void *address, unsigned long ip)
{
if (!kasan_byte_accessible(address)) {
- kasan_report(address, 1, false, ip);
+ kasan_report(address, 1, 0, ip);
return false;
}
return true;
diff --git a/mm/kasan/generic.c b/mm/kasan/generic.c
index 2b8e73f5f6a7..9efd6fbbb7c3 100644
--- a/mm/kasan/generic.c
+++ b/mm/kasan/generic.c
@@ -173,7 +173,7 @@ static __always_inline bool memory_is_poisoned(const void *addr, size_t size)
}
static __always_inline bool check_region_inline(const void *addr,
- size_t size, bool write,
+ size_t size, unsigned int flags,
unsigned long ret_ip)
{
if (!kasan_enabled())
@@ -183,21 +183,21 @@ static __always_inline bool check_region_inline(const void *addr,
return true;
if (unlikely(addr + size < addr))
- return !kasan_report(addr, size, write, ret_ip);
+ return !kasan_report(addr, size, flags, ret_ip);
if (unlikely(!addr_has_metadata(addr)))
- return !kasan_report(addr, size, write, ret_ip);
+ return !kasan_report(addr, size, flags, ret_ip);
if (likely(!memory_is_poisoned(addr, size)))
return true;
- return !kasan_report(addr, size, write, ret_ip);
+ return !kasan_report(addr, size, flags, ret_ip);
}
-bool kasan_check_range(const void *addr, size_t size, bool write,
+bool kasan_check_range(const void *addr, size_t size, unsigned int flags,
unsigned long ret_ip)
{
- return check_region_inline(addr, size, write, ret_ip);
+ return check_region_inline(addr, size, flags, ret_ip);
}
bool kasan_byte_accessible(const void *addr)
@@ -252,7 +252,7 @@ EXPORT_SYMBOL(__asan_unregister_globals);
#define DEFINE_ASAN_LOAD_STORE(size) \
void __asan_load##size(void *addr) \
{ \
- check_region_inline(addr, size, false, _RET_IP_); \
+ check_region_inline(addr, size, 0, _RET_IP_); \
} \
EXPORT_SYMBOL(__asan_load##size); \
__alias(__asan_load##size) \
@@ -260,7 +260,7 @@ EXPORT_SYMBOL(__asan_unregister_globals);
EXPORT_SYMBOL(__asan_load##size##_noabort); \
void __asan_store##size(void *addr) \
{ \
- check_region_inline(addr, size, true, _RET_IP_); \
+ check_region_inline(addr, size, KASAN_TYPE_WRITE, _RET_IP_); \
} \
EXPORT_SYMBOL(__asan_store##size); \
__alias(__asan_store##size) \
@@ -275,7 +275,7 @@ DEFINE_ASAN_LOAD_STORE(16);
void __asan_loadN(void *addr, ssize_t size)
{
- kasan_check_range(addr, size, false, _RET_IP_);
+ kasan_check_range(addr, size, 0, _RET_IP_);
}
EXPORT_SYMBOL(__asan_loadN);
@@ -285,7 +285,7 @@ EXPORT_SYMBOL(__asan_loadN_noabort);
void __asan_storeN(void *addr, ssize_t size)
{
- kasan_check_range(addr, size, true, _RET_IP_);
+ kasan_check_range(addr, size, KASAN_TYPE_WRITE, _RET_IP_);
}
EXPORT_SYMBOL(__asan_storeN);
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index fc9169a54766..c833bd44e3cc 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -339,11 +339,11 @@ static __always_inline bool addr_has_metadata(const void *addr)
* kasan_check_range - Check memory region, and report if invalid access.
* @addr: the accessed address
* @size: the accessed size
- * @write: true if access is a write access
+ * @flags: bitmask, can contain KASAN_TYPE_* flags
* @ret_ip: return address
* @return: true if access was valid, false if invalid
*/
-bool kasan_check_range(const void *addr, size_t size, bool write,
+bool kasan_check_range(const void *addr, size_t size, unsigned int flags,
unsigned long ret_ip);
#else /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
@@ -379,7 +379,7 @@ static inline void kasan_print_aux_stacks(struct kmem_cache *cache, const void *
#endif
bool kasan_report(const void *addr, size_t size,
- bool is_write, unsigned long ip);
+ unsigned int flags, unsigned long ip);
void kasan_report_invalid_free(void *object, unsigned long ip, enum kasan_report_type type);
struct slab *kasan_addr_to_slab(const void *addr);
diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index e804b1e1f886..cfe00ebb98ec 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -568,13 +568,14 @@ void kasan_report_invalid_free(void *ptr, unsigned long ip, enum kasan_report_ty
* user_access_save/restore(): kasan_report_invalid_free() cannot be called
* from a UACCESS region, and kasan_report_async() is not used on x86.
*/
-bool kasan_report(const void *addr, size_t size, bool is_write,
+bool kasan_report(const void *addr, size_t size, unsigned int flags,
unsigned long ip)
{
bool ret = true;
unsigned long ua_flags = user_access_save();
unsigned long irq_flags;
struct kasan_report_info info;
+ bool is_write = (flags & KASAN_TYPE_WRITE);
if (unlikely(report_suppressed_sw()) || unlikely(!report_enabled())) {
ret = false;
diff --git a/mm/kasan/report_generic.c b/mm/kasan/report_generic.c
index f5b8e37b3805..445183e2f4d3 100644
--- a/mm/kasan/report_generic.c
+++ b/mm/kasan/report_generic.c
@@ -364,14 +364,14 @@ void kasan_print_address_stack_frame(const void *addr)
#define DEFINE_ASAN_REPORT_LOAD(size) \
void __asan_report_load##size##_noabort(void *addr) \
{ \
- kasan_report(addr, size, false, _RET_IP_); \
+ kasan_report(addr, size, 0, _RET_IP_); \
} \
EXPORT_SYMBOL(__asan_report_load##size##_noabort)
#define DEFINE_ASAN_REPORT_STORE(size) \
void __asan_report_store##size##_noabort(void *addr) \
{ \
- kasan_report(addr, size, true, _RET_IP_); \
+ kasan_report(addr, size, KASAN_TYPE_WRITE, _RET_IP_); \
} \
EXPORT_SYMBOL(__asan_report_store##size##_noabort)
@@ -388,12 +388,12 @@ DEFINE_ASAN_REPORT_STORE(16);
void __asan_report_load_n_noabort(void *addr, ssize_t size)
{
- kasan_report(addr, size, false, _RET_IP_);
+ kasan_report(addr, size, 0, _RET_IP_);
}
EXPORT_SYMBOL(__asan_report_load_n_noabort);
void __asan_report_store_n_noabort(void *addr, ssize_t size)
{
- kasan_report(addr, size, true, _RET_IP_);
+ kasan_report(addr, size, KASAN_TYPE_WRITE, _RET_IP_);
}
EXPORT_SYMBOL(__asan_report_store_n_noabort);
diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c
index d286e0a04543..a24f1225dd88 100644
--- a/mm/kasan/shadow.c
+++ b/mm/kasan/shadow.c
@@ -28,13 +28,13 @@
bool __kasan_check_read(const volatile void *p, unsigned int size)
{
- return kasan_check_range((void *)p, size, false, _RET_IP_);
+ return kasan_check_range((void *)p, size, 0, _RET_IP_);
}
EXPORT_SYMBOL(__kasan_check_read);
bool __kasan_check_write(const volatile void *p, unsigned int size)
{
- return kasan_check_range((void *)p, size, true, _RET_IP_);
+ return kasan_check_range((void *)p, size, KASAN_TYPE_WRITE, _RET_IP_);
}
EXPORT_SYMBOL(__kasan_check_write);
@@ -50,7 +50,7 @@ EXPORT_SYMBOL(__kasan_check_write);
#undef memset
void *memset(void *addr, int c, size_t len)
{
- if (!kasan_check_range(addr, len, true, _RET_IP_))
+ if (!kasan_check_range(addr, len, KASAN_TYPE_WRITE, _RET_IP_))
return NULL;
return __memset(addr, c, len);
@@ -60,8 +60,8 @@ void *memset(void *addr, int c, size_t len)
#undef memmove
void *memmove(void *dest, const void *src, size_t len)
{
- if (!kasan_check_range(src, len, false, _RET_IP_) ||
- !kasan_check_range(dest, len, true, _RET_IP_))
+ if (!kasan_check_range(src, len, 0, _RET_IP_) ||
+ !kasan_check_range(dest, len, KASAN_TYPE_WRITE, _RET_IP_))
return NULL;
return __memmove(dest, src, len);
@@ -71,8 +71,8 @@ void *memmove(void *dest, const void *src, size_t len)
#undef memcpy
void *memcpy(void *dest, const void *src, size_t len)
{
- if (!kasan_check_range(src, len, false, _RET_IP_) ||
- !kasan_check_range(dest, len, true, _RET_IP_))
+ if (!kasan_check_range(src, len, 0, _RET_IP_) ||
+ !kasan_check_range(dest, len, KASAN_TYPE_WRITE, _RET_IP_))
return NULL;
return __memcpy(dest, src, len);
@@ -81,7 +81,7 @@ void *memcpy(void *dest, const void *src, size_t len)
void *__asan_memset(void *addr, int c, ssize_t len)
{
- if (!kasan_check_range(addr, len, true, _RET_IP_))
+ if (!kasan_check_range(addr, len, KASAN_TYPE_WRITE, _RET_IP_))
return NULL;
return __memset(addr, c, len);
@@ -91,8 +91,8 @@ EXPORT_SYMBOL(__asan_memset);
#ifdef __HAVE_ARCH_MEMMOVE
void *__asan_memmove(void *dest, const void *src, ssize_t len)
{
- if (!kasan_check_range(src, len, false, _RET_IP_) ||
- !kasan_check_range(dest, len, true, _RET_IP_))
+ if (!kasan_check_range(src, len, 0, _RET_IP_) ||
+ !kasan_check_range(dest, len, KASAN_TYPE_WRITE, _RET_IP_))
return NULL;
return __memmove(dest, src, len);
@@ -102,8 +102,8 @@ EXPORT_SYMBOL(__asan_memmove);
void *__asan_memcpy(void *dest, const void *src, ssize_t len)
{
- if (!kasan_check_range(src, len, false, _RET_IP_) ||
- !kasan_check_range(dest, len, true, _RET_IP_))
+ if (!kasan_check_range(src, len, 0, _RET_IP_) ||
+ !kasan_check_range(dest, len, KASAN_TYPE_WRITE, _RET_IP_))
return NULL;
return __memcpy(dest, src, len);
diff --git a/mm/kasan/sw_tags.c b/mm/kasan/sw_tags.c
index c75741a74602..af77b642ede7 100644
--- a/mm/kasan/sw_tags.c
+++ b/mm/kasan/sw_tags.c
@@ -72,7 +72,7 @@ u8 kasan_random_tag(void)
return (u8)(state % (KASAN_TAG_MAX + 1));
}
-bool kasan_check_range(const void *addr, size_t size, bool write,
+bool kasan_check_range(const void *addr, size_t size, unsigned int flags,
unsigned long ret_ip)
{
u8 tag;
@@ -83,7 +83,7 @@ bool kasan_check_range(const void *addr, size_t size, bool write,
return true;
if (unlikely(addr + size < addr))
- return !kasan_report(addr, size, write, ret_ip);
+ return !kasan_report(addr, size, flags, ret_ip);
tag = get_tag((const void *)addr);
@@ -109,12 +109,12 @@ bool kasan_check_range(const void *addr, size_t size, bool write,
untagged_addr = kasan_reset_tag((const void *)addr);
if (unlikely(!addr_has_metadata(untagged_addr)))
- return !kasan_report(addr, size, write, ret_ip);
+ return !kasan_report(addr, size, flags, ret_ip);
shadow_first = kasan_mem_to_shadow(untagged_addr);
shadow_last = kasan_mem_to_shadow(untagged_addr + size - 1);
for (shadow = shadow_first; shadow <= shadow_last; shadow++) {
if (*shadow != tag) {
- return !kasan_report(addr, size, write, ret_ip);
+ return !kasan_report(addr, size, flags, ret_ip);
}
}
@@ -137,12 +137,12 @@ bool kasan_byte_accessible(const void *addr)
#define DEFINE_HWASAN_LOAD_STORE(size) \
void __hwasan_load##size##_noabort(void *addr) \
{ \
- kasan_check_range(addr, size, false, _RET_IP_); \
+ kasan_check_range(addr, size, 0, _RET_IP_); \
} \
EXPORT_SYMBOL(__hwasan_load##size##_noabort); \
void __hwasan_store##size##_noabort(void *addr) \
{ \
- kasan_check_range(addr, size, true, _RET_IP_); \
+ kasan_check_range(addr, size, KASAN_TYPE_WRITE, _RET_IP_); \
} \
EXPORT_SYMBOL(__hwasan_store##size##_noabort)
@@ -154,13 +154,13 @@ DEFINE_HWASAN_LOAD_STORE(16);
void __hwasan_loadN_noabort(void *addr, ssize_t size)
{
- kasan_check_range(addr, size, false, _RET_IP_);
+ kasan_check_range(addr, size, 0, _RET_IP_);
}
EXPORT_SYMBOL(__hwasan_loadN_noabort);
void __hwasan_storeN_noabort(void *addr, ssize_t size)
{
- kasan_check_range(addr, size, true, _RET_IP_);
+ kasan_check_range(addr, size, KASAN_TYPE_WRITE, _RET_IP_);
}
EXPORT_SYMBOL(__hwasan_storeN_noabort);
@@ -173,6 +173,6 @@ EXPORT_SYMBOL(__hwasan_tag_memory);
void kasan_tag_mismatch(void *addr, unsigned long access_info,
unsigned long ret_ip)
{
- kasan_report(addr, 1 << (access_info & 0xf), access_info & 0x10,
- ret_ip);
+ kasan_report(addr, 1 << (access_info & 0xf),
+ (access_info & 0x10) ? KASAN_TYPE_WRITE : 0, ret_ip);
}
--
2.55.0.979.g7e5102b832-goog
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH RFC v3 06/12] kcov: introduce memory access tracing
2026-09-08 16:54 [PATCH RFC v3 00/12] KCOV: entry/exit records, memory access records, and delay injection Jann Horn
` (4 preceding siblings ...)
2026-09-08 16:54 ` [PATCH RFC v3 05/12] kasan: refactor write/is_write arguments to flags Jann Horn
@ 2026-09-08 16:54 ` Jann Horn
2026-09-08 16:54 ` [PATCH RFC v3 07/12] kasan: provide memory access information to KCOV Jann Horn
` (5 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Jann Horn @ 2026-09-08 16:54 UTC (permalink / raw)
To: Dmitry Vyukov, Andrey Konovalov, Alexander Potapenko
Cc: Nathan Chancellor, Nick Desaulniers, Bill Wendling, Justin Stitt,
linux-kernel, kasan-dev, llvm, Jann Horn
This commit only introduces tracing of memory accesses that are
instrumented at the source level with instrument_*(); a followup commit
will additionally provide data based on ASAN compiler instrumentation.
I am adding the instrumentation hook definitions directly in
include/linux/instrumented.h instead of adding separate headers; this
way the compiler won't have to read yet another header file for almost
every compilation unit.
To avoid instrumenting files that shouldn't be instrumented, reuse
KASAN's __SANITIZE_ADDRESS__.
Signed-off-by: Jann Horn <jannh@google.com>
---
include/linux/instrumented.h | 30 +++++++++++++++++
include/linux/kcov.h | 11 ++++++
include/uapi/linux/kcov.h | 24 +++++++++++++
kernel/kcov.c | 80 +++++++++++++++++++++++++++++++++++++++++---
lib/Kconfig.debug | 11 ++++++
5 files changed, 152 insertions(+), 4 deletions(-)
diff --git a/include/linux/instrumented.h b/include/linux/instrumented.h
index a1b4cf81adc2..940776dff616 100644
--- a/include/linux/instrumented.h
+++ b/include/linux/instrumented.h
@@ -13,6 +13,26 @@
#include <linux/kcsan-checks.h>
#include <linux/kmsan-checks.h>
#include <linux/types.h>
+#ifdef CONFIG_KCOV_MEMORY
+/* For build speed, only include this header in builds that actually need it. */
+#include <uapi/linux/kcov.h>
+#endif
+
+#ifdef CONFIG_KCOV_MEMORY
+void _kcov_handle_memaccess(const volatile void *p, size_t size, unsigned int type);
+#else
+static __always_inline void _kcov_handle_memaccess(const volatile void *p,
+ size_t size, unsigned int type) {}
+/* Discard type argument to avoid depending on kcov header. */
+#define _kcov_handle_memaccess(p, size, type) _kcov_handle_memaccess((p), (size), 0)
+#endif
+
+#if defined(__SANITIZE_ADDRESS__) || !defined(CONFIG_KCOV_MEMORY)
+#define kcov_handle_memaccess _kcov_handle_memaccess
+#else
+static __always_inline void kcov_handle_memaccess(const volatile void *p,
+ size_t size, unsigned int type) {}
+#endif
/**
* instrument_read - instrument regular read access
@@ -24,6 +44,7 @@
*/
static __always_inline void instrument_read(const volatile void *v, size_t size)
{
+ kcov_handle_memaccess(v, size, 0);
kasan_check_read(v, size);
kcsan_check_read(v, size);
}
@@ -38,6 +59,7 @@ static __always_inline void instrument_read(const volatile void *v, size_t size)
*/
static __always_inline void instrument_write(const volatile void *v, size_t size)
{
+ kcov_handle_memaccess(v, size, MEMORY_ACCESS_RECORD_WRITE);
kasan_check_write(v, size);
kcsan_check_write(v, size);
}
@@ -52,6 +74,7 @@ static __always_inline void instrument_write(const volatile void *v, size_t size
*/
static __always_inline void instrument_read_write(const volatile void *v, size_t size)
{
+ kcov_handle_memaccess(v, size, MEMORY_ACCESS_RECORD_RMW);
kasan_check_write(v, size);
kcsan_check_read_write(v, size);
}
@@ -79,6 +102,7 @@ static __always_inline void instrument_atomic_check_alignment(const volatile voi
*/
static __always_inline void instrument_atomic_read(const volatile void *v, size_t size)
{
+ kcov_handle_memaccess(v, size, MEMORY_ACCESS_RECORD_ATOMIC);
kasan_check_read(v, size);
kcsan_check_atomic_read(v, size);
instrument_atomic_check_alignment(v, size);
@@ -94,6 +118,7 @@ static __always_inline void instrument_atomic_read(const volatile void *v, size_
*/
static __always_inline void instrument_atomic_write(const volatile void *v, size_t size)
{
+ kcov_handle_memaccess(v, size, MEMORY_ACCESS_RECORD_WRITE|MEMORY_ACCESS_RECORD_ATOMIC);
kasan_check_write(v, size);
kcsan_check_atomic_write(v, size);
instrument_atomic_check_alignment(v, size);
@@ -109,6 +134,7 @@ static __always_inline void instrument_atomic_write(const volatile void *v, size
*/
static __always_inline void instrument_atomic_read_write(const volatile void *v, size_t size)
{
+ kcov_handle_memaccess(v, size, MEMORY_ACCESS_RECORD_RMW|MEMORY_ACCESS_RECORD_ATOMIC);
kasan_check_write(v, size);
kcsan_check_atomic_read_write(v, size);
instrument_atomic_check_alignment(v, size);
@@ -126,6 +152,7 @@ static __always_inline void instrument_atomic_read_write(const volatile void *v,
static __always_inline void
instrument_copy_to_user(void __user *to, const void *from, unsigned long n)
{
+ kcov_handle_memaccess(from, n, 0);
kasan_check_read(from, n);
kcsan_check_read(from, n);
kmsan_copy_to_user(to, from, n, 0);
@@ -143,6 +170,7 @@ instrument_copy_to_user(void __user *to, const void *from, unsigned long n)
static __always_inline void
instrument_copy_from_user_before(const void *to, const void __user *from, unsigned long n)
{
+ kcov_handle_memaccess(to, n, MEMORY_ACCESS_RECORD_WRITE);
kasan_check_write(to, n);
kcsan_check_write(to, n);
}
@@ -176,6 +204,8 @@ instrument_copy_from_user_after(const void *to, const void __user *from,
static __always_inline void instrument_memcpy_before(void *to, const void *from,
unsigned long n)
{
+ kcov_handle_memaccess(from, n, 0);
+ kcov_handle_memaccess(to, n, MEMORY_ACCESS_RECORD_WRITE);
kasan_check_write(to, n);
kasan_check_read(from, n);
kcsan_check_write(to, n);
diff --git a/include/linux/kcov.h b/include/linux/kcov.h
index 357f4de8790a..e4b818df189e 100644
--- a/include/linux/kcov.h
+++ b/include/linux/kcov.h
@@ -23,6 +23,7 @@ enum kcov_mode {
KCOV_MODE_TRACE_CMP = 3,
};
+#define KCOV_ENABLE_MEMORY (1 << 28)
/*
* Modifier for KCOV_MODE_TRACE_PC to record function entry/exit marked with
* metadata bits.
@@ -31,6 +32,7 @@ enum kcov_mode {
#define KCOV_IN_CTXSW (1 << 30)
#define KCOV_MODE_TRACE_PC_EXT (KCOV_MODE_TRACE_PC | KCOV_EXT_FORMAT)
+#define KCOV_MODE_TRACE_PC_AND_MEM (KCOV_MODE_TRACE_PC_EXT | KCOV_ENABLE_MEMORY)
void kcov_task_init(struct task_struct *t);
void kcov_task_exit(struct task_struct *t);
@@ -109,4 +111,13 @@ static inline void kcov_remote_start_usb_softirq(u64 id) {}
static inline void kcov_remote_stop_softirq(void) {}
#endif /* CONFIG_KCOV */
+
+#ifdef CONFIG_KCOV_MEMORY
+void __kcov_handle_memaccess(const volatile void *p, size_t size, unsigned int type,
+ unsigned long ret_ip);
+#else /* CONFIG_KCOV_MEMORY */
+static inline void __kcov_handle_memaccess(const volatile void *p, size_t size,
+ unsigned int type, unsigned long ret_ip) {}
+#endif /* CONFIG_KCOV_MEMORY */
+
#endif /* _LINUX_KCOV_H */
diff --git a/include/uapi/linux/kcov.h b/include/uapi/linux/kcov.h
index 75c582784055..7d7147e7b427 100644
--- a/include/uapi/linux/kcov.h
+++ b/include/uapi/linux/kcov.h
@@ -22,6 +22,7 @@ struct kcov_remote_arg {
#define KCOV_ENABLE _IO('c', 100)
#define KCOV_DISABLE _IO('c', 101)
#define KCOV_REMOTE_ENABLE _IOW('c', 102, struct kcov_remote_arg)
+#define KCOV_GET_MEMORY_RECORD_SIZE _IO('c', 103)
enum {
/*
@@ -41,6 +42,8 @@ enum {
* (KCOV_RECORDFLAG_*).
*/
KCOV_TRACE_PC_EXT = 2,
+ /* Extended PC coverage mode with tracing of memory accesses. */
+ KCOV_TRACE_MEMORY_ACCESS = 3,
};
#define KCOV_RECORD_IP_MASK 0x00ffffffffffffff
@@ -50,6 +53,7 @@ enum {
#define KCOV_RECORDFLAG_TYPE_EXIT 0x1000000000000000
/* Summarized entry/exit events that occurred in an untraced region. */
#define KCOV_RECORDFLAG_TYPE_EESUM 0x2000000000000000
+#define KCOV_RECORDFLAG_TYPE_MEMORY 0x3000000000000000
/*
* The format for the types of collected comparisons.
@@ -74,4 +78,24 @@ static inline __u64 kcov_remote_handle(__u64 subsys, __u64 inst)
return subsys | inst;
}
+/*
+ * Data format for memory access tracing mode.
+ * This is an extensible struct (it can be extended by appending elements);
+ * userspace can query the struct size used by the running kernel with
+ * KCOV_GET_MEMORY_ACCESS_RECORD_SIZE.
+ */
+#define MEMORY_ACCESS_RECORD_TYPE_MASK 0xf
+#define MEMORY_ACCESS_RECORD_TYPE_ACCESS 0x0
+/* flags for MEMORY_ACCESS_RECORD_TYPE_ACCESS */
+#define MEMORY_ACCESS_RECORD_WRITE 0x10
+#define MEMORY_ACCESS_RECORD_RMW 0x20
+#define MEMORY_ACCESS_RECORD_ATOMIC 0x40
+struct memory_access_record {
+ __aligned_u64 ip_address_and_kcov_flags;
+ __aligned_u64 data_address;
+ __u32 size;
+ __u32 flags; /* MEMORY_ACCESS_RECORD_* */
+ __aligned_u64 time;
+} __attribute__((aligned(8)));
+
#endif /* _LINUX_KCOV_IOCTLS_H */
diff --git a/kernel/kcov.c b/kernel/kcov.c
index 712f0f744ec5..83e05aa61728 100644
--- a/kernel/kcov.c
+++ b/kernel/kcov.c
@@ -235,7 +235,8 @@ void notrace __sanitizer_cov_trace_pc(void)
{
struct task_struct *cur = current;
- if ((READ_ONCE(cur->kcov_mode) & ~KCOV_EXT_FORMAT) != KCOV_MODE_TRACE_PC)
+ if ((READ_ONCE(cur->kcov_mode) & ~(KCOV_ENABLE_MEMORY|KCOV_EXT_FORMAT))
+ != KCOV_MODE_TRACE_PC)
return;
/*
* No bitops are needed here for setting the record type because
@@ -258,7 +259,7 @@ void notrace __sanitizer_cov_trace_pc_entry(void)
* This hook replaces __sanitizer_cov_trace_pc() for the function entry
* basic block; it should still emit a record even in classic kcov mode.
*/
- if ((kcov_mode & ~(KCOV_EXT_FORMAT|KCOV_IN_CTXSW)) != KCOV_MODE_TRACE_PC)
+ if ((kcov_mode & ~(KCOV_ENABLE_MEMORY|KCOV_EXT_FORMAT|KCOV_IN_CTXSW)) != KCOV_MODE_TRACE_PC)
return;
if (kcov_mode & KCOV_IN_CTXSW) {
cur->kcov->suppressed_stack_delta++;
@@ -281,7 +282,7 @@ void notrace __sanitizer_cov_trace_pc_exit(void)
* So unlike __sanitizer_cov_trace_pc_entry(), this PC should only be
* reported in extended mode, where function exit events are recorded.
*/
- if ((kcov_mode & ~KCOV_IN_CTXSW) != KCOV_MODE_TRACE_PC_EXT)
+ if ((kcov_mode & ~(KCOV_ENABLE_MEMORY|KCOV_IN_CTXSW)) != KCOV_MODE_TRACE_PC_EXT)
return;
if (kcov_mode & KCOV_IN_CTXSW) {
struct kcov *kcov = cur->kcov;
@@ -663,6 +664,8 @@ static int kcov_get_mode(unsigned long arg)
#endif
else if (arg == KCOV_TRACE_PC_EXT)
return IS_ENABLED(CONFIG_KCOV_EXT_RECORDS) ? KCOV_MODE_TRACE_PC_EXT : -ENOTSUPP;
+ else if (arg == KCOV_TRACE_MEMORY_ACCESS)
+ return IS_ENABLED(CONFIG_KCOV_MEMORY) ? KCOV_MODE_TRACE_PC_AND_MEM : -ENOTSUPP;
else
return -EINVAL;
}
@@ -803,6 +806,10 @@ static int kcov_ioctl_locked(struct kcov *kcov, unsigned int cmd,
/* Put either in kcov_task_exit() or in KCOV_DISABLE. */
kcov_get(kcov);
return 0;
+ case KCOV_GET_MEMORY_RECORD_SIZE:
+ if (!IS_ENABLED(CONFIG_KCOV_MEMORY))
+ return -ENOTSUPP;
+ return sizeof(struct memory_access_record);
default:
return -ENOTTY;
}
@@ -1171,7 +1178,8 @@ void kcov_remote_stop(void)
* and kcov_remote_stop(), hence the sequence check.
*/
if (sequence == kcov->sequence && kcov->remote)
- kcov_move_area(kcov->mode & ~KCOV_EXT_FORMAT, kcov->area, kcov->size, area);
+ kcov_move_area(kcov->mode & ~(KCOV_ENABLE_MEMORY|KCOV_EXT_FORMAT),
+ kcov->area, kcov->size, area);
spin_unlock(&kcov->lock);
spin_lock(&kcov_remote_lock);
@@ -1194,6 +1202,70 @@ struct kcov_common_handle_id kcov_common_handle(void)
}
EXPORT_SYMBOL(kcov_common_handle);
+#ifdef CONFIG_KCOV_MEMORY
+static notrace bool kcov_get_memaccess_record(struct task_struct *t,
+ struct memory_access_record **recordp)
+{
+ u64 *area = (u64 *)t->kcov_area;
+ /* The buffer was allocated for t->kcov_size unsigned longs. */
+ u64 max_pos = t->kcov_size * sizeof(unsigned long);
+ u64 count = READ_ONCE(area[0]);
+ u64 start_pos = sizeof(unsigned long) + count * sizeof(unsigned long);
+ u64 end_pos = start_pos + sizeof(struct memory_access_record);
+
+ if (unlikely(end_pos > max_pos))
+ return false;
+
+ /* See comment in kcov_add_pc_record(). */
+ WRITE_ONCE(area[0], count + sizeof(struct memory_access_record)/sizeof(unsigned long));
+ barrier();
+ *recordp = (void *)area + start_pos;
+ return true;
+}
+
+/*
+ * Memory ordering doesn't matter a lot here because timestamps aren't
+ * collected atomically with memory accesses anyway.
+ * The important things are that the clock access has to be uaccess-safe,
+ * notrace, and have high granularity.
+ */
+static notrace __always_inline u64 kcov_get_time(void)
+{
+#ifdef CONFIG_X86
+ return rdtsc_ordered();
+#else
+ return 0;
+#endif
+}
+
+void notrace __kcov_handle_memaccess(const volatile void *p, size_t size, unsigned int type,
+ unsigned long ret_ip)
+{
+ struct task_struct *t = current;
+ struct memory_access_record *record;
+ unsigned int kcov_mode = READ_ONCE(t->kcov_mode);
+
+ if (kcov_mode != KCOV_MODE_TRACE_PC_AND_MEM || !check_kcov_context(t))
+ return;
+ if (!kcov_get_memaccess_record(t, &record))
+ return;
+ *record = (struct memory_access_record) {
+ .ip_address_and_kcov_flags =
+ (ret_ip & KCOV_RECORD_IP_MASK) | KCOV_RECORDFLAG_TYPE_MEMORY,
+ .data_address = (u64)p,
+ .size = size,
+ .flags = type,
+ .time = kcov_get_time()
+ };
+}
+
+void notrace _kcov_handle_memaccess(const volatile void *p, size_t size, unsigned int type)
+{
+ __kcov_handle_memaccess(p, size, type, _RET_IP_);
+}
+EXPORT_SYMBOL(_kcov_handle_memaccess);
+#endif /* CONFIG_KCOV_MEMORY */
+
#ifdef CONFIG_KCOV_SELFTEST
static void __init selftest(void)
{
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 6ddf58692b09..5de427ccc42d 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -2217,6 +2217,17 @@ config KCOV_ENABLE_COMPARISONS
These operands can be used by fuzzing engines to improve the quality
of fuzzing coverage.
+config KCOV_MEMORY
+ bool "Enable memory access trace collection by KCOV"
+ depends on KCOV
+ depends on KCOV_EXT_RECORDS
+ help
+ Provide a KCOV mode which records memory access operations and allows
+ userspace to inject execution delays to impose constraints on the
+ order in which multithreaded execution happens.
+
+ This is mainly useful for testing race condition bugs.
+
config KCOV_INSTRUMENT_ALL
bool "Instrument all code by default"
depends on KCOV
--
2.55.0.979.g7e5102b832-goog
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH RFC v3 07/12] kasan: provide memory access information to KCOV
2026-09-08 16:54 [PATCH RFC v3 00/12] KCOV: entry/exit records, memory access records, and delay injection Jann Horn
` (5 preceding siblings ...)
2026-09-08 16:54 ` [PATCH RFC v3 06/12] kcov: introduce memory access tracing Jann Horn
@ 2026-09-08 16:54 ` Jann Horn
2026-09-08 16:54 ` [PATCH RFC v3 08/12] kcov: log freeing of SLUB objects and pages Jann Horn
` (4 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Jann Horn @ 2026-09-08 16:54 UTC (permalink / raw)
To: Dmitry Vyukov, Andrey Konovalov, Alexander Potapenko
Cc: Nathan Chancellor, Nick Desaulniers, Bill Wendling, Justin Stitt,
linux-kernel, kasan-dev, llvm, Jann Horn
In CONFIG_KCOV_MEMORY builds, let KASAN provide information about memory
accesses to KCOV.
Since KCOV already receives information from instrument_*() directly,
filter out those accesses by introducing KASAN_TYPE_EXPLICIT.
Because the KCOV usecase requires seeing ~every memory access (including
accesses to globals and repeated accesses to the same memory location
within a basic block), gate this on CC_IS_CLANG and let it flip some hidden
LLVM flags that disable ASAN optimizations.
Signed-off-by: Jann Horn <jannh@google.com>
---
include/linux/kasan.h | 1 +
lib/Kconfig.debug | 2 ++
lib/Kconfig.kasan | 9 +++++++++
mm/kasan/generic.c | 15 +++++++++++++++
mm/kasan/shadow.c | 5 +++--
scripts/Makefile.kasan | 17 +++++++++++++++++
tools/objtool/check.c | 1 +
7 files changed, 48 insertions(+), 2 deletions(-)
diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index 03c7ac79345d..4b915e0c51bc 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -34,6 +34,7 @@ typedef unsigned int __bitwise kasan_vmalloc_flags_t;
#define KASAN_VMALLOC_TLB_FLUSH 0x2 /* TLB flush */
#define KASAN_TYPE_WRITE 0x1
+#define KASAN_TYPE_EXPLICIT 0x2
#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 5de427ccc42d..f763f0504f62 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -2221,6 +2221,8 @@ config KCOV_MEMORY
bool "Enable memory access trace collection by KCOV"
depends on KCOV
depends on KCOV_EXT_RECORDS
+ depends on HAVE_KASAN_REPORT_EVERY_ACCESS
+ select KASAN_REPORT_EVERY_ACCESS
help
Provide a KCOV mode which records memory access operations and allows
userspace to inject execution delays to impose constraints on the
diff --git a/lib/Kconfig.kasan b/lib/Kconfig.kasan
index a4bb610a7a6f..9a43f6269c5c 100644
--- a/lib/Kconfig.kasan
+++ b/lib/Kconfig.kasan
@@ -228,4 +228,13 @@ config KASAN_EXTRA_INFO
boot parameter, it will add 8 * stack_ring_size bytes of additional
memory consumption.
+# Is KASAN_REPORT_EVERY_ACCESS allowed?
+config HAVE_KASAN_REPORT_EVERY_ACCESS
+ def_bool y
+ depends on KASAN_OUTLINE
+ depends on CC_IS_CLANG
+
+config KASAN_REPORT_EVERY_ACCESS
+ bool
+
endif # KASAN
diff --git a/mm/kasan/generic.c b/mm/kasan/generic.c
index 9efd6fbbb7c3..6cf88f569e64 100644
--- a/mm/kasan/generic.c
+++ b/mm/kasan/generic.c
@@ -32,6 +32,8 @@
#include <linux/types.h>
#include <linux/vmalloc.h>
#include <linux/bug.h>
+#include <linux/kcov.h>
+#include <uapi/linux/kcov.h>
#include "kasan.h"
#include "../slab.h"
@@ -182,6 +184,19 @@ static __always_inline bool check_region_inline(const void *addr,
if (unlikely(size == 0))
return true;
+ /*
+ * Do not route information about an access to KCOV if we got called
+ * through the instrument_*() path - KCOV can get those accesses
+ * directly from instrument_*(), and get a bit more metadata about the
+ * access that way.
+ */
+ if (likely((flags & KASAN_TYPE_EXPLICIT) == 0)) {
+ unsigned int kcov_flags =
+ (flags & KASAN_TYPE_WRITE) ? MEMORY_ACCESS_RECORD_WRITE : 0;
+
+ __kcov_handle_memaccess(addr, size, kcov_flags, ret_ip);
+ }
+
if (unlikely(addr + size < addr))
return !kasan_report(addr, size, flags, ret_ip);
diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c
index a24f1225dd88..84f8567d4f2c 100644
--- a/mm/kasan/shadow.c
+++ b/mm/kasan/shadow.c
@@ -28,13 +28,14 @@
bool __kasan_check_read(const volatile void *p, unsigned int size)
{
- return kasan_check_range((void *)p, size, 0, _RET_IP_);
+ return kasan_check_range((void *)p, size, KASAN_TYPE_EXPLICIT, _RET_IP_);
}
EXPORT_SYMBOL(__kasan_check_read);
bool __kasan_check_write(const volatile void *p, unsigned int size)
{
- return kasan_check_range((void *)p, size, KASAN_TYPE_WRITE, _RET_IP_);
+ return kasan_check_range((void *)p, size,
+ KASAN_TYPE_WRITE|KASAN_TYPE_EXPLICIT, _RET_IP_);
}
EXPORT_SYMBOL(__kasan_check_write);
diff --git a/scripts/Makefile.kasan b/scripts/Makefile.kasan
index 91504e81247a..82e88c5fb9bc 100644
--- a/scripts/Makefile.kasan
+++ b/scripts/Makefile.kasan
@@ -60,6 +60,23 @@ kasan_params += asan-instrumentation-with-call-threshold=$(call_threshold) \
asan-instrument-allocas=1 \
asan-globals=1
+# When we piggyback tracing of memory accesses for race condition testing on top
+# of KASAN, we want the compiler to report memory accesses even when KASAN can
+# prove that no UAF/OOB can occur; in particular, these optimizations must be
+# inhibited:
+#
+# - suppression of ASAN hook calls for global variables
+# - merging of multiple accesses in a basic block into a single ASAN hook call
+#
+# For now, known stack variable accesses are still ignored as a performance
+# tradeoff, though that will probably make a small number of races (where
+# another task concurrently accesses stuff on our stack) invisible to the
+# instrumentation. (Stack access instrumentation is gated on
+# asan-use-stack-safety and asan-skip-promotable-allocas.)
+ifdef CONFIG_KASAN_REPORT_EVERY_ACCESS
+kasan_params += asan-opt-globals=0 asan-opt-same-temp=0
+endif # CONFIG_KASAN_REPORT_EVERY_ACCESS
+
# Instrument memcpy/memset/memmove calls by using instrumented __asan_mem*()
# instead. With compilers that don't support this option, compiler-inserted
# memintrinsics won't be checked by KASAN on GENERIC_ENTRY architectures.
diff --git a/tools/objtool/check.c b/tools/objtool/check.c
index d70cb640e2ec..08ebfe1f3fac 100644
--- a/tools/objtool/check.c
+++ b/tools/objtool/check.c
@@ -1219,6 +1219,7 @@ static const char *uaccess_safe_builtin[] = {
/* KCOV */
"write_comp_data",
"check_kcov_mode",
+ "__kcov_handle_memaccess",
"__sanitizer_cov_trace_pc",
"__sanitizer_cov_trace_pc_entry",
"__sanitizer_cov_trace_pc_exit",
--
2.55.0.979.g7e5102b832-goog
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH RFC v3 08/12] kcov: log freeing of SLUB objects and pages
2026-09-08 16:54 [PATCH RFC v3 00/12] KCOV: entry/exit records, memory access records, and delay injection Jann Horn
` (6 preceding siblings ...)
2026-09-08 16:54 ` [PATCH RFC v3 07/12] kasan: provide memory access information to KCOV Jann Horn
@ 2026-09-08 16:54 ` Jann Horn
2026-09-08 17:04 ` Jann Horn
2026-09-08 16:54 ` [PATCH RFC v3 09/12] kcov: record return address on function entry Jann Horn
` (3 subsequent siblings)
11 siblings, 1 reply; 14+ messages in thread
From: Jann Horn @ 2026-09-08 16:54 UTC (permalink / raw)
To: Dmitry Vyukov, Andrey Konovalov, Alexander Potapenko
Cc: Nathan Chancellor, Nick Desaulniers, Bill Wendling, Justin Stitt,
linux-kernel, kasan-dev, llvm, Jann Horn
To help with using CONFIG_KCOV_MEMORY for detecting use-after-free issues,
log when memory (SLUB objects or page allocations) is being freed.
This should happen after KASAN has already marked the memory as freed;
this will become important if we allow driving delay injection off this
in the future.
Signed-off-by: Jann Horn <jannh@google.com>
---
include/uapi/linux/kcov.h | 1 +
mm/kasan/common.c | 2 ++
mm/page_alloc.c | 3 +++
3 files changed, 6 insertions(+)
diff --git a/include/uapi/linux/kcov.h b/include/uapi/linux/kcov.h
index 7d7147e7b427..76822d1c119a 100644
--- a/include/uapi/linux/kcov.h
+++ b/include/uapi/linux/kcov.h
@@ -90,6 +90,7 @@ static inline __u64 kcov_remote_handle(__u64 subsys, __u64 inst)
#define MEMORY_ACCESS_RECORD_WRITE 0x10
#define MEMORY_ACCESS_RECORD_RMW 0x20
#define MEMORY_ACCESS_RECORD_ATOMIC 0x40
+#define MEMORY_ACCESS_RECORD_FREE 0x80
struct memory_access_record {
__aligned_u64 ip_address_and_kcov_flags;
__aligned_u64 data_address;
diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 1ab77ac9719c..3a648eaad024 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -283,6 +283,8 @@ bool __kasan_slab_free(struct kmem_cache *cache, void *object, bool init,
return false;
poison_slab_object(cache, object, init);
+ _kcov_handle_memaccess(object, cache->object_size,
+ MEMORY_ACCESS_RECORD_WRITE|MEMORY_ACCESS_RECORD_FREE);
if (no_quarantine)
return false;
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 083cbcb5bdde..910d14925b84 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1446,6 +1446,9 @@ static __always_inline bool __free_pages_prepare(struct page *page,
if (init)
clear_highpages_kasan_tagged(page, 1 << order);
+ _kcov_handle_memaccess(page_address(page), (1 << order)*PAGE_SIZE,
+ MEMORY_ACCESS_RECORD_WRITE|MEMORY_ACCESS_RECORD_FREE);
+
/*
* arch_free_page() can make the page's contents inaccessible. s390
* does this. So nothing which can access the page's contents should
--
2.55.0.979.g7e5102b832-goog
^ permalink raw reply related [flat|nested] 14+ messages in thread* Re: [PATCH RFC v3 08/12] kcov: log freeing of SLUB objects and pages
2026-09-08 16:54 ` [PATCH RFC v3 08/12] kcov: log freeing of SLUB objects and pages Jann Horn
@ 2026-09-08 17:04 ` Jann Horn
0 siblings, 0 replies; 14+ messages in thread
From: Jann Horn @ 2026-09-08 17:04 UTC (permalink / raw)
To: Dmitry Vyukov, Andrey Konovalov, Alexander Potapenko
Cc: Nathan Chancellor, Nick Desaulniers, Bill Wendling, Justin Stitt,
linux-kernel, kasan-dev, llvm, Andrew Morton, Vlastimil Babka,
Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
Johannes Weiner, Zi Yan, Harry Yoo, Hao Li, Christoph Lameter,
David Rientjes, Roman Gushchin, Linux-MM
On Tue, Sep 8, 2026 at 6:55 PM Jann Horn <jannh@google.com> wrote:
> To help with using CONFIG_KCOV_MEMORY for detecting use-after-free issues,
> log when memory (SLUB objects or page allocations) is being freed.
>
> This should happen after KASAN has already marked the memory as freed;
> this will become important if we allow driving delay injection off this
> in the future.
Oops, I should include the page allocator and SLUB people on this one
patch in the series, since I'm adding a hook in their code.
> Signed-off-by: Jann Horn <jannh@google.com>
> ---
> include/uapi/linux/kcov.h | 1 +
> mm/kasan/common.c | 2 ++
> mm/page_alloc.c | 3 +++
> 3 files changed, 6 insertions(+)
>
> diff --git a/include/uapi/linux/kcov.h b/include/uapi/linux/kcov.h
> index 7d7147e7b427..76822d1c119a 100644
> --- a/include/uapi/linux/kcov.h
> +++ b/include/uapi/linux/kcov.h
> @@ -90,6 +90,7 @@ static inline __u64 kcov_remote_handle(__u64 subsys, __u64 inst)
> #define MEMORY_ACCESS_RECORD_WRITE 0x10
> #define MEMORY_ACCESS_RECORD_RMW 0x20
> #define MEMORY_ACCESS_RECORD_ATOMIC 0x40
> +#define MEMORY_ACCESS_RECORD_FREE 0x80
> struct memory_access_record {
> __aligned_u64 ip_address_and_kcov_flags;
> __aligned_u64 data_address;
> diff --git a/mm/kasan/common.c b/mm/kasan/common.c
> index 1ab77ac9719c..3a648eaad024 100644
> --- a/mm/kasan/common.c
> +++ b/mm/kasan/common.c
> @@ -283,6 +283,8 @@ bool __kasan_slab_free(struct kmem_cache *cache, void *object, bool init,
> return false;
>
> poison_slab_object(cache, object, init);
> + _kcov_handle_memaccess(object, cache->object_size,
> + MEMORY_ACCESS_RECORD_WRITE|MEMORY_ACCESS_RECORD_FREE);
>
> if (no_quarantine)
> return false;
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 083cbcb5bdde..910d14925b84 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -1446,6 +1446,9 @@ static __always_inline bool __free_pages_prepare(struct page *page,
> if (init)
> clear_highpages_kasan_tagged(page, 1 << order);
>
> + _kcov_handle_memaccess(page_address(page), (1 << order)*PAGE_SIZE,
> + MEMORY_ACCESS_RECORD_WRITE|MEMORY_ACCESS_RECORD_FREE);
> +
> /*
> * arch_free_page() can make the page's contents inaccessible. s390
> * does this. So nothing which can access the page's contents should
>
> --
> 2.55.0.979.g7e5102b832-goog
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH RFC v3 09/12] kcov: record return address on function entry
2026-09-08 16:54 [PATCH RFC v3 00/12] KCOV: entry/exit records, memory access records, and delay injection Jann Horn
` (7 preceding siblings ...)
2026-09-08 16:54 ` [PATCH RFC v3 08/12] kcov: log freeing of SLUB objects and pages Jann Horn
@ 2026-09-08 16:54 ` Jann Horn
2026-09-08 16:54 ` [PATCH RFC v3 10/12] kcov: log old value Jann Horn
` (2 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Jann Horn @ 2026-09-08 16:54 UTC (permalink / raw)
To: Dmitry Vyukov, Andrey Konovalov, Alexander Potapenko
Cc: Nathan Chancellor, Nick Desaulniers, Bill Wendling, Justin Stitt,
linux-kernel, kasan-dev, llvm, Jann Horn
It is helpful to know which code location a function was called from for:
- attributing calls to source locations in the callee
- attributing calls to inlined functions
For this purpose, make function entry records bigger, and record the caller
instruction address in them.
Signed-off-by: Jann Horn <jannh@google.com>
---
kernel/kcov.c | 26 ++++++++++++++++++--------
lib/Kconfig.debug | 2 ++
2 files changed, 20 insertions(+), 8 deletions(-)
diff --git a/kernel/kcov.c b/kernel/kcov.c
index 83e05aa61728..88aedaf41a9e 100644
--- a/kernel/kcov.c
+++ b/kernel/kcov.c
@@ -202,7 +202,8 @@ static notrace unsigned long canonicalize_ip(unsigned long ip)
return ip;
}
-static __always_inline void notrace kcov_add_pc_record(struct task_struct *t, unsigned long record)
+static __always_inline notrace
+void kcov_add_pc_record(struct task_struct *t, unsigned long record, bool hasext, unsigned long ext)
{
unsigned long *area;
unsigned long pos;
@@ -213,7 +214,7 @@ static __always_inline void notrace kcov_add_pc_record(struct task_struct *t, un
area = t->kcov_area;
/* The first 64-bit word is the number of subsequent PCs. */
pos = READ_ONCE(area[0]) + 1;
- if (likely(pos < t->kcov_size)) {
+ if (likely(pos + (hasext?1:0) < t->kcov_size)) {
/* Previously we write pc before updating pos. However, some
* early interrupt code could bypass check_kcov_context() check
* and invoke __sanitizer_cov_trace_pc(). If such interrupt is
@@ -221,9 +222,11 @@ static __always_inline void notrace kcov_add_pc_record(struct task_struct *t, un
* overitten by the recursive __sanitizer_cov_trace_pc().
* Update pos before writing pc to avoid such interleaving.
*/
- WRITE_ONCE(area[0], pos);
+ WRITE_ONCE(area[0], pos + (hasext?1:0));
barrier();
area[pos] = record;
+ if (hasext)
+ area[pos+1] = ext;
}
}
@@ -244,7 +247,7 @@ void notrace __sanitizer_cov_trace_pc(void)
* This relies on userspace not caring about the rest of the top byte
* for KCOV_RECORDFLAG_TYPE_NORMAL records.
*/
- kcov_add_pc_record(cur, canonicalize_ip(_RET_IP_));
+ kcov_add_pc_record(cur, canonicalize_ip(_RET_IP_), false, 0);
}
EXPORT_SYMBOL(__sanitizer_cov_trace_pc);
@@ -254,6 +257,7 @@ void notrace __sanitizer_cov_trace_pc_entry(void)
struct task_struct *cur = current;
unsigned long record = canonicalize_ip(_RET_IP_);
unsigned int kcov_mode = READ_ONCE(cur->kcov_mode);
+ bool ext_format;
/*
* This hook replaces __sanitizer_cov_trace_pc() for the function entry
@@ -265,9 +269,15 @@ void notrace __sanitizer_cov_trace_pc_entry(void)
cur->kcov->suppressed_stack_delta++;
return;
}
- if ((kcov_mode & KCOV_EXT_FORMAT) != 0)
+ ext_format = (kcov_mode & KCOV_EXT_FORMAT) != 0;
+ if (ext_format)
record = (record & KCOV_RECORD_IP_MASK) | KCOV_RECORDFLAG_TYPE_ENTRY;
- kcov_add_pc_record(cur, record);
+ /*
+ * __builtin_return_address(1) is safe because this function is only
+ * called from C functions, which are compiled with frame pointers
+ * enabled
+ */
+ kcov_add_pc_record(cur, record, ext_format, (unsigned long)__builtin_return_address(1));
}
void notrace __sanitizer_cov_trace_pc_exit(void)
{
@@ -293,7 +303,7 @@ void notrace __sanitizer_cov_trace_pc_exit(void)
return;
}
record = (canonicalize_ip(_RET_IP_) & KCOV_RECORD_IP_MASK) | KCOV_RECORDFLAG_TYPE_EXIT;
- kcov_add_pc_record(cur, record);
+ kcov_add_pc_record(cur, record, false, 0);
}
#endif
@@ -441,7 +451,7 @@ void kcov_finish_switch(struct task_struct *cur)
record = KCOV_RECORDFLAG_TYPE_EESUM |
(((u16)(s16)kcov->suppressed_stack_mindelta)<<16) |
(((u16)(s16)kcov->suppressed_stack_delta)<<16);
- kcov_add_pc_record(cur, record);
+ kcov_add_pc_record(cur, record, false, 0);
}
static void kcov_start(struct task_struct *t, struct kcov *kcov,
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index f763f0504f62..55c786a373b5 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -2200,6 +2200,8 @@ config KCOV_EXT_RECORDS
depends on KCOV
depends on 64BIT
depends on $(cc-option,-fsanitize-coverage=trace-pc-entry-exit)
+ select ARCH_WANT_FRAME_POINTERS
+ select FRAME_POINTER
help
Extended KCOV records allow distinguishing between multiple types of
records: Normal edge coverage, function entry, and function exit.
--
2.55.0.979.g7e5102b832-goog
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH RFC v3 10/12] kcov: log old value
2026-09-08 16:54 [PATCH RFC v3 00/12] KCOV: entry/exit records, memory access records, and delay injection Jann Horn
` (8 preceding siblings ...)
2026-09-08 16:54 ` [PATCH RFC v3 09/12] kcov: record return address on function entry Jann Horn
@ 2026-09-08 16:54 ` Jann Horn
2026-09-08 16:54 ` [PATCH RFC v3 11/12] kcov: introduce delay injection Jann Horn
2026-09-08 16:54 ` [PATCH RFC v3 12/12] Documentation/kcov: add documentation for EXT_RECORDS and KCOV_MEMORY Jann Horn
11 siblings, 0 replies; 14+ messages in thread
From: Jann Horn @ 2026-09-08 16:54 UTC (permalink / raw)
To: Dmitry Vyukov, Andrey Konovalov, Alexander Potapenko
Cc: Nathan Chancellor, Nick Desaulniers, Bill Wendling, Justin Stitt,
linux-kernel, kasan-dev, llvm, Jann Horn
For manual analysis of traces, log the old value at the memory location as
part of the memory_access_record.
This is best-effort; in particular:
- the value may not be recorded if it has an unusual size
- the recorded value may not match the value observed by the instrumented
memory operation in cases where the value is modified concurrently
Signed-off-by: Jann Horn <jannh@google.com>
---
include/uapi/linux/kcov.h | 2 ++
kernel/kcov.c | 20 ++++++++++++++++++++
2 files changed, 22 insertions(+)
diff --git a/include/uapi/linux/kcov.h b/include/uapi/linux/kcov.h
index 76822d1c119a..235f73e11d59 100644
--- a/include/uapi/linux/kcov.h
+++ b/include/uapi/linux/kcov.h
@@ -91,12 +91,14 @@ static inline __u64 kcov_remote_handle(__u64 subsys, __u64 inst)
#define MEMORY_ACCESS_RECORD_RMW 0x20
#define MEMORY_ACCESS_RECORD_ATOMIC 0x40
#define MEMORY_ACCESS_RECORD_FREE 0x80
+#define MEMORY_ACCESS_RECORD_VALUE 0x100 /* value field is valid */
struct memory_access_record {
__aligned_u64 ip_address_and_kcov_flags;
__aligned_u64 data_address;
__u32 size;
__u32 flags; /* MEMORY_ACCESS_RECORD_* */
__aligned_u64 time;
+ __aligned_u64 value;
} __attribute__((aligned(8)));
#endif /* _LINUX_KCOV_IOCTLS_H */
diff --git a/kernel/kcov.c b/kernel/kcov.c
index 88aedaf41a9e..ef405940a2cb 100644
--- a/kernel/kcov.c
+++ b/kernel/kcov.c
@@ -1267,6 +1267,26 @@ void notrace __kcov_handle_memaccess(const volatile void *p, size_t size, unsign
.flags = type,
.time = kcov_get_time()
};
+
+ switch (size) {
+ case 1:
+ __get_kernel_nofault((u8 *)&record->value, p, u8, handle_fault);
+ record->flags |= MEMORY_ACCESS_RECORD_VALUE;
+ break;
+ case 2:
+ __get_kernel_nofault((u16 *)&record->value, p, u16, handle_fault);
+ record->flags |= MEMORY_ACCESS_RECORD_VALUE;
+ break;
+ case 4:
+ __get_kernel_nofault((u32 *)&record->value, p, u32, handle_fault);
+ record->flags |= MEMORY_ACCESS_RECORD_VALUE;
+ break;
+ case 8:
+ __get_kernel_nofault((u64 *)&record->value, p, u64, handle_fault);
+ record->flags |= MEMORY_ACCESS_RECORD_VALUE;
+ break;
+ }
+handle_fault:;
}
void notrace _kcov_handle_memaccess(const volatile void *p, size_t size, unsigned int type)
--
2.55.0.979.g7e5102b832-goog
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH RFC v3 11/12] kcov: introduce delay injection
2026-09-08 16:54 [PATCH RFC v3 00/12] KCOV: entry/exit records, memory access records, and delay injection Jann Horn
` (9 preceding siblings ...)
2026-09-08 16:54 ` [PATCH RFC v3 10/12] kcov: log old value Jann Horn
@ 2026-09-08 16:54 ` Jann Horn
2026-09-08 16:54 ` [PATCH RFC v3 12/12] Documentation/kcov: add documentation for EXT_RECORDS and KCOV_MEMORY Jann Horn
11 siblings, 0 replies; 14+ messages in thread
From: Jann Horn @ 2026-09-08 16:54 UTC (permalink / raw)
To: Dmitry Vyukov, Andrey Konovalov, Alexander Potapenko
Cc: Nathan Chancellor, Nick Desaulniers, Bill Wendling, Justin Stitt,
linux-kernel, kasan-dev, llvm, Jann Horn
Introduce KCOV-based delay injection, which is intended for
deterministically testing race condition bugs.
Preceding patches allow userspace to record a trace of function entry/exit
events and memory access events in a multi-threaded test case.
After userspace identifies kernel memory accesses that could be part of a
race condition, userspace can use KCOV_SET_DI on the participating threads
to instruct KCOV to ensure that pairs of memory accesses execute in the
desired order.
A kcov_di_stack with type DI_STACK_WAIT instructs KCOV to spin-wait for
another kcov_di_stack (normally installed on another thread) with type
DI_STACK_WAKE and with the same flagidx.
Signed-off-by: Jann Horn <jannh@google.com>
---
include/uapi/linux/kcov.h | 45 ++++++
kernel/kcov.c | 380 +++++++++++++++++++++++++++++++++++++++++++++-
tools/objtool/check.c | 1 +
3 files changed, 422 insertions(+), 4 deletions(-)
diff --git a/include/uapi/linux/kcov.h b/include/uapi/linux/kcov.h
index 235f73e11d59..88bd11d7d108 100644
--- a/include/uapi/linux/kcov.h
+++ b/include/uapi/linux/kcov.h
@@ -3,6 +3,7 @@
#define _LINUX_KCOV_IOCTLS_H
#include <linux/types.h>
+#include <linux/ioctl.h>
/*
* Argument for KCOV_REMOTE_ENABLE ioctl, see Documentation/dev-tools/kcov.rst
@@ -23,6 +24,10 @@ struct kcov_remote_arg {
#define KCOV_DISABLE _IO('c', 101)
#define KCOV_REMOTE_ENABLE _IOW('c', 102, struct kcov_remote_arg)
#define KCOV_GET_MEMORY_RECORD_SIZE _IO('c', 103)
+#define KCOV_SET_DI _IOW('c', 104, struct kcov_set_di_arg)
+#define KCOV_RESET_DI_FLAGS _IO('c', 105)
+#define KCOV_WAKE_DI_FLAG _IO('c', 106)
+#define KCOV_SPINWAIT_DI_FLAG _IO('c', 107)
enum {
/*
@@ -54,6 +59,11 @@ enum {
/* Summarized entry/exit events that occurred in an untraced region. */
#define KCOV_RECORDFLAG_TYPE_EESUM 0x2000000000000000
#define KCOV_RECORDFLAG_TYPE_MEMORY 0x3000000000000000
+/* these two record types have a flag index in the low bits */
+#define KCOV_RECORDFLAG_TYPE_WAIT 0x4000000000000000
+#define KCOV_RECORDFLAG_TYPE_WAKE 0x5000000000000000
+/* set in KCOV_RECORDFLAG_TYPE_WAIT record to mark that the wait timed out */
+#define KCOV_WAIT_TIMEOUT 0x0010000000000000
/*
* The format for the types of collected comparisons.
@@ -101,4 +111,39 @@ struct memory_access_record {
__aligned_u64 value;
} __attribute__((aligned(8)));
+
+/*
+ * Delay Injection API
+ */
+struct kcov_di_stack_elem {
+ __aligned_u64 ip;
+ __aligned_u64 parent_idx;
+};
+enum di_stack_type {
+ DI_STACK_WAIT = 0,
+ DI_STACK_WAKE_PRE,
+ DI_STACK_WAKE_POST
+};
+struct kcov_di_stack {
+ __aligned_u64 elems;
+ __u32 num_elems;
+ enum di_stack_type type;
+ __u32 flagidx;
+};
+struct kcov_set_di_arg {
+ /*
+ * Pointer to array of struct kcov_di_stack.
+ * The array consists of function entry instruction addresses, with a
+ * memory access instruction address at the end.
+ * These must be addresses as reported in KCOV_RECORDFLAG_TYPE_ENTRY and
+ * KCOV_RECORDFLAG_TYPE_MEMORY events (so they are not the addresses
+ * where functions begin).
+ */
+ __aligned_u64 stacks;
+ __u32 num_stacks;
+
+ int sync_bits_fd;
+ __aligned_u64 spin_limit;
+};
+
#endif /* _LINUX_KCOV_IOCTLS_H */
diff --git a/kernel/kcov.c b/kernel/kcov.c
index ef405940a2cb..318ae2c89891 100644
--- a/kernel/kcov.c
+++ b/kernel/kcov.c
@@ -32,6 +32,22 @@
/* Number of 64-bit words written per one comparison: */
#define KCOV_WORDS_PER_CMP 4
+#define NUM_SYNC_BITS 64
+
+struct di_stack_elem {
+ unsigned long ip;
+ unsigned long parent_idx;
+ unsigned long cur_parent_idx;
+};
+
+struct di_stack {
+ struct di_stack_elem *elems;
+ unsigned int num_elems;
+ enum di_stack_type type;
+ unsigned int flagidx;
+ unsigned int nomatch_depth;
+};
+
/*
* kcov descriptor (one per opened debugfs file).
* State transitions of the descriptor:
@@ -78,6 +94,18 @@ struct kcov {
int sequence;
int suppressed_stack_delta;
int suppressed_stack_mindelta;
+
+ /* delay injection */
+ struct {
+ DECLARE_BITMAP(sync_bits, NUM_SYNC_BITS);
+ struct di_stack *match_stacks;
+ unsigned int num_match_stacks;
+ u64 spin_limit;
+ unsigned int stack_used;
+ unsigned int shared_nomatch_depth;
+ unsigned int pending_sync_bit;
+ struct kcov *syncbits_owner;
+ } di;
};
struct kcov_remote_area {
@@ -252,12 +280,53 @@ void notrace __sanitizer_cov_trace_pc(void)
EXPORT_SYMBOL(__sanitizer_cov_trace_pc);
#ifdef CONFIG_KCOV_EXT_RECORDS
+static void notrace kcov_di_enter_slowpath(struct kcov *kcov, unsigned long ip)
+{
+ unsigned int i;
+ bool no_matches = true;
+
+ for (int need_increments = 0; need_increments < 2; need_increments++) {
+ for (i = 0; i < kcov->di.num_match_stacks; i++) {
+ struct di_stack *dis = &kcov->di.match_stacks[i];
+ struct di_stack_elem *next_elem;
+
+ if (dis->nomatch_depth || kcov->di.stack_used >= dis->num_elems-1) {
+no_match:
+ if (need_increments)
+ dis->nomatch_depth++;
+ continue;
+ }
+ next_elem = &dis->elems[kcov->di.stack_used];
+ if (next_elem->ip != ip)
+ goto no_match;
+ if (need_increments == 0)
+ next_elem->cur_parent_idx++;
+ if (next_elem->parent_idx != next_elem->cur_parent_idx-1)
+ goto no_match;
+
+ /* going a step down in the di_stack */
+ no_matches = false;
+ next_elem[1].cur_parent_idx = 0;
+ }
+
+ if (likely(need_increments == 0 && no_matches)) {
+ kcov->di.shared_nomatch_depth++;
+ return;
+ }
+ /* do second pass and increment individual nomatch counters */
+ }
+
+ kcov->di.stack_used++;
+}
+
void notrace __sanitizer_cov_trace_pc_entry(void)
{
struct task_struct *cur = current;
- unsigned long record = canonicalize_ip(_RET_IP_);
+ unsigned long ip = canonicalize_ip(_RET_IP_);
+ unsigned long record = ip;
unsigned int kcov_mode = READ_ONCE(cur->kcov_mode);
bool ext_format;
+ struct kcov *kcov;
/*
* This hook replaces __sanitizer_cov_trace_pc() for the function entry
@@ -267,7 +336,7 @@ void notrace __sanitizer_cov_trace_pc_entry(void)
return;
if (kcov_mode & KCOV_IN_CTXSW) {
cur->kcov->suppressed_stack_delta++;
- return;
+ goto handle_distack;
}
ext_format = (kcov_mode & KCOV_EXT_FORMAT) != 0;
if (ext_format)
@@ -278,12 +347,25 @@ void notrace __sanitizer_cov_trace_pc_entry(void)
* enabled
*/
kcov_add_pc_record(cur, record, ext_format, (unsigned long)__builtin_return_address(1));
+
+handle_distack:
+ if (IS_ENABLED(CONFIG_KCOV_MEMORY)) {
+ kcov = cur->kcov;
+ if (unlikely(kcov->di.num_match_stacks)) {
+ if (likely(kcov->di.shared_nomatch_depth > 0)) {
+ kcov->di.shared_nomatch_depth++;
+ } else {
+ kcov_di_enter_slowpath(kcov, ip);
+ }
+ }
+ }
}
void notrace __sanitizer_cov_trace_pc_exit(void)
{
struct task_struct *cur = current;
unsigned long record;
unsigned int kcov_mode = READ_ONCE(cur->kcov_mode);
+ struct kcov *kcov;
/*
* This hook is not called at the beginning of a basic block; the basic
@@ -300,10 +382,31 @@ void notrace __sanitizer_cov_trace_pc_exit(void)
if (kcov->suppressed_stack_mindelta == kcov->suppressed_stack_delta)
kcov->suppressed_stack_mindelta--;
kcov->suppressed_stack_delta--;
- return;
+ goto handle_distack;
}
record = (canonicalize_ip(_RET_IP_) & KCOV_RECORD_IP_MASK) | KCOV_RECORDFLAG_TYPE_EXIT;
kcov_add_pc_record(cur, record, false, 0);
+
+handle_distack:
+ if (IS_ENABLED(CONFIG_KCOV_MEMORY)) {
+ kcov = cur->kcov;
+ if (unlikely(kcov->di.num_match_stacks)) {
+ if (likely(kcov->di.shared_nomatch_depth > 0)) {
+ kcov->di.shared_nomatch_depth--;
+ } else {
+ unsigned int i;
+
+ if (kcov->di.stack_used)
+ kcov->di.stack_used--;
+ for (i = 0; i < kcov->di.num_match_stacks; i++) {
+ struct di_stack *dis = &kcov->di.match_stacks[i];
+
+ if (dis->nomatch_depth > 0)
+ dis->nomatch_depth--;
+ }
+ }
+ }
+ }
}
#endif
@@ -458,6 +561,17 @@ static void kcov_start(struct task_struct *t, struct kcov *kcov,
unsigned int size, void *area, unsigned int mode,
int sequence)
{
+ int i;
+
+ if (IS_ENABLED(CONFIG_KCOV_MEMORY)) {
+ /* delay injection */
+ kcov->di.stack_used = 0;
+ kcov->di.shared_nomatch_depth = 0;
+ kcov->di.pending_sync_bit = UINT_MAX;
+ for (i = 0; i < kcov->di.num_match_stacks; i++)
+ kcov->di.match_stacks[i].elems[0].cur_parent_idx = 0;
+ }
+
kcov_debug("t = %px, size = %u, area = %px\n", t, size, area);
t->kcov = kcov;
/* Cache in task struct for performance. */
@@ -547,6 +661,15 @@ static void kcov_get(struct kcov *kcov)
refcount_inc(&kcov->refcount);
}
+static void free_di_stacks(struct di_stack *di_stacks, unsigned int num_stacks)
+{
+ unsigned int i;
+
+ for (i = 0; i < num_stacks; i++)
+ kfree(di_stacks[i].elems);
+ kfree(di_stacks);
+}
+
static void kcov_put(struct kcov *kcov)
{
if (refcount_dec_and_test(&kcov->refcount)) {
@@ -555,6 +678,11 @@ static void kcov_put(struct kcov *kcov)
kcov_remote_reset(kcov);
vfree(kcov->area);
);
+ if (IS_ENABLED(CONFIG_KCOV_MEMORY)) {
+ free_di_stacks(kcov->di.match_stacks, kcov->di.num_match_stacks);
+ if (kcov->di.syncbits_owner && kcov->di.syncbits_owner != kcov)
+ kcov_put(kcov->di.syncbits_owner);
+ }
kfree(kcov);
}
}
@@ -825,9 +953,180 @@ static int kcov_ioctl_locked(struct kcov *kcov, unsigned int cmd,
}
}
+static const struct file_operations kcov_fops;
+
+static int kcov_set_delay_injection(struct kcov *kcov, unsigned long arg_uaddr)
+{
+ struct kcov_set_di_arg arg;
+ struct di_stack *di_stacks;
+ int i, j;
+ int ret;
+ unsigned long flags;
+ struct file *syncbits_owner_file;
+ struct kcov *syncbits_owner;
+
+ if (!IS_ENABLED(CONFIG_KCOV_MEMORY))
+ return -ENOTSUPP;
+ if (copy_from_user(&arg, (void __user *)arg_uaddr, sizeof(arg)))
+ return -EFAULT;
+ if (arg.num_stacks > 128)
+ return -ERANGE;
+
+ /*
+ * This feature *intentionally* allows forcing the kernel to spinloop
+ * for a long time, including in contexts in which that would normally
+ * be a terrible idea.
+ * To prevent the user from causing a persistent system hang with this,
+ * cap the number of spinloop iterations.
+ */
+ if (arg.spin_limit > 10000000000)
+ return -ERANGE;
+
+ di_stacks = kmalloc_array(arg.num_stacks, sizeof(struct di_stack), GFP_KERNEL|__GFP_ZERO);
+ if (!di_stacks)
+ return -ENOMEM;
+
+ if (arg.sync_bits_fd != -1) {
+ syncbits_owner_file = fget(arg.sync_bits_fd);
+ if (!syncbits_owner_file) {
+ ret = -EBADF;
+ goto out_freestacks;
+ }
+ if (syncbits_owner_file->f_op != &kcov_fops ||
+ syncbits_owner_file->private_data == kcov) {
+ ret = -EBADF;
+ fput(syncbits_owner_file);
+ goto out_freestacks;
+ }
+ syncbits_owner = syncbits_owner_file->private_data;
+ kcov_get(syncbits_owner);
+ fput(syncbits_owner_file);
+
+ /*
+ * Ensure that the syncbits_owner does not, and can never,
+ * point to yet another KCOV instance.
+ */
+ spin_lock_irqsave(&syncbits_owner->lock, flags);
+ if (syncbits_owner->di.syncbits_owner &&
+ syncbits_owner->di.syncbits_owner != syncbits_owner) {
+ spin_unlock_irqrestore(&syncbits_owner->lock, flags);
+ ret = -ELOOP;
+ goto out_put_syncbits_owner;
+ }
+ if (!syncbits_owner->di.syncbits_owner)
+ syncbits_owner->di.syncbits_owner = syncbits_owner;
+ spin_unlock_irqrestore(&syncbits_owner->lock, flags);
+ } else {
+ syncbits_owner = kcov;
+ kcov_get(syncbits_owner);
+ }
+
+ for (i = 0; i < arg.num_stacks; i++) {
+ struct kcov_di_stack __user *user_stackp =
+ ((struct kcov_di_stack __user *)u64_to_user_ptr(arg.stacks)) + i;
+ struct kcov_di_stack u_di_stack;
+
+ if (copy_from_user(&u_di_stack, user_stackp, sizeof(struct kcov_di_stack))) {
+ ret = -EFAULT;
+ goto out_put_syncbits_owner;
+ }
+ if (u_di_stack.num_elems < 2 || u_di_stack.num_elems > 32 ||
+ u_di_stack.flagidx >= NUM_SYNC_BITS) {
+ ret = -ERANGE;
+ goto out_put_syncbits_owner;
+ }
+ if (u_di_stack.type != DI_STACK_WAIT && u_di_stack.type != DI_STACK_WAKE_PRE &&
+ u_di_stack.type != DI_STACK_WAKE_POST) {
+ ret = -EINVAL;
+ goto out_put_syncbits_owner;
+ }
+ di_stacks[i] = (struct di_stack) {
+ .elems = kmalloc_array(u_di_stack.num_elems, sizeof(struct di_stack_elem),
+ GFP_KERNEL),
+ .num_elems = u_di_stack.num_elems,
+ .type = u_di_stack.type,
+ .flagidx = u_di_stack.flagidx
+ };
+ if (!di_stacks[i].elems) {
+ ret = -ENOMEM;
+ goto out_put_syncbits_owner;
+ }
+ for (j = 0; j < u_di_stack.num_elems; j++) {
+ struct kcov_di_stack_elem __user *user_elemp =
+ ((struct kcov_di_stack_elem __user *)u_di_stack.elems) + j;
+ struct kcov_di_stack_elem user_elem;
+
+ if (copy_from_user(&user_elem, user_elemp, sizeof(user_elem))) {
+ ret = -EFAULT;
+ goto out_put_syncbits_owner;
+ }
+ di_stacks[i].elems[j] = (struct di_stack_elem) {
+ .ip = user_elem.ip,
+ .parent_idx = user_elem.parent_idx
+ };
+ }
+ }
+
+ spin_lock_irqsave(&kcov->lock, flags);
+ if (kcov->t) {
+ ret = -EBUSY;
+ } else if (kcov->di.syncbits_owner && kcov->di.syncbits_owner != syncbits_owner) {
+ ret = -EBADFD;
+ } else {
+ /* load config */
+ free_di_stacks(kcov->di.match_stacks, kcov->di.num_match_stacks);
+ kcov->di.match_stacks = di_stacks;
+ kcov->di.num_match_stacks = arg.num_stacks;
+ kcov->di.spin_limit = arg.spin_limit;
+ if (!kcov->di.syncbits_owner) {
+ /* Avoid reference loop. */
+ if (syncbits_owner != kcov)
+ kcov_get(syncbits_owner);
+ kcov->di.syncbits_owner = syncbits_owner;
+ }
+
+ ret = 0;
+ }
+ spin_unlock_irqrestore(&kcov->lock, flags);
+
+out_put_syncbits_owner:
+ kcov_put(syncbits_owner);
+out_freestacks:
+ if (ret)
+ free_di_stacks(di_stacks, arg.num_stacks);
+ return ret;
+}
+
+static int notrace __kcov_spin_wait(struct kcov *kcov, unsigned int flagidx)
+{
+ while (!test_bit(flagidx, kcov->di.syncbits_owner->di.sync_bits)) {
+ u64 spin_limit = READ_ONCE(kcov->di.spin_limit);
+
+ if (spin_limit == 0) /* spin timeout */
+ return -ETIMEDOUT;
+ WRITE_ONCE(kcov->di.spin_limit, spin_limit - 1);
+ cpu_relax();
+ }
+ return 0;
+}
+
+/*
+ * Look up kcov->syncbits_owner in a way that is safe is @kcov is not active on
+ * the current task.
+ */
+static struct kcov *get_syncbits_owner(struct kcov *kcov)
+{
+ guard(spinlock_irqsave)(&kcov->lock);
+
+ if (!kcov->di.syncbits_owner)
+ return NULL;
+ kcov_get(kcov->di.syncbits_owner);
+ return kcov->di.syncbits_owner;
+}
+
static long kcov_ioctl(struct file *filep, unsigned int cmd, unsigned long arg)
{
- struct kcov *kcov;
+ struct kcov *kcov, *syncbits_owner;
int res;
struct kcov_remote_arg *remote_arg = NULL;
unsigned int remote_num_handles;
@@ -862,6 +1161,29 @@ static long kcov_ioctl(struct file *filep, unsigned int cmd, unsigned long arg)
kcov->mode = KCOV_MODE_INIT;
spin_unlock_irqrestore(&kcov->lock, flags);
return 0;
+ case KCOV_SET_DI:
+ return kcov_set_delay_injection(kcov, arg);
+ case KCOV_RESET_DI_FLAGS:
+ case KCOV_WAKE_DI_FLAG:
+ case KCOV_SPINWAIT_DI_FLAG:
+ if (!IS_ENABLED(CONFIG_KCOV_MEMORY))
+ return -ENOTSUPP;
+ if (arg >= NUM_SYNC_BITS)
+ return -EINVAL;
+ syncbits_owner = get_syncbits_owner(kcov);
+ if (!syncbits_owner)
+ return -EINVAL;
+ if (cmd == KCOV_RESET_DI_FLAGS) {
+ bitmap_clear(syncbits_owner->di.sync_bits, 0, NUM_SYNC_BITS);
+ res = 0;
+ } else if (cmd == KCOV_WAKE_DI_FLAG) {
+ set_bit(arg, syncbits_owner->di.sync_bits);
+ res = 0;
+ } else {
+ res = __kcov_spin_wait(syncbits_owner, arg);
+ }
+ kcov_put(syncbits_owner);
+ return res;
case KCOV_REMOTE_ENABLE:
if (get_user(remote_num_handles, (unsigned __user *)(arg +
offsetof(struct kcov_remote_arg, num_handles))))
@@ -1254,9 +1576,56 @@ void notrace __kcov_handle_memaccess(const volatile void *p, size_t size, unsign
struct task_struct *t = current;
struct memory_access_record *record;
unsigned int kcov_mode = READ_ONCE(t->kcov_mode);
+ struct kcov *kcov;
+ int di_wake_idx = -1;
if (kcov_mode != KCOV_MODE_TRACE_PC_AND_MEM || !check_kcov_context(t))
return;
+
+ kcov = t->kcov;
+ if (IS_ENABLED(CONFIG_KCOV_MEMORY) && unlikely(kcov->di.num_match_stacks)) {
+ if (unlikely(kcov->di.pending_sync_bit != UINT_MAX)) {
+ set_bit(kcov->di.pending_sync_bit, kcov->di.syncbits_owner->di.sync_bits);
+ kcov->di.pending_sync_bit = UINT_MAX;
+ }
+
+ if (unlikely(kcov->di.shared_nomatch_depth == 0)) {
+ /* similar to kcov_di_enter_slowpath */
+ unsigned int i;
+
+ for (i = 0; i < kcov->di.num_match_stacks; i++) {
+ struct di_stack *dis = &kcov->di.match_stacks[i];
+ struct di_stack_elem *elem;
+
+ if (dis->nomatch_depth || kcov->di.stack_used != dis->num_elems-1)
+ continue;
+ elem = &dis->elems[kcov->di.stack_used];
+ if (elem->ip != ret_ip)
+ continue;
+ if (elem->parent_idx != elem->cur_parent_idx++)
+ continue;
+ if (dis->type == DI_STACK_WAIT) {
+ unsigned long wait_record = KCOV_RECORDFLAG_TYPE_WAIT;
+
+ wait_record |= dis->flagidx;
+ if (__kcov_spin_wait(kcov->di.syncbits_owner, dis->flagidx))
+ wait_record |= KCOV_WAIT_TIMEOUT;
+ kcov_add_pc_record(t, wait_record, false, 0);
+ } else if (dis->type == DI_STACK_WAKE_PRE) {
+ kcov_add_pc_record(t,
+ KCOV_RECORDFLAG_TYPE_WAKE | dis->flagidx,
+ false, 0);
+ set_bit(dis->flagidx,
+ kcov->di.syncbits_owner->di.sync_bits);
+ } else {
+ /* DI_STACK_WAKE_POST */
+ di_wake_idx = dis->flagidx;
+ kcov->di.pending_sync_bit = dis->flagidx;
+ }
+ }
+ }
+ }
+
if (!kcov_get_memaccess_record(t, &record))
return;
*record = (struct memory_access_record) {
@@ -1287,6 +1656,9 @@ void notrace __kcov_handle_memaccess(const volatile void *p, size_t size, unsign
break;
}
handle_fault:;
+
+ if (unlikely(di_wake_idx != -1))
+ kcov_add_pc_record(t, KCOV_RECORDFLAG_TYPE_WAKE | di_wake_idx, false, 0);
}
void notrace _kcov_handle_memaccess(const volatile void *p, size_t size, unsigned int type)
diff --git a/tools/objtool/check.c b/tools/objtool/check.c
index 08ebfe1f3fac..30d748249b1c 100644
--- a/tools/objtool/check.c
+++ b/tools/objtool/check.c
@@ -1222,6 +1222,7 @@ static const char *uaccess_safe_builtin[] = {
"__kcov_handle_memaccess",
"__sanitizer_cov_trace_pc",
"__sanitizer_cov_trace_pc_entry",
+ "kcov_di_enter_slowpath",
"__sanitizer_cov_trace_pc_exit",
"__sanitizer_cov_trace_const_cmp1",
"__sanitizer_cov_trace_const_cmp2",
--
2.55.0.979.g7e5102b832-goog
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH RFC v3 12/12] Documentation/kcov: add documentation for EXT_RECORDS and KCOV_MEMORY
2026-09-08 16:54 [PATCH RFC v3 00/12] KCOV: entry/exit records, memory access records, and delay injection Jann Horn
` (10 preceding siblings ...)
2026-09-08 16:54 ` [PATCH RFC v3 11/12] kcov: introduce delay injection Jann Horn
@ 2026-09-08 16:54 ` Jann Horn
11 siblings, 0 replies; 14+ messages in thread
From: Jann Horn @ 2026-09-08 16:54 UTC (permalink / raw)
To: Dmitry Vyukov, Andrey Konovalov, Alexander Potapenko
Cc: Nathan Chancellor, Nick Desaulniers, Bill Wendling, Justin Stitt,
linux-kernel, kasan-dev, llvm, Jann Horn
Document the new KCOV features CONFIG_KCOV_EXT_RECORDS and
CONFIG_KCOV_MEMORY.
Signed-off-by: Jann Horn <jannh@google.com>
---
Documentation/dev-tools/kcov.rst | 70 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 70 insertions(+)
diff --git a/Documentation/dev-tools/kcov.rst b/Documentation/dev-tools/kcov.rst
index 1a739290c8ec..41eef656ed90 100644
--- a/Documentation/dev-tools/kcov.rst
+++ b/Documentation/dev-tools/kcov.rst
@@ -383,3 +383,73 @@ local tasks spawned by the process and the global task that handles USB bus #1:
perror("close"), exit(1);
return 0;
}
+
+Extended trace format
+---------------------
+
+If the kernel is built with ``CONFIG_KCOV_EXT_RECORDS=y`` (which requires LLVM
+>=23.1.0), the ``KCOV_TRACE_PC_EXT`` mode can be used instead of
+``KCOV_TRACE_PC``.
+
+``KCOV_TRACE_PC_EXT`` uses the top byte of recorded PCs to store a record type.
+The function entry block is recorded with type ``KCOV_RECORDFLAG_TYPE_ENTRY``,
+and an additional record with ``KCOV_RECORDFLAG_TYPE_EXIT`` is generated on
+function exit.
+
+``KCOV_RECORDFLAG_TYPE_ENTRY`` records are immediately followed by the PC from
+which the call occurred.
+
+After code sections which have to temporarily stop emitting KCOV trace events,
+a ``KCOV_RECORDFLAG_TYPE_EESUM`` record summarizes the entry/exit events that
+happened.
+
+Together, these record types allow keeping track of the current stack trace.
+
+Memory access tracing
+---------------------
+
+If the kernel is built with ``CONFIG_KCOV_MEMORY=y`` (which depends on
+``CONFIG_KCOV_EXT_RECORDS=y``), the ``KCOV_TRACE_MEMORY_ACCESS`` mode can be
+used to produce a trace similar to ``KCOV_TRACE_PC_EXT``, but with additional
+``KCOV_RECORDFLAG_TYPE_MEMORY`` records that are emitted for every memory
+access.
+
+In such a trace, when a record with type ``KCOV_RECORDFLAG_TYPE_MEMORY`` is
+encountered, the trace element is a ``struct memory_access_record`` with a
+size returned by the ioctl ``KCOV_GET_MEMORY_RECORD_SIZE``.
+
+Delay injection
+---------------
+
+If the kernel is built with ``CONFIG_KCOV_MEMORY=y``, userspace can configure
+soft ordering constraints (like "this load on thread A should happen before that
+write happens on thread B") through the ioctl ``KCOV_SET_DI``, with an argument
+pointing to a ``struct kcov_set_di_arg``.
+The kernel will attempt to fulfill these ordering constraints by spin-waiting,
+with a configurable timeout ``spin_limit`` after which the kernel gives up on
+forcing the specified ordering.
+
+For each thread, userspace supplies an array of ``struct kcov_di_stack``
+elements, each of which describes an action to take at a specific call stack
+ending at an instrumented memory access.
+An action is one of:
+
+ - ``DI_STACK_WAKE_PRE``: "set synchronization bit N before this memory access"
+ - ``DI_STACK_WAKE_POST``: "set synchronization bit N after this memory access"
+ - ``DI_STACK_WAIT``: "spin-wait for synchronization bit N"
+
+These are normally paired between two threads: One thread sets synchronization
+bit N after the access at call stack A, another thread spin-waits for
+synchronization bit N before the access at call stack B, and this establishes an
+A-happens-before-B ordering.
+
+Since this involves multiple threads (and therefore multiple KCOV instances),
+the member ``sync_bits_fd`` in ``struct kcov_set_di_arg`` informs the kernel
+which KCOV instance holds the shared synchronization bits (where -1 means the
+current instance).
+
+Userspace can also directly interact with these synchronization bits using:
+
+ - ``KCOV_RESET_DI_FLAGS`` for zeroing all bits
+ - ``KCOV_WAKE_DI_FLAG`` for setting a specific bit
+ - ``KCOV_SPINWAIT_DI_FLAG`` for spin-waiting on a specific bit
--
2.55.0.979.g7e5102b832-goog
^ permalink raw reply related [flat|nested] 14+ messages in thread