Linux Trace Kernel
 help / color / mirror / Atom feed
* [PATCH 0/3] sched/preempt: Check notrace preempt-count updates
@ 2026-11-16  7:54 Karl Mehltretter
  2026-11-16  7:54 ` [PATCH 1/3] arm64: Always inline raw preempt count updates Karl Mehltretter
                   ` (2 more replies)
  0 siblings, 3 replies; 5+ messages in thread
From: Karl Mehltretter @ 2026-11-16  7:54 UTC (permalink / raw)
  To: Peter Zijlstra, Ingo Molnar, Catalin Marinas, Will Deacon
  Cc: Karl Mehltretter, Mark Rutland, Hongyan Xia, Song Liu,
	Steven Rostedt, Juri Lelli, Vincent Guittot, Dietmar Eggemann,
	Valentin Schneider, linux-arm-kernel, linux-trace-kernel,
	linux-kernel

With CONFIG_DEBUG_PREEMPT, the notrace preemption macros bypass the
bounds checks used by ordinary operations. Add checked noinstr updates
for their existing callers and share validation with
preempt_count_add() and preempt_count_sub().

Patch 1 forces the arm64 raw updates called by the new helpers inline.
Patch 2 does the same for preempt_count(), which the helpers use, for
should_resched(), reached by the debug notrace enable path, and for the
existing non-debug __preempt_count_dec_and_test() path. The corresponding
x86, s390 and asm-generic operations are already __always_inline.

Patch 1 is extracted from Hongyan Xia's entry RFC. Song Liu also posted
notrace annotations for the same arm64 helpers:

  https://lore.kernel.org/r/20261023011049.1164625-10-song@kernel.org

Patch 3 disables branch profiling for core.o when DEBUG_PREEMPT and
TRACE_BRANCH_PROFILING are both enabled. DEBUG_LOCKS_WARN_ON() evaluates
its condition before entering its instrumentation region, so profiling
that condition would call ftrace_likely_update() from noinstr text. The
scheduler Makefile already uses this per-object exclusion for
build_policy.o and build_utility.o.

The cost is loss of branch-profile data for the rest of core.o in that
configuration. Scheduler tracepoints and function tracing are unchanged.
A dedicated non-profiled object is the fallback if this object-wide cost
is considered too broad.

This series is independently useful. It also supplies the checked count
update planned for follow-up IRQ-entry work; it does not change IRQ entry,
trace-recursion lifetime or TRACE_CTX_TRANSITION.

This addresses Peter's suggestion here:

  https://lore.kernel.org/r/20261029093219.GH1779021@noisy.programming.kicks-ass.net

The series is based on v7.4-rc2.

Testing:

- Clang 19.1.7 x86-64: a W=1 defconfig-derived build with
  DEBUG_PREEMPT, annotated branch profiling, DEBUG_ENTRY and noinstr
  validation passed. A nested-KVM boot passed the built-in and external
  module checks and all five tracer workloads. A DEBUG_PREEMPT=n control
  retained branch profiling for core.o and emitted no new helper symbols.
- GCC 14.2 arm64: a W=1 PREEMPT_RT build and allmodconfig passed. The
  latter's nVHE object had no undefined references to the new helpers.
  A native Axion boot passed the built-in and module checks and all five
  tracers.
- GCC 14.2 and 16.2 ARC, C-SKY and MIPS: performance, size, all-branches
  and forced-noinline core.o inspections passed without prohibited
  noinstr relocations or branch-profile records. Full baseline and patched
  kernels plus the external debug module built. Matching QEMU boots reached
  the expected balanced and invalid-count markers on ARC and C-SKY UP and
  MIPS UP/SMP. The patched kernels diagnosed every invalid operation;
  rejected decrements retained their counts. An ARC DEBUG_PREEMPT=n
  control retained core.o profiling without references to the new helpers.

The MIPS UP baseline and patched positive boots emitted the same three
existing smp_processor_id() diagnostics; the SMP boots emitted none.

I used an LLM to help extract and review the changes, write the test
harnesses and prepare the commit messages.

Hongyan Xia (1):
  arm64: Always inline raw preempt count updates

Karl Mehltretter (2):
  arm64: Always inline preempt count and reschedule checks
  sched/preempt: Check preempt counts in notrace helpers

 arch/arm64/include/asm/preempt.h | 10 ++++----
 include/linux/preempt.h          | 22 ++++++++++++++----
 kernel/sched/Makefile            |  3 +++
 kernel/sched/core.c              | 40 ++++++++++++++++++++++++++++----
 4 files changed, 62 insertions(+), 13 deletions(-)


base-commit: 7475ea8b58616b33c04afb976cf0fbaad9dd63fb

^ permalink raw reply	[flat|nested] 5+ messages in thread

* [PATCH 1/3] arm64: Always inline raw preempt count updates
  2026-11-16  7:54 [PATCH 0/3] sched/preempt: Check notrace preempt-count updates Karl Mehltretter
@ 2026-11-16  7:54 ` Karl Mehltretter
  2026-11-16  8:16   ` sashiko-bot
  2026-11-16  7:54 ` [PATCH 2/3] arm64: Always inline preempt count and reschedule checks Karl Mehltretter
  2026-11-16  7:54 ` [PATCH 3/3] sched/preempt: Check preempt counts in notrace helpers Karl Mehltretter
  2 siblings, 1 reply; 5+ messages in thread
From: Karl Mehltretter @ 2026-11-16  7:54 UTC (permalink / raw)
  To: Peter Zijlstra, Ingo Molnar, Catalin Marinas, Will Deacon
  Cc: Hongyan Xia, Mark Rutland, Song Liu, Steven Rostedt, Juri Lelli,
	Vincent Guittot, Dietmar Eggemann, Valentin Schneider,
	linux-arm-kernel, linux-trace-kernel, linux-kernel,
	Karl Mehltretter

From: Hongyan Xia <hongyan.xia@transsion.com>

Raw preempt count updates can be emitted out of line by the compiler.
For a noinstr caller, that puts the update in ordinary instrumentable
text. Force __preempt_count_add() and __preempt_count_sub() inline so
their code stays in the caller's section.

Link: https://lore.kernel.org/all/c3a90368dd6c096af9d918e036a27728696fb8a4.1786603168.git.hongyan.xia@transsion.com/
Signed-off-by: Hongyan Xia <hongyan.xia@transsion.com>
[kmehltretter: split out the arm64 preempt.h hunk and rewrite the changelog]
Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>
---
 arch/arm64/include/asm/preempt.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/include/asm/preempt.h b/arch/arm64/include/asm/preempt.h
index 75a592b7f47f7..f3a5c2cd1936d 100644
--- a/arch/arm64/include/asm/preempt.h
+++ b/arch/arm64/include/asm/preempt.h
@@ -41,14 +41,14 @@ static inline bool test_preempt_need_resched(void)
 	return !current_thread_info()->preempt.need_resched;
 }
 
-static inline void __preempt_count_add(int val)
+static __always_inline void __preempt_count_add(int val)
 {
 	u32 pc = READ_ONCE(current_thread_info()->preempt.count);
 	pc += val;
 	WRITE_ONCE(current_thread_info()->preempt.count, pc);
 }
 
-static inline void __preempt_count_sub(int val)
+static __always_inline void __preempt_count_sub(int val)
 {
 	u32 pc = READ_ONCE(current_thread_info()->preempt.count);
 	pc -= val;

^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH 2/3] arm64: Always inline preempt count and reschedule checks
  2026-11-16  7:54 [PATCH 0/3] sched/preempt: Check notrace preempt-count updates Karl Mehltretter
  2026-11-16  7:54 ` [PATCH 1/3] arm64: Always inline raw preempt count updates Karl Mehltretter
@ 2026-11-16  7:54 ` Karl Mehltretter
  2026-11-16  7:54 ` [PATCH 3/3] sched/preempt: Check preempt counts in notrace helpers Karl Mehltretter
  2 siblings, 0 replies; 5+ messages in thread
From: Karl Mehltretter @ 2026-11-16  7:54 UTC (permalink / raw)
  To: Peter Zijlstra, Ingo Molnar, Catalin Marinas, Will Deacon
  Cc: Karl Mehltretter, Mark Rutland, Hongyan Xia, Song Liu,
	Steven Rostedt, Juri Lelli, Vincent Guittot, Dietmar Eggemann,
	Valentin Schneider, linux-arm-kernel, linux-trace-kernel,
	linux-kernel

The preempt count and reschedule checks are plain inline functions,
which permits out-of-line copies in instrumentable text. The checked
notrace helpers added by the following patch use preempt_count(), while
preempt_enable_notrace() uses should_resched() with
CONFIG_DEBUG_PREEMPT and __preempt_count_dec_and_test() without it.

Force all three operations inline so noinstr callers keep them in their
own section.

Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>
---
 arch/arm64/include/asm/preempt.h | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/include/asm/preempt.h b/arch/arm64/include/asm/preempt.h
index f3a5c2cd1936d..805f7b33baf09 100644
--- a/arch/arm64/include/asm/preempt.h
+++ b/arch/arm64/include/asm/preempt.h
@@ -7,7 +7,7 @@
 #define PREEMPT_NEED_RESCHED	BIT(32)
 #define PREEMPT_ENABLED	(PREEMPT_NEED_RESCHED)
 
-static inline int preempt_count(void)
+static __always_inline int preempt_count(void)
 {
 	return READ_ONCE(current_thread_info()->preempt.count);
 }
@@ -75,13 +75,13 @@ static inline int __preempt_count_sub_return(int val)
 	return pc;
 }
 
-static inline bool should_resched(int preempt_offset)
+static __always_inline bool should_resched(int preempt_offset)
 {
 	u64 pc = READ_ONCE(current_thread_info()->preempt_count);
 	return unlikely(pc == preempt_offset);
 }
 
-static inline bool __preempt_count_dec_and_test(void)
+static __always_inline bool __preempt_count_dec_and_test(void)
 {
 	/*
 	 * We must load the combined 'preempt_count' after decrementing

^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH 3/3] sched/preempt: Check preempt counts in notrace helpers
  2026-11-16  7:54 [PATCH 0/3] sched/preempt: Check notrace preempt-count updates Karl Mehltretter
  2026-11-16  7:54 ` [PATCH 1/3] arm64: Always inline raw preempt count updates Karl Mehltretter
  2026-11-16  7:54 ` [PATCH 2/3] arm64: Always inline preempt count and reschedule checks Karl Mehltretter
@ 2026-11-16  7:54 ` Karl Mehltretter
  2 siblings, 0 replies; 5+ messages in thread
From: Karl Mehltretter @ 2026-11-16  7:54 UTC (permalink / raw)
  To: Peter Zijlstra, Ingo Molnar, Catalin Marinas, Will Deacon
  Cc: Karl Mehltretter, Mark Rutland, Hongyan Xia, Song Liu,
	Steven Rostedt, Juri Lelli, Vincent Guittot, Dietmar Eggemann,
	Valentin Schneider, linux-arm-kernel, linux-trace-kernel,
	linux-kernel

The notrace preemption macros use raw count updates even with
CONFIG_DEBUG_PREEMPT enabled. They therefore skip the bounds checks
which ordinary preemption operations provide.

Add checked noinstr preempt-count helpers and use them in
preempt_disable_notrace(), preempt_enable_notrace() and
preempt_enable_no_resched_notrace(). Export the out-of-line helpers
for module callers. Keep raw updates when CONFIG_DEBUG_PREEMPT is
disabled.

Share the validation with preempt_count_add() and preempt_count_sub(),
while leaving their preemption-latency tracing and ordering unchanged.
DEBUG_LOCKS_WARN_ON() brackets its diagnostic work with
instrumentation_begin() and instrumentation_end(), but evaluates its
condition before that region. Branch profiling would therefore call
ftrace_likely_update() from noinstr text.

Compile core.o with branch profiling disabled when DEBUG_PREEMPT and
TRACE_BRANCH_PROFILING are both enabled, as the scheduler Makefile already
does for build_policy.o and build_utility.o. This keeps valid notrace count
updates free of profiler calls while retaining the existing warning path.
The tradeoff is loss of branch-profile data for the rest of core.o in that
configuration; scheduler tracepoints and function tracing are unchanged.

Suggested-by: Peter Zijlstra <peterz@infradead.org>
Link: https://lore.kernel.org/r/20261029093219.GH1779021@noisy.programming.kicks-ass.net
Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>
---
 include/linux/preempt.h | 22 ++++++++++++++++++----
 kernel/sched/Makefile   |  3 +++
 kernel/sched/core.c     | 40 ++++++++++++++++++++++++++++++++++++----
 3 files changed, 57 insertions(+), 8 deletions(-)

diff --git a/include/linux/preempt.h b/include/linux/preempt.h
index 06ff44a4b8b6a..42c585fd91d45 100644
--- a/include/linux/preempt.h
+++ b/include/linux/preempt.h
@@ -217,6 +217,17 @@ static __always_inline unsigned char interrupt_context_level(void)
  */
 #define in_atomic_preempt_off() (preempt_count() != PREEMPT_DISABLE_OFFSET)
 
+#ifdef CONFIG_DEBUG_PREEMPT
+void preempt_count_add_notrace(int val);
+void preempt_count_sub_notrace(int val);
+#define preempt_count_dec_and_test_notrace() \
+	({ preempt_count_sub_notrace(1); should_resched(0); })
+#else
+#define preempt_count_add_notrace(val)	__preempt_count_add(val)
+#define preempt_count_sub_notrace(val)	__preempt_count_sub(val)
+#define preempt_count_dec_and_test_notrace() __preempt_count_dec_and_test()
+#endif
+
 #if defined(CONFIG_DEBUG_PREEMPT) || defined(CONFIG_TRACE_PREEMPT_TOGGLE)
 extern void preempt_count_add(int val);
 extern void preempt_count_sub(int val);
@@ -231,6 +242,9 @@ extern void preempt_count_sub(int val);
 #define __preempt_count_inc() __preempt_count_add(1)
 #define __preempt_count_dec() __preempt_count_sub(1)
 
+#define preempt_count_inc_notrace() preempt_count_add_notrace(1)
+#define preempt_count_dec_notrace() preempt_count_sub_notrace(1)
+
 #define preempt_count_inc() preempt_count_add(1)
 #define preempt_count_dec() preempt_count_sub(1)
 
@@ -263,7 +277,7 @@ do { \
 #define preempt_enable_notrace() \
 do { \
 	barrier(); \
-	if (unlikely(__preempt_count_dec_and_test())) \
+	if (unlikely(preempt_count_dec_and_test_notrace())) \
 		__preempt_schedule_notrace(); \
 } while (0)
 
@@ -283,7 +297,7 @@ do { \
 #define preempt_enable_notrace() \
 do { \
 	barrier(); \
-	__preempt_count_dec(); \
+	preempt_count_dec_notrace(); \
 } while (0)
 
 #define preempt_check_resched() do { } while (0)
@@ -291,14 +305,14 @@ do { \
 
 #define preempt_disable_notrace() \
 do { \
-	__preempt_count_inc(); \
+	preempt_count_inc_notrace(); \
 	barrier(); \
 } while (0)
 
 #define preempt_enable_no_resched_notrace() \
 do { \
 	barrier(); \
-	__preempt_count_dec(); \
+	preempt_count_dec_notrace(); \
 } while (0)
 
 #else /* !CONFIG_PREEMPT_COUNT */
diff --git a/kernel/sched/Makefile b/kernel/sched/Makefile
index b1f1a367034ff..d87f1bd3c28c3 100644
--- a/kernel/sched/Makefile
+++ b/kernel/sched/Makefile
@@ -27,6 +27,9 @@ endif
 
 # Branch profiling isn't noinstr-safe
 ifdef CONFIG_TRACE_BRANCH_PROFILING
+ifdef CONFIG_DEBUG_PREEMPT
+CFLAGS_core.o += -DDISABLE_BRANCH_PROFILING
+endif
 CFLAGS_build_policy.o += -DDISABLE_BRANCH_PROFILING
 CFLAGS_build_utility.o += -DDISABLE_BRANCH_PROFILING
 endif
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 1a088d4576f59..98ca93bc60887 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -6040,7 +6040,7 @@ static inline void preempt_latency_start(int val)
 	}
 }
 
-void preempt_count_add(int val)
+static __always_inline bool __preempt_count_add_checked(int val)
 {
 #ifdef CONFIG_DEBUG_PREEMPT
 	/*
@@ -6052,7 +6052,7 @@ void preempt_count_add(int val)
 	 */
 	if (!IS_ENABLED(CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS) &&
 	    DEBUG_LOCKS_WARN_ON((preempt_count() < 0)))
-		return;
+		return false;
 #endif
 	__preempt_count_add(val);
 #ifdef CONFIG_DEBUG_PREEMPT
@@ -6062,6 +6062,21 @@ void preempt_count_add(int val)
 	DEBUG_LOCKS_WARN_ON((preempt_count() & PREEMPT_MASK) >=
 				PREEMPT_MASK - 10);
 #endif
+	return true;
+}
+
+#ifdef CONFIG_DEBUG_PREEMPT
+void noinstr preempt_count_add_notrace(int val)
+{
+	__preempt_count_add_checked(val);
+}
+EXPORT_SYMBOL(preempt_count_add_notrace);
+#endif
+
+void preempt_count_add(int val)
+{
+	if (!__preempt_count_add_checked(val))
+		return;
 	preempt_latency_start(val);
 }
 EXPORT_SYMBOL(preempt_count_add);
@@ -6077,7 +6092,7 @@ static inline void preempt_latency_stop(int val)
 		trace_preempt_on(CALLER_ADDR0, get_lock_parent_ip());
 }
 
-void preempt_count_sub(int val)
+static __always_inline bool preempt_count_sub_debug_checks(int val)
 {
 #ifdef CONFIG_DEBUG_PREEMPT
 	/*
@@ -6087,15 +6102,32 @@ void preempt_count_sub(int val)
 	unsigned int pc = preempt_count();
 
 	if (DEBUG_LOCKS_WARN_ON(pc - uval > pc))
-		return;
+		return false;
 	/*
 	 * Is the spinlock portion underflowing?
 	 */
 	if (DEBUG_LOCKS_WARN_ON((val < PREEMPT_MASK) &&
 			!(preempt_count() & PREEMPT_MASK)))
+		return false;
+#endif
+	return true;
+}
+
+#ifdef CONFIG_DEBUG_PREEMPT
+void noinstr preempt_count_sub_notrace(int val)
+{
+	if (!preempt_count_sub_debug_checks(val))
 		return;
+	__preempt_count_sub(val);
+}
+EXPORT_SYMBOL(preempt_count_sub_notrace);
 #endif
 
+void preempt_count_sub(int val)
+{
+	if (!preempt_count_sub_debug_checks(val))
+		return;
+
 	preempt_latency_stop(val);
 	__preempt_count_sub(val);
 }

^ permalink raw reply related	[flat|nested] 5+ messages in thread

* Re: [PATCH 1/3] arm64: Always inline raw preempt count updates
  2026-11-16  7:54 ` [PATCH 1/3] arm64: Always inline raw preempt count updates Karl Mehltretter
@ 2026-11-16  8:16   ` sashiko-bot
  0 siblings, 0 replies; 5+ messages in thread
From: sashiko-bot @ 2026-11-16  8:16 UTC (permalink / raw)
  To: Karl Mehltretter; +Cc: linux-trace-kernel

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Low] arm64: preempt: missing Fixes trailer for commit 396244692232
--

commit 0e970117f0bbc0e074b2a0e827bc00ba6204a7cc
Author: Hongyan Xia <hongyan.xia@transsion.com>

> arm64: Always inline raw preempt count updates
>
> Raw preempt count updates can be emitted out of line by the compiler.
> For a noinstr caller, that puts the update in ordinary instrumentable
> text. Force __preempt_count_add() and __preempt_count_sub() inline so
> their code stays in the caller's section.
>
> Link: https://lore.kernel.org/all/c3a90368dd6c096af9d918e036a27728696fb8a4.1786603168.git.hongyan.xia@transsion.com/
> Signed-off-by: Hongyan Xia <hongyan.xia@transsion.com>
> [kmehltretter: split out the arm64 preempt.h hunk and rewrite the changelog]
> Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>

[Severity: Low]
__preempt_count_add() and __preempt_count_sub() were defined as plain
static inline when arm64 stopped using asm-generic/preempt.h.  Should this
include:

  Fixes: 396244692232 ("arm64: preempt: Provide our own implementation of asm/preempt.h")

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794814699.git.kmehltretter@gmail.com?part=1

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2026-11-16  8:16 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-16  7:54 [PATCH 0/3] sched/preempt: Check notrace preempt-count updates Karl Mehltretter
2026-11-16  7:54 ` [PATCH 1/3] arm64: Always inline raw preempt count updates Karl Mehltretter
2026-11-16  8:16   ` sashiko-bot
2026-11-16  7:54 ` [PATCH 2/3] arm64: Always inline preempt count and reschedule checks Karl Mehltretter
2026-11-16  7:54 ` [PATCH 3/3] sched/preempt: Check preempt counts in notrace helpers Karl Mehltretter

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox