All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 0/3] sched/preempt: Check notrace preempt-count updates
@ 2026-11-16  7:54 Karl Mehltretter
  2026-11-16  7:54 ` [PATCH 1/3] arm64: Always inline raw preempt count updates Karl Mehltretter
                   ` (2 more replies)
  0 siblings, 3 replies; 5+ messages in thread
From: Karl Mehltretter @ 2026-11-16  7:54 UTC (permalink / raw)
  To: Peter Zijlstra, Ingo Molnar, Catalin Marinas, Will Deacon
  Cc: Karl Mehltretter, Mark Rutland, Hongyan Xia, Song Liu,
	Steven Rostedt, Juri Lelli, Vincent Guittot, Dietmar Eggemann,
	Valentin Schneider, linux-arm-kernel, linux-trace-kernel,
	linux-kernel

With CONFIG_DEBUG_PREEMPT, the notrace preemption macros bypass the
bounds checks used by ordinary operations. Add checked noinstr updates
for their existing callers and share validation with
preempt_count_add() and preempt_count_sub().

Patch 1 forces the arm64 raw updates called by the new helpers inline.
Patch 2 does the same for preempt_count(), which the helpers use, for
should_resched(), reached by the debug notrace enable path, and for the
existing non-debug __preempt_count_dec_and_test() path. The corresponding
x86, s390 and asm-generic operations are already __always_inline.

Patch 1 is extracted from Hongyan Xia's entry RFC. Song Liu also posted
notrace annotations for the same arm64 helpers:

  https://lore.kernel.org/r/20261023011049.1164625-10-song@kernel.org

Patch 3 disables branch profiling for core.o when DEBUG_PREEMPT and
TRACE_BRANCH_PROFILING are both enabled. DEBUG_LOCKS_WARN_ON() evaluates
its condition before entering its instrumentation region, so profiling
that condition would call ftrace_likely_update() from noinstr text. The
scheduler Makefile already uses this per-object exclusion for
build_policy.o and build_utility.o.

The cost is loss of branch-profile data for the rest of core.o in that
configuration. Scheduler tracepoints and function tracing are unchanged.
A dedicated non-profiled object is the fallback if this object-wide cost
is considered too broad.

This series is independently useful. It also supplies the checked count
update planned for follow-up IRQ-entry work; it does not change IRQ entry,
trace-recursion lifetime or TRACE_CTX_TRANSITION.

This addresses Peter's suggestion here:

  https://lore.kernel.org/r/20261029093219.GH1779021@noisy.programming.kicks-ass.net

The series is based on v7.4-rc2.

Testing:

- Clang 19.1.7 x86-64: a W=1 defconfig-derived build with
  DEBUG_PREEMPT, annotated branch profiling, DEBUG_ENTRY and noinstr
  validation passed. A nested-KVM boot passed the built-in and external
  module checks and all five tracer workloads. A DEBUG_PREEMPT=n control
  retained branch profiling for core.o and emitted no new helper symbols.
- GCC 14.2 arm64: a W=1 PREEMPT_RT build and allmodconfig passed. The
  latter's nVHE object had no undefined references to the new helpers.
  A native Axion boot passed the built-in and module checks and all five
  tracers.
- GCC 14.2 and 16.2 ARC, C-SKY and MIPS: performance, size, all-branches
  and forced-noinline core.o inspections passed without prohibited
  noinstr relocations or branch-profile records. Full baseline and patched
  kernels plus the external debug module built. Matching QEMU boots reached
  the expected balanced and invalid-count markers on ARC and C-SKY UP and
  MIPS UP/SMP. The patched kernels diagnosed every invalid operation;
  rejected decrements retained their counts. An ARC DEBUG_PREEMPT=n
  control retained core.o profiling without references to the new helpers.

The MIPS UP baseline and patched positive boots emitted the same three
existing smp_processor_id() diagnostics; the SMP boots emitted none.

I used an LLM to help extract and review the changes, write the test
harnesses and prepare the commit messages.

Hongyan Xia (1):
  arm64: Always inline raw preempt count updates

Karl Mehltretter (2):
  arm64: Always inline preempt count and reschedule checks
  sched/preempt: Check preempt counts in notrace helpers

 arch/arm64/include/asm/preempt.h | 10 ++++----
 include/linux/preempt.h          | 22 ++++++++++++++----
 kernel/sched/Makefile            |  3 +++
 kernel/sched/core.c              | 40 ++++++++++++++++++++++++++++----
 4 files changed, 62 insertions(+), 13 deletions(-)


base-commit: 7475ea8b58616b33c04afb976cf0fbaad9dd63fb


^ permalink raw reply	[flat|nested] 5+ messages in thread

* [PATCH 1/3] arm64: Always inline raw preempt count updates
  2026-11-16  7:54 [PATCH 0/3] sched/preempt: Check notrace preempt-count updates Karl Mehltretter
@ 2026-11-16  7:54 ` Karl Mehltretter
  2026-11-16  8:16   ` sashiko-bot
  2026-11-16  7:54 ` [PATCH 2/3] arm64: Always inline preempt count and reschedule checks Karl Mehltretter
  2026-11-16  7:54 ` [PATCH 3/3] sched/preempt: Check preempt counts in notrace helpers Karl Mehltretter
  2 siblings, 1 reply; 5+ messages in thread
From: Karl Mehltretter @ 2026-11-16  7:54 UTC (permalink / raw)
  To: Peter Zijlstra, Ingo Molnar, Catalin Marinas, Will Deacon
  Cc: Hongyan Xia, Mark Rutland, Song Liu, Steven Rostedt, Juri Lelli,
	Vincent Guittot, Dietmar Eggemann, Valentin Schneider,
	linux-arm-kernel, linux-trace-kernel, linux-kernel,
	Karl Mehltretter

From: Hongyan Xia <hongyan.xia@transsion.com>

Raw preempt count updates can be emitted out of line by the compiler.
For a noinstr caller, that puts the update in ordinary instrumentable
text. Force __preempt_count_add() and __preempt_count_sub() inline so
their code stays in the caller's section.

Link: https://lore.kernel.org/all/c3a90368dd6c096af9d918e036a27728696fb8a4.1786603168.git.hongyan.xia@transsion.com/
Signed-off-by: Hongyan Xia <hongyan.xia@transsion.com>
[kmehltretter: split out the arm64 preempt.h hunk and rewrite the changelog]
Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>
---
 arch/arm64/include/asm/preempt.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/include/asm/preempt.h b/arch/arm64/include/asm/preempt.h
index 75a592b7f47f7..f3a5c2cd1936d 100644
--- a/arch/arm64/include/asm/preempt.h
+++ b/arch/arm64/include/asm/preempt.h
@@ -41,14 +41,14 @@ static inline bool test_preempt_need_resched(void)
 	return !current_thread_info()->preempt.need_resched;
 }
 
-static inline void __preempt_count_add(int val)
+static __always_inline void __preempt_count_add(int val)
 {
 	u32 pc = READ_ONCE(current_thread_info()->preempt.count);
 	pc += val;
 	WRITE_ONCE(current_thread_info()->preempt.count, pc);
 }
 
-static inline void __preempt_count_sub(int val)
+static __always_inline void __preempt_count_sub(int val)
 {
 	u32 pc = READ_ONCE(current_thread_info()->preempt.count);
 	pc -= val;


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH 2/3] arm64: Always inline preempt count and reschedule checks
  2026-11-16  7:54 [PATCH 0/3] sched/preempt: Check notrace preempt-count updates Karl Mehltretter
  2026-11-16  7:54 ` [PATCH 1/3] arm64: Always inline raw preempt count updates Karl Mehltretter
@ 2026-11-16  7:54 ` Karl Mehltretter
  2026-11-16  7:54 ` [PATCH 3/3] sched/preempt: Check preempt counts in notrace helpers Karl Mehltretter
  2 siblings, 0 replies; 5+ messages in thread
From: Karl Mehltretter @ 2026-11-16  7:54 UTC (permalink / raw)
  To: Peter Zijlstra, Ingo Molnar, Catalin Marinas, Will Deacon
  Cc: Karl Mehltretter, Mark Rutland, Hongyan Xia, Song Liu,
	Steven Rostedt, Juri Lelli, Vincent Guittot, Dietmar Eggemann,
	Valentin Schneider, linux-arm-kernel, linux-trace-kernel,
	linux-kernel

The preempt count and reschedule checks are plain inline functions,
which permits out-of-line copies in instrumentable text. The checked
notrace helpers added by the following patch use preempt_count(), while
preempt_enable_notrace() uses should_resched() with
CONFIG_DEBUG_PREEMPT and __preempt_count_dec_and_test() without it.

Force all three operations inline so noinstr callers keep them in their
own section.

Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>
---
 arch/arm64/include/asm/preempt.h | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/include/asm/preempt.h b/arch/arm64/include/asm/preempt.h
index f3a5c2cd1936d..805f7b33baf09 100644
--- a/arch/arm64/include/asm/preempt.h
+++ b/arch/arm64/include/asm/preempt.h
@@ -7,7 +7,7 @@
 #define PREEMPT_NEED_RESCHED	BIT(32)
 #define PREEMPT_ENABLED	(PREEMPT_NEED_RESCHED)
 
-static inline int preempt_count(void)
+static __always_inline int preempt_count(void)
 {
 	return READ_ONCE(current_thread_info()->preempt.count);
 }
@@ -75,13 +75,13 @@ static inline int __preempt_count_sub_return(int val)
 	return pc;
 }
 
-static inline bool should_resched(int preempt_offset)
+static __always_inline bool should_resched(int preempt_offset)
 {
 	u64 pc = READ_ONCE(current_thread_info()->preempt_count);
 	return unlikely(pc == preempt_offset);
 }
 
-static inline bool __preempt_count_dec_and_test(void)
+static __always_inline bool __preempt_count_dec_and_test(void)
 {
 	/*
 	 * We must load the combined 'preempt_count' after decrementing


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH 3/3] sched/preempt: Check preempt counts in notrace helpers
  2026-11-16  7:54 [PATCH 0/3] sched/preempt: Check notrace preempt-count updates Karl Mehltretter
  2026-11-16  7:54 ` [PATCH 1/3] arm64: Always inline raw preempt count updates Karl Mehltretter
  2026-11-16  7:54 ` [PATCH 2/3] arm64: Always inline preempt count and reschedule checks Karl Mehltretter
@ 2026-11-16  7:54 ` Karl Mehltretter
  2 siblings, 0 replies; 5+ messages in thread
From: Karl Mehltretter @ 2026-11-16  7:54 UTC (permalink / raw)
  To: Peter Zijlstra, Ingo Molnar, Catalin Marinas, Will Deacon
  Cc: Karl Mehltretter, Mark Rutland, Hongyan Xia, Song Liu,
	Steven Rostedt, Juri Lelli, Vincent Guittot, Dietmar Eggemann,
	Valentin Schneider, linux-arm-kernel, linux-trace-kernel,
	linux-kernel

The notrace preemption macros use raw count updates even with
CONFIG_DEBUG_PREEMPT enabled. They therefore skip the bounds checks
which ordinary preemption operations provide.

Add checked noinstr preempt-count helpers and use them in
preempt_disable_notrace(), preempt_enable_notrace() and
preempt_enable_no_resched_notrace(). Export the out-of-line helpers
for module callers. Keep raw updates when CONFIG_DEBUG_PREEMPT is
disabled.

Share the validation with preempt_count_add() and preempt_count_sub(),
while leaving their preemption-latency tracing and ordering unchanged.
DEBUG_LOCKS_WARN_ON() brackets its diagnostic work with
instrumentation_begin() and instrumentation_end(), but evaluates its
condition before that region. Branch profiling would therefore call
ftrace_likely_update() from noinstr text.

Compile core.o with branch profiling disabled when DEBUG_PREEMPT and
TRACE_BRANCH_PROFILING are both enabled, as the scheduler Makefile already
does for build_policy.o and build_utility.o. This keeps valid notrace count
updates free of profiler calls while retaining the existing warning path.
The tradeoff is loss of branch-profile data for the rest of core.o in that
configuration; scheduler tracepoints and function tracing are unchanged.

Suggested-by: Peter Zijlstra <peterz@infradead.org>
Link: https://lore.kernel.org/r/20261029093219.GH1779021@noisy.programming.kicks-ass.net
Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>
---
 include/linux/preempt.h | 22 ++++++++++++++++++----
 kernel/sched/Makefile   |  3 +++
 kernel/sched/core.c     | 40 ++++++++++++++++++++++++++++++++++++----
 3 files changed, 57 insertions(+), 8 deletions(-)

diff --git a/include/linux/preempt.h b/include/linux/preempt.h
index 06ff44a4b8b6a..42c585fd91d45 100644
--- a/include/linux/preempt.h
+++ b/include/linux/preempt.h
@@ -217,6 +217,17 @@ static __always_inline unsigned char interrupt_context_level(void)
  */
 #define in_atomic_preempt_off() (preempt_count() != PREEMPT_DISABLE_OFFSET)
 
+#ifdef CONFIG_DEBUG_PREEMPT
+void preempt_count_add_notrace(int val);
+void preempt_count_sub_notrace(int val);
+#define preempt_count_dec_and_test_notrace() \
+	({ preempt_count_sub_notrace(1); should_resched(0); })
+#else
+#define preempt_count_add_notrace(val)	__preempt_count_add(val)
+#define preempt_count_sub_notrace(val)	__preempt_count_sub(val)
+#define preempt_count_dec_and_test_notrace() __preempt_count_dec_and_test()
+#endif
+
 #if defined(CONFIG_DEBUG_PREEMPT) || defined(CONFIG_TRACE_PREEMPT_TOGGLE)
 extern void preempt_count_add(int val);
 extern void preempt_count_sub(int val);
@@ -231,6 +242,9 @@ extern void preempt_count_sub(int val);
 #define __preempt_count_inc() __preempt_count_add(1)
 #define __preempt_count_dec() __preempt_count_sub(1)
 
+#define preempt_count_inc_notrace() preempt_count_add_notrace(1)
+#define preempt_count_dec_notrace() preempt_count_sub_notrace(1)
+
 #define preempt_count_inc() preempt_count_add(1)
 #define preempt_count_dec() preempt_count_sub(1)
 
@@ -263,7 +277,7 @@ do { \
 #define preempt_enable_notrace() \
 do { \
 	barrier(); \
-	if (unlikely(__preempt_count_dec_and_test())) \
+	if (unlikely(preempt_count_dec_and_test_notrace())) \
 		__preempt_schedule_notrace(); \
 } while (0)
 
@@ -283,7 +297,7 @@ do { \
 #define preempt_enable_notrace() \
 do { \
 	barrier(); \
-	__preempt_count_dec(); \
+	preempt_count_dec_notrace(); \
 } while (0)
 
 #define preempt_check_resched() do { } while (0)
@@ -291,14 +305,14 @@ do { \
 
 #define preempt_disable_notrace() \
 do { \
-	__preempt_count_inc(); \
+	preempt_count_inc_notrace(); \
 	barrier(); \
 } while (0)
 
 #define preempt_enable_no_resched_notrace() \
 do { \
 	barrier(); \
-	__preempt_count_dec(); \
+	preempt_count_dec_notrace(); \
 } while (0)
 
 #else /* !CONFIG_PREEMPT_COUNT */
diff --git a/kernel/sched/Makefile b/kernel/sched/Makefile
index b1f1a367034ff..d87f1bd3c28c3 100644
--- a/kernel/sched/Makefile
+++ b/kernel/sched/Makefile
@@ -27,6 +27,9 @@ endif
 
 # Branch profiling isn't noinstr-safe
 ifdef CONFIG_TRACE_BRANCH_PROFILING
+ifdef CONFIG_DEBUG_PREEMPT
+CFLAGS_core.o += -DDISABLE_BRANCH_PROFILING
+endif
 CFLAGS_build_policy.o += -DDISABLE_BRANCH_PROFILING
 CFLAGS_build_utility.o += -DDISABLE_BRANCH_PROFILING
 endif
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 1a088d4576f59..98ca93bc60887 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -6040,7 +6040,7 @@ static inline void preempt_latency_start(int val)
 	}
 }
 
-void preempt_count_add(int val)
+static __always_inline bool __preempt_count_add_checked(int val)
 {
 #ifdef CONFIG_DEBUG_PREEMPT
 	/*
@@ -6052,7 +6052,7 @@ void preempt_count_add(int val)
 	 */
 	if (!IS_ENABLED(CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS) &&
 	    DEBUG_LOCKS_WARN_ON((preempt_count() < 0)))
-		return;
+		return false;
 #endif
 	__preempt_count_add(val);
 #ifdef CONFIG_DEBUG_PREEMPT
@@ -6062,6 +6062,21 @@ void preempt_count_add(int val)
 	DEBUG_LOCKS_WARN_ON((preempt_count() & PREEMPT_MASK) >=
 				PREEMPT_MASK - 10);
 #endif
+	return true;
+}
+
+#ifdef CONFIG_DEBUG_PREEMPT
+void noinstr preempt_count_add_notrace(int val)
+{
+	__preempt_count_add_checked(val);
+}
+EXPORT_SYMBOL(preempt_count_add_notrace);
+#endif
+
+void preempt_count_add(int val)
+{
+	if (!__preempt_count_add_checked(val))
+		return;
 	preempt_latency_start(val);
 }
 EXPORT_SYMBOL(preempt_count_add);
@@ -6077,7 +6092,7 @@ static inline void preempt_latency_stop(int val)
 		trace_preempt_on(CALLER_ADDR0, get_lock_parent_ip());
 }
 
-void preempt_count_sub(int val)
+static __always_inline bool preempt_count_sub_debug_checks(int val)
 {
 #ifdef CONFIG_DEBUG_PREEMPT
 	/*
@@ -6087,15 +6102,32 @@ void preempt_count_sub(int val)
 	unsigned int pc = preempt_count();
 
 	if (DEBUG_LOCKS_WARN_ON(pc - uval > pc))
-		return;
+		return false;
 	/*
 	 * Is the spinlock portion underflowing?
 	 */
 	if (DEBUG_LOCKS_WARN_ON((val < PREEMPT_MASK) &&
 			!(preempt_count() & PREEMPT_MASK)))
+		return false;
+#endif
+	return true;
+}
+
+#ifdef CONFIG_DEBUG_PREEMPT
+void noinstr preempt_count_sub_notrace(int val)
+{
+	if (!preempt_count_sub_debug_checks(val))
 		return;
+	__preempt_count_sub(val);
+}
+EXPORT_SYMBOL(preempt_count_sub_notrace);
 #endif
 
+void preempt_count_sub(int val)
+{
+	if (!preempt_count_sub_debug_checks(val))
+		return;
+
 	preempt_latency_stop(val);
 	__preempt_count_sub(val);
 }


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* Re: [PATCH 1/3] arm64: Always inline raw preempt count updates
  2026-11-16  7:54 ` [PATCH 1/3] arm64: Always inline raw preempt count updates Karl Mehltretter
@ 2026-11-16  8:16   ` sashiko-bot
  0 siblings, 0 replies; 5+ messages in thread
From: sashiko-bot @ 2026-11-16  8:16 UTC (permalink / raw)
  To: Karl Mehltretter; +Cc: linux-trace-kernel

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Low] arm64: preempt: missing Fixes trailer for commit 396244692232
--

commit 0e970117f0bbc0e074b2a0e827bc00ba6204a7cc
Author: Hongyan Xia <hongyan.xia@transsion.com>

> arm64: Always inline raw preempt count updates
>
> Raw preempt count updates can be emitted out of line by the compiler.
> For a noinstr caller, that puts the update in ordinary instrumentable
> text. Force __preempt_count_add() and __preempt_count_sub() inline so
> their code stays in the caller's section.
>
> Link: https://lore.kernel.org/all/c3a90368dd6c096af9d918e036a27728696fb8a4.1786603168.git.hongyan.xia@transsion.com/
> Signed-off-by: Hongyan Xia <hongyan.xia@transsion.com>
> [kmehltretter: split out the arm64 preempt.h hunk and rewrite the changelog]
> Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>

[Severity: Low]
__preempt_count_add() and __preempt_count_sub() were defined as plain
static inline when arm64 stopped using asm-generic/preempt.h.  Should this
include:

  Fixes: 396244692232 ("arm64: preempt: Provide our own implementation of asm/preempt.h")

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1794814699.git.kmehltretter@gmail.com?part=1

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2026-11-16  8:16 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-16  7:54 [PATCH 0/3] sched/preempt: Check notrace preempt-count updates Karl Mehltretter
2026-11-16  7:54 ` [PATCH 1/3] arm64: Always inline raw preempt count updates Karl Mehltretter
2026-11-16  8:16   ` sashiko-bot
2026-11-16  7:54 ` [PATCH 2/3] arm64: Always inline preempt count and reschedule checks Karl Mehltretter
2026-11-16  7:54 ` [PATCH 3/3] sched/preempt: Check preempt counts in notrace helpers Karl Mehltretter

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.