* Re: [PATCH v12 05/15] arm64: rqspinlock: Remove private copy of smp_cond_load_acquire_timewait()
From: bot+bpf-ci @ 2026-06-08 8:53 UTC (permalink / raw)
To: ankur.a.arora, linux-kernel, linux-arch, linux-arm-kernel,
linux-pm, bpf
Cc: arnd, catalin.marinas, will, peterz, akpm, mark.rutland, harisokn,
cl, ast, rafael, daniel.lezcano, memxor, zhenglifeng1, xueshuai,
rdunlap, david.laight.linux, broonie, joao.m.martins,
boris.ostrovsky, konrad.wilk, ashok.bhat, ankur.a.arora, ast,
andrii, daniel, martin.lau, eddyz87, yonghong.song, clm,
ihor.solodrai
In-Reply-To: <20260608080440.127491-6-ankur.a.arora@oracle.com>
[-- Attachment #1: Type: text/plain, Size: 3202 bytes --]
> diff --git a/arch/arm64/include/asm/rqspinlock.h b/arch/arm64/include/asm/rqspinlock.h
> index 9ea0a74e5892..a385603436e9 100644
> --- a/arch/arm64/include/asm/rqspinlock.h
> +++ b/arch/arm64/include/asm/rqspinlock.h
> @@ -3,91 +3,6 @@
> #define _ASM_RQSPINLOCK_H
>
> #include <asm/barrier.h>
> -
> -/*
> - * Hardcode res_smp_cond_load_acquire implementations for arm64 to a custom
> - * version based on [0]. In rqspinlock code, our conditional expression involves
> - * checking the value _and_ additionally a timeout. However, on arm64, the
> - * WFE-based implementation may never spin again if no stores occur to the
> - * locked byte in the lock word. As such, we may be stuck forever if
> - * event-stream based unblocking is not available on the platform for WFE spin
> - * loops (arch_timer_evtstrm_available).
> - *
[ ... ]
> -#endif
> -
> -#define res_smp_cond_load_acquire(v, c) smp_cond_load_acquire_timewait(v, c, 0, 1)
> -
> #include <asm-generic/rqspinlock.h>
With res_smp_cond_load_acquire removed here, does arm64 now fall back to
the generic definition in kernel/bpf/rqspinlock.c?
#ifndef res_smp_cond_load_acquire
#define res_smp_cond_load_acquire(v, c) smp_cond_load_acquire(v, c)
#endif
On arm64, smp_cond_load_acquire() uses __cmpwait_relaxed(), which issues a
WFE that wakes only on a store to the monitored cache line or on an
event-stream tick.
The consumer is resilient_queued_spin_lock_slowpath():
kernel/bpf/rqspinlock.c:
res_smp_cond_load_acquire(&lock->locked,
!VAL || RES_CHECK_TIMEOUT(ts, ret, _Q_LOCKED_MASK));
The RES_CHECK_TIMEOUT() escape is part of cond_expr, so it is only
re-evaluated when smp_cond_load_acquire() returns from its WFE.
On platforms where arch_timer_evtstrm_available() is false, if the current
lock owner is stuck or dead and issues no further store to lock->locked,
can the WFE ever wake, and can the RES_CHECK_TIMEOUT timeout ever fire?
This looks like the exact case the removed comment described:
> - * checking the value _and_ additionally a timeout. However, on arm64, the
> - * WFE-based implementation may never spin again if no stores occur to the
> - * locked byte in the lock word. As such, we may be stuck forever if
> - * event-stream based unblocking is not available on the platform for WFE spin
> - * loops (arch_timer_evtstrm_available).
The removed private copy avoided this by falling back to a cpu_relax()-based
spinwait that always re-checks the timeout when the event stream is
unavailable.
Forward search of the series shows this is resolved by the later commit
7abb03c21db2 ("bpf/rqspinlock: Use smp_cond_load_acquire_timeout()"), which
switches rqspinlock to smp_cond_load_acquire_timeout(); on arm64 that uses
a waiting implementation handling the timeout and event-stream-absent case.
Should the change-over happen in the same commit, or is the window where a
kernel built or bisected at this commit can block indefinitely acceptable?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/27125050324
^ permalink raw reply
* Re: [PATCH v12 01/15] asm-generic: barrier: Add smp_cond_load_relaxed_timeout()
From: bot+bpf-ci @ 2026-06-08 8:53 UTC (permalink / raw)
To: ankur.a.arora, linux-kernel, linux-arch, linux-arm-kernel,
linux-pm, bpf
Cc: arnd, catalin.marinas, will, peterz, akpm, mark.rutland, harisokn,
cl, ast, rafael, daniel.lezcano, memxor, zhenglifeng1, xueshuai,
rdunlap, david.laight.linux, broonie, joao.m.martins,
boris.ostrovsky, konrad.wilk, ashok.bhat, ankur.a.arora, ast,
andrii, daniel, martin.lau, eddyz87, yonghong.song, clm,
ihor.solodrai
In-Reply-To: <20260608080440.127491-2-ankur.a.arora@oracle.com>
[-- Attachment #1: Type: text/plain, Size: 2681 bytes --]
> diff --git a/include/asm-generic/barrier.h b/include/asm-generic/barrier.h
> index d4f581c1e21d..c56df9513a08 100644
> --- a/include/asm-generic/barrier.h
> +++ b/include/asm-generic/barrier.h
> @@ -273,6 +273,75 @@ do { \
[ ... ]
> +#ifndef smp_cond_load_relaxed_timeout
> +#define smp_cond_load_relaxed_timeout(ptr, cond_expr, \
> + time_expr_ns, timeout_ns) \
> +({ \
> + typeof(ptr) __PTR = (ptr); \
> + __unqual_scalar_typeof(*(ptr)) VAL; \
> + u32 __count = 0, __spin = SMP_TIMEOUT_POLL_COUNT; \
> + s64 __timeout = (s64)timeout_ns; \
> + s64 __time_now, __time_end = 0; \
> + \
> + for (;;) { \
> + VAL = READ_ONCE(*__PTR); \
> + if (cond_expr) \
> + break; \
> + cpu_poll_relax(__PTR, VAL, (u64)__timeout); \
__timeout is held as a signed s64 but is cast to u64 here. On the first
iterations __timeout still holds the raw timeout_ns argument before any
time-check has run. If a caller passes a negative or already-expired
timeout_ns, the (u64) cast turns it into a value close to U64_MAX.
This is harmless while cpu_poll_relax() ignores its timeout argument and
falls back to cpu_relax(). Would it misbehave on an architecture whose
cpu_poll_relax() consumes the value, such as an arm64 WFET variant that
uses it as an event-stream deadline?
David Laight raised this on v11
(https://lore.kernel.org/lkml/20260508093206.389d9af2@pumpkin/):
Yes, the code is ok if cpu_poll_relax() ignores the timeout. But for
WFE it is all broken. ... if you assume/require that the converted
time is either 0 or greater than 200 then it can be used as the
initialiser for __n.
The Sashiko bot repeated the same concern as a medium finding on v12.
The v12 cover note opts to document the valid range rather than clamp at
runtime, but the code still passes (u64)__timeout unguarded, so would it
be worth converting timeout_ns into an absolute end-time (or 0 when WFE
is unused) before it reaches cpu_poll_relax()?
> + if (++__count < __spin) \
> + continue; \
> + __time_now = (s64)(time_expr_ns); \
> + if (unlikely(__time_end == 0)) \
> + __time_end = __time_now + __timeout; \
> + __timeout = __time_end - __time_now; \
> + if (__time_now <= 0 || __timeout <= 0) { \
> + VAL = READ_ONCE(*__PTR); \
> + break; \
> + } \
> + __count = 0; \
> + } \
> + (typeof(*(ptr)))VAL; \
> +})
> +#endif
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/27125050324
^ permalink raw reply
* [PATCH v12 12/15] cpuidle/poll_state: Wait for need-resched via tif_need_resched_relaxed_wait()
From: Ankur Arora @ 2026-06-08 8:04 UTC (permalink / raw)
To: linux-kernel, linux-arch, linux-arm-kernel, linux-pm, bpf
Cc: arnd, catalin.marinas, will, peterz, akpm, mark.rutland, harisokn,
cl, ast, rafael, daniel.lezcano, memxor, zhenglifeng1, xueshuai,
rdunlap, david.laight.linux, broonie, joao.m.martins,
boris.ostrovsky, konrad.wilk, ashok.bhat, Ankur Arora
In-Reply-To: <20260608080440.127491-1-ankur.a.arora@oracle.com>
The inner loop in poll_idle() polls over the thread_info flags,
waiting to see if the thread has TIF_NEED_RESCHED set. The loop
exits once the condition is met, or if the poll time limit has
been exceeded.
To minimize the number of instructions executed in each iteration,
the time check is rate-limited. In addition, each loop iteration
executes cpu_relax() which on certain platforms provides a hint to
the pipeline that the loop busy-waits, allowing the processor to
reduce power consumption.
Switch over to tif_need_resched_relaxed_wait() instead, since that
provides exactly that.
However, since we want to minimize power consumption in idle, building
of cpuidle/poll_state.c continues to depend on CONFIG_ARCH_HAS_CPU_RELAX
as that serves as an indicator that the platform supports an optimized
version of tif_need_resched_relaxed_wait() (via
smp_cond_load_acquire_timeout()).
Cc: Rafael J. Wysocki <rafael@kernel.org>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: linux-pm@vger.kernel.org
Suggested-by: Rafael J. Wysocki <rafael@kernel.org>
Acked-by: Rafael J. Wysocki (Intel) <rafael@kernel.org>
Tested-by: Haris Okanovic <harisokn@amazon.com>
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
Notes:
Sashiko notes [1] that lazy initialization of the timeout deadline will
cause an overshoot of the wakeup deadline: this was discussed earlier
and shouldn't be a big concern [2]. Cpuidle ranges aren't meant to be
precise and in any case we are only waiting to go into a deeper idle
state.
[1] https://sashiko.dev/#/patchset/20260408122538.3610871-1-ankur.a.arora%40oracle.com
[2] https://lore.kernel.org/lkml/CAJZ5v0izSBR0_DeH5HVnSLFGRfV9WoSzbu9Mh5yvvuyrvw7fLg@mail.gmail.com/
---
drivers/cpuidle/poll_state.c | 21 +--------------------
1 file changed, 1 insertion(+), 20 deletions(-)
diff --git a/drivers/cpuidle/poll_state.c b/drivers/cpuidle/poll_state.c
index c7524e4c522a..7443b3e971ba 100644
--- a/drivers/cpuidle/poll_state.c
+++ b/drivers/cpuidle/poll_state.c
@@ -6,41 +6,22 @@
#include <linux/cpuidle.h>
#include <linux/export.h>
#include <linux/irqflags.h>
-#include <linux/sched.h>
-#include <linux/sched/clock.h>
#include <linux/sched/idle.h>
#include <linux/sprintf.h>
#include <linux/types.h>
-#define POLL_IDLE_RELAX_COUNT 200
-
static int __cpuidle poll_idle(struct cpuidle_device *dev,
struct cpuidle_driver *drv, int index)
{
- u64 time_start;
-
- time_start = local_clock_noinstr();
-
dev->poll_time_limit = false;
raw_local_irq_enable();
if (!current_set_polling_and_test()) {
- unsigned int loop_count = 0;
u64 limit;
limit = cpuidle_poll_time(drv, dev);
- while (!need_resched()) {
- cpu_relax();
- if (loop_count++ < POLL_IDLE_RELAX_COUNT)
- continue;
-
- loop_count = 0;
- if (local_clock_noinstr() - time_start > limit) {
- dev->poll_time_limit = true;
- break;
- }
- }
+ dev->poll_time_limit = !tif_need_resched_relaxed_wait(limit);
}
raw_local_irq_disable();
--
2.31.1
^ permalink raw reply related
* [PATCH v12 03/15] arm64/delay: move some constants out to a separate header
From: Ankur Arora @ 2026-06-08 8:04 UTC (permalink / raw)
To: linux-kernel, linux-arch, linux-arm-kernel, linux-pm, bpf
Cc: arnd, catalin.marinas, will, peterz, akpm, mark.rutland, harisokn,
cl, ast, rafael, daniel.lezcano, memxor, zhenglifeng1, xueshuai,
rdunlap, david.laight.linux, broonie, joao.m.martins,
boris.ostrovsky, konrad.wilk, ashok.bhat, Ankur Arora,
Bjorn Andersson, Konrad Dybcio, Christoph Lameter
In-Reply-To: <20260608080440.127491-1-ankur.a.arora@oracle.com>
Moves some constants and functions related to xloops, cycles computation
out to a new header. Also make __delay_cycles() available outside of
arch/arm64/lib/delay.c.
Rename some macros in qcom/rpmh-rsc.c which were occupying the same
namespace.
No functional change.
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Bjorn Andersson <andersson@kernel.org>
Cc: Konrad Dybcio <konradybcio@kernel.org>
Cc: linux-arm-kernel@lists.infradead.org
Reviewed-by: Christoph Lameter <cl@linux.com>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
Notes:
- include <linux/types.h> (flagged by sashiko)
---
arch/arm64/include/asm/delay-const.h | 28 ++++++++++++++++++++++++++++
arch/arm64/lib/delay.c | 15 ++++-----------
drivers/soc/qcom/rpmh-rsc.c | 8 ++++----
3 files changed, 36 insertions(+), 15 deletions(-)
create mode 100644 arch/arm64/include/asm/delay-const.h
diff --git a/arch/arm64/include/asm/delay-const.h b/arch/arm64/include/asm/delay-const.h
new file mode 100644
index 000000000000..2a5acfb7bff1
--- /dev/null
+++ b/arch/arm64/include/asm/delay-const.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef _ASM_DELAY_CONST_H
+#define _ASM_DELAY_CONST_H
+
+#include <linux/types.h>
+#include <asm/param.h> /* For HZ */
+
+/* 2**32 / 1000000 (rounded up) */
+#define __usecs_to_xloops_mult 0x10C7UL
+
+/* 2**32 / 1000000000 (rounded up) */
+#define __nsecs_to_xloops_mult 0x5UL
+
+extern unsigned long loops_per_jiffy;
+static inline unsigned long xloops_to_cycles(unsigned long xloops)
+{
+ return (xloops * loops_per_jiffy * HZ) >> 32;
+}
+
+#define USECS_TO_CYCLES(time_usecs) \
+ xloops_to_cycles((time_usecs) * __usecs_to_xloops_mult)
+
+#define NSECS_TO_CYCLES(time_nsecs) \
+ xloops_to_cycles((time_nsecs) * __nsecs_to_xloops_mult)
+
+u64 notrace __delay_cycles(void);
+
+#endif /* _ASM_DELAY_CONST_H */
diff --git a/arch/arm64/lib/delay.c b/arch/arm64/lib/delay.c
index e278e060e78a..c660a7ea26dd 100644
--- a/arch/arm64/lib/delay.c
+++ b/arch/arm64/lib/delay.c
@@ -12,17 +12,10 @@
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/timex.h>
+#include <asm/delay-const.h>
#include <clocksource/arm_arch_timer.h>
-#define USECS_TO_CYCLES(time_usecs) \
- xloops_to_cycles((time_usecs) * 0x10C7UL)
-
-static inline unsigned long xloops_to_cycles(unsigned long xloops)
-{
- return (xloops * loops_per_jiffy * HZ) >> 32;
-}
-
/*
* Force the use of CNTVCT_EL0 in order to have the same base as WFxT.
* This avoids some annoying issues when CNTVOFF_EL2 is not reset 0 on a
@@ -32,7 +25,7 @@ static inline unsigned long xloops_to_cycles(unsigned long xloops)
* Note that userspace cannot change the offset behind our back either,
* as the vcpu mutex is held as long as KVM_RUN is in progress.
*/
-static cycles_t notrace __delay_cycles(void)
+u64 notrace __delay_cycles(void)
{
guard(preempt_notrace)();
return __arch_counter_get_cntvct_stable();
@@ -73,12 +66,12 @@ EXPORT_SYMBOL(__const_udelay);
void __udelay(unsigned long usecs)
{
- __const_udelay(usecs * 0x10C7UL); /* 2**32 / 1000000 (rounded up) */
+ __const_udelay(usecs * __usecs_to_xloops_mult);
}
EXPORT_SYMBOL(__udelay);
void __ndelay(unsigned long nsecs)
{
- __const_udelay(nsecs * 0x5UL); /* 2**32 / 1000000000 (rounded up) */
+ __const_udelay(nsecs * __nsecs_to_xloops_mult);
}
EXPORT_SYMBOL(__ndelay);
diff --git a/drivers/soc/qcom/rpmh-rsc.c b/drivers/soc/qcom/rpmh-rsc.c
index c6f7d5c9c493..ad5ec5c0de0a 100644
--- a/drivers/soc/qcom/rpmh-rsc.c
+++ b/drivers/soc/qcom/rpmh-rsc.c
@@ -146,10 +146,10 @@ enum {
* +---------------------------------------------------+
*/
-#define USECS_TO_CYCLES(time_usecs) \
- xloops_to_cycles((time_usecs) * 0x10C7UL)
+#define RPMH_USECS_TO_CYCLES(time_usecs) \
+ rpmh_xloops_to_cycles((time_usecs) * 0x10C7UL)
-static inline unsigned long xloops_to_cycles(u64 xloops)
+static inline unsigned long rpmh_xloops_to_cycles(u64 xloops)
{
return (xloops * loops_per_jiffy * HZ) >> 32;
}
@@ -819,7 +819,7 @@ void rpmh_rsc_write_next_wakeup(struct rsc_drv *drv)
wakeup_us = ktime_to_us(wakeup);
/* Convert the wakeup to arch timer scale */
- wakeup_cycles = USECS_TO_CYCLES(wakeup_us);
+ wakeup_cycles = RPMH_USECS_TO_CYCLES(wakeup_us);
wakeup_cycles += arch_timer_read_counter();
exit:
--
2.31.1
^ permalink raw reply related
* [PATCH v12 15/15] barrier: add clock tests for smp_cond_load_relaxed_timeout()
From: Ankur Arora @ 2026-06-08 8:04 UTC (permalink / raw)
To: linux-kernel, linux-arch, linux-arm-kernel, linux-pm, bpf
Cc: arnd, catalin.marinas, will, peterz, akpm, mark.rutland, harisokn,
cl, ast, rafael, daniel.lezcano, memxor, zhenglifeng1, xueshuai,
rdunlap, david.laight.linux, broonie, joao.m.martins,
boris.ostrovsky, konrad.wilk, ashok.bhat, Ankur Arora
In-Reply-To: <20260608080440.127491-1-ankur.a.arora@oracle.com>
Add a few clock tests for smp_cond_load_relaxed_timeout(). These
ensure that the implementation doesn't do anything funny stuff with the
clock (like multiple accesses per iteration.)
Also ensure that we handle edge cases sanely. Note that two edge cases
fail: S64_MAX and U64_MAX. However, both of those are quite far out
and if needed, can be addressed in the implementation of the interface.
Also, this tests only smp_cond_load_relaxed_timeout(). The acquire
variant uses an identical clock path and testing wouldn't add anything.
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
lib/tests/barrier-timeout-test.c | 57 ++++++++++++++++++++++++++++++++
1 file changed, 57 insertions(+)
diff --git a/lib/tests/barrier-timeout-test.c b/lib/tests/barrier-timeout-test.c
index 2160844b27b8..ec9dc0aa65d1 100644
--- a/lib/tests/barrier-timeout-test.c
+++ b/lib/tests/barrier-timeout-test.c
@@ -19,6 +19,8 @@ MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING");
struct clock_state {
s64 start_time;
s64 end_time;
+ s64 extra;
+ u32 niters;
};
#define TIMEOUT_MSEC 2
@@ -112,8 +114,63 @@ static void test_smp_cond_timeout(struct kunit *test)
KUNIT_EXPECT_GE(test, runtime, timeout_ns);
}
+static s64 synthetic_clock(struct clock_state *clk)
+{
+ clk->end_time += clk->extra;
+ clk->niters++;
+
+ return clk->end_time;
+}
+
+
+struct smp_cond_expiry_params {
+ char *desc;
+ s64 timeout;
+ s64 clk_inc;
+ u32 niters;
+};
+
+static const struct smp_cond_expiry_params expiry_params_list[] = {
+ { .clk_inc = (0x1ULL << 28), .timeout = -1LL, .niters = 1, .desc = "-1LL", },
+ { .clk_inc = (0x1ULL << 28), .timeout = (0x1ULL << 30), .niters = 1 + (1 << (30-28)), .desc = "1<<30", },
+ { .clk_inc = (0x1ULL << 28), .timeout = S32_MAX, .niters = 1 + (1 << (31-28)), .desc = "S32_MAX", },
+ { .clk_inc = (0x1ULL << 28), .timeout = U32_MAX, .niters = 1 + (1 << (32-28)), .desc = "U32_MAX", },
+ { .clk_inc = (0x1ULL << 28), .timeout = (0x1ULL << 33), .niters = 1 + (1 << (33-28)), .desc = "1<<33", },
+};
+
+static void expiry_param_to_desc(const struct smp_cond_expiry_params *p, char *desc)
+{
+ snprintf(desc, KUNIT_PARAM_DESC_SIZE, "smp_cond_%s_timeout: clock-%s, timeout=%s, iterations=%u",
+ "relaxed", "synthetic", p->desc, p->niters);
+}
+
+static void test_smp_cond_expiry(struct kunit *test)
+{
+ const struct smp_cond_expiry_params *p = test->param_value;
+ struct clock_state clk = {
+ .start_time = 0,
+ .end_time = 0,
+ .extra = p->clk_inc,
+ .niters = 0,
+ };
+ s64 runtime;
+
+ flag = 0;
+ smp_cond_load_relaxed_timeout(&flag,
+ 0,
+ synthetic_clock(&clk),
+ p->timeout);
+
+ runtime = (u64)clk.end_time - (u64)clk.start_time;
+ KUNIT_EXPECT_EQ(test, clk.niters, p->niters);
+ KUNIT_EXPECT_GE(test, runtime, p->timeout);
+}
+
+
+KUNIT_ARRAY_PARAM(smp_cond_expiry_params, expiry_params_list, expiry_param_to_desc);
static struct kunit_case barrier_timeout_test_cases[] = {
KUNIT_CASE_PARAM(test_smp_cond_timeout, smp_cond_update_params_gen_params),
+ KUNIT_CASE_PARAM(test_smp_cond_expiry, smp_cond_expiry_params_gen_params),
{}
};
--
2.31.1
^ permalink raw reply related
* [PATCH v12 09/15] bpf/rqspinlock: switch check_timeout() to a clock interface
From: Ankur Arora @ 2026-06-08 8:04 UTC (permalink / raw)
To: linux-kernel, linux-arch, linux-arm-kernel, linux-pm, bpf
Cc: arnd, catalin.marinas, will, peterz, akpm, mark.rutland, harisokn,
cl, ast, rafael, daniel.lezcano, memxor, zhenglifeng1, xueshuai,
rdunlap, david.laight.linux, broonie, joao.m.martins,
boris.ostrovsky, konrad.wilk, ashok.bhat, Ankur Arora
In-Reply-To: <20260608080440.127491-1-ankur.a.arora@oracle.com>
check_timeout() gets the current time value and depending on how
much time has passed, checks for deadlock or times out, returning 0
or -errno on deadlock or timeout.
Switch this out to a clock style interface, where it functions as a
clock in the "lock-domain", returning the current time until a
deadlock or timeout occurs. Once a deadlock or timeout has occurred,
it stops functioning as a clock and returns error.
Also adjust the RES_CHECK_TIMEOUT macro to discard the clock value
when updating the explicit return status.
Cc: bpf@vger.kernel.org
Cc: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Acked-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
kernel/bpf/rqspinlock.c | 45 +++++++++++++++++++++++++++--------------
1 file changed, 30 insertions(+), 15 deletions(-)
diff --git a/kernel/bpf/rqspinlock.c b/kernel/bpf/rqspinlock.c
index e4e338cdb437..0ec17ebb67c1 100644
--- a/kernel/bpf/rqspinlock.c
+++ b/kernel/bpf/rqspinlock.c
@@ -196,8 +196,12 @@ static noinline int check_deadlock_ABBA(rqspinlock_t *lock, u32 mask)
return 0;
}
-static noinline int check_timeout(rqspinlock_t *lock, u32 mask,
- struct rqspinlock_timeout *ts)
+/*
+ * Returns current monotonic time in ns on success or, negative errno
+ * value on failure due to timeout expiration or detection of deadlock.
+ */
+static noinline s64 clock_deadlock(rqspinlock_t *lock, u32 mask,
+ struct rqspinlock_timeout *ts)
{
u64 prev = ts->cur;
u64 time;
@@ -207,7 +211,7 @@ static noinline int check_timeout(rqspinlock_t *lock, u32 mask,
return -EDEADLK;
ts->cur = ktime_get_mono_fast_ns();
ts->timeout_end = ts->cur + ts->duration;
- return 0;
+ return (s64)ts->cur;
}
time = ktime_get_mono_fast_ns();
@@ -219,11 +223,15 @@ static noinline int check_timeout(rqspinlock_t *lock, u32 mask,
* checks.
*/
if (prev + NSEC_PER_MSEC < time) {
+ int ret;
ts->cur = time;
- return check_deadlock_ABBA(lock, mask);
+ ret = check_deadlock_ABBA(lock, mask);
+ if (ret)
+ return ret;
+
}
- return 0;
+ return (s64)time;
}
/*
@@ -231,15 +239,22 @@ static noinline int check_timeout(rqspinlock_t *lock, u32 mask,
* as the macro does internal amortization for us.
*/
#ifndef res_smp_cond_load_acquire
-#define RES_CHECK_TIMEOUT(ts, ret, mask) \
- ({ \
- if (!(ts).spin++) \
- (ret) = check_timeout((lock), (mask), &(ts)); \
- (ret); \
+#define RES_CHECK_TIMEOUT(ts, ret, mask) \
+ ({ \
+ s64 __timeval_err = 0; \
+ if (!(ts).spin++) \
+ __timeval_err = clock_deadlock((lock), (mask), &(ts)); \
+ (ret) = __timeval_err < 0 ? __timeval_err : 0; \
+ __timeval_err; \
})
#else
-#define RES_CHECK_TIMEOUT(ts, ret, mask) \
- ({ (ret) = check_timeout((lock), (mask), &(ts)); })
+#define RES_CHECK_TIMEOUT(ts, ret, mask) \
+ ({ \
+ s64 __timeval_err; \
+ __timeval_err = clock_deadlock((lock), (mask), &(ts)); \
+ (ret) = __timeval_err < 0 ? __timeval_err : 0; \
+ __timeval_err; \
+ })
#endif
/*
@@ -281,7 +296,7 @@ int __lockfunc resilient_tas_spin_lock(rqspinlock_t *lock)
val = atomic_read(&lock->val);
if (val || !atomic_try_cmpxchg(&lock->val, &val, 1)) {
- if (RES_CHECK_TIMEOUT(ts, ret, ~0u))
+ if (RES_CHECK_TIMEOUT(ts, ret, ~0u) < 0)
goto out;
cpu_relax();
goto retry;
@@ -406,7 +421,7 @@ int __lockfunc resilient_queued_spin_lock_slowpath(rqspinlock_t *lock, u32 val)
*/
if (val & _Q_LOCKED_MASK) {
RES_RESET_TIMEOUT(ts, RES_DEF_TIMEOUT);
- res_smp_cond_load_acquire(&lock->locked, !VAL || RES_CHECK_TIMEOUT(ts, ret, _Q_LOCKED_MASK));
+ res_smp_cond_load_acquire(&lock->locked, !VAL || RES_CHECK_TIMEOUT(ts, ret, _Q_LOCKED_MASK) < 0);
}
if (ret) {
@@ -568,7 +583,7 @@ int __lockfunc resilient_queued_spin_lock_slowpath(rqspinlock_t *lock, u32 val)
*/
RES_RESET_TIMEOUT(ts, RES_DEF_TIMEOUT * 2);
val = res_atomic_cond_read_acquire(&lock->val, !(VAL & _Q_LOCKED_PENDING_MASK) ||
- RES_CHECK_TIMEOUT(ts, ret, _Q_LOCKED_PENDING_MASK));
+ RES_CHECK_TIMEOUT(ts, ret, _Q_LOCKED_PENDING_MASK) < 0);
/* Disable queue destruction when we detect deadlocks. */
if (ret == -EDEADLK) {
--
2.31.1
^ permalink raw reply related
* [PATCH v12 13/15] arm64/delay: enable testing smp_cond_load_relaxed_timeout()
From: Ankur Arora @ 2026-06-08 8:04 UTC (permalink / raw)
To: linux-kernel, linux-arch, linux-arm-kernel, linux-pm, bpf
Cc: arnd, catalin.marinas, will, peterz, akpm, mark.rutland, harisokn,
cl, ast, rafael, daniel.lezcano, memxor, zhenglifeng1, xueshuai,
rdunlap, david.laight.linux, broonie, joao.m.martins,
boris.ostrovsky, konrad.wilk, ashok.bhat, Ankur Arora
In-Reply-To: <20260608080440.127491-1-ankur.a.arora@oracle.com>
This enables the barrier tests to be built as a module.
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
arch/arm64/lib/delay.c | 2 ++
drivers/clocksource/arm_arch_timer.c | 2 ++
2 files changed, 4 insertions(+)
diff --git a/arch/arm64/lib/delay.c b/arch/arm64/lib/delay.c
index c660a7ea26dd..dfb102ce3009 100644
--- a/arch/arm64/lib/delay.c
+++ b/arch/arm64/lib/delay.c
@@ -12,6 +12,7 @@
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/timex.h>
+#include <kunit/visibility.h>
#include <asm/delay-const.h>
#include <clocksource/arm_arch_timer.h>
@@ -30,6 +31,7 @@ u64 notrace __delay_cycles(void)
guard(preempt_notrace)();
return __arch_counter_get_cntvct_stable();
}
+EXPORT_SYMBOL_IF_KUNIT(__delay_cycles);
void __delay(unsigned long cycles)
{
diff --git a/drivers/clocksource/arm_arch_timer.c b/drivers/clocksource/arm_arch_timer.c
index 90aeff44a276..1de63e1a2cd2 100644
--- a/drivers/clocksource/arm_arch_timer.c
+++ b/drivers/clocksource/arm_arch_timer.c
@@ -28,6 +28,7 @@
#include <linux/acpi.h>
#include <linux/arm-smccc.h>
#include <linux/ptp_kvm.h>
+#include <kunit/visibility.h>
#include <asm/arch_timer.h>
#include <asm/virt.h>
@@ -896,6 +897,7 @@ bool arch_timer_evtstrm_available(void)
*/
return cpumask_test_cpu(raw_smp_processor_id(), &evtstrm_available);
}
+EXPORT_SYMBOL_IF_KUNIT(arch_timer_evtstrm_available);
static struct arch_timer_kvm_info arch_timer_kvm_info;
--
2.31.1
^ permalink raw reply related
* [PATCH v12 10/15] bpf/rqspinlock: Use smp_cond_load_acquire_timeout()
From: Ankur Arora @ 2026-06-08 8:04 UTC (permalink / raw)
To: linux-kernel, linux-arch, linux-arm-kernel, linux-pm, bpf
Cc: arnd, catalin.marinas, will, peterz, akpm, mark.rutland, harisokn,
cl, ast, rafael, daniel.lezcano, memxor, zhenglifeng1, xueshuai,
rdunlap, david.laight.linux, broonie, joao.m.martins,
boris.ostrovsky, konrad.wilk, ashok.bhat, Ankur Arora
In-Reply-To: <20260608080440.127491-1-ankur.a.arora@oracle.com>
Switch out the conditional load interfaces used by rqspinlock
to smp_cond_read_acquire_timeout() and its wrapper,
atomic_cond_read_acquire_timeout().
Both these handle the timeout and amortize as needed, so use the
non-amortized RES_CHECK_TIMEOUT.
RES_CHECK_TIMEOUT does double duty here -- presenting the current
clock value, the timeout/deadlock error from clock_deadlock() to
the cond-load and, returning the error value via ret.
For correctness, we need to ensure that the error case of the
cond-load interface always agrees with that in clock_deadlock().
For the most part, this is fine because there's no independent clock,
or double reads from the clock in cond-load -- either of which could
lead to its internal state going out of sync from that of
clock_deadlock().
There is, however, an edge case where clock_deadlock() checks for:
if (time > ts->timeout_end)
return -ETIMEDOUT;
while smp_cond_load_acquire_timeout() checks for:
__time_now = (time_expr_ns);
if (__time_now <= 0 || __time_now >= __time_end) {
VAL = READ_ONCE(*__PTR);
break;
}
This runs into a problem when (__time_now == __time_end) since
clock_deadlock() does not treat it as a timeout condition but
the second clause in the conditional above does.
So, add an equality check in clock_deadlock().
Finally, redefine SMP_TIMEOUT_POLL_COUNT to be 16k to be similar to
the spin-count used in the amortized version. We only do this for
non-arm64 as that uses a waiting implementation.
Cc: bpf@vger.kernel.org
Cc: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Acked-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
kernel/bpf/rqspinlock.c | 40 +++++++++++++++++++++++-----------------
1 file changed, 23 insertions(+), 17 deletions(-)
diff --git a/kernel/bpf/rqspinlock.c b/kernel/bpf/rqspinlock.c
index 0ec17ebb67c1..e5e27266b813 100644
--- a/kernel/bpf/rqspinlock.c
+++ b/kernel/bpf/rqspinlock.c
@@ -215,7 +215,7 @@ static noinline s64 clock_deadlock(rqspinlock_t *lock, u32 mask,
}
time = ktime_get_mono_fast_ns();
- if (time > ts->timeout_end)
+ if (time >= ts->timeout_end)
return -ETIMEDOUT;
/*
@@ -235,11 +235,10 @@ static noinline s64 clock_deadlock(rqspinlock_t *lock, u32 mask,
}
/*
- * Do not amortize with spins when res_smp_cond_load_acquire is defined,
- * as the macro does internal amortization for us.
+ * Spin amortized version of RES_CHECK_TIMEOUT. Used when busy-waiting in
+ * atomic_try_cmpxchg().
*/
-#ifndef res_smp_cond_load_acquire
-#define RES_CHECK_TIMEOUT(ts, ret, mask) \
+#define RES_CHECK_TIMEOUT_AMORTIZED(ts, ret, mask) \
({ \
s64 __timeval_err = 0; \
if (!(ts).spin++) \
@@ -247,7 +246,7 @@ static noinline s64 clock_deadlock(rqspinlock_t *lock, u32 mask,
(ret) = __timeval_err < 0 ? __timeval_err : 0; \
__timeval_err; \
})
-#else
+
#define RES_CHECK_TIMEOUT(ts, ret, mask) \
({ \
s64 __timeval_err; \
@@ -255,7 +254,6 @@ static noinline s64 clock_deadlock(rqspinlock_t *lock, u32 mask,
(ret) = __timeval_err < 0 ? __timeval_err : 0; \
__timeval_err; \
})
-#endif
/*
* Initialize the 'spin' member.
@@ -269,6 +267,17 @@ static noinline s64 clock_deadlock(rqspinlock_t *lock, u32 mask,
*/
#define RES_RESET_TIMEOUT(ts, _duration) ({ (ts).timeout_end = 0; (ts).duration = _duration; })
+/*
+ * Limit how often we invoke clock_deadlock() while spin-waiting in
+ * smp_cond_load_acquire_timeout() or atomic_cond_read_acquire_timeout().
+ *
+ * We only override the default value not superceding ARM64's override.
+ */
+#ifndef CONFIG_ARM64
+#undef SMP_TIMEOUT_POLL_COUNT
+#define SMP_TIMEOUT_POLL_COUNT (16*1024)
+#endif
+
/*
* Provide a test-and-set fallback for cases when queued spin lock support is
* absent from the architecture.
@@ -296,7 +305,7 @@ int __lockfunc resilient_tas_spin_lock(rqspinlock_t *lock)
val = atomic_read(&lock->val);
if (val || !atomic_try_cmpxchg(&lock->val, &val, 1)) {
- if (RES_CHECK_TIMEOUT(ts, ret, ~0u) < 0)
+ if (RES_CHECK_TIMEOUT_AMORTIZED(ts, ret, ~0u) < 0)
goto out;
cpu_relax();
goto retry;
@@ -319,12 +328,6 @@ EXPORT_SYMBOL_GPL(resilient_tas_spin_lock);
*/
static DEFINE_PER_CPU_ALIGNED(struct qnode, rqnodes[_Q_MAX_NODES]);
-#ifndef res_smp_cond_load_acquire
-#define res_smp_cond_load_acquire(v, c) smp_cond_load_acquire(v, c)
-#endif
-
-#define res_atomic_cond_read_acquire(v, c) res_smp_cond_load_acquire(&(v)->counter, (c))
-
/**
* resilient_queued_spin_lock_slowpath - acquire the queued spinlock
* @lock: Pointer to queued spinlock structure
@@ -421,7 +424,9 @@ int __lockfunc resilient_queued_spin_lock_slowpath(rqspinlock_t *lock, u32 val)
*/
if (val & _Q_LOCKED_MASK) {
RES_RESET_TIMEOUT(ts, RES_DEF_TIMEOUT);
- res_smp_cond_load_acquire(&lock->locked, !VAL || RES_CHECK_TIMEOUT(ts, ret, _Q_LOCKED_MASK) < 0);
+ smp_cond_load_acquire_timeout(&lock->locked, !VAL,
+ RES_CHECK_TIMEOUT(ts, ret, _Q_LOCKED_MASK),
+ ts.duration);
}
if (ret) {
@@ -582,8 +587,9 @@ int __lockfunc resilient_queued_spin_lock_slowpath(rqspinlock_t *lock, u32 val)
* us.
*/
RES_RESET_TIMEOUT(ts, RES_DEF_TIMEOUT * 2);
- val = res_atomic_cond_read_acquire(&lock->val, !(VAL & _Q_LOCKED_PENDING_MASK) ||
- RES_CHECK_TIMEOUT(ts, ret, _Q_LOCKED_PENDING_MASK) < 0);
+ val = atomic_cond_read_acquire_timeout(&lock->val, !(VAL & _Q_LOCKED_PENDING_MASK),
+ RES_CHECK_TIMEOUT(ts, ret, _Q_LOCKED_PENDING_MASK),
+ ts.duration);
/* Disable queue destruction when we detect deadlocks. */
if (ret == -EDEADLK) {
--
2.31.1
^ permalink raw reply related
* [PATCH v12 14/15] barrier: add tests for smp_cond_load_*_timeout()
From: Ankur Arora @ 2026-06-08 8:04 UTC (permalink / raw)
To: linux-kernel, linux-arch, linux-arm-kernel, linux-pm, bpf
Cc: arnd, catalin.marinas, will, peterz, akpm, mark.rutland, harisokn,
cl, ast, rafael, daniel.lezcano, memxor, zhenglifeng1, xueshuai,
rdunlap, david.laight.linux, broonie, joao.m.martins,
boris.ostrovsky, konrad.wilk, ashok.bhat, Ankur Arora
In-Reply-To: <20260608080440.127491-1-ankur.a.arora@oracle.com>
Add success and failure case tests for smp_cond_load_*_timeout().
Success or failure cases depend on the expected bit being set (or not).
Additionally in failure cases smp_cond_load_*_timeout() cannot return
before timeout.
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
Note: This fixes an error in the test case reported by Mark Brown
in https://lore.kernel.org/lkml/agr_RxvNtfASfevg@sirena.org.uk/.
There are three changes:
- One of the test conditions used in the test was much too strict.
The test was treating:
success => (runtime <= timeout_ns).
Instead, it makes greater sense to treat:
!success => (runtime >= timeout_ns).
- The test can run in a wide variety of environments including
emulated qemu. To get rid of potential failures due to timing issues,
remove the kthreaded case.
- Parametrize the test cases.
---
lib/Kconfig.debug | 10 +++
lib/tests/Makefile | 1 +
lib/tests/barrier-timeout-test.c | 128 +++++++++++++++++++++++++++++++
3 files changed, 139 insertions(+)
create mode 100644 lib/tests/barrier-timeout-test.c
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 8ff5adcfe1e0..ad5131776f68 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -2406,6 +2406,16 @@ config FPROBE_SANITY_TEST
Say N if you are unsure.
+config BARRIER_TIMEOUT_TEST
+ tristate "KUnit tests for smp_cond_load_relaxed_timeout()"
+ depends on KUNIT
+ default KUNIT_ALL_TESTS
+ help
+ Builds KUnit tests that validate wake-up and timeout handling paths
+ in smp_cond_load_relaxed_timeout().
+
+ Say N if you are unsure.
+
config BACKTRACE_SELF_TEST
tristate "Self test for the backtrace code"
depends on DEBUG_KERNEL
diff --git a/lib/tests/Makefile b/lib/tests/Makefile
index 7e9c2fa52e35..19c1d6b17856 100644
--- a/lib/tests/Makefile
+++ b/lib/tests/Makefile
@@ -20,6 +20,7 @@ CFLAGS_fortify_kunit.o += $(DISABLE_STRUCTLEAK_PLUGIN)
obj-$(CONFIG_FORTIFY_KUNIT_TEST) += fortify_kunit.o
CFLAGS_test_fprobe.o += $(CC_FLAGS_FTRACE)
obj-$(CONFIG_FPROBE_SANITY_TEST) += test_fprobe.o
+obj-$(CONFIG_BARRIER_TIMEOUT_TEST) += barrier-timeout-test.o
obj-$(CONFIG_GLOB_KUNIT_TEST) += glob_kunit.o
obj-$(CONFIG_HASHTABLE_KUNIT_TEST) += hashtable_test.o
obj-$(CONFIG_HASH_KUNIT_TEST) += test_hash.o
diff --git a/lib/tests/barrier-timeout-test.c b/lib/tests/barrier-timeout-test.c
new file mode 100644
index 000000000000..2160844b27b8
--- /dev/null
+++ b/lib/tests/barrier-timeout-test.c
@@ -0,0 +1,128 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * KUnit tests exercising smp_cond_load_relaxed_timeout().
+ *
+ * Copyright (c) 2026, Oracle Corp.
+ * Author: Ankur Arora <ankur.a.arora@oracle.com>
+ */
+
+#include <linux/bitops.h>
+#include <linux/types.h>
+#include <linux/sched/clock.h>
+#include <linux/delay.h>
+#include <asm/barrier.h>
+#include <kunit/test.h>
+#include <kunit/visibility.h>
+
+MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING");
+
+struct clock_state {
+ s64 start_time;
+ s64 end_time;
+};
+
+#define TIMEOUT_MSEC 2
+#define TEST_FLAG_VAL BIT(2)
+static unsigned int flag;
+
+static s64 basic_clock(struct clock_state *clk)
+{
+ clk->end_time = local_clock();
+ return clk->end_time;
+}
+
+static void update_flags(void)
+{
+ WRITE_ONCE(flag, TEST_FLAG_VAL);
+}
+
+static s64 mocked_clock(struct clock_state *clk)
+{
+ s64 clk_mid = clk->start_time + (TIMEOUT_MSEC * NSEC_PER_MSEC)/2;
+
+ clk->end_time = local_clock();
+ if (clk->end_time >= clk_mid)
+ update_flags();
+ return clk->end_time;
+}
+
+typedef s64 (*clkfn_t)(struct clock_state *);
+struct smp_cond_update_params {
+ clkfn_t clock;
+ bool acquire;
+ bool succeeds;
+};
+
+static const struct smp_cond_update_params update_params_list[] = {
+ /* mocked-clock updates flag inline. */
+ { .clock = &mocked_clock, .succeeds = true, .acquire = false, },
+ { .clock = &mocked_clock, .succeeds = true, .acquire = true, },
+
+ /* basic-clock doesn't update flag. */
+ { .clock = &basic_clock, .succeeds = false, .acquire = true, },
+ { .clock = &basic_clock, .succeeds = false, .acquire = false, },
+};
+
+static void param_to_desc(const struct smp_cond_update_params *p, char *desc)
+{
+ char *clk, *update;
+
+ if (p->clock == &mocked_clock) {
+ clk = "mocked";
+ update = "inline";
+ } else if (p->clock == &basic_clock) {
+ clk = "basic";
+ update = "none";
+ }
+
+
+ snprintf(desc, KUNIT_PARAM_DESC_SIZE, "smp_cond_%s_timeout: clock-%s, update=%s",
+ p->acquire ? "acquire" : "relaxed", clk, update);
+}
+
+KUNIT_ARRAY_PARAM(smp_cond_update_params, update_params_list, param_to_desc);
+
+
+static void test_smp_cond_timeout(struct kunit *test)
+{
+ const struct smp_cond_update_params *p = test->param_value;
+ struct clock_state clk = {
+ .start_time = local_clock(),
+ .end_time = local_clock(),
+ };
+ s64 runtime, timeout_ns = TIMEOUT_MSEC * NSEC_PER_MSEC;
+ unsigned int result;
+
+ flag = 0;
+ if (p->acquire) {
+ result = smp_cond_load_acquire_timeout(&flag,
+ (VAL & TEST_FLAG_VAL),
+ p->clock(&clk),
+ timeout_ns);
+ } else {
+ result = smp_cond_load_relaxed_timeout(&flag,
+ (VAL & TEST_FLAG_VAL),
+ p->clock(&clk),
+ timeout_ns);
+ }
+
+ runtime = clk.end_time - clk.start_time;
+ KUNIT_EXPECT_EQ(test, (bool)(result & TEST_FLAG_VAL), p->succeeds);
+ if (!p->succeeds)
+ KUNIT_EXPECT_GE(test, runtime, timeout_ns);
+}
+
+static struct kunit_case barrier_timeout_test_cases[] = {
+ KUNIT_CASE_PARAM(test_smp_cond_timeout, smp_cond_update_params_gen_params),
+ {}
+};
+
+static struct kunit_suite barrier_timeout_test_suite = {
+ .name = "smp-cond-load-*-timeout",
+ .test_cases = barrier_timeout_test_cases,
+};
+
+kunit_test_suite(barrier_timeout_test_suite);
+
+MODULE_DESCRIPTION("KUnit tests for smp_cond_load_relaxed_timeout()");
+MODULE_LICENSE("GPL");
--
2.31.1
^ permalink raw reply related
* [PATCH v12 11/15] sched: add need-resched timed wait interface
From: Ankur Arora @ 2026-06-08 8:04 UTC (permalink / raw)
To: linux-kernel, linux-arch, linux-arm-kernel, linux-pm, bpf
Cc: arnd, catalin.marinas, will, peterz, akpm, mark.rutland, harisokn,
cl, ast, rafael, daniel.lezcano, memxor, zhenglifeng1, xueshuai,
rdunlap, david.laight.linux, broonie, joao.m.martins,
boris.ostrovsky, konrad.wilk, ashok.bhat, Ankur Arora,
Ingo Molnar
In-Reply-To: <20260608080440.127491-1-ankur.a.arora@oracle.com>
Add tif_bitset_relaxed_wait() (and tif_need_resched_relaxed_wait()
which wraps it) which takes the thread_info bit and timeout duration
as parameters and waits until the bit is set or for the expiration
of the timeout.
The wait is implemented via smp_cond_load_relaxed_timeout().
smp_cond_load_relaxed_timeout() essentially provides the pattern used
in poll_idle() where we spin in a loop waiting for the flag to change
until a timeout occurs.
tif_need_resched_relaxed_wait() allows us to abstract out the internals
of waiting, scheduler specific details etc.
Placed in linux/sched/idle.h instead of linux/thread_info.h to work
around recursive include hell.
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rafael J. Wysocki <rafael@kernel.org>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: linux-pm@vger.kernel.org
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
include/linux/sched/idle.h | 29 +++++++++++++++++++++++++++++
1 file changed, 29 insertions(+)
diff --git a/include/linux/sched/idle.h b/include/linux/sched/idle.h
index 8465ff1f20d1..ddee9b019895 100644
--- a/include/linux/sched/idle.h
+++ b/include/linux/sched/idle.h
@@ -3,6 +3,7 @@
#define _LINUX_SCHED_IDLE_H
#include <linux/sched.h>
+#include <linux/sched/clock.h>
enum cpu_idle_type {
__CPU_NOT_IDLE = 0,
@@ -113,4 +114,32 @@ static __always_inline void current_clr_polling(void)
}
#endif
+/*
+ * Caller needs to make sure that the thread context cannot be preempted
+ * or migrated, so current_thread_info() cannot change from under us.
+ *
+ * This also allows us to safely stay in the local_clock domain.
+ */
+static __always_inline bool tif_bitset_relaxed_wait(int tif, u64 timeout_ns)
+{
+ unsigned long flags;
+
+ flags = smp_cond_load_relaxed_timeout(¤t_thread_info()->flags,
+ (VAL & BIT(tif)),
+ local_clock_noinstr(),
+ timeout_ns);
+ return flags & BIT(tif);
+}
+
+/**
+ * tif_need_resched_relaxed_wait() - Wait for need-resched being set
+ * with no ordering guarantees until a timeout expires.
+ *
+ * @timeout_ns: timeout value.
+ */
+static __always_inline bool tif_need_resched_relaxed_wait(u64 timeout_ns)
+{
+ return tif_bitset_relaxed_wait(TIF_NEED_RESCHED, timeout_ns);
+}
+
#endif /* _LINUX_SCHED_IDLE_H */
--
2.31.1
^ permalink raw reply related
* [PATCH v12 08/15] locking/atomic: scripts: build atomic_long_cond_read_*_timeout()
From: Ankur Arora @ 2026-06-08 8:04 UTC (permalink / raw)
To: linux-kernel, linux-arch, linux-arm-kernel, linux-pm, bpf
Cc: arnd, catalin.marinas, will, peterz, akpm, mark.rutland, harisokn,
cl, ast, rafael, daniel.lezcano, memxor, zhenglifeng1, xueshuai,
rdunlap, david.laight.linux, broonie, joao.m.martins,
boris.ostrovsky, konrad.wilk, ashok.bhat, Ankur Arora, Boqun Feng
In-Reply-To: <20260608080440.127491-1-ankur.a.arora@oracle.com>
Add the atomic long wrappers for the cond-load timeout interfaces.
Cc: Will Deacon <will@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
include/linux/atomic/atomic-long.h | 18 +++++++++++-------
scripts/atomic/gen-atomic-long.sh | 16 ++++++++++------
2 files changed, 21 insertions(+), 13 deletions(-)
diff --git a/include/linux/atomic/atomic-long.h b/include/linux/atomic/atomic-long.h
index 6a4e47d2db35..553b6b0e0258 100644
--- a/include/linux/atomic/atomic-long.h
+++ b/include/linux/atomic/atomic-long.h
@@ -11,14 +11,18 @@
#ifdef CONFIG_64BIT
typedef atomic64_t atomic_long_t;
-#define ATOMIC_LONG_INIT(i) ATOMIC64_INIT(i)
-#define atomic_long_cond_read_acquire atomic64_cond_read_acquire
-#define atomic_long_cond_read_relaxed atomic64_cond_read_relaxed
+#define ATOMIC_LONG_INIT(i) ATOMIC64_INIT(i)
+#define atomic_long_cond_read_acquire atomic64_cond_read_acquire
+#define atomic_long_cond_read_relaxed atomic64_cond_read_relaxed
+#define atomic_long_cond_read_acquire_timeout atomic64_cond_read_acquire_timeout
+#define atomic_long_cond_read_relaxed_timeout atomic64_cond_read_relaxed_timeout
#else
typedef atomic_t atomic_long_t;
-#define ATOMIC_LONG_INIT(i) ATOMIC_INIT(i)
-#define atomic_long_cond_read_acquire atomic_cond_read_acquire
-#define atomic_long_cond_read_relaxed atomic_cond_read_relaxed
+#define ATOMIC_LONG_INIT(i) ATOMIC_INIT(i)
+#define atomic_long_cond_read_acquire atomic_cond_read_acquire
+#define atomic_long_cond_read_relaxed atomic_cond_read_relaxed
+#define atomic_long_cond_read_acquire_timeout atomic_cond_read_acquire_timeout
+#define atomic_long_cond_read_relaxed_timeout atomic_cond_read_relaxed_timeout
#endif
/**
@@ -1809,4 +1813,4 @@ raw_atomic_long_dec_if_positive(atomic_long_t *v)
}
#endif /* _LINUX_ATOMIC_LONG_H */
-// 4b882bf19018602c10816c52f8b4ae280adc887b
+// 79c1f4acb5774376ceed559843d5d9ed1348df99
diff --git a/scripts/atomic/gen-atomic-long.sh b/scripts/atomic/gen-atomic-long.sh
index 9826be3ba986..874643dc74bd 100755
--- a/scripts/atomic/gen-atomic-long.sh
+++ b/scripts/atomic/gen-atomic-long.sh
@@ -79,14 +79,18 @@ cat << EOF
#ifdef CONFIG_64BIT
typedef atomic64_t atomic_long_t;
-#define ATOMIC_LONG_INIT(i) ATOMIC64_INIT(i)
-#define atomic_long_cond_read_acquire atomic64_cond_read_acquire
-#define atomic_long_cond_read_relaxed atomic64_cond_read_relaxed
+#define ATOMIC_LONG_INIT(i) ATOMIC64_INIT(i)
+#define atomic_long_cond_read_acquire atomic64_cond_read_acquire
+#define atomic_long_cond_read_relaxed atomic64_cond_read_relaxed
+#define atomic_long_cond_read_acquire_timeout atomic64_cond_read_acquire_timeout
+#define atomic_long_cond_read_relaxed_timeout atomic64_cond_read_relaxed_timeout
#else
typedef atomic_t atomic_long_t;
-#define ATOMIC_LONG_INIT(i) ATOMIC_INIT(i)
-#define atomic_long_cond_read_acquire atomic_cond_read_acquire
-#define atomic_long_cond_read_relaxed atomic_cond_read_relaxed
+#define ATOMIC_LONG_INIT(i) ATOMIC_INIT(i)
+#define atomic_long_cond_read_acquire atomic_cond_read_acquire
+#define atomic_long_cond_read_relaxed atomic_cond_read_relaxed
+#define atomic_long_cond_read_acquire_timeout atomic_cond_read_acquire_timeout
+#define atomic_long_cond_read_relaxed_timeout atomic_cond_read_relaxed_timeout
#endif
EOF
--
2.31.1
^ permalink raw reply related
* [PATCH v12 07/15] atomic: Add atomic_cond_read_*_timeout()
From: Ankur Arora @ 2026-06-08 8:04 UTC (permalink / raw)
To: linux-kernel, linux-arch, linux-arm-kernel, linux-pm, bpf
Cc: arnd, catalin.marinas, will, peterz, akpm, mark.rutland, harisokn,
cl, ast, rafael, daniel.lezcano, memxor, zhenglifeng1, xueshuai,
rdunlap, david.laight.linux, broonie, joao.m.martins,
boris.ostrovsky, konrad.wilk, ashok.bhat, Ankur Arora, Boqun Feng
In-Reply-To: <20260608080440.127491-1-ankur.a.arora@oracle.com>
Add atomic load wrappers, atomic_cond_read_*_timeout() and
atomic64_cond_read_*_timeout() for the cond-load timeout interfaces.
Also add a short description for the atomic_cond_read_{relaxed,acquire}(),
and the atomic_cond_read_{relaxed,acquire}_timeout() interfaces.
Cc: Will Deacon <will@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
Documentation/atomic_t.txt | 14 +++++++++-----
include/linux/atomic.h | 10 ++++++++++
2 files changed, 19 insertions(+), 5 deletions(-)
diff --git a/Documentation/atomic_t.txt b/Documentation/atomic_t.txt
index bee3b1bca9a7..0e53f6ccb558 100644
--- a/Documentation/atomic_t.txt
+++ b/Documentation/atomic_t.txt
@@ -16,6 +16,10 @@ Non-RMW ops:
atomic_read(), atomic_set()
atomic_read_acquire(), atomic_set_release()
+Non-RMW, non-atomic_t ops:
+
+ atomic_cond_read_{relaxed,acquire}()
+ atomic_cond_read_{relaxed,acquire}_timeout()
RMW atomic operations:
@@ -79,11 +83,11 @@ SEMANTICS
Non-RMW ops:
-The non-RMW ops are (typically) regular LOADs and STOREs and are canonically
-implemented using READ_ONCE(), WRITE_ONCE(), smp_load_acquire() and
-smp_store_release() respectively. Therefore, if you find yourself only using
-the Non-RMW operations of atomic_t, you do not in fact need atomic_t at all
-and are doing it wrong.
+The non-RMW ops are (typically) regular, or conditional LOADs and STOREs and
+are canonically implemented using READ_ONCE(), WRITE_ONCE(),
+smp_load_acquire() and smp_store_release() respectively. Therefore, if you
+find yourself only using the Non-RMW operations of atomic_t, you do not in
+fact need atomic_t at all and are doing it wrong.
A note for the implementation of atomic_set{}() is that it must not break the
atomicity of the RMW ops. That is:
diff --git a/include/linux/atomic.h b/include/linux/atomic.h
index 8dd57c3a99e9..5bcb86e07784 100644
--- a/include/linux/atomic.h
+++ b/include/linux/atomic.h
@@ -31,6 +31,16 @@
#define atomic64_cond_read_acquire(v, c) smp_cond_load_acquire(&(v)->counter, (c))
#define atomic64_cond_read_relaxed(v, c) smp_cond_load_relaxed(&(v)->counter, (c))
+#define atomic_cond_read_acquire_timeout(v, c, e, t) \
+ smp_cond_load_acquire_timeout(&(v)->counter, (c), (e), (t))
+#define atomic_cond_read_relaxed_timeout(v, c, e, t) \
+ smp_cond_load_relaxed_timeout(&(v)->counter, (c), (e), (t))
+
+#define atomic64_cond_read_acquire_timeout(v, c, e, t) \
+ smp_cond_load_acquire_timeout(&(v)->counter, (c), (e), (t))
+#define atomic64_cond_read_relaxed_timeout(v, c, e, t) \
+ smp_cond_load_relaxed_timeout(&(v)->counter, (c), (e), (t))
+
/*
* The idea here is to build acquire/release variants by adding explicit
* barriers on top of the relaxed variant. In the case where the relaxed
--
2.31.1
^ permalink raw reply related
* [PATCH v12 00/15] barrier: Add smp_cond_load_{relaxed,acquire}_timeout()
From: Ankur Arora @ 2026-06-08 8:04 UTC (permalink / raw)
To: linux-kernel, linux-arch, linux-arm-kernel, linux-pm, bpf
Cc: arnd, catalin.marinas, will, peterz, akpm, mark.rutland, harisokn,
cl, ast, rafael, daniel.lezcano, memxor, zhenglifeng1, xueshuai,
rdunlap, david.laight.linux, broonie, joao.m.martins,
boris.ostrovsky, konrad.wilk, ashok.bhat, Ankur Arora
Hi,
Main change in this version:
- addressed some review comments from sashiko (see commit notes)
- The one notable change is to the implementation of
smp_cond_load_acquire_timeout() where there was a missed
control dependency in the timeout case.
All the others are minor.
- fixed a low probability race in the kunit test added in v11.
- added a bunch of kunit tests validating the implementation's
use of the clock.
Andrew, if the changes look okay, could we take this in your mm-nomm
tree as before?
The core kernel often uses smp_cond_load_{relaxed,acquire}() to spin
on condition variables with architectural primitives used to avoid
hammering the relevant cachelines.
(This primitive can vary greatly across architectures: on x86 it's a
cpu_relax() to slow down the pipeline. On arm64, this is a __cmpwait()
which waits for a cacheline to change state in a time limited fashion.)
Regardless of architectural details, typical smp_cond_load*() usage
does not allow for termination until the condition change occurs.
Beyond the core kernel, there are cases where it is useful to additionally
terminate on a timeout. Two cases:
- cpuidle poll_idle(): wait for need-resched until the cpuidle polling
duration expires.
- rqspinlock: nested qspinlock acquisition that terminates on timeout
or deadlock.
Accordingly add two interfaces (with their generic and arm64 specific
implementations):
smp_cond_load_relaxed_timeout(ptr, cond_expr, time_expr, timeout)
smp_cond_load_acquire_timeout(ptr, cond_expr, time_expr, timeout)
Also add tif_need_resched_relaxed_wait() which wraps the polling
pattern and its scheduler specific details in poll_idle().
In addition add atomic_cond_read_*_timeout(),
atomic64_cond_read_*_timeout(), and atomic_long wrappers.
Structurally, both the smp_cond_load_*_timeout() interfaces are similar
to smp_cond_load*(), with the addition of a rate-limited time-check.
Usage
==
These interfaces drop straight-forwardly into the rqspinlock logic
since qspinlock already uses smp_cond_load*(), and the time-check
extension can now be used for timeout and deadlock handling.
Using tif_need_resched_relaxed_wait() in poll_idle() removes any
architectural details allowing arm64 to straight-forwardly support
that path.
(However, for efficiency reasons cpuidle/poll_state.c continues to
depend on ARCH_HAS_CPU_RELAX since that is defined on architectures
with an optimized architectural primitive.)
Performance
==
Apart from simplifications due to this change, supporting polling in
cpuidle on arm64 helps improve wakeup latency (needs a few cpuidle/acpi
patches):
# perf stat -r 5 --cpu 4,5 -e task-clock,cycles,instructions,sched:sched_wake_idle_without_ipi \
perf bench sched pipe -l 1000000 -c 4
# No haltpoll (and, no TIF_POLLING_NRFLAG):
Performance counter stats for 'CPU(s) 4,5' (5 runs):
25,229.57 msec task-clock # 2.000 CPUs utilized ( +- 7.75% )
45,821,250,284 cycles # 1.816 GHz ( +- 10.07% )
26,557,496,665 instructions # 0.58 insn per cycle ( +- 0.21% )
0 sched:sched_wake_idle_without_ipi # 0.000 /sec
12.615 +- 0.977 seconds time elapsed ( +- 7.75% )
# Haltpoll:
Performance counter stats for 'CPU(s) 4,5' (5 runs):
15,131.58 msec task-clock # 2.000 CPUs utilized ( +- 10.00% )
34,158,188,839 cycles # 2.257 GHz ( +- 6.91% )
20,824,950,916 instructions # 0.61 insn per cycle ( +- 0.09% )
1,983,822 sched:sched_wake_idle_without_ipi # 131.105 K/sec ( +- 0.78% )
7.566 +- 0.756 seconds time elapsed ( +- 10.00% )
We get improved latency because we don't switch in and out of a
deeper sleep state or from the hypervisor. This also causes us to
execute ~20% fewer instructions.
Haris Okanovic also saw improvement in real workloads due to the
cpuidle changes: "observed 4-6% improvements in memcahed, cassandra,
mysql, and postgresql under certain loads. Other applications likely
benefit too." [12]
Changelog:
v11 [13] (as listed above):
- addressed some review comments from sashiko (see commit notes)
- The one notable change is to the implementation of
smp_cond_load_acquire_timeout() where there was a missed
control dependency in the timeout case.
All the others are minor.
- fixed a low probability race in the kunit test added in v11.
- added a bunch of kunit tests validating the implementation's
use of the clock.
v10 [10]:
- add a comment mentioning that smp_cond_load_relaxed_timeout() might
be using architectural primitives that don't support MMIO.
(David Laight, Catalin Marinas)
- added a kunit test for smp_cond_load_relaxed_timeout() (Andrew
Morton.)
v9 [9]:
- s/@cond/@cond_expr/ (Randy Dunlap)
- Clarify that SMP_TIMEOUT_POLL_COUNT is only around memory
addresses. (David Laight)
- Add the missing config ARCH_HAS_CPU_RELAX in arch/arm64/Kconfig.
(Catalin Marinas).
- Switch to arch_counter_get_cntvct_stable() (via __delay_cycles())
in the cmpwait path instead of using arch_timer_read_counter().
(Catalin Marinas)
v8 [0]:
- Defer evaluation of @time_expr_ns to when we hit the slowpath.
(comment from Alexei Starovoitov).
- Mention that cpu_poll_relax() is better than raw CPU polling
only where ARCH_HAS_CPU_RELAX is defined.
- also define ARCH_HAS_CPU_RELAX for arm64.
(Came out of a discussion with Will Deacon.)
- Split out WFET and WFE handling. I was doing both of these
in a common handler.
(From Will Deacon and in an earlier revision by Catalin Marinas.)
- Add mentions of atomic_cond_read_{relaxed,acquire}(),
atomic_cond_read_{relaxed,acquire}_timeout() in
Documentation/atomic_t.txt.
- Use the BIT() macro to do the checking in tif_bitset_relaxed_wait().
- Cleanup unnecessary assignments, casts etc in poll_idle().
(From Rafael Wysocki.)
- Fixup warnings from kernel build robot
v7 [1]:
- change the interface to separately provide the timeout. This is
useful for supporting WFET and similar primitives which can do
timed waiting (suggested by Arnd Bergmann).
- Adapting rqspinlock code to this changed interface also
necessitated allowing time_expr to fail.
- rqspinlock changes to adapt to the new smp_cond_load_acquire_timeout().
- add WFET support (suggested by Arnd Bergmann).
- add support for atomic-long wrappers.
- add a new scheduler interface tif_need_resched_relaxed_wait() which
encapsulates the polling logic used by poll_idle().
- interface suggested by (Rafael J. Wysocki).
v6 [2]:
- fixup missing timeout parameters in atomic64_cond_read_*_timeout()
- remove a race between setting of TIF_NEED_RESCHED and the call to
smp_cond_load_relaxed_timeout(). This would mean that dev->poll_time_limit
would be set even if we hadn't spent any time waiting.
(The original check compared against local_clock(), which would have been
fine, but I was instead using a cheaper check against _TIF_NEED_RESCHED.)
(Both from meta-CI bot)
v5 [3]:
- use cpu_poll_relax() instead of cpu_relax().
- instead of defining an arm64 specific
smp_cond_load_relaxed_timeout(), just define the appropriate
cpu_poll_relax().
- re-read the target pointer when we exit due to the time-check.
- s/SMP_TIMEOUT_SPIN_COUNT/SMP_TIMEOUT_POLL_COUNT/
(Suggested by Will Deacon)
- add atomic_cond_read_*_timeout() and atomic64_cond_read_*_timeout()
interfaces.
- rqspinlock: use atomic_cond_read_acquire_timeout().
- cpuidle: use smp_cond_load_relaxed_tiemout() for polling.
(Suggested by Catalin Marinas)
- rqspinlock: define SMP_TIMEOUT_POLL_COUNT to be 16k for non arm64
v4 [4]:
- naming change 's/timewait/timeout/'
- resilient spinlocks: get rid of res_smp_cond_load_acquire_waiting()
and fixup use of RES_CHECK_TIMEOUT().
(Both suggested by Catalin Marinas)
v3 [5]:
- further interface simplifications (suggested by Catalin Marinas)
v2 [6]:
- simplified the interface (suggested by Catalin Marinas)
- get rid of wait_policy, and a multitude of constants
- adds a slack parameter
This helped remove a fair amount of duplicated code duplication and in
hindsight unnecessary constants.
v1 [7]:
- add wait_policy (coarse and fine)
- derive spin-count etc at runtime instead of using arbitrary
constants.
Haris Okanovic tested v4 of this series with poll_idle()/haltpoll patches. [8]
Comments appreciated!
Thanks
Ankur
[0] https://lore.kernel.org/lkml/20251215044919.460086-1-ankur.a.arora@oracle.com/
[1] https://lore.kernel.org/lkml/20251028053136.692462-1-ankur.a.arora@oracle.com/
[2] https://lore.kernel.org/lkml/20250911034655.3916002-1-ankur.a.arora@oracle.com/
[3] https://lore.kernel.org/lkml/20250911034655.3916002-1-ankur.a.arora@oracle.com/
[4] https://lore.kernel.org/lkml/20250829080735.3598416-1-ankur.a.arora@oracle.com/
[5] https://lore.kernel.org/lkml/20250627044805.945491-1-ankur.a.arora@oracle.com/
[6] https://lore.kernel.org/lkml/20250502085223.1316925-1-ankur.a.arora@oracle.com/
[7] https://lore.kernel.org/lkml/20250203214911.898276-1-ankur.a.arora@oracle.com/
[8] https://lore.kernel.org/lkml/2cecbf7fb23ee83a4ce027e1be3f46f97efd585c.camel@amazon.com/
[9] https://lore.kernel.org/lkml/20260209023153.2661784-1-ankur.a.arora@oracle.com/
[10] https://lore.kernel.org/lkml/20260316013651.3225328-1-ankur.a.arora@oracle.com/
[11] https://lore.kernel.org/lkml/20230809134837.GM212435@hirez.programming.kicks-ass.net/
[12] https://lore.kernel.org/lkml/c6f3c8d3f1f2e89a9dc7ae22482973b5a51b08cb.camel@amazon.com/
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Will Deacon <will@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Rafael J. Wysocki" <rafael@kernel.org>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: bpf@vger.kernel.org
Cc: linux-arch@vger.kernel.org
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-pm@vger.kernel.org
Ankur Arora (15):
asm-generic: barrier: Add smp_cond_load_relaxed_timeout()
arm64: barrier: Support smp_cond_load_relaxed_timeout()
arm64/delay: move some constants out to a separate header
arm64: support WFET in smp_cond_load_relaxed_timeout()
arm64: rqspinlock: Remove private copy of
smp_cond_load_acquire_timewait()
asm-generic: barrier: Add smp_cond_load_acquire_timeout()
atomic: Add atomic_cond_read_*_timeout()
locking/atomic: scripts: build atomic_long_cond_read_*_timeout()
bpf/rqspinlock: switch check_timeout() to a clock interface
bpf/rqspinlock: Use smp_cond_load_acquire_timeout()
sched: add need-resched timed wait interface
cpuidle/poll_state: Wait for need-resched via
tif_need_resched_relaxed_wait()
arm64/delay: enable testing smp_cond_load_relaxed_timeout()
barrier: add tests for smp_cond_load_*_timeout()
barrier: add clock tests for smp_cond_load_relaxed_timeout()
Documentation/atomic_t.txt | 14 +-
arch/arm64/Kconfig | 3 +
arch/arm64/include/asm/barrier.h | 23 ++++
arch/arm64/include/asm/cmpxchg.h | 62 +++++++--
arch/arm64/include/asm/delay-const.h | 28 ++++
arch/arm64/include/asm/rqspinlock.h | 85 ------------
arch/arm64/lib/delay.c | 17 +--
drivers/clocksource/arm_arch_timer.c | 2 +
drivers/cpuidle/poll_state.c | 21 +--
drivers/soc/qcom/rpmh-rsc.c | 8 +-
include/asm-generic/barrier.h | 97 ++++++++++++++
include/linux/atomic.h | 10 ++
include/linux/atomic/atomic-long.h | 18 ++-
include/linux/sched/idle.h | 29 +++++
kernel/bpf/rqspinlock.c | 77 +++++++----
lib/Kconfig.debug | 10 ++
lib/tests/Makefile | 1 +
lib/tests/barrier-timeout-test.c | 185 +++++++++++++++++++++++++++
scripts/atomic/gen-atomic-long.sh | 16 ++-
19 files changed, 528 insertions(+), 178 deletions(-)
create mode 100644 arch/arm64/include/asm/delay-const.h
create mode 100644 lib/tests/barrier-timeout-test.c
--
2.31.1
^ permalink raw reply
* [PATCH v12 05/15] arm64: rqspinlock: Remove private copy of smp_cond_load_acquire_timewait()
From: Ankur Arora @ 2026-06-08 8:04 UTC (permalink / raw)
To: linux-kernel, linux-arch, linux-arm-kernel, linux-pm, bpf
Cc: arnd, catalin.marinas, will, peterz, akpm, mark.rutland, harisokn,
cl, ast, rafael, daniel.lezcano, memxor, zhenglifeng1, xueshuai,
rdunlap, david.laight.linux, broonie, joao.m.martins,
boris.ostrovsky, konrad.wilk, ashok.bhat, Ankur Arora
In-Reply-To: <20260608080440.127491-1-ankur.a.arora@oracle.com>
In preparation for defining smp_cond_load_acquire_timeout(), remove
the private copy. Lacking this, the rqspinlock code falls back to using
smp_cond_load_acquire().
Cc: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: bpf@vger.kernel.org
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Haris Okanovic <harisokn@amazon.com>
Acked-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
Notes:
Sashiko mentions that this introduces a bisection hole: this has been
discussed in prior versions and doing it this way seems the cleanest.
---
arch/arm64/include/asm/rqspinlock.h | 85 -----------------------------
1 file changed, 85 deletions(-)
diff --git a/arch/arm64/include/asm/rqspinlock.h b/arch/arm64/include/asm/rqspinlock.h
index 9ea0a74e5892..a385603436e9 100644
--- a/arch/arm64/include/asm/rqspinlock.h
+++ b/arch/arm64/include/asm/rqspinlock.h
@@ -3,91 +3,6 @@
#define _ASM_RQSPINLOCK_H
#include <asm/barrier.h>
-
-/*
- * Hardcode res_smp_cond_load_acquire implementations for arm64 to a custom
- * version based on [0]. In rqspinlock code, our conditional expression involves
- * checking the value _and_ additionally a timeout. However, on arm64, the
- * WFE-based implementation may never spin again if no stores occur to the
- * locked byte in the lock word. As such, we may be stuck forever if
- * event-stream based unblocking is not available on the platform for WFE spin
- * loops (arch_timer_evtstrm_available).
- *
- * Once support for smp_cond_load_acquire_timewait [0] lands, we can drop this
- * copy-paste.
- *
- * While we rely on the implementation to amortize the cost of sampling
- * cond_expr for us, it will not happen when event stream support is
- * unavailable, time_expr check is amortized. This is not the common case, and
- * it would be difficult to fit our logic in the time_expr_ns >= time_limit_ns
- * comparison, hence just let it be. In case of event-stream, the loop is woken
- * up at microsecond granularity.
- *
- * [0]: https://lore.kernel.org/lkml/20250203214911.898276-1-ankur.a.arora@oracle.com
- */
-
-#ifndef smp_cond_load_acquire_timewait
-
-#define smp_cond_time_check_count 200
-
-#define __smp_cond_load_relaxed_spinwait(ptr, cond_expr, time_expr_ns, \
- time_limit_ns) ({ \
- typeof(ptr) __PTR = (ptr); \
- __unqual_scalar_typeof(*ptr) VAL; \
- unsigned int __count = 0; \
- for (;;) { \
- VAL = READ_ONCE(*__PTR); \
- if (cond_expr) \
- break; \
- cpu_relax(); \
- if (__count++ < smp_cond_time_check_count) \
- continue; \
- if ((time_expr_ns) >= (time_limit_ns)) \
- break; \
- __count = 0; \
- } \
- (typeof(*ptr))VAL; \
-})
-
-#define __smp_cond_load_acquire_timewait(ptr, cond_expr, \
- time_expr_ns, time_limit_ns) \
-({ \
- typeof(ptr) __PTR = (ptr); \
- __unqual_scalar_typeof(*ptr) VAL; \
- for (;;) { \
- VAL = smp_load_acquire(__PTR); \
- if (cond_expr) \
- break; \
- __cmpwait_relaxed(__PTR, VAL); \
- if ((time_expr_ns) >= (time_limit_ns)) \
- break; \
- } \
- (typeof(*ptr))VAL; \
-})
-
-#define smp_cond_load_acquire_timewait(ptr, cond_expr, \
- time_expr_ns, time_limit_ns) \
-({ \
- __unqual_scalar_typeof(*ptr) _val; \
- int __wfe = arch_timer_evtstrm_available(); \
- \
- if (likely(__wfe)) { \
- _val = __smp_cond_load_acquire_timewait(ptr, cond_expr, \
- time_expr_ns, \
- time_limit_ns); \
- } else { \
- _val = __smp_cond_load_relaxed_spinwait(ptr, cond_expr, \
- time_expr_ns, \
- time_limit_ns); \
- smp_acquire__after_ctrl_dep(); \
- } \
- (typeof(*ptr))_val; \
-})
-
-#endif
-
-#define res_smp_cond_load_acquire(v, c) smp_cond_load_acquire_timewait(v, c, 0, 1)
-
#include <asm-generic/rqspinlock.h>
#endif /* _ASM_RQSPINLOCK_H */
--
2.31.1
^ permalink raw reply related
* [PATCH v12 01/15] asm-generic: barrier: Add smp_cond_load_relaxed_timeout()
From: Ankur Arora @ 2026-06-08 8:04 UTC (permalink / raw)
To: linux-kernel, linux-arch, linux-arm-kernel, linux-pm, bpf
Cc: arnd, catalin.marinas, will, peterz, akpm, mark.rutland, harisokn,
cl, ast, rafael, daniel.lezcano, memxor, zhenglifeng1, xueshuai,
rdunlap, david.laight.linux, broonie, joao.m.martins,
boris.ostrovsky, konrad.wilk, ashok.bhat, Ankur Arora
In-Reply-To: <20260608080440.127491-1-ankur.a.arora@oracle.com>
Add smp_cond_load_relaxed_timeout(), which extends
smp_cond_load_relaxed() to allow waiting for a duration.
We loop around waiting for the condition variable to change while
peridically doing a time-check. The loop uses cpu_poll_relax() to slow
down the busy-wait, which, unless overridden by the architecture
code, amounts to a cpu_relax().
Note that there are two ways for the time-check to fail: the timeout
case or, @time_expr_ns returning an invalid value (negative or zero).
The second failure mode allows for clocks attached to the clock-domain
of @cond_expr -- which might cease to operate meaningfully once some
state internal to @cond_expr has changed -- to fail.
Evaluation of @time_expr_ns: in the fastpath we want to keep the
performance close to smp_cond_load_relaxed(). So defer evaluation
of the potentially costly @time_expr_ns to the slowpath.
This also means that there will always be some hardware dependent
duration that has passed in cpu_poll_relax() iterations at the time
of first evaluation. Additionally cpu_poll_relax() is not guaranteed
to return at timeout boundary. In sum, expect timeout overshoot when
we exit due to expiration of the timeout.
The number of spin iterations before time-check, SMP_TIMEOUT_POLL_COUNT
is chosen to be 200 by default. With a cpu_poll_relax() iteration
taking ~20-30 cycles (measured on a variety of x86 platforms), we
expect a time-check every ~4000-6000 cycles.
The outer limit of the overshoot is double that when working with the
parameters above. This might be higher or lower depending on the
implementation of cpu_poll_relax() across architectures.
Lastly, config option ARCH_HAS_CPU_RELAX indicates availability of a
cpu_poll_relax() that is cheaper than polling. This might be relevant
for cases with a long timeout.
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Will Deacon <will@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: linux-arch@vger.kernel.org
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
Note: addresses a few Sashiko comments from [1].
We leave unaddressed a couple of potential timeout range issues (around
S64_MAX, or during early boot). I had proposed a version earlier that
would address those in [2]. Since then, however, I've come to the view
that these issues are best handled in code review instead of
overcomplicating the implementation.
[1] https://sashiko.dev/#/patchset/20260408122538.3610871-1-ankur.a.arora%40oracle.com
[2] https://lore.kernel.org/lkml/874iklm1uy.fsf@oracle.com/
---
include/asm-generic/barrier.h | 69 +++++++++++++++++++++++++++++++++++
1 file changed, 69 insertions(+)
diff --git a/include/asm-generic/barrier.h b/include/asm-generic/barrier.h
index d4f581c1e21d..c56df9513a08 100644
--- a/include/asm-generic/barrier.h
+++ b/include/asm-generic/barrier.h
@@ -273,6 +273,75 @@ do { \
})
#endif
+/*
+ * Number of times we iterate in the loop before doing the time check.
+ * Note that the iteration count assumes that the loop condition is
+ * relatively cheap.
+ */
+#ifndef SMP_TIMEOUT_POLL_COUNT
+#define SMP_TIMEOUT_POLL_COUNT 200
+#endif
+
+/*
+ * Platforms with ARCH_HAS_CPU_RELAX have a cpu_poll_relax() implementation
+ * that is expected to be cheaper (lower power) than pure polling.
+ */
+#ifndef cpu_poll_relax
+#define cpu_poll_relax(ptr, val, timeout_ns) cpu_relax()
+#endif
+
+/**
+ * smp_cond_load_relaxed_timeout() - (Spin) wait for cond with no ordering
+ * guarantees until a timeout expires.
+ * @ptr: pointer to the variable to wait on.
+ * @cond_expr: boolean expression to wait for.
+ * @time_expr_ns: expression that evaluates to monotonic time (in ns) or,
+ * on failure, returns a negative value.
+ * @timeout_ns: timeout value in ns
+ * Both of the above are assumed to be compatible with s64; the signed
+ * value is used to handle the failure case in @time_expr_ns.
+ *
+ * Equivalent to using READ_ONCE() on the condition variable.
+ *
+ * Callers that expect to wait for prolonged durations might want
+ * to take into account the availability of ARCH_HAS_CPU_RELAX.
+ *
+ * Note that @ptr is expected to point to a memory address. Using this
+ * interface with MMIO will be slower (since SMP_TIMEOUT_POLL_COUNT is
+ * tuned for memory) and might also break in interesting architecture
+ * dependent ways.
+ */
+#ifndef smp_cond_load_relaxed_timeout
+#define smp_cond_load_relaxed_timeout(ptr, cond_expr, \
+ time_expr_ns, timeout_ns) \
+({ \
+ typeof(ptr) __PTR = (ptr); \
+ __unqual_scalar_typeof(*(ptr)) VAL; \
+ u32 __count = 0, __spin = SMP_TIMEOUT_POLL_COUNT; \
+ s64 __timeout = (s64)timeout_ns; \
+ s64 __time_now, __time_end = 0; \
+ \
+ for (;;) { \
+ VAL = READ_ONCE(*__PTR); \
+ if (cond_expr) \
+ break; \
+ cpu_poll_relax(__PTR, VAL, (u64)__timeout); \
+ if (++__count < __spin) \
+ continue; \
+ __time_now = (s64)(time_expr_ns); \
+ if (unlikely(__time_end == 0)) \
+ __time_end = __time_now + __timeout; \
+ __timeout = __time_end - __time_now; \
+ if (__time_now <= 0 || __timeout <= 0) { \
+ VAL = READ_ONCE(*__PTR); \
+ break; \
+ } \
+ __count = 0; \
+ } \
+ (typeof(*(ptr)))VAL; \
+})
+#endif
+
/*
* pmem_wmb() ensures that all stores for which the modification
* are written to persistent storage by preceding instructions have
--
2.31.1
^ permalink raw reply related
* [PATCH v12 04/15] arm64: support WFET in smp_cond_load_relaxed_timeout()
From: Ankur Arora @ 2026-06-08 8:04 UTC (permalink / raw)
To: linux-kernel, linux-arch, linux-arm-kernel, linux-pm, bpf
Cc: arnd, catalin.marinas, will, peterz, akpm, mark.rutland, harisokn,
cl, ast, rafael, daniel.lezcano, memxor, zhenglifeng1, xueshuai,
rdunlap, david.laight.linux, broonie, joao.m.martins,
boris.ostrovsky, konrad.wilk, ashok.bhat, Ankur Arora
In-Reply-To: <20260608080440.127491-1-ankur.a.arora@oracle.com>
To handle WFET use __cmpwait_timeout() similarly to __cmpwait(). These
call out to the respective __cmpwait_case_timeout_##sz(),
__cmpwait_case_##sz() functions.
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: linux-arm-kernel@lists.infradead.org
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
Notes:
Does not address sashiko [1] comments on:
- timeout overshoot: intentional, not meant to be a precise
interface.
- range edge cases: as mentioned before, better addressed in
code review.
- loadable kernel modules using this will fail to build: niche
interface, not worth fixing unless necessary.
[1] https://sashiko.dev/#/patchset/20260408122538.3610871-1-ankur.a.arora%40oracle.com
---
arch/arm64/include/asm/barrier.h | 8 +++--
arch/arm64/include/asm/cmpxchg.h | 62 +++++++++++++++++++++++++-------
2 files changed, 55 insertions(+), 15 deletions(-)
diff --git a/arch/arm64/include/asm/barrier.h b/arch/arm64/include/asm/barrier.h
index 6190e178db51..fbd71cd4ef4e 100644
--- a/arch/arm64/include/asm/barrier.h
+++ b/arch/arm64/include/asm/barrier.h
@@ -224,8 +224,8 @@ do { \
extern bool arch_timer_evtstrm_available(void);
/*
- * In the common case, cpu_poll_relax() sits waiting in __cmpwait_relaxed()
- * for the ptr value to change.
+ * In the common case, cpu_poll_relax() sits waiting in __cmpwait_relaxed()/
+ * __cmpwait_relaxed_timeout() for the ptr value to change.
*
* Since this period is reasonably long, choose SMP_TIMEOUT_POLL_COUNT
* to be 1, so smp_cond_load_{relaxed,acquire}_timeout() does a
@@ -234,7 +234,9 @@ extern bool arch_timer_evtstrm_available(void);
#define SMP_TIMEOUT_POLL_COUNT 1
#define cpu_poll_relax(ptr, val, timeout_ns) do { \
- if (arch_timer_evtstrm_available()) \
+ if (alternative_has_cap_unlikely(ARM64_HAS_WFXT)) \
+ __cmpwait_relaxed_timeout(ptr, val, timeout_ns); \
+ else if (arch_timer_evtstrm_available()) \
__cmpwait_relaxed(ptr, val); \
else \
cpu_relax(); \
diff --git a/arch/arm64/include/asm/cmpxchg.h b/arch/arm64/include/asm/cmpxchg.h
index 6cf3cd6873f5..9e4cdc9e41d1 100644
--- a/arch/arm64/include/asm/cmpxchg.h
+++ b/arch/arm64/include/asm/cmpxchg.h
@@ -12,6 +12,7 @@
#include <asm/barrier.h>
#include <asm/lse.h>
+#include <asm/delay-const.h>
/*
* We need separate acquire parameters for ll/sc and lse, since the full
@@ -212,7 +213,8 @@ __CMPXCHG_GEN(_mb)
#define __CMPWAIT_CASE(w, sfx, sz) \
static inline void __cmpwait_case_##sz(volatile void *ptr, \
- unsigned long val) \
+ unsigned long val, \
+ u64 __maybe_unused timeout_ns) \
{ \
unsigned long tmp; \
\
@@ -235,20 +237,52 @@ __CMPWAIT_CASE( , , 64);
#undef __CMPWAIT_CASE
-#define __CMPWAIT_GEN(sfx) \
-static __always_inline void __cmpwait##sfx(volatile void *ptr, \
- unsigned long val, \
- int size) \
+#define __CMPWAIT_TIMEOUT_CASE(w, sfx, sz) \
+static inline void __cmpwait_case_timeout_##sz(volatile void *ptr, \
+ unsigned long val, \
+ u64 timeout_ns) \
+{ \
+ unsigned long tmp; \
+ u64 ecycles = __delay_cycles() + \
+ NSECS_TO_CYCLES(timeout_ns); \
+ asm volatile( \
+ " sevl\n" \
+ " wfe\n" \
+ " ldxr" #sfx "\t%" #w "[tmp], %[v]\n" \
+ " eor %" #w "[tmp], %" #w "[tmp], %" #w "[val]\n" \
+ " cbnz %" #w "[tmp], 2f\n" \
+ " msr s0_3_c1_c0_0, %[ecycles]\n" \
+ "2:" \
+ : [tmp] "=&r" (tmp), [v] "+Q" (*(u##sz *)ptr) \
+ : [val] "r" (val), [ecycles] "r" (ecycles)); \
+}
+
+__CMPWAIT_TIMEOUT_CASE(w, b, 8);
+__CMPWAIT_TIMEOUT_CASE(w, h, 16);
+__CMPWAIT_TIMEOUT_CASE(w, , 32);
+__CMPWAIT_TIMEOUT_CASE( , , 64);
+
+#undef __CMPWAIT_TIMEOUT_CASE
+
+#define __CMPWAIT_GEN(timeout, sfx) \
+static __always_inline void __cmpwait##timeout##sfx(volatile void *ptr, \
+ unsigned long val, \
+ u64 timeout_ns, \
+ int size) \
{ \
switch (size) { \
case 1: \
- return __cmpwait_case##sfx##_8(ptr, (u8)val); \
+ return __cmpwait_case##timeout##sfx##_8(ptr, (u8)val, \
+ timeout_ns); \
case 2: \
- return __cmpwait_case##sfx##_16(ptr, (u16)val); \
+ return __cmpwait_case##timeout##sfx##_16(ptr, (u16)val, \
+ timeout_ns); \
case 4: \
- return __cmpwait_case##sfx##_32(ptr, val); \
+ return __cmpwait_case##timeout##sfx##_32(ptr, val, \
+ timeout_ns); \
case 8: \
- return __cmpwait_case##sfx##_64(ptr, val); \
+ return __cmpwait_case##timeout##sfx##_64(ptr, val, \
+ timeout_ns); \
default: \
BUILD_BUG(); \
} \
@@ -256,11 +290,15 @@ static __always_inline void __cmpwait##sfx(volatile void *ptr, \
unreachable(); \
}
-__CMPWAIT_GEN()
+__CMPWAIT_GEN( , )
+__CMPWAIT_GEN(_timeout, )
#undef __CMPWAIT_GEN
-#define __cmpwait_relaxed(ptr, val) \
- __cmpwait((ptr), (unsigned long)(val), sizeof(*(ptr)))
+#define __cmpwait_relaxed_timeout(ptr, val, timeout_ns) \
+ __cmpwait_timeout((ptr), (unsigned long)(val), timeout_ns, sizeof(*(ptr)))
+
+#define __cmpwait_relaxed(ptr, val) \
+ __cmpwait((ptr), (unsigned long)(val), 0, sizeof(*(ptr)))
#endif /* __ASM_CMPXCHG_H */
--
2.31.1
^ permalink raw reply related
* [PATCH v12 06/15] asm-generic: barrier: Add smp_cond_load_acquire_timeout()
From: Ankur Arora @ 2026-06-08 8:04 UTC (permalink / raw)
To: linux-kernel, linux-arch, linux-arm-kernel, linux-pm, bpf
Cc: arnd, catalin.marinas, will, peterz, akpm, mark.rutland, harisokn,
cl, ast, rafael, daniel.lezcano, memxor, zhenglifeng1, xueshuai,
rdunlap, david.laight.linux, broonie, joao.m.martins,
boris.ostrovsky, konrad.wilk, ashok.bhat, Ankur Arora
In-Reply-To: <20260608080440.127491-1-ankur.a.arora@oracle.com>
Add the acquire variant of smp_cond_load_relaxed_timeout(). This
reuses the relaxed variant, with additional LOAD->LOAD ordering
via smp_acquire__after_ctrl_dep().
To ensure that the necessary control dependency on the dereference
of @ptr exists (which does not in the timeout path), introduce
an empty evaluation of the cond_expr branch.
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Will Deacon <will@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: linux-arch@vger.kernel.org
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Haris Okanovic <harisokn@amazon.com>
Tested-by: Haris Okanovic <harisokn@amazon.com>
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
Notes:
Sashiko notes [1] that there's a missed control dependency in the
timeout path. Fix that by forcing evaluation of the cond_expr branch.
Catalin, Haris: I've kept your R-by on this. Please let me know if
you aren't okay with this change.
[1] https://sashiko.dev/#/patchset/20260408122538.3610871-1-ankur.a.arora%40oracle.com
---
include/asm-generic/barrier.h | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/include/asm-generic/barrier.h b/include/asm-generic/barrier.h
index c56df9513a08..0ab26e98842c 100644
--- a/include/asm-generic/barrier.h
+++ b/include/asm-generic/barrier.h
@@ -342,6 +342,34 @@ do { \
})
#endif
+/**
+ * smp_cond_load_acquire_timeout() - (Spin) wait for cond with ACQUIRE ordering
+ * until a timeout expires.
+ * @ptr: pointer to the variable to wait on.
+ * @cond_expr: boolean expression to wait for.
+ * @time_expr_ns: monotonic expression that evaluates to time in ns or,
+ * on failure, returns a negative value.
+ * @timeout_ns: timeout value in ns
+ * (Both of the above are assumed to be compatible with s64.)
+ *
+ * Equivalent to using smp_cond_load_acquire() on the condition variable with
+ * a timeout.
+ */
+#ifndef smp_cond_load_acquire_timeout
+#define smp_cond_load_acquire_timeout(ptr, cond_expr, \
+ time_expr_ns, timeout_ns) \
+({ \
+ __unqual_scalar_typeof(*(ptr)) VAL; \
+ VAL = smp_cond_load_relaxed_timeout(ptr, cond_expr, \
+ time_expr_ns, \
+ timeout_ns); \
+ if (cond_expr) \
+ barrier(); \
+ smp_acquire__after_ctrl_dep(); \
+ (typeof(*(ptr)))VAL; \
+})
+#endif
+
/*
* pmem_wmb() ensures that all stores for which the modification
* are written to persistent storage by preceding instructions have
--
2.31.1
^ permalink raw reply related
* [PATCH v12 02/15] arm64: barrier: Support smp_cond_load_relaxed_timeout()
From: Ankur Arora @ 2026-06-08 8:04 UTC (permalink / raw)
To: linux-kernel, linux-arch, linux-arm-kernel, linux-pm, bpf
Cc: arnd, catalin.marinas, will, peterz, akpm, mark.rutland, harisokn,
cl, ast, rafael, daniel.lezcano, memxor, zhenglifeng1, xueshuai,
rdunlap, david.laight.linux, broonie, joao.m.martins,
boris.ostrovsky, konrad.wilk, ashok.bhat, Ankur Arora
In-Reply-To: <20260608080440.127491-1-ankur.a.arora@oracle.com>
Support waiting in smp_cond_load_relaxed_timeout() via
__cmpwait_relaxed(). To ensure that we wake from waiting in
WFE periodically and don't block forever if there are no stores
to ptr, this path is only used when the event-stream is enabled.
Note that when using __cmpwait_relaxed() we ignore the timeout
value, allowing an overshoot by up to the event-stream period.
And, in the unlikely event that the event-stream is unavailable,
fallback to spin-waiting.
Also set SMP_TIMEOUT_POLL_COUNT to 1 so we do the time-check in
each iteration of smp_cond_load_relaxed_timeout().
And finally define ARCH_HAS_CPU_RELAX to indicate that we have
an optimized implementation of cpu_poll_relax().
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Will Deacon <will@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: linux-arm-kernel@lists.infradead.org
Suggested-by: Will Deacon <will@kernel.org>
Acked-by: Will Deacon <will@kernel.org>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
Notes: Doesn't address sashiko comments [1]:
- event stream being disabled or broken: the event stream being
disabled is not a production use case; doesn't make sense to
add a fastpath check for this.
- arch_timer_evtstrm_available() isn't exported. On arm64, this
means that smp_cond_load_relaxed_timeout() won't be available
to modules: tight now the use of this interface (and others of
this ilk) is fairly limited. Not worth solving until there's a
use case.
[1] https://sashiko.dev/#/patchset/20260408122538.3610871-1-ankur.a.arora%40oracle.com
---
arch/arm64/Kconfig | 3 +++
arch/arm64/include/asm/barrier.h | 21 +++++++++++++++++++++
2 files changed, 24 insertions(+)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index fe60738e5943..fa676428ec3f 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -1606,6 +1606,9 @@ config ARCH_SUPPORTS_CRASH_DUMP
config ARCH_DEFAULT_CRASH_DUMP
def_bool y
+config ARCH_HAS_CPU_RELAX
+ def_bool y
+
config ARCH_HAS_GENERIC_CRASHKERNEL_RESERVATION
def_bool CRASH_RESERVE
diff --git a/arch/arm64/include/asm/barrier.h b/arch/arm64/include/asm/barrier.h
index 9495c4441a46..6190e178db51 100644
--- a/arch/arm64/include/asm/barrier.h
+++ b/arch/arm64/include/asm/barrier.h
@@ -12,6 +12,7 @@
#include <linux/kasan-checks.h>
#include <asm/alternative-macros.h>
+#include <asm/vdso/processor.h>
#define __nops(n) ".rept " #n "\nnop\n.endr\n"
#define nops(n) asm volatile(__nops(n))
@@ -219,6 +220,26 @@ do { \
(typeof(*ptr))VAL; \
})
+/* Re-declared here to avoid include dependency. */
+extern bool arch_timer_evtstrm_available(void);
+
+/*
+ * In the common case, cpu_poll_relax() sits waiting in __cmpwait_relaxed()
+ * for the ptr value to change.
+ *
+ * Since this period is reasonably long, choose SMP_TIMEOUT_POLL_COUNT
+ * to be 1, so smp_cond_load_{relaxed,acquire}_timeout() does a
+ * time-check in each iteration.
+ */
+#define SMP_TIMEOUT_POLL_COUNT 1
+
+#define cpu_poll_relax(ptr, val, timeout_ns) do { \
+ if (arch_timer_evtstrm_available()) \
+ __cmpwait_relaxed(ptr, val); \
+ else \
+ cpu_relax(); \
+} while (0)
+
#include <asm-generic/barrier.h>
#endif /* __ASSEMBLER__ */
--
2.31.1
^ permalink raw reply related
* Re: [PATCH v4 6/8] string: introduce memcpy_streaming() helpers
From: Borislav Petkov @ 2026-06-07 19:08 UTC (permalink / raw)
To: Li Zhe
Cc: akpm, apopple, arnd, dave.hansen, david, kees, mingo, rppt, tglx,
linux-arch, linux-hardening, linux-kernel, linux-mm, x86
In-Reply-To: <20260603080152.64728-7-lizhe.67@bytedance.com>
On Wed, Jun 03, 2026 at 04:01:50PM +0800, Li Zhe wrote:
> Introduce a generic memcpy_streaming() interface for write-once copy
> sites that can fall back to memcpy() when no architecture-specific
> optimization is available, or when an architecture-specific backend
> cannot safely handle a given transfer.
>
> Add memcpy_streaming_drain() alongside it so callers can separate the
> copy primitive from any required ordering point. On x86, use
> memcpy_flushcache() and sfence only for aligned transfers that can stay
> entirely on the non-temporal store path; otherwise fall back to memcpy()
So you throwing "streaming", "non-temporal" and "flush-cache" wildly around
here and this is adding unnecessary confusion where it shouldn't. I'd suggest
you stick to "non-temporal" which you can abbreviate short'n'sweet to "nt" and
that's it. Keep it simple.
> so the generic API does not expose flushcache semantics on cached
> head/tail fragments.
>
> Callers are responsible for invoking memcpy_streaming_drain() before
> later normal stores that must be ordered after the streaming copy.
>
> Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
> ---
> arch/x86/include/asm/string_64.h | 32 ++++++++++++++++++++++++++++++++
> include/linux/string.h | 20 ++++++++++++++++++++
> 2 files changed, 52 insertions(+)
>
> diff --git a/arch/x86/include/asm/string_64.h b/arch/x86/include/asm/string_64.h
> index 4635616863f5..aee63108577f 100644
> --- a/arch/x86/include/asm/string_64.h
> +++ b/arch/x86/include/asm/string_64.h
There's arch/x86/include/asm/string.h. Why are those here, in the _64 variant?
> @@ -4,6 +4,7 @@
>
> #ifdef __KERNEL__
> #include <linux/jump_label.h>
> +#include <linux/align.h>
>
> /* Written 2002 by Andi Kleen */
>
> @@ -100,6 +101,37 @@ static __always_inline void memcpy_flushcache(void *dst, const void *src, size_t
> }
> __memcpy_flushcache(dst, src, cnt);
> }
> +
> +/*
> + * Only map memcpy_streaming() to memcpy_flushcache() when the destination
> + * is already 8-byte aligned and the size can be handled without cached
> + * head/tail fragments in __memcpy_flushcache().
> + */
> +static __always_inline bool memcpy_flushcache_nt_safe(const void *dst,
> + size_t cnt)
This is checking alignment. Then call it that.
> +{
> + unsigned long d = (unsigned long)dst;
Useless.
> +
> + return cnt && IS_ALIGNED(d, 8) && IS_ALIGNED(cnt, 4);
> +}
AFAICT, this helper is used only once. Zap it completely.
> +
> +#define __HAVE_ARCH_MEMCPY_STREAMING 1
> +static __always_inline void memcpy_streaming(void *dst, const void *src,
memcpy_nt()
> + size_t cnt)
> +{
> + if (!cnt)
> + return;
> +
> + if (memcpy_flushcache_nt_safe(dst, cnt))
That branch can cost. Why is that alignment checking so necessary? Why can't
you simply DTRT by handling the misaligned parts like __memcpy_flushcache().
What does this bring you? None of that is explained in the commit message so
why do I want this patch at all?
The commit message is basically telling me what the patch does but I can kinda
read that from the diff itself. What it is not telling me is *why* it exists.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
^ permalink raw reply
* Re: [RFC PATCH v1 00/13] exec: add spawn templates for repeated executable startup
From: Li Chen @ 2026-06-07 13:22 UTC (permalink / raw)
To: Gabriel Krisman Bertazi
Cc: Christian Brauner, Kees Cook, Alexander Viro, linux-fsdevel,
linux-api, linux-kernel, linux-mm, linux-arch, linux-doc,
linux-kselftest, x86, Arnd Bergmann, Andy Lutomirski,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Jan Kara, Jonathan Corbet, Shuah Khan
In-Reply-To: <87fr31xdz3.fsf@mailhost.krisman.be>
Hi Gabriel,
Yes, I looked at Josh's slides and your RFC a few days ago.
I agree that io_uring is a very interesting direction, and I can see why it
fits the "ordered setup operations before exec" model.
My current preference is still to first explore a pidfd/pidfs-based builder,
modeled roughly like fsconfig(). Process creation feels like a core process
lifecycle API, and I think a normal fd-based syscall interface may be easier
for libc, language runtimes, shells,and sandboxing tools to adopt.
My hesitation is practical rather than conceptual.Some important
deployments still disable io_uring entirely; Docker's default seccomp
profile blocks the io_uring syscalls, and Google has disabled or restricted
io_uring in ChromeOS, Android app processes, and production servers.
I will study your io_uring work more carefully and compare the two directions.
One possible outcome is that io_uring can drive/share the same builder object later;
I do not know that yet.
Thanks for pointing this out.
---- On Fri, 05 Jun 2026 22:24:00 +0800 Gabriel Krisman Bertazi <krisman@suse.de> wrote ---
> Li Chen <me@linux.beauty> writes:
>
> > Hi,
> >
> > This is an early RFC for an idea that is probably still rough in both the
> > UAPI and implementation details. Sorry for the rough edges; I am sending
> > it now to check whether this direction is worth pursuing and to get
> > feedback on the kernel/userspace boundary.
> >
> > The series is based on linux-next version 20260518.
> >
> > This RFC adds spawn_template, a userspace-controlled exec acceleration
> > mechanism for runtimes that repeatedly start the same executable with
> > different argv, envp, and per-spawn file descriptor setup.
>
> Have you looked at Josh's proposal to do this over io_uring [1] and my
> implementation of it at [2]? I think io_uring is a very natural
> interface for something like this, it will avoid adding a larger API,
> since you could, in theory, set up the entire new task context using
> regular io_uring operations in an io workqueue and then starting it would
> be a matter of forking the pre-configured io thread with a new io_uring
> operation.
>
> [1]
> https://lpc.events/event/16/contributions/1213/attachments/1012/1945/io-uring-spawn.pdf
> [2] https://lwn.net/Articles/1001622/
>
> >
> > The main target is agent runtimes. Modern coding agents repeatedly start
> > short-lived helper tools such as rg, git, sed, awk, python, node, and
> > shell wrappers while they inspect and edit a workspace. Those runtimes
> > already know which tools are hot, and they are also the right place to
> > decide policy. The kernel does not choose names such as rg, git, or sed.
> > Userspace opts in by creating a template fd for one executable, then uses
> > that fd for later spawns. Launchers, shells, and build systems have a
> > similar repeated-startup shape and could use the same primitive, but the
> > agent runtime case is the main motivation for this RFC.
> >
> > The mechanism applies to the executable that userspace asks the kernel to
> > start. If an agent runtime directly starts /usr/bin/rg, the rg executable
> > is the template target. If the runtime starts /usr/bin/bash -c "rg ... |
> > head", the shell is the template target unless the shell itself opts in
> > when it starts rg and head. The kernel does not parse the shell command
> > string or rewrite inner commands into template spawns. Userspace has to
> > call spawn_template for those inner commands explicitly:
> >
> > direct exec shell wrapper
> > ----------- -------------
> > agent agent
> > template("/usr/bin/rg") template("/usr/bin/bash")
> > spawn rg argv spawn bash -c "rg ... | head"
> >
> > kernel target: rg kernel target: bash
> > rg startup benefits rg/head need shell opt-in
> >
> > Several agent runtime discussions are moving toward direct argv-style
> > exec tools for both security and policy clarity. For example, opencode
> > issue #2206 proposes an exec tool as a safer alternative to a shell-only
> > bash tool:
> >
> > https://github.com/anomalyco/opencode/issues/2206
> >
> > spawn_template is meant to support both models. Direct exec users can
> > cache the actual hot tool. Shell-wrapper users can cache the shell and
> > still reduce shell startup cost. If a shell or an agent runtime later
> > uses the same API for commands started inside a shell command, those
> > inner tools can benefit too.
> >
> > Each spawn still goes through the normal exec path. The template reuses
> > only metadata that can be revalidated before use. Credential preparation,
> > permission checks, binary handler checks, secure-exec handling, and LSM
> > hooks remain on the normal execve path.
> >
> > The UAPI has two operations. spawn_template_create() creates an
> > anonymous-inode template fd from either an executable fd or an absolute
> > executable path. spawn_template_spawn() starts one child from that
> > template, applies per-spawn fd, cwd, and signal actions, and returns both
> > pid and pidfd.
> >
> > fd inheritance is deliberately conservative. By default, after the
> > requested per-spawn actions have run, the child closes fds above stderr.
> > An agent runtime can still request traditional inheritance explicitly,
> > but helper tools do not inherit unrelated secret files or sockets by
> > accident. The create-time actions fields are reserved and rejected in
> > this RFC because fd numbers are per-process state, not stable reusable
> > objects. The caller supplies fd actions for each spawn instead.
> >
> > A typical agent runtime would keep one template per hot executable and
> > still build argv, envp, cwd, and pipe wiring for each tool call:
> >
> > rg_tmpl = spawn_template_create("/usr/bin/rg");
> >
> > for each search request:
> > out_r, out_w = pipe_cloexec();
> > err_r, err_w = pipe_cloexec();
> > actions = [
> > FCHDIR(worktree_fd),
> > DUP2(out_w, STDOUT_FILENO),
> > DUP2(err_w, STDERR_FILENO),
> > ];
> > child = spawn_template_spawn(rg_tmpl, rg_argv, envp, actions);
> > close(out_w);
> > close(err_w);
> > read out_r and err_r;
> > waitid(P_PIDFD, child.pidfd, ...);
> >
> > A shell-wrapper runtime would use the same shape with a template for
> > /usr/bin/bash and argv such as ["/usr/bin/bash", "-c", command]. That
> > reduces shell startup cost, but it does not cache rg or head inside that
> > command unless the shell also opts into spawn_template for commands it
> > starts internally.
> >
> > The template pins the executable and denies writes to that file while the
> > template fd is alive, so cached executable metadata cannot race with a
> > writer changing the same inode. This means direct in-place writes to the
> > executable can fail while a runtime keeps a template open. It does not
> > block the common package-manager update pattern where a new inode is
> > written and then atomically renamed over the old path. In that case the
> > old path-created template becomes stale, spawn_template_spawn() rejects
> > it with ESTALE, and the runtime should close and recreate the template
> > for the new executable.
> >
> > in-place write package-manager update
> > -------------- ----------------------
> > template pins old inode write new inode
> > write(old inode) denied rename(new, "/usr/bin/rg")
> >
> > cached metadata safe old template sees path mismatch
> > spawn_template_spawn() = -ESTALE
> > recreate template for new inode
> >
> > Each spawn revalidates executable identity before cached metadata is
> > used. Path-created templates only accept absolute paths: a relative path
> > such as ./tool depends on cwd, and the same string can name a different
> > file after chdir. For an absolute path template, each spawn reopens the
> > path and checks that it still resolves to the executable recorded when
> > the template was created. If the path now names a replaced file, the
> > template is stale and userspace should close and recreate it.
> >
> > A template fd can be passed over SCM_RIGHTS like any other fd, but this
> > RFC does not treat that as delegation. spawn_template_spawn() only works
> > while the caller still has the same struct cred object that created the
> > template. If another task, or the same task after a credential change,
> > receives the fd, spawn fails instead of running the executable using the
> > creator's launch authority:
> >
> > ordinary fd spawn_template fd
> > ----------- -----------------
> > A: open log A: create rg template
> > A -> B: SCM_RIGHTS(fd) A -> B: SCM_RIGHTS(tfd)
> >
> > B: read(fd) = ok B: spawn(tfd) = -EACCES
> > B: create own rg template
> > B: spawn(own_tfd) = ok
> >
> > open-file use is delegated spawn authority is not delegated
> >
> > The cached state is intentionally small. The template fd keeps the opened
> > main executable file, an optional absolute path string, the creator
> > credential pointer, and the deny-write state. The executable identity key
> > records device, inode, size, mode, owner, ctime, and mtime, and is
> > rechecked before cached metadata is used. The ELF cache keeps only the
> > main executable's ELF header, program header table, and program header
> > count.
> >
> > cached in this RFC not cached in this RFC
> > ------------------ ----------------------
> > opened main executable PT_INTERP metadata
> > executable identity key shared-library graph
> > main ELF header VMA layout metadata
> > main ELF program headers cross-process metadata sharing
> > creator cred pointer
> > deny-write state
> >
> > This RFC does not cache ELF interpreter metadata, shared-library
> > dependency state, or derived mapping-layout state. Shared-library
> > resolution is dynamic linker policy and depends on LD_LIBRARY_PATH,
> > RPATH, RUNPATH, /etc/ld.so.cache, mount namespaces, and secure-exec
> > state. It also does not share cached executable metadata between template
> > fds created by different processes. Each template owns its small cached
> > metadata object in this RFC.
> >
> > Performance
> > ===========
> >
> > The numbers below come from my separate local autogen-bench project.
> > autogen-bench uses AutoGen [1] Core as the agent harness: RoutedAgent
> > instances run under SingleThreadedAgentRuntime, and RPC-style dispatch
> > fans out concurrent tool-call requests to worker agents. The workload
> > definitions, generated test files, and subprocess/spawn_template backends
> > are local to autogen-bench.
> >
> > The agent-tools preset includes direct tool calls and shell-wrapper forms
> > for:
> >
> > rg, grep, sed, awk, cat, head, tail, find, stat, ls, git-status, git-diff,
> > python-small, node-small, sh-c, and bash-c.
> >
> > The benchmark is launch-heavy but not no-op: it searches generated
> > Python-like source files, reads sample files, runs small Python and
> > Node.js programs, and runs git status and git diff in a small repository.
> > It does not include model inference or long-running tool work, so the
> > numbers mainly describe the short-tool regime.
> >
> > The subprocess column starts each tool call through the existing
> > userspace launch path. The spawn_template column creates templates for
> > hot executables and uses spawn_template_spawn() for later calls.
> >
> > Total in-flight tool calls stay at 16; only the worker-process split
> > changes. For example, 4x4 means 4 worker processes with 4 in-flight tool
> > calls each. The two time_s values are subprocess/spawn_template wall
> > times.
> >
> > Workload Calls subprocess spawn_template time_s Delta
> > (workers) calls calls/s calls/s seconds
> > 1x16 6144 411.04 420.32 14.95/14.62 +2.26%
> > 2x8 6144 666.78 690.08 9.21/8.90 +3.49%
> > 4x4 6144 955.61 1003.25 6.43/6.12 +4.99%
> > 8x2 6144 1048.25 1069.18 5.86/5.75 +2.00%
> >
> > The table measures the whole mixed workload, including both process
> > startup and the short tool work done after exec. Since this workload is
> > launch-heavy, the possible launch-side savings include:
> >
> > - the template fd keeps an opened executable, avoiding repeated ordinary
> > open/path setup for that executable;
> > - the kernel can reuse cached main-executable ELF header and program
> > header metadata after revalidation;
> > - the fork-and-exec-style launch is submitted as one
> > spawn_template_spawn() operation;
> > - fd, cwd, and signal actions run in the child kernel path instead of
> > being driven one syscall at a time by userspace child glue;
> > - pid and pidfd are returned by the same operation, reducing some
> > runtime-side bookkeeping.
> >
> > In local experiments before this RFC, I also tried caching ELF
> > interpreter metadata and derived ELF mapping-layout metadata. A focused
> > repeated-exec benchmark did not show a stable standalone throughput gain
> > for those two optimizations, so this RFC leaves them out and keeps only
> > the main executable metadata cache.
> >
> > I also tried sharing main-executable ELF metadata across template fds
> > created by different processes for the same executable identity. That can
> > reduce duplicated metadata memory when many agent worker processes create
> > their own templates for /usr/bin/rg, /usr/bin/git, and similar tools, but
> > it did not show a stable throughput win in local multi-agent tests. It
> > also adds cache keying, lifetime, invalidation, credential, and namespace
> > questions to the RFC. This version therefore keeps per-template metadata
> > ownership and leaves cross-process sharing out.
> >
> > Sorry again for the rough edges in this RFC. I would appreciate feedback
> > on whether this direction is useful and what the right API boundary
> > should be.
> >
> > Thanks,
> > Li
> >
> > [1]: https://github.com/microsoft/autogen
> >
> > Li Chen (13):
> > exec: factor argument setup out of do_execveat_common()
> > exec: add an internal helper for opened executables
> > file: expose helpers for in-kernel fd actions
> > exec: add spawn template UAPI definitions
> > exec: add spawn template file descriptors
> > exec: add spawn_template_spawn()
> > exec: validate spawn template executable identity
> > binfmt_elf: cache ELF metadata for spawn templates
> > Documentation: describe spawn templates
> > exec: require absolute paths for path-created templates
> > exec: let close-range actions target the max fd
> > syscalls: add generic spawn template entries
> > selftests/exec: cover spawn template basics
> >
> > Documentation/userspace-api/index.rst | 1 +
> > .../userspace-api/spawn_template.rst | 153 +++
> > MAINTAINERS | 6 +
> > arch/x86/entry/syscalls/syscall_64.tbl | 3 +-
> > fs/Makefile | 2 +-
> > fs/binfmt_elf.c | 104 +-
> > fs/exec.c | 162 ++-
> > fs/file.c | 11 +-
> > fs/spawn_template.c | 619 +++++++++++
> > include/linux/binfmts.h | 10 +
> > include/linux/fdtable.h | 2 +
> > include/linux/spawn_template.h | 72 ++
> > include/linux/syscalls.h | 7 +
> > include/uapi/asm-generic/unistd.h | 7 +-
> > include/uapi/linux/spawn_template.h | 62 ++
> > scripts/syscall.tbl | 2 +
> > tools/testing/selftests/exec/Makefile | 1 +
> > tools/testing/selftests/exec/spawn_template.c | 997 ++++++++++++++++++
> > 18 files changed, 2179 insertions(+), 42 deletions(-)
> > create mode 100644 Documentation/userspace-api/spawn_template.rst
> > create mode 100644 fs/spawn_template.c
> > create mode 100644 include/linux/spawn_template.h
> > create mode 100644 include/uapi/linux/spawn_template.h
> > create mode 100644 tools/testing/selftests/exec/spawn_template.c
>
> --
> Gabriel Krisman Bertazi
>
Regards,
Li
^ permalink raw reply
* Re: [RFC PATCH v1 00/13] exec: add spawn templates for repeated executable startup
From: Gabriel Krisman Bertazi @ 2026-06-05 14:24 UTC (permalink / raw)
To: Li Chen
Cc: Christian Brauner, Kees Cook, Alexander Viro, linux-fsdevel,
linux-api, linux-kernel, linux-mm, linux-arch, linux-doc,
linux-kselftest, x86, Arnd Bergmann, Andy Lutomirski,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Jan Kara, Jonathan Corbet, Shuah Khan
In-Reply-To: <20260528095235.2491226-1-me@linux.beauty>
Li Chen <me@linux.beauty> writes:
> Hi,
>
> This is an early RFC for an idea that is probably still rough in both the
> UAPI and implementation details. Sorry for the rough edges; I am sending
> it now to check whether this direction is worth pursuing and to get
> feedback on the kernel/userspace boundary.
>
> The series is based on linux-next version 20260518.
>
> This RFC adds spawn_template, a userspace-controlled exec acceleration
> mechanism for runtimes that repeatedly start the same executable with
> different argv, envp, and per-spawn file descriptor setup.
Have you looked at Josh's proposal to do this over io_uring [1] and my
implementation of it at [2]? I think io_uring is a very natural
interface for something like this, it will avoid adding a larger API,
since you could, in theory, set up the entire new task context using
regular io_uring operations in an io workqueue and then starting it would
be a matter of forking the pre-configured io thread with a new io_uring
operation.
[1]
https://lpc.events/event/16/contributions/1213/attachments/1012/1945/io-uring-spawn.pdf
[2] https://lwn.net/Articles/1001622/
>
> The main target is agent runtimes. Modern coding agents repeatedly start
> short-lived helper tools such as rg, git, sed, awk, python, node, and
> shell wrappers while they inspect and edit a workspace. Those runtimes
> already know which tools are hot, and they are also the right place to
> decide policy. The kernel does not choose names such as rg, git, or sed.
> Userspace opts in by creating a template fd for one executable, then uses
> that fd for later spawns. Launchers, shells, and build systems have a
> similar repeated-startup shape and could use the same primitive, but the
> agent runtime case is the main motivation for this RFC.
>
> The mechanism applies to the executable that userspace asks the kernel to
> start. If an agent runtime directly starts /usr/bin/rg, the rg executable
> is the template target. If the runtime starts /usr/bin/bash -c "rg ... |
> head", the shell is the template target unless the shell itself opts in
> when it starts rg and head. The kernel does not parse the shell command
> string or rewrite inner commands into template spawns. Userspace has to
> call spawn_template for those inner commands explicitly:
>
> direct exec shell wrapper
> ----------- -------------
> agent agent
> template("/usr/bin/rg") template("/usr/bin/bash")
> spawn rg argv spawn bash -c "rg ... | head"
>
> kernel target: rg kernel target: bash
> rg startup benefits rg/head need shell opt-in
>
> Several agent runtime discussions are moving toward direct argv-style
> exec tools for both security and policy clarity. For example, opencode
> issue #2206 proposes an exec tool as a safer alternative to a shell-only
> bash tool:
>
> https://github.com/anomalyco/opencode/issues/2206
>
> spawn_template is meant to support both models. Direct exec users can
> cache the actual hot tool. Shell-wrapper users can cache the shell and
> still reduce shell startup cost. If a shell or an agent runtime later
> uses the same API for commands started inside a shell command, those
> inner tools can benefit too.
>
> Each spawn still goes through the normal exec path. The template reuses
> only metadata that can be revalidated before use. Credential preparation,
> permission checks, binary handler checks, secure-exec handling, and LSM
> hooks remain on the normal execve path.
>
> The UAPI has two operations. spawn_template_create() creates an
> anonymous-inode template fd from either an executable fd or an absolute
> executable path. spawn_template_spawn() starts one child from that
> template, applies per-spawn fd, cwd, and signal actions, and returns both
> pid and pidfd.
>
> fd inheritance is deliberately conservative. By default, after the
> requested per-spawn actions have run, the child closes fds above stderr.
> An agent runtime can still request traditional inheritance explicitly,
> but helper tools do not inherit unrelated secret files or sockets by
> accident. The create-time actions fields are reserved and rejected in
> this RFC because fd numbers are per-process state, not stable reusable
> objects. The caller supplies fd actions for each spawn instead.
>
> A typical agent runtime would keep one template per hot executable and
> still build argv, envp, cwd, and pipe wiring for each tool call:
>
> rg_tmpl = spawn_template_create("/usr/bin/rg");
>
> for each search request:
> out_r, out_w = pipe_cloexec();
> err_r, err_w = pipe_cloexec();
> actions = [
> FCHDIR(worktree_fd),
> DUP2(out_w, STDOUT_FILENO),
> DUP2(err_w, STDERR_FILENO),
> ];
> child = spawn_template_spawn(rg_tmpl, rg_argv, envp, actions);
> close(out_w);
> close(err_w);
> read out_r and err_r;
> waitid(P_PIDFD, child.pidfd, ...);
>
> A shell-wrapper runtime would use the same shape with a template for
> /usr/bin/bash and argv such as ["/usr/bin/bash", "-c", command]. That
> reduces shell startup cost, but it does not cache rg or head inside that
> command unless the shell also opts into spawn_template for commands it
> starts internally.
>
> The template pins the executable and denies writes to that file while the
> template fd is alive, so cached executable metadata cannot race with a
> writer changing the same inode. This means direct in-place writes to the
> executable can fail while a runtime keeps a template open. It does not
> block the common package-manager update pattern where a new inode is
> written and then atomically renamed over the old path. In that case the
> old path-created template becomes stale, spawn_template_spawn() rejects
> it with ESTALE, and the runtime should close and recreate the template
> for the new executable.
>
> in-place write package-manager update
> -------------- ----------------------
> template pins old inode write new inode
> write(old inode) denied rename(new, "/usr/bin/rg")
>
> cached metadata safe old template sees path mismatch
> spawn_template_spawn() = -ESTALE
> recreate template for new inode
>
> Each spawn revalidates executable identity before cached metadata is
> used. Path-created templates only accept absolute paths: a relative path
> such as ./tool depends on cwd, and the same string can name a different
> file after chdir. For an absolute path template, each spawn reopens the
> path and checks that it still resolves to the executable recorded when
> the template was created. If the path now names a replaced file, the
> template is stale and userspace should close and recreate it.
>
> A template fd can be passed over SCM_RIGHTS like any other fd, but this
> RFC does not treat that as delegation. spawn_template_spawn() only works
> while the caller still has the same struct cred object that created the
> template. If another task, or the same task after a credential change,
> receives the fd, spawn fails instead of running the executable using the
> creator's launch authority:
>
> ordinary fd spawn_template fd
> ----------- -----------------
> A: open log A: create rg template
> A -> B: SCM_RIGHTS(fd) A -> B: SCM_RIGHTS(tfd)
>
> B: read(fd) = ok B: spawn(tfd) = -EACCES
> B: create own rg template
> B: spawn(own_tfd) = ok
>
> open-file use is delegated spawn authority is not delegated
>
> The cached state is intentionally small. The template fd keeps the opened
> main executable file, an optional absolute path string, the creator
> credential pointer, and the deny-write state. The executable identity key
> records device, inode, size, mode, owner, ctime, and mtime, and is
> rechecked before cached metadata is used. The ELF cache keeps only the
> main executable's ELF header, program header table, and program header
> count.
>
> cached in this RFC not cached in this RFC
> ------------------ ----------------------
> opened main executable PT_INTERP metadata
> executable identity key shared-library graph
> main ELF header VMA layout metadata
> main ELF program headers cross-process metadata sharing
> creator cred pointer
> deny-write state
>
> This RFC does not cache ELF interpreter metadata, shared-library
> dependency state, or derived mapping-layout state. Shared-library
> resolution is dynamic linker policy and depends on LD_LIBRARY_PATH,
> RPATH, RUNPATH, /etc/ld.so.cache, mount namespaces, and secure-exec
> state. It also does not share cached executable metadata between template
> fds created by different processes. Each template owns its small cached
> metadata object in this RFC.
>
> Performance
> ===========
>
> The numbers below come from my separate local autogen-bench project.
> autogen-bench uses AutoGen [1] Core as the agent harness: RoutedAgent
> instances run under SingleThreadedAgentRuntime, and RPC-style dispatch
> fans out concurrent tool-call requests to worker agents. The workload
> definitions, generated test files, and subprocess/spawn_template backends
> are local to autogen-bench.
>
> The agent-tools preset includes direct tool calls and shell-wrapper forms
> for:
>
> rg, grep, sed, awk, cat, head, tail, find, stat, ls, git-status, git-diff,
> python-small, node-small, sh-c, and bash-c.
>
> The benchmark is launch-heavy but not no-op: it searches generated
> Python-like source files, reads sample files, runs small Python and
> Node.js programs, and runs git status and git diff in a small repository.
> It does not include model inference or long-running tool work, so the
> numbers mainly describe the short-tool regime.
>
> The subprocess column starts each tool call through the existing
> userspace launch path. The spawn_template column creates templates for
> hot executables and uses spawn_template_spawn() for later calls.
>
> Total in-flight tool calls stay at 16; only the worker-process split
> changes. For example, 4x4 means 4 worker processes with 4 in-flight tool
> calls each. The two time_s values are subprocess/spawn_template wall
> times.
>
> Workload Calls subprocess spawn_template time_s Delta
> (workers) calls calls/s calls/s seconds
> 1x16 6144 411.04 420.32 14.95/14.62 +2.26%
> 2x8 6144 666.78 690.08 9.21/8.90 +3.49%
> 4x4 6144 955.61 1003.25 6.43/6.12 +4.99%
> 8x2 6144 1048.25 1069.18 5.86/5.75 +2.00%
>
> The table measures the whole mixed workload, including both process
> startup and the short tool work done after exec. Since this workload is
> launch-heavy, the possible launch-side savings include:
>
> - the template fd keeps an opened executable, avoiding repeated ordinary
> open/path setup for that executable;
> - the kernel can reuse cached main-executable ELF header and program
> header metadata after revalidation;
> - the fork-and-exec-style launch is submitted as one
> spawn_template_spawn() operation;
> - fd, cwd, and signal actions run in the child kernel path instead of
> being driven one syscall at a time by userspace child glue;
> - pid and pidfd are returned by the same operation, reducing some
> runtime-side bookkeeping.
>
> In local experiments before this RFC, I also tried caching ELF
> interpreter metadata and derived ELF mapping-layout metadata. A focused
> repeated-exec benchmark did not show a stable standalone throughput gain
> for those two optimizations, so this RFC leaves them out and keeps only
> the main executable metadata cache.
>
> I also tried sharing main-executable ELF metadata across template fds
> created by different processes for the same executable identity. That can
> reduce duplicated metadata memory when many agent worker processes create
> their own templates for /usr/bin/rg, /usr/bin/git, and similar tools, but
> it did not show a stable throughput win in local multi-agent tests. It
> also adds cache keying, lifetime, invalidation, credential, and namespace
> questions to the RFC. This version therefore keeps per-template metadata
> ownership and leaves cross-process sharing out.
>
> Sorry again for the rough edges in this RFC. I would appreciate feedback
> on whether this direction is useful and what the right API boundary
> should be.
>
> Thanks,
> Li
>
> [1]: https://github.com/microsoft/autogen
>
> Li Chen (13):
> exec: factor argument setup out of do_execveat_common()
> exec: add an internal helper for opened executables
> file: expose helpers for in-kernel fd actions
> exec: add spawn template UAPI definitions
> exec: add spawn template file descriptors
> exec: add spawn_template_spawn()
> exec: validate spawn template executable identity
> binfmt_elf: cache ELF metadata for spawn templates
> Documentation: describe spawn templates
> exec: require absolute paths for path-created templates
> exec: let close-range actions target the max fd
> syscalls: add generic spawn template entries
> selftests/exec: cover spawn template basics
>
> Documentation/userspace-api/index.rst | 1 +
> .../userspace-api/spawn_template.rst | 153 +++
> MAINTAINERS | 6 +
> arch/x86/entry/syscalls/syscall_64.tbl | 3 +-
> fs/Makefile | 2 +-
> fs/binfmt_elf.c | 104 +-
> fs/exec.c | 162 ++-
> fs/file.c | 11 +-
> fs/spawn_template.c | 619 +++++++++++
> include/linux/binfmts.h | 10 +
> include/linux/fdtable.h | 2 +
> include/linux/spawn_template.h | 72 ++
> include/linux/syscalls.h | 7 +
> include/uapi/asm-generic/unistd.h | 7 +-
> include/uapi/linux/spawn_template.h | 62 ++
> scripts/syscall.tbl | 2 +
> tools/testing/selftests/exec/Makefile | 1 +
> tools/testing/selftests/exec/spawn_template.c | 997 ++++++++++++++++++
> 18 files changed, 2179 insertions(+), 42 deletions(-)
> create mode 100644 Documentation/userspace-api/spawn_template.rst
> create mode 100644 fs/spawn_template.c
> create mode 100644 include/linux/spawn_template.h
> create mode 100644 include/uapi/linux/spawn_template.h
> create mode 100644 tools/testing/selftests/exec/spawn_template.c
--
Gabriel Krisman Bertazi
^ permalink raw reply
* Re: [PATCH v4 0/8] mm: speed up ZONE_DEVICE memmap initialization
From: Li Zhe @ 2026-06-05 9:52 UTC (permalink / raw)
To: apopple
Cc: akpm, arnd, bp, dave.hansen, david, kees, linux-arch,
linux-hardening, linux-kernel, linux-mm, lizhe.67, mingo, rppt,
tglx, x86
In-Reply-To: <aiEoByaQdRR3xtM5@nvdebian.thelocal>
On Thu, 4 Jun 2026 18:14:05 +1000, apopple@nvidia.com wrote:
> On 2026-06-03 at 18:01 +1000, Li Zhe <lizhe.67@bytedance.com> wrote...
> > memmap_init_zone_device() can spend a substantial amount of time
> > initializing large ZONE_DEVICE ranges because it repeats nearly
> > identical struct page setup for every PFN.
> >
> > This series reduces that overhead in eight steps.
> >
> > The first patch fixes a stale comment in __init_zone_device_page() so
> > the documented refcount policy matches the current ZONE_DEVICE code.
> >
> > The second patch factors the reusable pieces out of
> > __init_zone_device_page() so later patches can share the same logic
> > without changing the existing slow path.
> >
> > The third patch adds set_page_section_from_pfn(), so callers that want
> > to refresh section bits from a PFN no longer need to open-code
> > SECTION_IN_PAGE_FLAGS handling.
> >
> > The fourth patch adds a template-based fast path for ZONE_DEVICE head
> > pages. Instead of rebuilding the same struct page state for every PFN,
> > it prepares one reusable template through the existing slow path,
> > refreshes the PFN-dependent fields in that template, and copies it to
> > each destination page.
> >
> > The fifth patch extends the same template-based approach to compound
> > tails, so pfns_per_compound > 1 can also benefit from the fast path.
> >
> > The sixth patch introduces memcpy_streaming() and
> > memcpy_streaming_drain() as a generic interface for write-once copies.
> > Architectures that do not provide a specialized backend, or cases that
> > cannot safely use one, fall back to memcpy().
> >
> > The seventh patch extends x86 memcpy_flushcache() small fixed-size
> > fastpaths so struct-page-sized streaming copies can stay on the inline
> > path when alignment permits.
> >
> > The last patch switches the ZONE_DEVICE template-copy path over to
> > memcpy_streaming(). It keeps pageblock-aligned PFNs on regular memcpy(),
> > uses memcpy_streaming() for the remaining write-once copies, and drains
> > streaming stores before later metadata updates that may depend on them.
> >
> > This is not intended as a steady-state data-path optimization. Its
> > benefit is in pmem bring-up paths where memmap_init_zone_device()
> > dominates device online / rebind latency, such as:
> > - fsdax or devdax namespace creation and reconfiguration
> > - nd_pmem / dax_pmem driver bind or rebind
> >
> > In those paths, the kernel initializes a large vmemmap range once and
> > does not immediately benefit from keeping the copied struct page state
> > hot in cache. Reducing write-allocate traffic in that one-time setup
> > path can therefore reduce end-to-end device bring-up latency.
> >
> > The optimized path is disabled when the page_ref_set tracepoint is
> > enabled, and sanitized builds remain on the slow path so their
> > instrumented stores are preserved.
> >
> > Testing
> > =======
> >
> > Tests were run in a VM on an Intel Ice Lake server.
> >
> > Two PMEM configurations were used:
> > - a 100 GB fsdax namespace configured with map=dev, which exercises
> > the nd_pmem rebind path (pfns_per_compound == 1)
> > - a 100 GB devdax namespace configured with align=2097152, which
> > exercises the dax_pmem rebind path (pfns_per_compound > 1)
> >
> > For each configuration, the corresponding driver was unbound and
> > rebound 30 times. Memmap initialization latency was collected from the
> > pr_debug() output of memmap_init_zone_device().
> >
> > The first bind is reported separately, and the average of subsequent
> > rebinds is used as the steady-state result.
> >
> > Performance
> > ===========
> >
> > nd_pmem rebind, 100 GB fsdax namespace, map=dev
> > Base(v7.1-rc6):
> > First binding: 1466 ms
> > Average of subsequent rebinds: 262.12 ms
> > Full series:
> > First binding: 1359 ms
> > Average of subsequent rebinds: 108.36 ms
> >
> > dax_pmem rebind, 100 GB devdax namespace, align=2097152
> > Base(v7.1-rc6):
> > First binding: 1430 ms
> > Average of subsequent rebinds: 229.12 ms
> > Full series:
> > First binding: 1273 ms
> > Average of subsequent rebinds: 100.17 ms
>
> The results here are impressive, but I've been having trouble replicating them
> with hmm_test on my local development machines. Both an older AMD machine and
> a newer Arrow Lake based machine shows ~3% worse performance with this series
> applied doing ZONE_DEVICE_PRIVATE.
>
> This is based on measuring the memremap_pages() call when inserting test_hmm.ko
> in a VM using the following hack to measure 10 64GB memremaps. Is there an easy
> way for me to replicate your results in a VM? Or is there something in my
> testing that I'm missing here?
>
> ---
>
> diff --git a/lib/test_hmm.c b/lib/test_hmm.c
> index 213504915737..a1d5463dbc86 100644
> --- a/lib/test_hmm.c
> +++ b/lib/test_hmm.c
> @@ -34,7 +34,7 @@
>
> #define DMIRROR_NDEVICES 4
> #define DMIRROR_RANGE_FAULT_TIMEOUT 1000
> -#define DEVMEM_CHUNK_SIZE (256 * 1024 * 1024U)
> +#define DEVMEM_CHUNK_SIZE (64 * 1024 * 1024 * 1024UL)
> #define DEVMEM_CHUNKS_RESERVE 16
>
> /*
> @@ -565,6 +565,8 @@ static int dmirror_allocate_chunk(struct dmirror_device *mdevice,
> unsigned long pfn_last;
> void *ptr;
> int ret = -ENOMEM;
> + int i;
> + u64 t0, total = 0;
>
> devmem = kzalloc_obj(*devmem);
> if (!devmem)
> @@ -613,6 +615,22 @@ static int dmirror_allocate_chunk(struct dmirror_device *mdevice,
> mdevice->devmem_capacity = new_capacity;
> mdevice->devmem_chunks = new_chunks;
> }
> +
> + for (i = 0; i < 10; i++) {
> + t0 = ktime_get_ns();
> + ptr = memremap_pages(&devmem->pagemap, numa_node_id());
> + total += ktime_get_ns() - t0;
> + if (IS_ERR_OR_NULL(ptr)) {
> + if (ptr)
> + ret = PTR_ERR(ptr);
> + else
> + ret = -EFAULT;
> + goto err_release;
> + }
> + memunmap_pages(&devmem->pagemap);
> + }
> + pr_info("avg memremap %llu ns\n", total / i);
> +
> ptr = memremap_pages(&devmem->pagemap, numa_node_id());
> if (IS_ERR_OR_NULL(ptr)) {
> if (ptr)
> @@ -629,7 +647,7 @@ static int dmirror_allocate_chunk(struct dmirror_device *mdevice,
>
> mutex_unlock(&mdevice->devmem_lock);
>
> - pr_info("added new %u MB chunk (total %u chunks, %u MB) PFNs [0x%lx 0x%lx)\n",
> + pr_info("added new %lu MB chunk (total %u chunks, %lu MB) PFNs [0x%lx 0x%lx)\n",
> DEVMEM_CHUNK_SIZE / (1024 * 1024),
> mdevice->devmem_count,
> mdevice->devmem_count * (DEVMEM_CHUNK_SIZE / (1024 * 1024)),
Thanks for the feedback and for sharing your test results.
I reran the measurements on my side using two setups. I do not
currently have access to physical PMEM hardware on my side, and the
target use case for this work is a virtualized environment. So my
measurements were taken in a guest using a 100G emulated pmem device
backed by a regular file on the host filesystem.
First, I followed your modified test_hmm.c approach, i.e. looping
over memremap_pages() / memunmap_pages() and measuring the average
memremap time in the MEMORY_DEVICE_PRIVATE case, where the vmemmap
backing comes from normal system RAM. On this setup, I got:
- base kernel: avg memremap 222.0 ms
- patches 1-5 only: avg memremap 206.9 ms
- full 8-patch series: avg memremap 264.1 ms
I also enabled the pr_debug() timing inside memmap_init_zone_device()
for the same setup, and the numbers tracked that closely:
- base kernel: 221.0 ms
- patches 1-5 only: 206.0 ms
- full 8-patch series: 260.1 ms
So on this path, patches 1-5 seem to help, but the full 8-patch series
does not.
Second, I also tested a benchmark-only setup corresponding to the
FS_DAX map=dev case, where the memmap itself is allocated from the dax
altmap range rather than normal DRAM. On that setup, I got:
- base kernel: avg memremap 200.8 ms, pr_debug 196.4 ms
- full 8-patch series: avg memremap 117.2 ms, pr_debug 113.5 ms
So on my side, the full series shows a clear gain in the
FS_DAX + altmap case, but not in the MEMORY_DEVICE_PRIVATE / DRAM-backed
vmemmap case.
If convenient, could you also try the same kind of measurement from my
cover letter, or at least enable the pr_debug() in
memmap_init_zone_device(), to check whether the delta is visible there
on your setup as well?
Also, if you have time, could you please try your modified test_hmm.c
setup with patches 1-5 only? On my side that configuration still shows
a measurable improvement.
Given these results, I would also appreciate your advice on how best
to evolve the series. My current understanding is that patches 1-5 are
a more generic optimization, while patches 6-8 are only beneficial in
some cases. Do you think patches 1-5 alone would already be a
reasonable candidate for upstreaming?
For patches 6-8, I am not yet sure what the right direction is. Would
it make more sense to expose some explicit opt-in mechanism so that
the movnti-based path is selected only when desired, or does it make
more sense to use that path unconditionally for FS_DAX map=dev case?
I would also be interested in your view on why the FS_DAX + altmap
case shows a large gain while the DRAM-backed vmemmap case shows a
regression with the full series. I do not think I fully understand
that difference yet.
Thanks,
Zhe
^ permalink raw reply
* Re: [PATCH v3 03/13] preempt: Introduce __preempt_count_{sub, add}_return()
From: Boqun Feng @ 2026-06-05 6:45 UTC (permalink / raw)
To: bot+bpf-ci
Cc: peterz, catalin.marinas, will, jonas, stefan.kristiansson, shorne,
hca, gor, agordeev, borntraeger, svens, tglx, mingo, bp,
dave.hansen, x86, hpa, arnd, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, longman, akpm, andrii, eddyz87, ast, daniel,
martin.lau, memxor, song, yonghong.song, jolsa, shuah, ojeda,
gary, bjorn3_gh, lossin, a.hindborg, aliceryhl, tmgross, dakr,
ruanjinjie, lyude, thuth, sohil.mehta, pawan.kumar.gupta, seanjc,
nikunj, xin, joelagnelf, andriy.shevchenko, rdunlap, ynorov,
bigeasy, linux-kernel, linux-openrisc, linux-s390, linux-arch,
bpf, linux-kselftest, rust-for-linux, boqun.feng, martin.lau, clm,
ihor.solodrai
In-Reply-To: <d3e91d227b0e1b777a85806e54b03d46581e4460f893e5b45baa9168f5319fbf@mail.kernel.org>
On Fri, Jun 05, 2026 at 06:30:14AM +0000, bot+bpf-ci@kernel.org wrote:
> > preempt: Introduce __preempt_count_{sub, add}_return()
> >
> > In order to use preempt_count() to tracking the interrupt disable
> > nesting level, __preempt_count_{add,sub}_return() are introduced, as
> > their name suggest, these primitives return the new value of the
> > preempt_count() after changing it. The following example shows the usage
> > of it in local_interrupt_disable():
> >
> > // increase the HARDIRQ_DISABLE bit
> > new_count = __preempt_count_add_return(HARDIRQ_DISABLE_OFFSET);
> >
> > // if it's the first-time increment, then disable the interrupt
> > // at hardware level.
> > if (new_count & HARDIRQ_DISABLE_MASK == HARDIRQ_DISABLE_OFFSET) {
> ^^
>
> The operator precedence in this example appears incorrect. In C, the equality
> operator == has higher precedence than the bitwise AND operator &, so this
> evaluates as:
>
> new_count & (HARDIRQ_DISABLE_MASK == HARDIRQ_DISABLE_OFFSET)
>
> Since the mask does not equal the offset, the equality evaluates to 0, making
> the entire condition always false.
>
> Should this be written with explicit parentheses to prevent readers from
> copying flawed logic into downstream implementations?
>
Will fix this but this code is in commit log rather than real code.
Regards,
Boqun
> if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET) {
>
> This was raised by sashiko-bot@kernel.org in v3 review:
> https://lore.kernel.org/all/20260605055933.EDF3D1F00898@smtp.kernel.org/
>
>
> ---
> AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
> See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
>
> CI run summary: https://github.com/kernel-patches/bpf/actions/runs/26998319662
^ permalink raw reply
* Re: [PATCH v3 05/13] irq & spin_lock: Add counted interrupt disabling/enabling
From: Boqun Feng @ 2026-06-05 6:40 UTC (permalink / raw)
To: bot+bpf-ci
Cc: peterz, catalin.marinas, will, jonas, stefan.kristiansson, shorne,
hca, gor, agordeev, borntraeger, svens, tglx, mingo, bp,
dave.hansen, x86, hpa, arnd, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, longman, akpm, andrii, eddyz87, ast, daniel,
martin.lau, memxor, song, yonghong.song, jolsa, shuah, ojeda,
gary, bjorn3_gh, lossin, a.hindborg, aliceryhl, tmgross, dakr,
ruanjinjie, lyude, thuth, sohil.mehta, pawan.kumar.gupta, seanjc,
nikunj, xin, joelagnelf, andriy.shevchenko, rdunlap, ynorov,
bigeasy, linux-kernel, linux-openrisc, linux-s390, linux-arch,
bpf, linux-kselftest, rust-for-linux, boqun.feng, martin.lau, clm,
ihor.solodrai
In-Reply-To: <cc4b5fb3f63b8b508334edd883765bf81c9ab1d2045f414a74c8712c6cc705e9@mail.kernel.org>
On Fri, Jun 05, 2026 at 06:30:16AM +0000, bot+bpf-ci@kernel.org wrote:
> > diff --git a/include/linux/interrupt_rc.h b/include/linux/interrupt_rc.h
> > new file mode 100644
> > index 000000000000..dd4444c61330
> > --- /dev/null
> > +++ b/include/linux/interrupt_rc.h
> > @@ -0,0 +1,67 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/*
> > + * include/linux/interrupt_rc.h - refcounted local processor interrupt
> > + * management.
> > + *
> > + * Since the implementation of this API currently depends on
> > + * local_irq_save()/local_irq_restore(), we split this into it's own header to
> > + * make it easier to include without hitting circular header dependencies.
> > + */
> > +
> > +#ifndef __LINUX_INTERRUPT_RC_H
> > +#define __LINUX_INTERRUPT_RC_H
> > +
> > +#include <linux/irqflags.h>
> > +#include <asm/processor.h>
> > +#ifdef CONFIG_SMP
> > +#include <asm/smp.h>
> > +#endif
> > +
> > +/* Per-cpu interrupt disabling state for local_interrupt_{disable,enable}() */
> > +struct interrupt_disable_state {
> > + unsigned long flags;
> > +};
> > +
> > +DECLARE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_state);
> > +
> > +static inline void local_interrupt_disable(void)
> > +{
> > + unsigned long flags;
> > + int new_count;
> > +
> > + WARN_ON_ONCE(in_nmi());
> > +
> > + new_count = hardirq_disable_enter();
> > +
> > + /* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */
> > +
> > + if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET) {
> > + local_irq_save(flags);
> > + raw_cpu_write(local_interrupt_disable_state.flags, flags);
> > + }
> > +}
> > +
> > +static inline void local_interrupt_enable(void)
> > +{
> > + int new_count;
> > +
> > + new_count = hardirq_disable_exit();
> > +
> > + if ((new_count & HARDIRQ_DISABLE_MASK) == 0) {
> > + unsigned long flags;
> > +
> > + flags = raw_cpu_read(local_interrupt_disable_state.flags);
> > + local_irq_restore(flags);
>
> Could this lead to indefinite softirq starvation? When __irq_exit_rcu()
Yes and no. There is no behavioral difference between here and
local_irq_enable(), neither will trigger softirqs.
> skips invoke_softirq() due to hardirq_disable_count() > 0, any scheduled
> softirqs (timers, RCU callbacks, networking) remain pending. When the
> outermost critical section ends and local_interrupt_enable() restores
> hardware interrupts, it doesn't check local_softirq_pending() to flush
> these deferred softirqs.
>
> Will these pending softirqs stall until another unrelated hardware
> interrupt fires? In low-activity periods, this could potentially cause
> RCU stalls or network hangs. This concern was raised by the sashiko bot in
> both v2 and v3 but remains unaddressed:
>
> https://lore.kernel.org/all/20260526175409.7CFCC1F000E9@smtp.kernel.org/
>
> > + /*
> > + * TODO: re-read preempt count can be avoided, but it needs
> > + * should_resched() taking another parameter as the current
> > + * preempt count
> > + */
> > +#ifdef CONFIG_PREEMPTION
> > + if (should_resched(0))
> > + __preempt_schedule();
>
> Peter Zijlstra suggested in v2 replacing should_resched(0) with a direct
> check of new_count for preemption. His suggestion was "can't you simply
> write: if (!new_count) __preempt_schedule();" The author acknowledged this
> in discussion but hasn't implemented it in v3:
>
> https://lore.kernel.org/all/20260528104322.GF343181@noisy.programming.kicks-ass.net/
>
> Is there a reason to keep the more complex should_resched(0) approach, or
> would the direct check be more appropriate here?
>
I have an optimization/cleanup patch in patch #13.
Regards,
Boqun
> > +#endif
> > + }
> > +}
> > +
> > +#endif /* !__LINUX_INTERRUPT_RC_H */
>
> [ ... ]
>
> > diff --git a/kernel/softirq.c b/kernel/softirq.c
> > index 10af5ed859e7..d1ab1799794c 100644
> > --- a/kernel/softirq.c
> > +++ b/kernel/softirq.c
> > @@ -88,6 +88,9 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled);
> > EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context);
> > #endif
> >
> > +DEFINE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_state);
> > +EXPORT_PER_CPU_SYMBOL_GPL(local_interrupt_disable_state);
> > +
> > DEFINE_PER_CPU(unsigned int, nmi_nesting);
> >
> > /*
> > @@ -728,7 +731,16 @@ static inline void __irq_exit_rcu(void)
> > #endif
> > account_hardirq_exit(current);
> > preempt_count_sub(HARDIRQ_OFFSET);
> > - if (!in_interrupt() && local_softirq_pending()) {
> > + /*
> > + * Interrupts may happen between hardirq_disable_enter() and
> > + * local_irq_save() in local_interrupt_disable(), if irq_exit() invokes
> > + * softirq here, we may have a softirq handler calling
> > + * local_interrupt_disable() but it won't disable the irq because
> > + * hardirq disabling count is already 1, hence we need to prevent
> > + * invoking softirq when a local_interrupt_disable() is ongoing.
> > + */
> > + if (!in_interrupt() && !hardirq_disable_count() &&
> > + local_softirq_pending()) {
> > /*
> > * If we left hrtimers unarmed, make sure to arm them now,
> > * before enabling interrupts to run SoftIRQ.
>
>
> ---
> AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
> See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
>
> CI run summary: https://github.com/kernel-patches/bpf/actions/runs/26998319662
^ permalink raw reply
* Re: [PATCH v3 10/13] preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITS
From: bot+bpf-ci @ 2026-06-05 6:30 UTC (permalink / raw)
To: boqun, peterz
Cc: catalin.marinas, will, jonas, stefan.kristiansson, shorne, hca,
gor, agordeev, borntraeger, svens, tglx, mingo, bp, dave.hansen,
x86, hpa, arnd, juri.lelli, vincent.guittot, dietmar.eggemann,
rostedt, bsegall, mgorman, vschneid, kprateek.nayak, boqun,
longman, akpm, andrii, eddyz87, ast, daniel, martin.lau, memxor,
song, yonghong.song, jolsa, shuah, ojeda, gary, bjorn3_gh, lossin,
a.hindborg, aliceryhl, tmgross, dakr, ruanjinjie, lyude, thuth,
sohil.mehta, pawan.kumar.gupta, seanjc, nikunj, xin, joelagnelf,
andriy.shevchenko, rdunlap, ynorov, bigeasy, linux-kernel,
linux-openrisc, linux-s390, linux-arch, bpf, linux-kselftest,
rust-for-linux, ast, andrii, daniel, martin.lau, eddyz87,
yonghong.song, clm, ihor.solodrai
In-Reply-To: <20260605054128.5925-11-boqun@kernel.org>
[-- Attachment #1: Type: text/plain, Size: 3239 bytes --]
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index f3f7cb01d69d..bf8288b3d52b 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -327,6 +327,7 @@ config X86
> select USER_STACKTRACE_SUPPORT
> select HAVE_ARCH_KCSAN if X86_64
> select PROC_PID_ARCH_STATUS if PROC_FS
> + select HAS_SEPARATE_PREEMPT_RESCHED_BITS if X86_64
> select HAVE_ARCH_NODE_DEV_GROUP if X86_SGX
A subsystem pattern flags this as potentially concerning: selecting
HAS_SEPARATE_PREEMPT_RESCHED_BITS with only the X86_64 condition may
bypass the PREEMPT_COUNT dependency.
Looking at the definition in kernel/Kconfig.preempt, the config symbol
HAS_SEPARATE_PREEMPT_RESCHED_BITS explicitly depends on both PREEMPT_COUNT
and 64BIT:
kernel/Kconfig.preempt:
config HAS_SEPARATE_PREEMPT_RESCHED_BITS
bool
depends on PREEMPT_COUNT && 64BIT
PREEMPT_COUNT is not always enabled. It gets selected by PREEMPTION, and
by a few debug configs (CONFIG_DEBUG_PREEMPT, DEBUG_ATOMIC_SLEEP,
PREEMPT_TRACER, etc.), but a standard CONFIG_PREEMPT_NONE=y configuration
without any of those debug options will have PREEMPT_COUNT=n.
In such a configuration, the select statement (gated only by X86_64)
will force HAS_SEPARATE_PREEMPT_RESCHED_BITS=y while its depends on
PREEMPT_COUNT is unmet. This will produce a Kconfig unmet-dependency
warning: "WARNING: unmet direct dependencies detected for
HAS_SEPARATE_PREEMPT_RESCHED_BITS / Depends on [n]: PREEMPT_COUNT &&
64BIT / Selected by [y]: - X86 [=y]".
Sashiko-bot raised this concern in both v2 and v3 of this patch (message
IDs <20260526195703.C90E31F000E9@smtp.kernel.org> and
<20260605055956.46EC81F00893@smtp.kernel.org>), escalating it from Low to
Medium severity:
https://lore.kernel.org/all/20260605055956.46EC81F00893@smtp.kernel.org/
Should this select be conditional on PREEMPT_COUNT? For example:
select HAS_SEPARATE_PREEMPT_RESCHED_BITS if X86_64 && PREEMPT_COUNT
[ ... ]
> diff --git a/kernel/Kconfig.preempt b/kernel/Kconfig.preempt
> index 88c594c6d7fc..35f546a042b1 100644
> --- a/kernel/Kconfig.preempt
> +++ b/kernel/Kconfig.preempt
> @@ -122,6 +122,10 @@ config PREEMPT_RT_NEEDS_BH_LOCK
> config PREEMPT_COUNT
> bool
>
> +config HAS_SEPARATE_PREEMPT_RESCHED_BITS
> + bool
> + depends on PREEMPT_COUNT && 64BIT
> +
This commit fixes a bug but lacks a Fixes: tag. The commit corrects
underflow detection logic in preempt_count_sub() by changing from a signed
comparison to unsigned wraparound detection. The bug was introduced when
commit 7c346d37f86b expanded the preempt_count bit layout by adding 8-bit
HARDIRQ_DISABLE_BITS, increasing total bit usage from 21 to 29 bits.
This expansion highlighted the flaw in the existing signed comparison
underflow check (val > preempt_count()), which doesn't correctly detect
unsigned underflow. The fix properly uses unsigned arithmetic (pc - uval >
pc) to detect wraparound.
Should this include:
Fixes: 7c346d37f86b ("preempt: Introduce HARDIRQ_DISABLE_BITS")
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/26998319662
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox