* [PATCH v14 01/16] preempt: Introduce HARDIRQ_DISABLE_BITS
2025-11-20 21:45 [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust Lyude Paul
@ 2025-11-20 21:45 ` Lyude Paul
2025-11-20 21:45 ` [PATCH v14 02/16] preempt: Track NMI nesting to separate per-CPU counter Lyude Paul
` (15 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Lyude Paul @ 2025-11-20 21:45 UTC (permalink / raw)
To: rust-for-linux, linux-kernel, Thomas Gleixner
Cc: Boqun Feng, Daniel Almeida, Miguel Ojeda, Alex Gaynor, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Andrew Morton, Peter Zijlstra,
Ingo Molnar, Will Deacon, Waiman Long
From: Boqun Feng <boqun.feng@gmail.com>
In order to support preempt_disable()-like interrupt disabling, that is,
using part of preempt_count() to track interrupt disabling nested level,
change the preempt_count() layout to contain 8-bit HARDIRQ_DISABLE
count.
Note that HARDIRQ_BITS and NMI_BITS are reduced by 1 because of this,
and it changes the maximum of their (hardirq and nmi) nesting level.
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
Signed-off-by: Lyude Paul <lyude@redhat.com>
---
V14:
* Fix HARDIRQ_DISABLE_MASK definition
include/linux/preempt.h | 16 +++++++++++-----
1 file changed, 11 insertions(+), 5 deletions(-)
diff --git a/include/linux/preempt.h b/include/linux/preempt.h
index 102202185d7a2..94ebdd98b7a94 100644
--- a/include/linux/preempt.h
+++ b/include/linux/preempt.h
@@ -17,6 +17,7 @@
*
* - bits 0-7 are the preemption count (max preemption depth: 256)
* - bits 8-15 are the softirq count (max # of softirqs: 256)
+ * - bits 16-23 are the hardirq disable count (max # of hardirq disable: 256)
*
* The hardirq count could in theory be the same as the number of
* interrupts in the system, but we run all interrupt handlers with
@@ -26,29 +27,34 @@
*
* PREEMPT_MASK: 0x000000ff
* SOFTIRQ_MASK: 0x0000ff00
- * HARDIRQ_MASK: 0x000f0000
- * NMI_MASK: 0x00f00000
+ * HARDIRQ_DISABLE_MASK: 0x00ff0000
+ * HARDIRQ_MASK: 0x07000000
+ * NMI_MASK: 0x38000000
* PREEMPT_NEED_RESCHED: 0x80000000
*/
#define PREEMPT_BITS 8
#define SOFTIRQ_BITS 8
-#define HARDIRQ_BITS 4
-#define NMI_BITS 4
+#define HARDIRQ_DISABLE_BITS 8
+#define HARDIRQ_BITS 3
+#define NMI_BITS 3
#define PREEMPT_SHIFT 0
#define SOFTIRQ_SHIFT (PREEMPT_SHIFT + PREEMPT_BITS)
-#define HARDIRQ_SHIFT (SOFTIRQ_SHIFT + SOFTIRQ_BITS)
+#define HARDIRQ_DISABLE_SHIFT (SOFTIRQ_SHIFT + SOFTIRQ_BITS)
+#define HARDIRQ_SHIFT (HARDIRQ_DISABLE_SHIFT + HARDIRQ_DISABLE_BITS)
#define NMI_SHIFT (HARDIRQ_SHIFT + HARDIRQ_BITS)
#define __IRQ_MASK(x) ((1UL << (x))-1)
#define PREEMPT_MASK (__IRQ_MASK(PREEMPT_BITS) << PREEMPT_SHIFT)
#define SOFTIRQ_MASK (__IRQ_MASK(SOFTIRQ_BITS) << SOFTIRQ_SHIFT)
+#define HARDIRQ_DISABLE_MASK (__IRQ_MASK(HARDIRQ_DISABLE_BITS) << HARDIRQ_DISABLE_SHIFT)
#define HARDIRQ_MASK (__IRQ_MASK(HARDIRQ_BITS) << HARDIRQ_SHIFT)
#define NMI_MASK (__IRQ_MASK(NMI_BITS) << NMI_SHIFT)
#define PREEMPT_OFFSET (1UL << PREEMPT_SHIFT)
#define SOFTIRQ_OFFSET (1UL << SOFTIRQ_SHIFT)
+#define HARDIRQ_DISABLE_OFFSET (1UL << HARDIRQ_DISABLE_SHIFT)
#define HARDIRQ_OFFSET (1UL << HARDIRQ_SHIFT)
#define NMI_OFFSET (1UL << NMI_SHIFT)
--
2.51.1
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v14 02/16] preempt: Track NMI nesting to separate per-CPU counter
2025-11-20 21:45 [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust Lyude Paul
2025-11-20 21:45 ` [PATCH v14 01/16] preempt: Introduce HARDIRQ_DISABLE_BITS Lyude Paul
@ 2025-11-20 21:45 ` Lyude Paul
2025-11-20 21:45 ` [PATCH v14 03/16] preempt: Introduce __preempt_count_{sub, add}_return() Lyude Paul
` (14 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Lyude Paul @ 2025-11-20 21:45 UTC (permalink / raw)
To: rust-for-linux, linux-kernel, Thomas Gleixner
Cc: Boqun Feng, Daniel Almeida, Miguel Ojeda, Alex Gaynor, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Andrew Morton, Peter Zijlstra,
Ingo Molnar, Will Deacon, Waiman Long, Joel Fernandes
From: Joel Fernandes <joelagnelf@nvidia.com>
Move NMI nesting tracking from the preempt_count bits to a separate per-CPU
counter (nmi_nesting). This is to free up the NMI bits in the preempt_count,
allowing those bits to be repurposed for other uses. This also has the benefit
of tracking more than 16-levels deep if there is ever a need.
Reduce multiple bits in preempt_count for NMI tracking. Reduce NMI_BITS
from 3 to 1, using it only to detect if we're in an NMI.
Suggested-by: Boqun Feng <boqun.feng@gmail.com>
Signed-off-by: Joel Fernandes <joelaf@google.com>
Signed-off-by: Lyude Paul <lyude@redhat.com>
---
include/linux/hardirq.h | 16 ++++++++++++----
include/linux/preempt.h | 13 +++++++++----
kernel/softirq.c | 2 ++
3 files changed, 23 insertions(+), 8 deletions(-)
diff --git a/include/linux/hardirq.h b/include/linux/hardirq.h
index d57cab4d4c06f..cc06bda52c3e5 100644
--- a/include/linux/hardirq.h
+++ b/include/linux/hardirq.h
@@ -10,6 +10,8 @@
#include <linux/vtime.h>
#include <asm/hardirq.h>
+DECLARE_PER_CPU(unsigned int, nmi_nesting);
+
extern void synchronize_irq(unsigned int irq);
extern bool synchronize_hardirq(unsigned int irq);
@@ -102,14 +104,16 @@ void irq_exit_rcu(void);
*/
/*
- * nmi_enter() can nest up to 15 times; see NMI_BITS.
+ * nmi_enter() can nest - nesting is tracked in a per-CPU counter.
*/
#define __nmi_enter() \
do { \
lockdep_off(); \
arch_nmi_enter(); \
- BUG_ON(in_nmi() == NMI_MASK); \
- __preempt_count_add(NMI_OFFSET + HARDIRQ_OFFSET); \
+ BUG_ON(__this_cpu_read(nmi_nesting) == UINT_MAX); \
+ __this_cpu_inc(nmi_nesting); \
+ __preempt_count_add(HARDIRQ_OFFSET); \
+ preempt_count_set(preempt_count() | NMI_MASK); \
} while (0)
#define nmi_enter() \
@@ -124,8 +128,12 @@ void irq_exit_rcu(void);
#define __nmi_exit() \
do { \
+ unsigned int nesting; \
BUG_ON(!in_nmi()); \
- __preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET); \
+ __preempt_count_sub(HARDIRQ_OFFSET); \
+ nesting = __this_cpu_dec_return(nmi_nesting); \
+ if (!nesting) \
+ __preempt_count_sub(NMI_OFFSET); \
arch_nmi_exit(); \
lockdep_on(); \
} while (0)
diff --git a/include/linux/preempt.h b/include/linux/preempt.h
index 94ebdd98b7a94..860769a717c10 100644
--- a/include/linux/preempt.h
+++ b/include/linux/preempt.h
@@ -18,6 +18,8 @@
* - bits 0-7 are the preemption count (max preemption depth: 256)
* - bits 8-15 are the softirq count (max # of softirqs: 256)
* - bits 16-23 are the hardirq disable count (max # of hardirq disable: 256)
+ * - bits 24-27 are the hardirq count (max # of hardirqs: 16)
+ * - bit 28 is the NMI flag (no nesting count, tracked separately)
*
* The hardirq count could in theory be the same as the number of
* interrupts in the system, but we run all interrupt handlers with
@@ -25,18 +27,21 @@
* there are a few palaeontologic drivers which reenable interrupts in
* the handler, so we need more than one bit here.
*
+ * NMI nesting depth is tracked in a separate per-CPU variable
+ * (nmi_nesting) to save bits in preempt_count.
+ *
* PREEMPT_MASK: 0x000000ff
* SOFTIRQ_MASK: 0x0000ff00
* HARDIRQ_DISABLE_MASK: 0x00ff0000
- * HARDIRQ_MASK: 0x07000000
- * NMI_MASK: 0x38000000
+ * HARDIRQ_MASK: 0x0f000000
+ * NMI_MASK: 0x10000000
* PREEMPT_NEED_RESCHED: 0x80000000
*/
#define PREEMPT_BITS 8
#define SOFTIRQ_BITS 8
#define HARDIRQ_DISABLE_BITS 8
-#define HARDIRQ_BITS 3
-#define NMI_BITS 3
+#define HARDIRQ_BITS 4
+#define NMI_BITS 1
#define PREEMPT_SHIFT 0
#define SOFTIRQ_SHIFT (PREEMPT_SHIFT + PREEMPT_BITS)
diff --git a/kernel/softirq.c b/kernel/softirq.c
index 77198911b8dd4..af47ea23aba3b 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -88,6 +88,8 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled);
EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context);
#endif
+DEFINE_PER_CPU(unsigned int, nmi_nesting);
+
/*
* SOFTIRQ_OFFSET usage:
*
--
2.51.1
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v14 03/16] preempt: Introduce __preempt_count_{sub, add}_return()
2025-11-20 21:45 [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust Lyude Paul
2025-11-20 21:45 ` [PATCH v14 01/16] preempt: Introduce HARDIRQ_DISABLE_BITS Lyude Paul
2025-11-20 21:45 ` [PATCH v14 02/16] preempt: Track NMI nesting to separate per-CPU counter Lyude Paul
@ 2025-11-20 21:45 ` Lyude Paul
2025-11-20 21:45 ` [PATCH v14 04/16] irq & spin_lock: Add counted interrupt disabling/enabling Lyude Paul
` (13 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Lyude Paul @ 2025-11-20 21:45 UTC (permalink / raw)
To: rust-for-linux, linux-kernel, Thomas Gleixner
Cc: Boqun Feng, Daniel Almeida, Miguel Ojeda, Alex Gaynor, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Andrew Morton, Peter Zijlstra,
Ingo Molnar, Will Deacon, Waiman Long
From: Boqun Feng <boqun.feng@gmail.com>
In order to use preempt_count() to tracking the interrupt disable
nesting level, __preempt_count_{add,sub}_return() are introduced, as
their name suggest, these primitives return the new value of the
preempt_count() after changing it. The following example shows the usage
of it in local_interrupt_disable():
// increase the HARDIRQ_DISABLE bit
new_count = __preempt_count_add_return(HARDIRQ_DISABLE_OFFSET);
// if it's the first-time increment, then disable the interrupt
// at hardware level.
if (new_count & HARDIRQ_DISABLE_MASK == HARDIRQ_DISABLE_OFFSET) {
local_irq_save(flags);
raw_cpu_write(local_interrupt_disable_state.flags, flags);
}
Having these primitives will avoid a read of preempt_count() after
changing preempt_count() on certain architectures.
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
---
V10:
* Add commit message I forgot
* Rebase against latest pcpu_hot changes
V11:
* Remove CONFIG_PROFILE_ALL_BRANCHES workaround from
__preempt_count_add_return()
arch/arm64/include/asm/preempt.h | 18 ++++++++++++++++++
arch/s390/include/asm/preempt.h | 10 ++++++++++
arch/x86/include/asm/preempt.h | 10 ++++++++++
include/asm-generic/preempt.h | 14 ++++++++++++++
4 files changed, 52 insertions(+)
diff --git a/arch/arm64/include/asm/preempt.h b/arch/arm64/include/asm/preempt.h
index 932ea4b620428..0dd8221d1bef7 100644
--- a/arch/arm64/include/asm/preempt.h
+++ b/arch/arm64/include/asm/preempt.h
@@ -55,6 +55,24 @@ static inline void __preempt_count_sub(int val)
WRITE_ONCE(current_thread_info()->preempt.count, pc);
}
+static inline int __preempt_count_add_return(int val)
+{
+ u32 pc = READ_ONCE(current_thread_info()->preempt.count);
+ pc += val;
+ WRITE_ONCE(current_thread_info()->preempt.count, pc);
+
+ return pc;
+}
+
+static inline int __preempt_count_sub_return(int val)
+{
+ u32 pc = READ_ONCE(current_thread_info()->preempt.count);
+ pc -= val;
+ WRITE_ONCE(current_thread_info()->preempt.count, pc);
+
+ return pc;
+}
+
static inline bool __preempt_count_dec_and_test(void)
{
struct thread_info *ti = current_thread_info();
diff --git a/arch/s390/include/asm/preempt.h b/arch/s390/include/asm/preempt.h
index 6ccd033acfe52..5ae366e26c57d 100644
--- a/arch/s390/include/asm/preempt.h
+++ b/arch/s390/include/asm/preempt.h
@@ -98,6 +98,16 @@ static __always_inline bool should_resched(int preempt_offset)
return unlikely(READ_ONCE(get_lowcore()->preempt_count) == preempt_offset);
}
+static __always_inline int __preempt_count_add_return(int val)
+{
+ return val + __atomic_add(val, &get_lowcore()->preempt_count);
+}
+
+static __always_inline int __preempt_count_sub_return(int val)
+{
+ return __preempt_count_add_return(-val);
+}
+
#define init_task_preempt_count(p) do { } while (0)
/* Deferred to CPU bringup time */
#define init_idle_preempt_count(p, cpu) do { } while (0)
diff --git a/arch/x86/include/asm/preempt.h b/arch/x86/include/asm/preempt.h
index 578441db09f0b..1220656f3370b 100644
--- a/arch/x86/include/asm/preempt.h
+++ b/arch/x86/include/asm/preempt.h
@@ -85,6 +85,16 @@ static __always_inline void __preempt_count_sub(int val)
raw_cpu_add_4(__preempt_count, -val);
}
+static __always_inline int __preempt_count_add_return(int val)
+{
+ return raw_cpu_add_return_4(__preempt_count, val);
+}
+
+static __always_inline int __preempt_count_sub_return(int val)
+{
+ return raw_cpu_add_return_4(__preempt_count, -val);
+}
+
/*
* Because we keep PREEMPT_NEED_RESCHED set when we do _not_ need to reschedule
* a decrement which hits zero means we have no preempt_count and should
diff --git a/include/asm-generic/preempt.h b/include/asm-generic/preempt.h
index 51f8f3881523a..c8683c046615d 100644
--- a/include/asm-generic/preempt.h
+++ b/include/asm-generic/preempt.h
@@ -59,6 +59,20 @@ static __always_inline void __preempt_count_sub(int val)
*preempt_count_ptr() -= val;
}
+static __always_inline int __preempt_count_add_return(int val)
+{
+ *preempt_count_ptr() += val;
+
+ return *preempt_count_ptr();
+}
+
+static __always_inline int __preempt_count_sub_return(int val)
+{
+ *preempt_count_ptr() -= val;
+
+ return *preempt_count_ptr();
+}
+
static __always_inline bool __preempt_count_dec_and_test(void)
{
/*
--
2.51.1
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v14 04/16] irq & spin_lock: Add counted interrupt disabling/enabling
2025-11-20 21:45 [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust Lyude Paul
` (2 preceding siblings ...)
2025-11-20 21:45 ` [PATCH v14 03/16] preempt: Introduce __preempt_count_{sub, add}_return() Lyude Paul
@ 2025-11-20 21:45 ` Lyude Paul
2025-11-20 21:45 ` [PATCH v14 05/16] irq: Add KUnit test for refcounted interrupt enable/disable Lyude Paul
` (12 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Lyude Paul @ 2025-11-20 21:45 UTC (permalink / raw)
To: rust-for-linux, linux-kernel, Thomas Gleixner
Cc: Boqun Feng, Daniel Almeida, Miguel Ojeda, Alex Gaynor, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Andrew Morton, Peter Zijlstra,
Ingo Molnar, Will Deacon, Waiman Long
From: Boqun Feng <boqun.feng@gmail.com>
Currently the nested interrupt disabling and enabling is present by
_irqsave() and _irqrestore() APIs, which are relatively unsafe, for
example:
<interrupts are enabled as beginning>
spin_lock_irqsave(l1, flag1);
spin_lock_irqsave(l2, flag2);
spin_unlock_irqrestore(l1, flags1);
<l2 is still held but interrupts are enabled>
// accesses to interrupt-disable protect data will cause races.
This is even easier to triggered with guard facilities:
unsigned long flag2;
scoped_guard(spin_lock_irqsave, l1) {
spin_lock_irqsave(l2, flag2);
}
// l2 locked but interrupts are enabled.
spin_unlock_irqrestore(l2, flag2);
(Hand-to-hand locking critical sections are not uncommon for a
fine-grained lock design)
And because this unsafety, Rust cannot easily wrap the
interrupt-disabling locks in a safe API, which complicates the design.
To resolve this, introduce a new set of interrupt disabling APIs:
* local_interrupt_disable();
* local_interrupt_enable();
They work like local_irq_save() and local_irq_restore() except that 1)
the outermost local_interrupt_disable() call save the interrupt state
into a percpu variable, so that the outermost local_interrupt_enable()
can restore the state, and 2) a percpu counter is added to record the
nest level of these calls, so that interrupts are not accidentally
enabled inside the outermost critical section.
Also add the corresponding spin_lock primitives: spin_lock_irq_disable()
and spin_unlock_irq_enable(), as a result, code as follow:
spin_lock_irq_disable(l1);
spin_lock_irq_disable(l2);
spin_unlock_irq_enable(l1);
// Interrupts are still disabled.
spin_unlock_irq_enable(l2);
doesn't have the issue that interrupts are accidentally enabled.
This also makes the wrapper of interrupt-disabling locks on Rust easier
to design.
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
Signed-off-by: Lyude Paul <lyude@redhat.com>
---
V10:
* Add missing __raw_spin_lock_irq_disable() definition in spinlock.c
V11:
* Move definition of spin_trylock_irq_disable() into this commit
* Get rid of leftover space
* Remove unneeded preempt_disable()/preempt_enable()
V12:
* Move local_interrupt_enable()/local_interrupt_disable() out of
include/linux/spinlock.h, into include/linux/irqflags.h
V14:
* Move local_interrupt_enable()/disable() again, this time into its own
header, interrupt_rc.h, in order to fix a hexagon-specific build issue
caught by the CKI bot.
The reason this is needed is because on most architectures, irqflags.h
ends up including <arch/smp.h>. This provides a definition for the
raw_smp_processor_id() function which we depend on like so:
<linux/percpu-defs.h> <arch/smp.h>
local_interrupt_disable() → raw_cpu_write() → raw_smp_processor_id()
Unfortunately, hexagon appears to be one such architecture which does
not pull in <arch/smp.h> by default here - causing kernel builds to
fail and claim that raw_smp_processor_id() is undefined:
In file included from kernel/sched/rq-offsets.c:5:
In file included from kernel/sched/sched.h:8:
In file included from include/linux/sched/affinity.h:1:
In file included from include/linux/sched.h:37:
In file included from include/linux/spinlock.h:59:
>> include/linux/irqflags.h:277:3: error: call to undeclared function
'raw_smp_processor_id'; ISO C99 and later do not support implicit
function declarations [-Wimplicit-function-declaration]
277 | raw_cpu_write(local_interrupt_disable_state.flags, flags);
| ^
include/linux/percpu-defs.h:413:34: note: expanded from macro
'raw_cpu_write'
While including <arch/smp.h> in <linux/irqflags.h> does fix the build
on hexagon, it ends up breaking the build on x86_64:
In file included from kernel/sched/rq-offsets.c:5:
In file included from kernel/sched/sched.h:8:
In file included from ./include/linux/sched/affinity.h:1:
In file included from ./include/linux/sched.h:13:
In file included from ./arch/x86/include/asm/processor.h:25:
In file included from ./arch/x86/include/asm/special_insns.h:10:
In file included from ./include/linux/irqflags.h:22:
In file included from ./arch/x86/include/asm/smp.h:6:
In file included from ./include/linux/thread_info.h:60:
In file included from ./arch/x86/include/asm/thread_info.h:59:
./arch/x86/include/asm/cpufeature.h:110:40: error: use of undeclared
identifier 'boot_cpu_data'
[cap_byte] "i" (&((const char *)boot_cpu_data.x86_capability)[bit >> 3])
^
While boot_cpu_data is defined in <asm/processor.h>, it's not possible
for us to include that header in irqflags.h because we're already
inside of <asm/processor.h>.
As a result, I just concluded there's no reasonable way of having these
functions in <linux/irqflags.h> because of how many low level ASM
headers depend on it. So, we go with the solution of simply giving
ourselves our own header file.
include/linux/interrupt_rc.h | 61 ++++++++++++++++++++++++++++++++
include/linux/preempt.h | 4 +++
include/linux/spinlock.h | 25 +++++++++++++
include/linux/spinlock_api_smp.h | 27 ++++++++++++++
include/linux/spinlock_api_up.h | 8 +++++
include/linux/spinlock_rt.h | 15 ++++++++
kernel/locking/spinlock.c | 29 +++++++++++++++
kernel/softirq.c | 3 ++
8 files changed, 172 insertions(+)
create mode 100644 include/linux/interrupt_rc.h
diff --git a/include/linux/interrupt_rc.h b/include/linux/interrupt_rc.h
new file mode 100644
index 0000000000000..2f131f8ef1d61
--- /dev/null
+++ b/include/linux/interrupt_rc.h
@@ -0,0 +1,61 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * include/linux/interrupt_rc.h - refcounted local processor interrupt
+ * management.
+ *
+ * Since the implementation of this API currently depends on
+ * local_irq_save()/local_irq_restore(), we split this into it's own header to
+ * make it easier to include without hitting circular header dependencies.
+ */
+
+#ifndef __LINUX_INTERRUPT_RC_H
+#define __LINUX_INTERRUPT_RC_H
+
+#include <linux/irqflags.h>
+#include <asm/processor.h>
+#include <asm/smp.h>
+
+/* Per-cpu interrupt disabling state for local_interrupt_{disable,enable}() */
+struct interrupt_disable_state {
+ unsigned long flags;
+};
+
+DECLARE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_state);
+
+static inline void local_interrupt_disable(void)
+{
+ unsigned long flags;
+ int new_count;
+
+ new_count = hardirq_disable_enter();
+
+ if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET) {
+ local_irq_save(flags);
+ raw_cpu_write(local_interrupt_disable_state.flags, flags);
+ }
+}
+
+static inline void local_interrupt_enable(void)
+{
+ int new_count;
+
+ new_count = hardirq_disable_exit();
+
+ if ((new_count & HARDIRQ_DISABLE_MASK) == 0) {
+ unsigned long flags;
+
+ flags = raw_cpu_read(local_interrupt_disable_state.flags);
+ local_irq_restore(flags);
+ /*
+ * TODO: re-read preempt count can be avoided, but it needs
+ * should_resched() taking another parameter as the current
+ * preempt count
+ */
+#ifdef PREEMPTION
+ if (should_resched(0))
+ __preempt_schedule();
+#endif
+ }
+}
+
+#endif /* !__LINUX_INTERRUPT_RC_H */
diff --git a/include/linux/preempt.h b/include/linux/preempt.h
index 860769a717c10..2f2ee9006f544 100644
--- a/include/linux/preempt.h
+++ b/include/linux/preempt.h
@@ -153,6 +153,10 @@ static __always_inline unsigned char interrupt_context_level(void)
#define in_softirq() (softirq_count())
#define in_interrupt() (irq_count())
+#define hardirq_disable_count() ((preempt_count() & HARDIRQ_DISABLE_MASK) >> HARDIRQ_DISABLE_SHIFT)
+#define hardirq_disable_enter() __preempt_count_add_return(HARDIRQ_DISABLE_OFFSET)
+#define hardirq_disable_exit() __preempt_count_sub_return(HARDIRQ_DISABLE_OFFSET)
+
/*
* The preempt_count offset after preempt_disable();
*/
diff --git a/include/linux/spinlock.h b/include/linux/spinlock.h
index d3561c4a080e2..bbbee61c6f5df 100644
--- a/include/linux/spinlock.h
+++ b/include/linux/spinlock.h
@@ -57,6 +57,7 @@
#include <linux/linkage.h>
#include <linux/compiler.h>
#include <linux/irqflags.h>
+#include <linux/interrupt_rc.h>
#include <linux/thread_info.h>
#include <linux/stringify.h>
#include <linux/bottom_half.h>
@@ -272,9 +273,11 @@ static inline void do_raw_spin_unlock(raw_spinlock_t *lock) __releases(lock)
#endif
#define raw_spin_lock_irq(lock) _raw_spin_lock_irq(lock)
+#define raw_spin_lock_irq_disable(lock) _raw_spin_lock_irq_disable(lock)
#define raw_spin_lock_bh(lock) _raw_spin_lock_bh(lock)
#define raw_spin_unlock(lock) _raw_spin_unlock(lock)
#define raw_spin_unlock_irq(lock) _raw_spin_unlock_irq(lock)
+#define raw_spin_unlock_irq_enable(lock) _raw_spin_unlock_irq_enable(lock)
#define raw_spin_unlock_irqrestore(lock, flags) \
do { \
@@ -300,6 +303,13 @@ static inline void do_raw_spin_unlock(raw_spinlock_t *lock) __releases(lock)
1 : ({ local_irq_restore(flags); 0; }); \
})
+#define raw_spin_trylock_irq_disable(lock) \
+({ \
+ local_interrupt_disable(); \
+ raw_spin_trylock(lock) ? \
+ 1 : ({ local_interrupt_enable(); 0; }); \
+})
+
#ifndef CONFIG_PREEMPT_RT
/* Include rwlock functions for !RT */
#include <linux/rwlock.h>
@@ -376,6 +386,11 @@ static __always_inline void spin_lock_irq(spinlock_t *lock)
raw_spin_lock_irq(&lock->rlock);
}
+static __always_inline void spin_lock_irq_disable(spinlock_t *lock)
+{
+ raw_spin_lock_irq_disable(&lock->rlock);
+}
+
#define spin_lock_irqsave(lock, flags) \
do { \
raw_spin_lock_irqsave(spinlock_check(lock), flags); \
@@ -401,6 +416,11 @@ static __always_inline void spin_unlock_irq(spinlock_t *lock)
raw_spin_unlock_irq(&lock->rlock);
}
+static __always_inline void spin_unlock_irq_enable(spinlock_t *lock)
+{
+ raw_spin_unlock_irq_enable(&lock->rlock);
+}
+
static __always_inline void spin_unlock_irqrestore(spinlock_t *lock, unsigned long flags)
{
raw_spin_unlock_irqrestore(&lock->rlock, flags);
@@ -421,6 +441,11 @@ static __always_inline int spin_trylock_irq(spinlock_t *lock)
raw_spin_trylock_irqsave(spinlock_check(lock), flags); \
})
+static __always_inline int spin_trylock_irq_disable(spinlock_t *lock)
+{
+ return raw_spin_trylock_irq_disable(&lock->rlock);
+}
+
/**
* spin_is_locked() - Check whether a spinlock is locked.
* @lock: Pointer to the spinlock.
diff --git a/include/linux/spinlock_api_smp.h b/include/linux/spinlock_api_smp.h
index 9ecb0ab504e32..92532103b9eaa 100644
--- a/include/linux/spinlock_api_smp.h
+++ b/include/linux/spinlock_api_smp.h
@@ -28,6 +28,8 @@ _raw_spin_lock_nest_lock(raw_spinlock_t *lock, struct lockdep_map *map)
void __lockfunc _raw_spin_lock_bh(raw_spinlock_t *lock) __acquires(lock);
void __lockfunc _raw_spin_lock_irq(raw_spinlock_t *lock)
__acquires(lock);
+void __lockfunc _raw_spin_lock_irq_disable(raw_spinlock_t *lock)
+ __acquires(lock);
unsigned long __lockfunc _raw_spin_lock_irqsave(raw_spinlock_t *lock)
__acquires(lock);
@@ -39,6 +41,7 @@ int __lockfunc _raw_spin_trylock_bh(raw_spinlock_t *lock);
void __lockfunc _raw_spin_unlock(raw_spinlock_t *lock) __releases(lock);
void __lockfunc _raw_spin_unlock_bh(raw_spinlock_t *lock) __releases(lock);
void __lockfunc _raw_spin_unlock_irq(raw_spinlock_t *lock) __releases(lock);
+void __lockfunc _raw_spin_unlock_irq_enable(raw_spinlock_t *lock) __releases(lock);
void __lockfunc
_raw_spin_unlock_irqrestore(raw_spinlock_t *lock, unsigned long flags)
__releases(lock);
@@ -55,6 +58,11 @@ _raw_spin_unlock_irqrestore(raw_spinlock_t *lock, unsigned long flags)
#define _raw_spin_lock_irq(lock) __raw_spin_lock_irq(lock)
#endif
+/* Use the same config as spin_lock_irq() temporarily. */
+#ifdef CONFIG_INLINE_SPIN_LOCK_IRQ
+#define _raw_spin_lock_irq_disable(lock) __raw_spin_lock_irq_disable(lock)
+#endif
+
#ifdef CONFIG_INLINE_SPIN_LOCK_IRQSAVE
#define _raw_spin_lock_irqsave(lock) __raw_spin_lock_irqsave(lock)
#endif
@@ -79,6 +87,11 @@ _raw_spin_unlock_irqrestore(raw_spinlock_t *lock, unsigned long flags)
#define _raw_spin_unlock_irq(lock) __raw_spin_unlock_irq(lock)
#endif
+/* Use the same config as spin_unlock_irq() temporarily. */
+#ifdef CONFIG_INLINE_SPIN_UNLOCK_IRQ
+#define _raw_spin_unlock_irq_enable(lock) __raw_spin_unlock_irq_enable(lock)
+#endif
+
#ifdef CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE
#define _raw_spin_unlock_irqrestore(lock, flags) __raw_spin_unlock_irqrestore(lock, flags)
#endif
@@ -120,6 +133,13 @@ static inline void __raw_spin_lock_irq(raw_spinlock_t *lock)
LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock);
}
+static inline void __raw_spin_lock_irq_disable(raw_spinlock_t *lock)
+{
+ local_interrupt_disable();
+ spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
+ LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock);
+}
+
static inline void __raw_spin_lock_bh(raw_spinlock_t *lock)
{
__local_bh_disable_ip(_RET_IP_, SOFTIRQ_LOCK_OFFSET);
@@ -160,6 +180,13 @@ static inline void __raw_spin_unlock_irq(raw_spinlock_t *lock)
preempt_enable();
}
+static inline void __raw_spin_unlock_irq_enable(raw_spinlock_t *lock)
+{
+ spin_release(&lock->dep_map, _RET_IP_);
+ do_raw_spin_unlock(lock);
+ local_interrupt_enable();
+}
+
static inline void __raw_spin_unlock_bh(raw_spinlock_t *lock)
{
spin_release(&lock->dep_map, _RET_IP_);
diff --git a/include/linux/spinlock_api_up.h b/include/linux/spinlock_api_up.h
index 819aeba1c87e6..d02a73671713b 100644
--- a/include/linux/spinlock_api_up.h
+++ b/include/linux/spinlock_api_up.h
@@ -36,6 +36,9 @@
#define __LOCK_IRQ(lock) \
do { local_irq_disable(); __LOCK(lock); } while (0)
+#define __LOCK_IRQ_DISABLE(lock) \
+ do { local_interrupt_disable(); __LOCK(lock); } while (0)
+
#define __LOCK_IRQSAVE(lock, flags) \
do { local_irq_save(flags); __LOCK(lock); } while (0)
@@ -52,6 +55,9 @@
#define __UNLOCK_IRQ(lock) \
do { local_irq_enable(); __UNLOCK(lock); } while (0)
+#define __UNLOCK_IRQ_ENABLE(lock) \
+ do { __UNLOCK(lock); local_interrupt_enable(); } while (0)
+
#define __UNLOCK_IRQRESTORE(lock, flags) \
do { local_irq_restore(flags); __UNLOCK(lock); } while (0)
@@ -64,6 +70,7 @@
#define _raw_read_lock_bh(lock) __LOCK_BH(lock)
#define _raw_write_lock_bh(lock) __LOCK_BH(lock)
#define _raw_spin_lock_irq(lock) __LOCK_IRQ(lock)
+#define _raw_spin_lock_irq_disable(lock) __LOCK_IRQ_DISABLE(lock)
#define _raw_read_lock_irq(lock) __LOCK_IRQ(lock)
#define _raw_write_lock_irq(lock) __LOCK_IRQ(lock)
#define _raw_spin_lock_irqsave(lock, flags) __LOCK_IRQSAVE(lock, flags)
@@ -80,6 +87,7 @@
#define _raw_write_unlock_bh(lock) __UNLOCK_BH(lock)
#define _raw_read_unlock_bh(lock) __UNLOCK_BH(lock)
#define _raw_spin_unlock_irq(lock) __UNLOCK_IRQ(lock)
+#define _raw_spin_unlock_irq_enable(lock) __UNLOCK_IRQ_ENABLE(lock)
#define _raw_read_unlock_irq(lock) __UNLOCK_IRQ(lock)
#define _raw_write_unlock_irq(lock) __UNLOCK_IRQ(lock)
#define _raw_spin_unlock_irqrestore(lock, flags) \
diff --git a/include/linux/spinlock_rt.h b/include/linux/spinlock_rt.h
index f6499c37157df..074182f7cfeea 100644
--- a/include/linux/spinlock_rt.h
+++ b/include/linux/spinlock_rt.h
@@ -93,6 +93,11 @@ static __always_inline void spin_lock_irq(spinlock_t *lock)
rt_spin_lock(lock);
}
+static __always_inline void spin_lock_irq_disable(spinlock_t *lock)
+{
+ rt_spin_lock(lock);
+}
+
#define spin_lock_irqsave(lock, flags) \
do { \
typecheck(unsigned long, flags); \
@@ -116,12 +121,22 @@ static __always_inline void spin_unlock_irq(spinlock_t *lock)
rt_spin_unlock(lock);
}
+static __always_inline void spin_unlock_irq_enable(spinlock_t *lock)
+{
+ rt_spin_unlock(lock);
+}
+
static __always_inline void spin_unlock_irqrestore(spinlock_t *lock,
unsigned long flags)
{
rt_spin_unlock(lock);
}
+static __always_inline int spin_trylock_irq_disable(spinlock_t *lock)
+{
+ return rt_spin_trylock(lock);
+}
+
#define spin_trylock(lock) \
__cond_lock(lock, rt_spin_trylock(lock))
diff --git a/kernel/locking/spinlock.c b/kernel/locking/spinlock.c
index 7685defd7c526..da54b220b5a45 100644
--- a/kernel/locking/spinlock.c
+++ b/kernel/locking/spinlock.c
@@ -125,6 +125,19 @@ static void __lockfunc __raw_##op##_lock_bh(locktype##_t *lock) \
*/
BUILD_LOCK_OPS(spin, raw_spinlock);
+/* No rwlock_t variants for now, so just build this function by hand */
+static void __lockfunc __raw_spin_lock_irq_disable(raw_spinlock_t *lock)
+{
+ for (;;) {
+ local_interrupt_disable();
+ if (likely(do_raw_spin_trylock(lock)))
+ break;
+ local_interrupt_enable();
+
+ arch_spin_relax(&lock->raw_lock);
+ }
+}
+
#ifndef CONFIG_PREEMPT_RT
BUILD_LOCK_OPS(read, rwlock);
BUILD_LOCK_OPS(write, rwlock);
@@ -172,6 +185,14 @@ noinline void __lockfunc _raw_spin_lock_irq(raw_spinlock_t *lock)
EXPORT_SYMBOL(_raw_spin_lock_irq);
#endif
+#ifndef CONFIG_INLINE_SPIN_LOCK_IRQ
+noinline void __lockfunc _raw_spin_lock_irq_disable(raw_spinlock_t *lock)
+{
+ __raw_spin_lock_irq_disable(lock);
+}
+EXPORT_SYMBOL_GPL(_raw_spin_lock_irq_disable);
+#endif
+
#ifndef CONFIG_INLINE_SPIN_LOCK_BH
noinline void __lockfunc _raw_spin_lock_bh(raw_spinlock_t *lock)
{
@@ -204,6 +225,14 @@ noinline void __lockfunc _raw_spin_unlock_irq(raw_spinlock_t *lock)
EXPORT_SYMBOL(_raw_spin_unlock_irq);
#endif
+#ifndef CONFIG_INLINE_SPIN_UNLOCK_IRQ
+noinline void __lockfunc _raw_spin_unlock_irq_enable(raw_spinlock_t *lock)
+{
+ __raw_spin_unlock_irq_enable(lock);
+}
+EXPORT_SYMBOL_GPL(_raw_spin_unlock_irq_enable);
+#endif
+
#ifndef CONFIG_INLINE_SPIN_UNLOCK_BH
noinline void __lockfunc _raw_spin_unlock_bh(raw_spinlock_t *lock)
{
diff --git a/kernel/softirq.c b/kernel/softirq.c
index af47ea23aba3b..b681545eabbbe 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -88,6 +88,9 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled);
EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context);
#endif
+DEFINE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_state);
+EXPORT_PER_CPU_SYMBOL_GPL(local_interrupt_disable_state);
+
DEFINE_PER_CPU(unsigned int, nmi_nesting);
/*
--
2.51.1
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v14 05/16] irq: Add KUnit test for refcounted interrupt enable/disable
2025-11-20 21:45 [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust Lyude Paul
` (3 preceding siblings ...)
2025-11-20 21:45 ` [PATCH v14 04/16] irq & spin_lock: Add counted interrupt disabling/enabling Lyude Paul
@ 2025-11-20 21:45 ` Lyude Paul
2025-11-20 21:45 ` [PATCH v14 06/16] rust: Introduce interrupt module Lyude Paul
` (11 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Lyude Paul @ 2025-11-20 21:45 UTC (permalink / raw)
To: rust-for-linux, linux-kernel, Thomas Gleixner
Cc: Boqun Feng, Daniel Almeida, Miguel Ojeda, Alex Gaynor, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Andrew Morton, Peter Zijlstra,
Ingo Molnar, Will Deacon, Waiman Long
While making changes to the refcounted interrupt patch series, at some
point on my local branch I broke something and ended up writing some kunit
tests for testing refcounted interrupts as a result. So, let's include
these tests now that we have refcounted interrupts.
Signed-off-by: Lyude Paul <lyude@redhat.com>
---
V13:
* Add missing MODULE_DESCRIPTION/MODULE_LICENSE lines
* Switch from kunit_test_suites(…) to kunit_test_suite(…)
kernel/irq/Makefile | 1 +
kernel/irq/refcount_interrupt_test.c | 109 +++++++++++++++++++++++++++
2 files changed, 110 insertions(+)
create mode 100644 kernel/irq/refcount_interrupt_test.c
diff --git a/kernel/irq/Makefile b/kernel/irq/Makefile
index 6ab3a40556670..7b5bb5510b110 100644
--- a/kernel/irq/Makefile
+++ b/kernel/irq/Makefile
@@ -20,3 +20,4 @@ obj-$(CONFIG_SMP) += affinity.o
obj-$(CONFIG_GENERIC_IRQ_DEBUGFS) += debugfs.o
obj-$(CONFIG_GENERIC_IRQ_MATRIX_ALLOCATOR) += matrix.o
obj-$(CONFIG_IRQ_KUNIT_TEST) += irq_test.o
+obj-$(CONFIG_KUNIT) += refcount_interrupt_test.o
diff --git a/kernel/irq/refcount_interrupt_test.c b/kernel/irq/refcount_interrupt_test.c
new file mode 100644
index 0000000000000..b4f224595f261
--- /dev/null
+++ b/kernel/irq/refcount_interrupt_test.c
@@ -0,0 +1,109 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * KUnit test for refcounted interrupt enable/disables.
+ */
+
+#include <kunit/test.h>
+#include <linux/interrupt_rc.h>
+
+#define TEST_IRQ_ON() KUNIT_EXPECT_FALSE(test, irqs_disabled())
+#define TEST_IRQ_OFF() KUNIT_EXPECT_TRUE(test, irqs_disabled())
+
+/* ===== Test cases ===== */
+static void test_single_irq_change(struct kunit *test)
+{
+ local_interrupt_disable();
+ TEST_IRQ_OFF();
+ local_interrupt_enable();
+}
+
+static void test_nested_irq_change(struct kunit *test)
+{
+ local_interrupt_disable();
+ TEST_IRQ_OFF();
+ local_interrupt_disable();
+ TEST_IRQ_OFF();
+ local_interrupt_disable();
+ TEST_IRQ_OFF();
+
+ local_interrupt_enable();
+ TEST_IRQ_OFF();
+ local_interrupt_enable();
+ TEST_IRQ_OFF();
+ local_interrupt_enable();
+ TEST_IRQ_ON();
+}
+
+static void test_multiple_irq_change(struct kunit *test)
+{
+ local_interrupt_disable();
+ TEST_IRQ_OFF();
+ local_interrupt_disable();
+ TEST_IRQ_OFF();
+
+ local_interrupt_enable();
+ TEST_IRQ_OFF();
+ local_interrupt_enable();
+ TEST_IRQ_ON();
+
+ local_interrupt_disable();
+ TEST_IRQ_OFF();
+ local_interrupt_enable();
+ TEST_IRQ_ON();
+}
+
+static void test_irq_save(struct kunit *test)
+{
+ unsigned long flags;
+
+ local_irq_save(flags);
+ TEST_IRQ_OFF();
+ local_interrupt_disable();
+ TEST_IRQ_OFF();
+ local_interrupt_enable();
+ TEST_IRQ_OFF();
+ local_irq_restore(flags);
+ TEST_IRQ_ON();
+
+ local_interrupt_disable();
+ TEST_IRQ_OFF();
+ local_irq_save(flags);
+ TEST_IRQ_OFF();
+ local_irq_restore(flags);
+ TEST_IRQ_OFF();
+ local_interrupt_enable();
+ TEST_IRQ_ON();
+}
+
+static struct kunit_case test_cases[] = {
+ KUNIT_CASE(test_single_irq_change),
+ KUNIT_CASE(test_nested_irq_change),
+ KUNIT_CASE(test_multiple_irq_change),
+ KUNIT_CASE(test_irq_save),
+ {},
+};
+
+/* (init and exit are the same */
+static int test_init(struct kunit *test)
+{
+ TEST_IRQ_ON();
+
+ return 0;
+}
+
+static void test_exit(struct kunit *test)
+{
+ TEST_IRQ_ON();
+}
+
+static struct kunit_suite refcount_interrupt_test_suite = {
+ .name = "refcount_interrupt",
+ .test_cases = test_cases,
+ .init = test_init,
+ .exit = test_exit,
+};
+
+kunit_test_suite(refcount_interrupt_test_suite);
+MODULE_AUTHOR("Lyude Paul <lyude@redhat.com>");
+MODULE_DESCRIPTION("Refcounted interrupt unit test suite");
+MODULE_LICENSE("GPL");
--
2.51.1
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v14 06/16] rust: Introduce interrupt module
2025-11-20 21:45 [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust Lyude Paul
` (4 preceding siblings ...)
2025-11-20 21:45 ` [PATCH v14 05/16] irq: Add KUnit test for refcounted interrupt enable/disable Lyude Paul
@ 2025-11-20 21:45 ` Lyude Paul
2025-11-20 21:45 ` [PATCH v14 07/16] rust: helper: Add spin_{un,}lock_irq_{enable,disable}() helpers Lyude Paul
` (10 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Lyude Paul @ 2025-11-20 21:45 UTC (permalink / raw)
To: rust-for-linux, linux-kernel, Thomas Gleixner
Cc: Boqun Feng, Daniel Almeida, Miguel Ojeda, Alex Gaynor, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Andrew Morton, Peter Zijlstra,
Ingo Molnar, Will Deacon, Waiman Long
This introduces a module for dealing with interrupt-disabled contexts,
including the ability to enable and disable interrupts along with the
ability to annotate functions as expecting that IRQs are already
disabled on the local CPU.
[Boqun: This is based on Lyude's work on interrupt disable abstraction,
I port to the new local_interrupt_disable() mechanism to make it work
as a guard type. I cannot even take the credit of this design, since
Lyude also brought up the same idea in zulip. Anyway, this is only for
POC purpose, and of course all bugs are mine]
Signed-off-by: Lyude Paul <lyude@redhat.com>
Co-developed-by: Boqun Feng <boqun.feng@gmail.com>
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
Reviewed-by: Benno Lossin <lossin@kernel.org>
Reviewed-by: Andreas Hindborg <a.hindborg@kernel.org>
---
V10:
* Fix documentation typos
V11:
* Get rid of unneeded `use bindings;`
* Move ASSUME_DISABLED into assume_disabled()
* Confirm using lockdep_assert_irqs_disabled() that local interrupts are in
fact disabled when LocalInterruptDisabled::assume_disabled() is called.
rust/helpers/helpers.c | 1 +
rust/helpers/interrupt.c | 18 +++++++++
rust/helpers/sync.c | 5 +++
rust/kernel/interrupt.rs | 86 ++++++++++++++++++++++++++++++++++++++++
rust/kernel/lib.rs | 1 +
5 files changed, 111 insertions(+)
create mode 100644 rust/helpers/interrupt.c
create mode 100644 rust/kernel/interrupt.rs
diff --git a/rust/helpers/helpers.c b/rust/helpers/helpers.c
index 551da6c9b5064..01ffade6c0389 100644
--- a/rust/helpers/helpers.c
+++ b/rust/helpers/helpers.c
@@ -29,6 +29,7 @@
#include "err.c"
#include "irq.c"
#include "fs.c"
+#include "interrupt.c"
#include "io.c"
#include "jump_label.c"
#include "kunit.c"
diff --git a/rust/helpers/interrupt.c b/rust/helpers/interrupt.c
new file mode 100644
index 0000000000000..f2380dd461ca5
--- /dev/null
+++ b/rust/helpers/interrupt.c
@@ -0,0 +1,18 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/spinlock.h>
+
+void rust_helper_local_interrupt_disable(void)
+{
+ local_interrupt_disable();
+}
+
+void rust_helper_local_interrupt_enable(void)
+{
+ local_interrupt_enable();
+}
+
+bool rust_helper_irqs_disabled(void)
+{
+ return irqs_disabled();
+}
diff --git a/rust/helpers/sync.c b/rust/helpers/sync.c
index ff7e68b488101..45b2f519f4e2e 100644
--- a/rust/helpers/sync.c
+++ b/rust/helpers/sync.c
@@ -11,3 +11,8 @@ void rust_helper_lockdep_unregister_key(struct lock_class_key *k)
{
lockdep_unregister_key(k);
}
+
+void rust_helper_lockdep_assert_irqs_disabled(void)
+{
+ lockdep_assert_irqs_disabled();
+}
diff --git a/rust/kernel/interrupt.rs b/rust/kernel/interrupt.rs
new file mode 100644
index 0000000000000..6c8d2f58bca70
--- /dev/null
+++ b/rust/kernel/interrupt.rs
@@ -0,0 +1,86 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! Interrupt controls
+//!
+//! This module allows Rust code to annotate areas of code where local processor interrupts should
+//! be disabled, along with actually disabling local processor interrupts.
+//!
+//! # ⚠️ Warning! ⚠️
+//!
+//! The usage of this module can be more complicated than meets the eye, especially surrounding
+//! [preemptible kernels]. It's recommended to take care when using the functions and types defined
+//! here and familiarize yourself with the various documentation we have before using them, along
+//! with the various documents we link to here.
+//!
+//! # Reading material
+//!
+//! - [Software interrupts and realtime (LWN)](https://lwn.net/Articles/520076)
+//!
+//! [preemptible kernels]: https://www.kernel.org/doc/html/latest/locking/preempt-locking.html
+
+use kernel::types::NotThreadSafe;
+
+/// A guard that represents local processor interrupt disablement on preemptible kernels.
+///
+/// [`LocalInterruptDisabled`] is a guard type that represents that local processor interrupts have
+/// been disabled on a preemptible kernel.
+///
+/// Certain functions take an immutable reference of [`LocalInterruptDisabled`] in order to require
+/// that they may only be run in local-interrupt-disabled contexts on preemptible kernels.
+///
+/// This is a marker type; it has no size, and is simply used as a compile-time guarantee that local
+/// processor interrupts are disabled on preemptible kernels. Note that no guarantees about the
+/// state of interrupts are made by this type on non-preemptible kernels.
+///
+/// # Invariants
+///
+/// Local processor interrupts are disabled on preemptible kernels for as long as an object of this
+/// type exists.
+pub struct LocalInterruptDisabled(NotThreadSafe);
+
+/// Disable local processor interrupts on a preemptible kernel.
+///
+/// This function disables local processor interrupts on a preemptible kernel, and returns a
+/// [`LocalInterruptDisabled`] token as proof of this. On non-preemptible kernels, this function is
+/// a no-op.
+///
+/// **Usage of this function is discouraged** unless you are absolutely sure you know what you are
+/// doing, as kernel interfaces for rust that deal with interrupt state will typically handle local
+/// processor interrupt state management on their own and managing this by hand is quite error
+/// prone.
+pub fn local_interrupt_disable() -> LocalInterruptDisabled {
+ // SAFETY: It's always safe to call `local_interrupt_disable()`.
+ unsafe { bindings::local_interrupt_disable() };
+
+ LocalInterruptDisabled(NotThreadSafe)
+}
+
+impl Drop for LocalInterruptDisabled {
+ fn drop(&mut self) {
+ // SAFETY: Per type invariants, a `local_interrupt_disable()` must be called to create this
+ // object, hence call the corresponding `local_interrupt_enable()` is safe.
+ unsafe { bindings::local_interrupt_enable() };
+ }
+}
+
+impl LocalInterruptDisabled {
+ /// Assume that local processor interrupts are disabled on preemptible kernels.
+ ///
+ /// This can be used for annotating code that is known to be run in contexts where local
+ /// processor interrupts are disabled on preemptible kernels. It makes no changes to the local
+ /// interrupt state on its own.
+ ///
+ /// # Safety
+ ///
+ /// For the whole life `'a`, local interrupts must be disabled on preemptible kernels. This
+ /// could be a context like for example, an interrupt handler.
+ pub unsafe fn assume_disabled<'a>() -> &'a LocalInterruptDisabled {
+ const ASSUME_DISABLED: &LocalInterruptDisabled = &LocalInterruptDisabled(NotThreadSafe);
+
+ // Confirm they're actually disabled if lockdep is available
+ // SAFETY: It's always safe to call `lockdep_assert_irqs_disabled()`
+ unsafe { bindings::lockdep_assert_irqs_disabled() };
+
+ ASSUME_DISABLED
+ }
+}
diff --git a/rust/kernel/lib.rs b/rust/kernel/lib.rs
index 3dd7bebe78882..75a466d9319b2 100644
--- a/rust/kernel/lib.rs
+++ b/rust/kernel/lib.rs
@@ -96,6 +96,7 @@
pub mod fs;
pub mod id_pool;
pub mod init;
+pub mod interrupt;
pub mod io;
pub mod ioctl;
pub mod iov;
--
2.51.1
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v14 07/16] rust: helper: Add spin_{un,}lock_irq_{enable,disable}() helpers
2025-11-20 21:45 [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust Lyude Paul
` (5 preceding siblings ...)
2025-11-20 21:45 ` [PATCH v14 06/16] rust: Introduce interrupt module Lyude Paul
@ 2025-11-20 21:45 ` Lyude Paul
2025-11-20 21:46 ` [PATCH v14 08/16] rust: sync: Add SpinLockIrq Lyude Paul
` (9 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Lyude Paul @ 2025-11-20 21:45 UTC (permalink / raw)
To: rust-for-linux, linux-kernel, Thomas Gleixner
Cc: Boqun Feng, Daniel Almeida, Miguel Ojeda, Alex Gaynor, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Andrew Morton, Peter Zijlstra,
Ingo Molnar, Will Deacon, Waiman Long
From: Boqun Feng <boqun.feng@gmail.com>
spin_lock_irq_disable() and spin_unlock_irq_enable() are inline
functions, to use them in Rust, helpers are introduced. This is for
interrupt disabling lock abstraction in Rust.
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
Reviewed-by: Andreas Hindborg <a.hindborg@kernel.org>
Signed-off-by: Lyude Paul <lyude@redhat.com>
---
rust/helpers/spinlock.c | 15 +++++++++++++++
1 file changed, 15 insertions(+)
diff --git a/rust/helpers/spinlock.c b/rust/helpers/spinlock.c
index 42c4bf01a23e4..d4e61057c2a7a 100644
--- a/rust/helpers/spinlock.c
+++ b/rust/helpers/spinlock.c
@@ -35,3 +35,18 @@ void rust_helper_spin_assert_is_held(spinlock_t *lock)
{
lockdep_assert_held(lock);
}
+
+void rust_helper_spin_lock_irq_disable(spinlock_t *lock)
+{
+ spin_lock_irq_disable(lock);
+}
+
+void rust_helper_spin_unlock_irq_enable(spinlock_t *lock)
+{
+ spin_unlock_irq_enable(lock);
+}
+
+int rust_helper_spin_trylock_irq_disable(spinlock_t *lock)
+{
+ return spin_trylock_irq_disable(lock);
+}
--
2.51.1
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v14 08/16] rust: sync: Add SpinLockIrq
2025-11-20 21:45 [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust Lyude Paul
` (6 preceding siblings ...)
2025-11-20 21:45 ` [PATCH v14 07/16] rust: helper: Add spin_{un,}lock_irq_{enable,disable}() helpers Lyude Paul
@ 2025-11-20 21:46 ` Lyude Paul
2025-11-20 21:46 ` [PATCH v14 09/16] rust: sync: Introduce lock::Backend::Context Lyude Paul
` (8 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Lyude Paul @ 2025-11-20 21:46 UTC (permalink / raw)
To: rust-for-linux, linux-kernel, Thomas Gleixner
Cc: Boqun Feng, Daniel Almeida, Miguel Ojeda, Alex Gaynor, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Andrew Morton, Peter Zijlstra,
Ingo Molnar, Will Deacon, Waiman Long
A variant of SpinLock that is expected to be used in noirq contexts, so
lock() will disable interrupts and unlock() (i.e. `Guard::drop()` will
undo the interrupt disable.
[Boqun: Port to use spin_lock_irq_disable() and
spin_unlock_irq_enable()]
Signed-off-by: Lyude Paul <lyude@redhat.com>
Co-developed-by: Boqun Feng <boqun.feng@gmail.com>
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
Reviewed-by: Andreas Hindborg <a.hindborg@kernel.org>
---
V10:
* Also add support to GlobalLock
* Documentation fixes from Dirk
V11:
* Add unit test requested by Daniel Almeida
V14:
- Improve rustdoc for SpinLockIrqBackend
rust/kernel/sync.rs | 4 +-
rust/kernel/sync/lock/global.rs | 3 +
rust/kernel/sync/lock/spinlock.rs | 229 ++++++++++++++++++++++++++++++
3 files changed, 235 insertions(+), 1 deletion(-)
diff --git a/rust/kernel/sync.rs b/rust/kernel/sync.rs
index cf5b638a097d9..f293bbe13e855 100644
--- a/rust/kernel/sync.rs
+++ b/rust/kernel/sync.rs
@@ -26,7 +26,9 @@
pub use condvar::{new_condvar, CondVar, CondVarTimeoutResult};
pub use lock::global::{global_lock, GlobalGuard, GlobalLock, GlobalLockBackend, GlobalLockedBy};
pub use lock::mutex::{new_mutex, Mutex, MutexGuard};
-pub use lock::spinlock::{new_spinlock, SpinLock, SpinLockGuard};
+pub use lock::spinlock::{
+ new_spinlock, new_spinlock_irq, SpinLock, SpinLockGuard, SpinLockIrq, SpinLockIrqGuard,
+};
pub use locked_by::LockedBy;
pub use refcount::Refcount;
diff --git a/rust/kernel/sync/lock/global.rs b/rust/kernel/sync/lock/global.rs
index 79d0ef7fda867..b8d78ca4c0cd3 100644
--- a/rust/kernel/sync/lock/global.rs
+++ b/rust/kernel/sync/lock/global.rs
@@ -299,4 +299,7 @@ macro_rules! global_lock_inner {
(backend SpinLock) => {
$crate::sync::lock::spinlock::SpinLockBackend
};
+ (backend SpinLockIrq) => {
+ $crate::sync::lock::spinlock::SpinLockIrqBackend
+ };
}
diff --git a/rust/kernel/sync/lock/spinlock.rs b/rust/kernel/sync/lock/spinlock.rs
index d7be38ccbdc7d..3fdfb0a8a0ab1 100644
--- a/rust/kernel/sync/lock/spinlock.rs
+++ b/rust/kernel/sync/lock/spinlock.rs
@@ -3,6 +3,7 @@
//! A kernel spinlock.
//!
//! This module allows Rust code to use the kernel's `spinlock_t`.
+use crate::prelude::*;
/// Creates a [`SpinLock`] initialiser with the given name and a newly-created lock class.
///
@@ -139,3 +140,231 @@ unsafe fn assert_is_held(ptr: *mut Self::State) {
unsafe { bindings::spin_assert_is_held(ptr) }
}
}
+
+/// Creates a [`SpinLockIrq`] initialiser with the given name and a newly-created lock class.
+///
+/// It uses the name if one is given, otherwise it generates one based on the file name and line
+/// number.
+#[macro_export]
+macro_rules! new_spinlock_irq {
+ ($inner:expr $(, $name:literal)? $(,)?) => {
+ $crate::sync::SpinLockIrq::new(
+ $inner, $crate::optional_name!($($name)?), $crate::static_lock_class!())
+ };
+}
+pub use new_spinlock_irq;
+
+/// A spinlock that may be acquired when local processor interrupts are disabled.
+///
+/// This is a version of [`SpinLock`] that can only be used in contexts where interrupts for the
+/// local CPU are disabled. It can be acquired in two ways:
+///
+/// - Using [`lock()`] like any other type of lock, in which case the bindings will modify the
+/// interrupt state to ensure that local processor interrupts remain disabled for at least as long
+/// as the [`SpinLockIrqGuard`] exists.
+/// - Using [`lock_with()`] in contexts where a [`LocalInterruptDisabled`] token is present and
+/// local processor interrupts are already known to be disabled, in which case the local interrupt
+/// state will not be touched. This method should be preferred if a [`LocalInterruptDisabled`]
+/// token is present in the scope.
+///
+/// For more info on spinlocks, see [`SpinLock`]. For more information on interrupts,
+/// [see the interrupt module](kernel::interrupt).
+///
+/// # Examples
+///
+/// The following example shows how to declare, allocate initialise and access a struct (`Example`)
+/// that contains an inner struct (`Inner`) that is protected by a spinlock that requires local
+/// processor interrupts to be disabled.
+///
+/// ```
+/// use kernel::sync::{new_spinlock_irq, SpinLockIrq};
+///
+/// struct Inner {
+/// a: u32,
+/// b: u32,
+/// }
+///
+/// #[pin_data]
+/// struct Example {
+/// #[pin]
+/// c: SpinLockIrq<Inner>,
+/// #[pin]
+/// d: SpinLockIrq<Inner>,
+/// }
+///
+/// impl Example {
+/// fn new() -> impl PinInit<Self> {
+/// pin_init!(Self {
+/// c <- new_spinlock_irq!(Inner { a: 0, b: 10 }),
+/// d <- new_spinlock_irq!(Inner { a: 20, b: 30 }),
+/// })
+/// }
+/// }
+///
+/// // Allocate a boxed `Example`
+/// let e = KBox::pin_init(Example::new(), GFP_KERNEL)?;
+///
+/// // Accessing an `Example` from a context where interrupts may not be disabled already.
+/// let c_guard = e.c.lock(); // interrupts are disabled now, +1 interrupt disable refcount
+/// let d_guard = e.d.lock(); // no interrupt state change, +1 interrupt disable refcount
+///
+/// assert_eq!(c_guard.a, 0);
+/// assert_eq!(c_guard.b, 10);
+/// assert_eq!(d_guard.a, 20);
+/// assert_eq!(d_guard.b, 30);
+///
+/// drop(c_guard); // Dropping c_guard will not re-enable interrupts just yet, since d_guard is
+/// // still in scope.
+/// drop(d_guard); // Last interrupt disable reference dropped here, so interrupts are re-enabled
+/// // now
+/// # Ok::<(), Error>(())
+/// ```
+///
+/// [`lock()`]: SpinLockIrq::lock
+/// [`lock_with()`]: SpinLockIrq::lock_with
+pub type SpinLockIrq<T> = super::Lock<T, SpinLockIrqBackend>;
+
+/// A kernel `spinlock_t` lock backend that can only be acquired in interrupt disabled contexts.
+pub struct SpinLockIrqBackend;
+
+/// A [`Guard`] acquired from locking a [`SpinLockIrq`] using [`lock()`].
+///
+/// This is simply a type alias for a [`Guard`] returned from locking a [`SpinLockIrq`] using
+/// [`lock_with()`]. It will unlock the [`SpinLockIrq`] and decrement the local processor's
+/// interrupt disablement refcount upon being dropped.
+///
+/// [`Guard`]: super::Guard
+/// [`lock()`]: SpinLockIrq::lock
+/// [`lock_with()`]: SpinLockIrq::lock_with
+pub type SpinLockIrqGuard<'a, T> = super::Guard<'a, T, SpinLockIrqBackend>;
+
+// SAFETY: The underlying kernel `spinlock_t` object ensures mutual exclusion. `relock` uses the
+// default implementation that always calls the same locking method.
+unsafe impl super::Backend for SpinLockIrqBackend {
+ type State = bindings::spinlock_t;
+ type GuardState = ();
+
+ unsafe fn init(
+ ptr: *mut Self::State,
+ name: *const crate::ffi::c_char,
+ key: *mut bindings::lock_class_key,
+ ) {
+ // SAFETY: The safety requirements ensure that `ptr` is valid for writes, and `name` and
+ // `key` are valid for read indefinitely.
+ unsafe { bindings::__spin_lock_init(ptr, name, key) }
+ }
+
+ unsafe fn lock(ptr: *mut Self::State) -> Self::GuardState {
+ // SAFETY: The safety requirements of this function ensure that `ptr` points to valid
+ // memory, and that it has been initialised before.
+ unsafe { bindings::spin_lock_irq_disable(ptr) }
+ }
+
+ unsafe fn unlock(ptr: *mut Self::State, _guard_state: &Self::GuardState) {
+ // SAFETY: The safety requirements of this function ensure that `ptr` is valid and that the
+ // caller is the owner of the spinlock.
+ unsafe { bindings::spin_unlock_irq_enable(ptr) }
+ }
+
+ unsafe fn try_lock(ptr: *mut Self::State) -> Option<Self::GuardState> {
+ // SAFETY: The `ptr` pointer is guaranteed to be valid and initialized before use.
+ let result = unsafe { bindings::spin_trylock_irq_disable(ptr) };
+
+ if result != 0 {
+ Some(())
+ } else {
+ None
+ }
+ }
+
+ unsafe fn assert_is_held(ptr: *mut Self::State) {
+ // SAFETY: The `ptr` pointer is guaranteed to be valid and initialized before use.
+ unsafe { bindings::spin_assert_is_held(ptr) }
+ }
+}
+
+#[kunit_tests(rust_spinlock_irq_condvar)]
+mod tests {
+ use super::*;
+ use crate::{
+ sync::*,
+ workqueue::{self, impl_has_work, new_work, Work, WorkItem},
+ };
+
+ struct TestState {
+ value: u32,
+ waiter_ready: bool,
+ }
+
+ #[pin_data]
+ struct Test {
+ #[pin]
+ state: SpinLockIrq<TestState>,
+
+ #[pin]
+ state_changed: CondVar,
+
+ #[pin]
+ waiter_state_changed: CondVar,
+
+ #[pin]
+ wait_work: Work<Self>,
+ }
+
+ impl_has_work! {
+ impl HasWork<Self> for Test { self.wait_work }
+ }
+
+ impl Test {
+ pub(crate) fn new() -> Result<Arc<Self>> {
+ Arc::try_pin_init(
+ try_pin_init!(
+ Self {
+ state <- new_spinlock_irq!(TestState {
+ value: 1,
+ waiter_ready: false
+ }),
+ state_changed <- new_condvar!(),
+ waiter_state_changed <- new_condvar!(),
+ wait_work <- new_work!("IrqCondvarTest::wait_work")
+ }
+ ),
+ GFP_KERNEL,
+ )
+ }
+ }
+
+ impl WorkItem for Test {
+ type Pointer = Arc<Self>;
+
+ fn run(this: Arc<Self>) {
+ // Wait for the test to be ready to wait for us
+ let mut state = this.state.lock();
+
+ while !state.waiter_ready {
+ this.waiter_state_changed.wait(&mut state);
+ }
+
+ // Deliver the exciting value update our test has been waiting for
+ state.value += 1;
+ this.state_changed.notify_sync();
+ }
+ }
+
+ #[test]
+ fn spinlock_irq_condvar() -> Result {
+ let testdata = Test::new()?;
+
+ let _ = workqueue::system().enqueue(testdata.clone());
+
+ // Let the updater know when we're ready to wait
+ let mut state = testdata.state.lock();
+ state.waiter_ready = true;
+ testdata.waiter_state_changed.notify_sync();
+
+ // Wait for the exciting value update
+ testdata.state_changed.wait(&mut state);
+ assert_eq!(state.value, 2);
+ Ok(())
+ }
+}
--
2.51.1
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v14 09/16] rust: sync: Introduce lock::Backend::Context
2025-11-20 21:45 [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust Lyude Paul
` (7 preceding siblings ...)
2025-11-20 21:46 ` [PATCH v14 08/16] rust: sync: Add SpinLockIrq Lyude Paul
@ 2025-11-20 21:46 ` Lyude Paul
2025-11-20 21:46 ` [PATCH v14 10/16] rust: sync: lock: Add `Backend::BackendInContext` Lyude Paul
` (7 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Lyude Paul @ 2025-11-20 21:46 UTC (permalink / raw)
To: rust-for-linux, linux-kernel, Thomas Gleixner
Cc: Boqun Feng, Daniel Almeida, Miguel Ojeda, Alex Gaynor, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Andrew Morton, Peter Zijlstra,
Ingo Molnar, Will Deacon, Waiman Long
Now that we've introduced an `InterruptDisabled` token for marking
contexts in which IRQs are disabled, we can have a way to avoid
`SpinLockIrq` disabling interrupts if the interrupts have already been
disabled. Basically, a `SpinLockIrq` should work like a `SpinLock` if
interrupts are disabled. So a function:
(&'a SpinLockIrq, &'a InterruptDisabled) -> Guard<'a, .., SpinLockBackend>
makes senses. Note that due to `Guard` and `InterruptDisabled` having the
same lifetime, interrupts cannot be enabled while the Guard exists.
Add a `lock_with()` interface for `Lock`, and an associate type of
`Backend` to describe the context.
Signed-off-by: Lyude Paul <lyude@redhat.com>
Co-developed-by: Boqun Feng <boqun.feng@gmail.com>
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
---
V10:
- Fix typos - Dirk
rust/kernel/sync/lock.rs | 12 +++++++++++-
rust/kernel/sync/lock/mutex.rs | 1 +
rust/kernel/sync/lock/spinlock.rs | 4 +++-
3 files changed, 15 insertions(+), 2 deletions(-)
diff --git a/rust/kernel/sync/lock.rs b/rust/kernel/sync/lock.rs
index 5d7991e6d3736..7df62923608b7 100644
--- a/rust/kernel/sync/lock.rs
+++ b/rust/kernel/sync/lock.rs
@@ -44,6 +44,9 @@ pub unsafe trait Backend {
/// [`unlock`]: Backend::unlock
type GuardState;
+ /// The context which can be provided to acquire the lock with a different backend.
+ type Context<'a>;
+
/// Initialises the lock.
///
/// # Safety
@@ -163,8 +166,15 @@ pub unsafe fn from_raw<'a>(ptr: *mut B::State) -> &'a Self {
}
impl<T: ?Sized, B: Backend> Lock<T, B> {
+ /// Acquires the lock with the given context and gives the caller access to the data protected
+ /// by it.
+ pub fn lock_with<'a>(&'a self, _context: B::Context<'a>) -> Guard<'a, T, B> {
+ todo!()
+ }
+
/// Acquires the lock and gives the caller access to the data protected by it.
- pub fn lock(&self) -> Guard<'_, T, B> {
+ #[inline]
+ pub fn lock<'a>(&'a self) -> Guard<'a, T, B> {
// SAFETY: The constructor of the type calls `init`, so the existence of the object proves
// that `init` was called.
let state = unsafe { B::lock(self.state.get()) };
diff --git a/rust/kernel/sync/lock/mutex.rs b/rust/kernel/sync/lock/mutex.rs
index 581cee7ab842a..be1e2e18cf42d 100644
--- a/rust/kernel/sync/lock/mutex.rs
+++ b/rust/kernel/sync/lock/mutex.rs
@@ -101,6 +101,7 @@ macro_rules! new_mutex {
unsafe impl super::Backend for MutexBackend {
type State = bindings::mutex;
type GuardState = ();
+ type Context<'a> = ();
unsafe fn init(
ptr: *mut Self::State,
diff --git a/rust/kernel/sync/lock/spinlock.rs b/rust/kernel/sync/lock/spinlock.rs
index 3fdfb0a8a0ab1..70d19a2636afe 100644
--- a/rust/kernel/sync/lock/spinlock.rs
+++ b/rust/kernel/sync/lock/spinlock.rs
@@ -3,7 +3,7 @@
//! A kernel spinlock.
//!
//! This module allows Rust code to use the kernel's `spinlock_t`.
-use crate::prelude::*;
+use crate::{interrupt::LocalInterruptDisabled, prelude::*};
/// Creates a [`SpinLock`] initialiser with the given name and a newly-created lock class.
///
@@ -101,6 +101,7 @@ macro_rules! new_spinlock {
unsafe impl super::Backend for SpinLockBackend {
type State = bindings::spinlock_t;
type GuardState = ();
+ type Context<'a> = ();
unsafe fn init(
ptr: *mut Self::State,
@@ -243,6 +244,7 @@ macro_rules! new_spinlock_irq {
unsafe impl super::Backend for SpinLockIrqBackend {
type State = bindings::spinlock_t;
type GuardState = ();
+ type Context<'a> = &'a LocalInterruptDisabled;
unsafe fn init(
ptr: *mut Self::State,
--
2.51.1
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v14 10/16] rust: sync: lock: Add `Backend::BackendInContext`
2025-11-20 21:45 [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust Lyude Paul
` (8 preceding siblings ...)
2025-11-20 21:46 ` [PATCH v14 09/16] rust: sync: Introduce lock::Backend::Context Lyude Paul
@ 2025-11-20 21:46 ` Lyude Paul
2025-11-20 21:46 ` [PATCH v14 11/16] rust: sync: lock/global: Rename B to G in trait bounds Lyude Paul
` (6 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Lyude Paul @ 2025-11-20 21:46 UTC (permalink / raw)
To: rust-for-linux, linux-kernel, Thomas Gleixner
Cc: Boqun Feng, Daniel Almeida, Miguel Ojeda, Alex Gaynor, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Andrew Morton, Peter Zijlstra,
Ingo Molnar, Will Deacon, Waiman Long
From: Boqun Feng <boqun.feng@gmail.com>
`SpinLockIrq` and `SpinLock` use the exact same underlying C structure,
with the only real difference being that the former uses the irq_disable()
and irq_enable() variants for locking/unlocking. These variants can
introduce some minor overhead in contexts where we already know that
local processor interrupts are disabled, and as such we want a way to be
able to skip modifying processor interrupt state in said contexts in order
to avoid some overhead - just like the current C API allows us to do. So,
`BackendInContext` allows us to cast a lock into it's contextless version
for situations where we already have whatever guarantees would be provided
by `Backend::Context` in place.
In some hacked-together benchmarks we ran, most of the time this did
actually seem to lead to a noticeable difference in overhead:
From an aarch64 VM running on a MacBook M4:
lock() when irq is disabled, 100 times cost Delta { nanos: 500 }
lock_with() when irq is disabled, 100 times cost Delta { nanos: 292 }
lock() when irq is enabled, 100 times cost Delta { nanos: 834 }
lock() when irq is disabled, 100 times cost Delta { nanos: 459 }
lock_with() when irq is disabled, 100 times cost Delta { nanos: 291 }
lock() when irq is enabled, 100 times cost Delta { nanos: 709 }
From an x86_64 VM (qemu/kvm) running on a i7-13700H
lock() when irq is disabled, 100 times cost Delta { nanos: 1002 }
lock_with() when irq is disabled, 100 times cost Delta { nanos: 729 }
lock() when irq is enabled, 100 times cost Delta { nanos: 1516 }
lock() when irq is disabled, 100 times cost Delta { nanos: 754 }
lock_with() when irq is disabled, 100 times cost Delta { nanos: 966 }
lock() when irq is enabled, 100 times cost Delta { nanos: 1227 }
(note that there were some runs on x86_64 where lock() on irq disabled
vs. lock_with() on irq disabled had equivalent benchmarks, but it very
much appeared to be a minority of test runs.
While it's not clear how this affects real-world workloads yet, let's add
this for the time being so we can find out.
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
Co-developed-by: Lyude Paul <lyude@redhat.com>
Signed-off-by: Lyude Paul <lyude@redhat.com>
---
V10:
* Fix typos - Dirk/Lyude
* Since we're adding support for context locks to GlobalLock as well, let's
also make sure to cover try_lock while we're at it and add try_lock_with
* Add a private function as_lock_in_context() for handling casting from a
Lock<T, B> to Lock<T, B::BackendInContext> so we don't have to duplicate
safety comments
V11:
* Fix clippy::ref_as_ptr error in Lock::as_lock_in_context()
V14:
* Add benchmark results, rewrite commit message
rust/kernel/sync/lock.rs | 61 ++++++++++++++++++++++++++++++-
rust/kernel/sync/lock/mutex.rs | 1 +
rust/kernel/sync/lock/spinlock.rs | 41 +++++++++++++++++++++
3 files changed, 101 insertions(+), 2 deletions(-)
diff --git a/rust/kernel/sync/lock.rs b/rust/kernel/sync/lock.rs
index 7df62923608b7..abc7ceff35994 100644
--- a/rust/kernel/sync/lock.rs
+++ b/rust/kernel/sync/lock.rs
@@ -30,10 +30,15 @@
/// is owned, that is, between calls to [`lock`] and [`unlock`].
/// - Implementers must also ensure that [`relock`] uses the same locking method as the original
/// lock operation.
+/// - Implementers must ensure if [`BackendInContext`] is a [`Backend`], it's safe to acquire the
+/// lock under the [`Context`], the [`State`] of two backends must be the same.
///
/// [`lock`]: Backend::lock
/// [`unlock`]: Backend::unlock
/// [`relock`]: Backend::relock
+/// [`BackendInContext`]: Backend::BackendInContext
+/// [`Context`]: Backend::Context
+/// [`State`]: Backend::State
pub unsafe trait Backend {
/// The state required by the lock.
type State;
@@ -47,6 +52,9 @@ pub unsafe trait Backend {
/// The context which can be provided to acquire the lock with a different backend.
type Context<'a>;
+ /// The alternative backend we can use if a [`Context`](Backend::Context) is provided.
+ type BackendInContext: Sized;
+
/// Initialises the lock.
///
/// # Safety
@@ -166,10 +174,59 @@ pub unsafe fn from_raw<'a>(ptr: *mut B::State) -> &'a Self {
}
impl<T: ?Sized, B: Backend> Lock<T, B> {
+ /// Casts the lock as a `Lock<T, B::BackendInContext>`.
+ fn as_lock_in_context<'a>(
+ &'a self,
+ _context: B::Context<'a>,
+ ) -> &'a Lock<T, B::BackendInContext>
+ where
+ B::BackendInContext: Backend,
+ {
+ // SAFETY:
+ // - Per the safety guarantee of `Backend`, if `B::BackendInContext` and `B` should
+ // have the same state, the layout of the lock is the same so it's safe to convert one to
+ // another.
+ // - The caller provided `B::Context<'a>`, so it is safe to recast and return this lock.
+ unsafe { &*(core::ptr::from_ref(self) as *const _) }
+ }
+
/// Acquires the lock with the given context and gives the caller access to the data protected
/// by it.
- pub fn lock_with<'a>(&'a self, _context: B::Context<'a>) -> Guard<'a, T, B> {
- todo!()
+ pub fn lock_with<'a>(&'a self, context: B::Context<'a>) -> Guard<'a, T, B::BackendInContext>
+ where
+ B::BackendInContext: Backend,
+ {
+ let lock = self.as_lock_in_context(context);
+
+ // SAFETY: The constructor of the type calls `init`, so the existence of the object proves
+ // that `init` was called. Plus the safety guarantee of `Backend` guarantees that `B::State`
+ // is the same as `B::BackendInContext::State`, also it's safe to call another backend
+ // because there is `B::Context<'a>`.
+ let state = unsafe { B::BackendInContext::lock(lock.state.get()) };
+
+ // SAFETY: The lock was just acquired.
+ unsafe { Guard::new(lock, state) }
+ }
+
+ /// Tries to acquire the lock with the given context.
+ ///
+ /// Returns a guard that can be used to access the data protected by the lock if successful.
+ pub fn try_lock_with<'a>(
+ &'a self,
+ context: B::Context<'a>,
+ ) -> Option<Guard<'a, T, B::BackendInContext>>
+ where
+ B::BackendInContext: Backend,
+ {
+ let lock = self.as_lock_in_context(context);
+
+ // SAFETY: The constructor of the type calls `init`, so the existence of the object proves
+ // that `init` was called. Plus the safety guarantee of `Backend` guarantees that `B::State`
+ // is the same as `B::BackendInContext::State`, also it's safe to call another backend
+ // because there is `B::Context<'a>`.
+ unsafe {
+ B::BackendInContext::try_lock(lock.state.get()).map(|state| Guard::new(lock, state))
+ }
}
/// Acquires the lock and gives the caller access to the data protected by it.
diff --git a/rust/kernel/sync/lock/mutex.rs b/rust/kernel/sync/lock/mutex.rs
index be1e2e18cf42d..662a530750703 100644
--- a/rust/kernel/sync/lock/mutex.rs
+++ b/rust/kernel/sync/lock/mutex.rs
@@ -102,6 +102,7 @@ unsafe impl super::Backend for MutexBackend {
type State = bindings::mutex;
type GuardState = ();
type Context<'a> = ();
+ type BackendInContext = ();
unsafe fn init(
ptr: *mut Self::State,
diff --git a/rust/kernel/sync/lock/spinlock.rs b/rust/kernel/sync/lock/spinlock.rs
index 70d19a2636afe..81384ea239955 100644
--- a/rust/kernel/sync/lock/spinlock.rs
+++ b/rust/kernel/sync/lock/spinlock.rs
@@ -102,6 +102,7 @@ unsafe impl super::Backend for SpinLockBackend {
type State = bindings::spinlock_t;
type GuardState = ();
type Context<'a> = ();
+ type BackendInContext = ();
unsafe fn init(
ptr: *mut Self::State,
@@ -221,6 +222,45 @@ macro_rules! new_spinlock_irq {
/// # Ok::<(), Error>(())
/// ```
///
+/// The next example demonstrates locking a [`SpinLockIrq`] using [`lock_with()`] in a function
+/// which can only be called when local processor interrupts are already disabled.
+///
+/// ```
+/// use kernel::sync::{new_spinlock_irq, SpinLockIrq};
+/// use kernel::interrupt::*;
+///
+/// struct Inner {
+/// a: u32,
+/// }
+///
+/// #[pin_data]
+/// struct Example {
+/// #[pin]
+/// inner: SpinLockIrq<Inner>,
+/// }
+///
+/// impl Example {
+/// fn new() -> impl PinInit<Self> {
+/// pin_init!(Self {
+/// inner <- new_spinlock_irq!(Inner { a: 20 }),
+/// })
+/// }
+/// }
+///
+/// // Accessing an `Example` from a function that can only be called in no-interrupt contexts.
+/// fn noirq_work(e: &Example, interrupt_disabled: &LocalInterruptDisabled) {
+/// // Because we know interrupts are disabled from interrupt_disable, we can skip toggling
+/// // interrupt state using lock_with() and the provided token
+/// assert_eq!(e.inner.lock_with(interrupt_disabled).a, 20);
+/// }
+///
+/// # let e = KBox::pin_init(Example::new(), GFP_KERNEL)?;
+/// # let interrupt_guard = local_interrupt_disable();
+/// # noirq_work(&e, &interrupt_guard);
+/// #
+/// # Ok::<(), Error>(())
+/// ```
+///
/// [`lock()`]: SpinLockIrq::lock
/// [`lock_with()`]: SpinLockIrq::lock_with
pub type SpinLockIrq<T> = super::Lock<T, SpinLockIrqBackend>;
@@ -245,6 +285,7 @@ unsafe impl super::Backend for SpinLockIrqBackend {
type State = bindings::spinlock_t;
type GuardState = ();
type Context<'a> = &'a LocalInterruptDisabled;
+ type BackendInContext = SpinLockBackend;
unsafe fn init(
ptr: *mut Self::State,
--
2.51.1
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v14 11/16] rust: sync: lock/global: Rename B to G in trait bounds
2025-11-20 21:45 [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust Lyude Paul
` (9 preceding siblings ...)
2025-11-20 21:46 ` [PATCH v14 10/16] rust: sync: lock: Add `Backend::BackendInContext` Lyude Paul
@ 2025-11-20 21:46 ` Lyude Paul
2025-11-20 21:46 ` [PATCH v14 12/16] rust: sync: Add a lifetime parameter to lock::global::GlobalGuard Lyude Paul
` (5 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Lyude Paul @ 2025-11-20 21:46 UTC (permalink / raw)
To: rust-for-linux, linux-kernel, Thomas Gleixner
Cc: Boqun Feng, Daniel Almeida, Miguel Ojeda, Alex Gaynor, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Andrew Morton, Peter Zijlstra,
Ingo Molnar, Will Deacon, Waiman Long
Due to the introduction of Backend::BackendInContext, if we want to be able
support Lock types with a Context we need to be able to handle the fact
that the Backend for a returned Guard may not exactly match the Backend for
the lock. Before we add this though, rename B to G in all of our trait
bounds to make sure things don't become more difficult to understand once
we add a Backend bound.
There should be no functional changes in this patch.
Signed-off-by: Lyude Paul <lyude@redhat.com>
---
rust/kernel/sync/lock/global.rs | 56 ++++++++++++++++-----------------
1 file changed, 28 insertions(+), 28 deletions(-)
diff --git a/rust/kernel/sync/lock/global.rs b/rust/kernel/sync/lock/global.rs
index b8d78ca4c0cd3..c2349347976d8 100644
--- a/rust/kernel/sync/lock/global.rs
+++ b/rust/kernel/sync/lock/global.rs
@@ -33,18 +33,18 @@ pub trait GlobalLockBackend {
/// Type used for global locks.
///
/// See [`global_lock!`] for examples.
-pub struct GlobalLock<B: GlobalLockBackend> {
- inner: Lock<B::Item, B::Backend>,
+pub struct GlobalLock<G: GlobalLockBackend> {
+ inner: Lock<G::Item, G::Backend>,
}
-impl<B: GlobalLockBackend> GlobalLock<B> {
+impl<G: GlobalLockBackend> GlobalLock<G> {
/// Creates a global lock.
///
/// # Safety
///
/// * Before any other method on this lock is called, [`Self::init`] must be called.
- /// * The type `B` must not be used with any other lock.
- pub const unsafe fn new(data: B::Item) -> Self {
+ /// * The type `G` must not be used with any other lock.
+ pub const unsafe fn new(data: G::Item) -> Self {
Self {
inner: Lock {
state: Opaque::uninit(),
@@ -68,23 +68,23 @@ pub unsafe fn init(&'static self) {
// `init` before using any other methods. As `init` can only be called once, all other
// uses of this lock must happen after this call.
unsafe {
- B::Backend::init(
+ G::Backend::init(
self.inner.state.get(),
- B::NAME.as_char_ptr(),
- B::get_lock_class().as_ptr(),
+ G::NAME.as_char_ptr(),
+ G::get_lock_class().as_ptr(),
)
}
}
/// Lock this global lock.
- pub fn lock(&'static self) -> GlobalGuard<B> {
+ pub fn lock(&'static self) -> GlobalGuard<G> {
GlobalGuard {
inner: self.inner.lock(),
}
}
/// Try to lock this global lock.
- pub fn try_lock(&'static self) -> Option<GlobalGuard<B>> {
+ pub fn try_lock(&'static self) -> Option<GlobalGuard<G>> {
Some(GlobalGuard {
inner: self.inner.try_lock()?,
})
@@ -94,19 +94,19 @@ pub fn try_lock(&'static self) -> Option<GlobalGuard<B>> {
/// A guard for a [`GlobalLock`].
///
/// See [`global_lock!`] for examples.
-pub struct GlobalGuard<B: GlobalLockBackend> {
- inner: Guard<'static, B::Item, B::Backend>,
+pub struct GlobalGuard<G: GlobalLockBackend> {
+ inner: Guard<'static, G::Item, G::Backend>,
}
-impl<B: GlobalLockBackend> core::ops::Deref for GlobalGuard<B> {
- type Target = B::Item;
+impl<G: GlobalLockBackend> core::ops::Deref for GlobalGuard<G> {
+ type Target = G::Item;
fn deref(&self) -> &Self::Target {
&self.inner
}
}
-impl<B: GlobalLockBackend> core::ops::DerefMut for GlobalGuard<B> {
+impl<G: GlobalLockBackend> core::ops::DerefMut for GlobalGuard<G> {
fn deref_mut(&mut self) -> &mut Self::Target {
&mut self.inner
}
@@ -115,33 +115,33 @@ fn deref_mut(&mut self) -> &mut Self::Target {
/// A version of [`LockedBy`] for a [`GlobalLock`].
///
/// See [`global_lock!`] for examples.
-pub struct GlobalLockedBy<T: ?Sized, B: GlobalLockBackend> {
- _backend: PhantomData<B>,
+pub struct GlobalLockedBy<T: ?Sized, G: GlobalLockBackend> {
+ _backend: PhantomData<G>,
value: UnsafeCell<T>,
}
// SAFETY: The same thread-safety rules as `LockedBy` apply to `GlobalLockedBy`.
-unsafe impl<T, B> Send for GlobalLockedBy<T, B>
+unsafe impl<T, G> Send for GlobalLockedBy<T, G>
where
T: ?Sized,
- B: GlobalLockBackend,
- LockedBy<T, B::Item>: Send,
+ G: GlobalLockBackend,
+ LockedBy<T, G::Item>: Send,
{
}
// SAFETY: The same thread-safety rules as `LockedBy` apply to `GlobalLockedBy`.
-unsafe impl<T, B> Sync for GlobalLockedBy<T, B>
+unsafe impl<T, G> Sync for GlobalLockedBy<T, G>
where
T: ?Sized,
- B: GlobalLockBackend,
- LockedBy<T, B::Item>: Sync,
+ G: GlobalLockBackend,
+ LockedBy<T, G::Item>: Sync,
{
}
-impl<T, B: GlobalLockBackend> GlobalLockedBy<T, B> {
+impl<T, G: GlobalLockBackend> GlobalLockedBy<T, G> {
/// Create a new [`GlobalLockedBy`].
///
- /// The provided value will be protected by the global lock indicated by `B`.
+ /// The provided value will be protected by the global lock indicated by `G`.
pub fn new(val: T) -> Self {
Self {
value: UnsafeCell::new(val),
@@ -150,11 +150,11 @@ pub fn new(val: T) -> Self {
}
}
-impl<T: ?Sized, B: GlobalLockBackend> GlobalLockedBy<T, B> {
+impl<T: ?Sized, G: GlobalLockBackend> GlobalLockedBy<T, G> {
/// Access the value immutably.
///
/// The caller must prove shared access to the lock.
- pub fn as_ref<'a>(&'a self, _guard: &'a GlobalGuard<B>) -> &'a T {
+ pub fn as_ref<'a>(&'a self, _guard: &'a GlobalGuard<G>) -> &'a T {
// SAFETY: The lock is globally unique, so there can only be one guard.
unsafe { &*self.value.get() }
}
@@ -162,7 +162,7 @@ pub fn as_ref<'a>(&'a self, _guard: &'a GlobalGuard<B>) -> &'a T {
/// Access the value mutably.
///
/// The caller must prove shared exclusive to the lock.
- pub fn as_mut<'a>(&'a self, _guard: &'a mut GlobalGuard<B>) -> &'a mut T {
+ pub fn as_mut<'a>(&'a self, _guard: &'a mut GlobalGuard<G>) -> &'a mut T {
// SAFETY: The lock is globally unique, so there can only be one guard.
unsafe { &mut *self.value.get() }
}
--
2.51.1
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v14 12/16] rust: sync: Add a lifetime parameter to lock::global::GlobalGuard
2025-11-20 21:45 [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust Lyude Paul
` (10 preceding siblings ...)
2025-11-20 21:46 ` [PATCH v14 11/16] rust: sync: lock/global: Rename B to G in trait bounds Lyude Paul
@ 2025-11-20 21:46 ` Lyude Paul
2025-11-20 21:46 ` [PATCH v14 13/16] rust: sync: Expose lock::Backend Lyude Paul
` (4 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Lyude Paul @ 2025-11-20 21:46 UTC (permalink / raw)
To: rust-for-linux, linux-kernel, Thomas Gleixner
Cc: Boqun Feng, Daniel Almeida, Miguel Ojeda, Alex Gaynor, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Andrew Morton, Peter Zijlstra,
Ingo Molnar, Will Deacon, Waiman Long
While a GlobalLock is always going to be static, in the case of locks with
explicit backend contexts the GlobalGuard will not be 'static and will
instead share the lifetime of the context. So, add a lifetime parameter to
GlobalGuard to allow for this so we can implement GlobalGuard support for
SpinlockIrq.
Signed-off-by: Lyude Paul <lyude@redhat.com>
---
rust/kernel/sync/lock/global.rs | 18 +++++++++---------
1 file changed, 9 insertions(+), 9 deletions(-)
diff --git a/rust/kernel/sync/lock/global.rs b/rust/kernel/sync/lock/global.rs
index c2349347976d8..1d279be1a2eca 100644
--- a/rust/kernel/sync/lock/global.rs
+++ b/rust/kernel/sync/lock/global.rs
@@ -77,14 +77,14 @@ pub unsafe fn init(&'static self) {
}
/// Lock this global lock.
- pub fn lock(&'static self) -> GlobalGuard<G> {
+ pub fn lock(&'static self) -> GlobalGuard<'static, G> {
GlobalGuard {
inner: self.inner.lock(),
}
}
/// Try to lock this global lock.
- pub fn try_lock(&'static self) -> Option<GlobalGuard<G>> {
+ pub fn try_lock(&'static self) -> Option<GlobalGuard<'static, G>> {
Some(GlobalGuard {
inner: self.inner.try_lock()?,
})
@@ -94,11 +94,11 @@ pub fn try_lock(&'static self) -> Option<GlobalGuard<G>> {
/// A guard for a [`GlobalLock`].
///
/// See [`global_lock!`] for examples.
-pub struct GlobalGuard<G: GlobalLockBackend> {
- inner: Guard<'static, G::Item, G::Backend>,
+pub struct GlobalGuard<'a, G: GlobalLockBackend> {
+ inner: Guard<'a, G::Item, G::Backend>,
}
-impl<G: GlobalLockBackend> core::ops::Deref for GlobalGuard<G> {
+impl<'a, G: GlobalLockBackend> core::ops::Deref for GlobalGuard<'a, G> {
type Target = G::Item;
fn deref(&self) -> &Self::Target {
@@ -106,7 +106,7 @@ fn deref(&self) -> &Self::Target {
}
}
-impl<G: GlobalLockBackend> core::ops::DerefMut for GlobalGuard<G> {
+impl<'a, G: GlobalLockBackend> core::ops::DerefMut for GlobalGuard<'a, G> {
fn deref_mut(&mut self) -> &mut Self::Target {
&mut self.inner
}
@@ -154,7 +154,7 @@ impl<T: ?Sized, G: GlobalLockBackend> GlobalLockedBy<T, G> {
/// Access the value immutably.
///
/// The caller must prove shared access to the lock.
- pub fn as_ref<'a>(&'a self, _guard: &'a GlobalGuard<G>) -> &'a T {
+ pub fn as_ref<'a>(&'a self, _guard: &'a GlobalGuard<'_, G>) -> &'a T {
// SAFETY: The lock is globally unique, so there can only be one guard.
unsafe { &*self.value.get() }
}
@@ -162,7 +162,7 @@ pub fn as_ref<'a>(&'a self, _guard: &'a GlobalGuard<G>) -> &'a T {
/// Access the value mutably.
///
/// The caller must prove shared exclusive to the lock.
- pub fn as_mut<'a>(&'a self, _guard: &'a mut GlobalGuard<G>) -> &'a mut T {
+ pub fn as_mut<'a>(&'a self, _guard: &'a mut GlobalGuard<'_, G>) -> &'a mut T {
// SAFETY: The lock is globally unique, so there can only be one guard.
unsafe { &mut *self.value.get() }
}
@@ -232,7 +232,7 @@ pub fn get_mut(&mut self) -> &mut T {
/// /// Increment the counter in this instance.
/// ///
/// /// The caller must hold the `MY_MUTEX` mutex.
-/// fn increment(&self, guard: &mut GlobalGuard<MY_MUTEX>) -> u32 {
+/// fn increment(&self, guard: &mut GlobalGuard<'_, MY_MUTEX>) -> u32 {
/// let my_counter = self.my_counter.as_mut(guard);
/// *my_counter += 1;
/// *my_counter
--
2.51.1
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v14 13/16] rust: sync: Expose lock::Backend
2025-11-20 21:45 [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust Lyude Paul
` (11 preceding siblings ...)
2025-11-20 21:46 ` [PATCH v14 12/16] rust: sync: Add a lifetime parameter to lock::global::GlobalGuard Lyude Paul
@ 2025-11-20 21:46 ` Lyude Paul
2025-11-20 21:46 ` [PATCH v14 14/16] rust: sync: lock/global: Add Backend parameter to GlobalGuard Lyude Paul
` (3 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Lyude Paul @ 2025-11-20 21:46 UTC (permalink / raw)
To: rust-for-linux, linux-kernel, Thomas Gleixner
Cc: Boqun Feng, Daniel Almeida, Miguel Ojeda, Alex Gaynor, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Andrew Morton, Peter Zijlstra,
Ingo Molnar, Will Deacon, Waiman Long
Due to the addition of sync::lock::Backend::Context, lock guards can be
returned with a different Backend than their respective lock. Since we'll
be adding a trait bound for Backend to GlobalGuard in order to support
this, users will need to be able to directly refer to Backend so that they
can use it in trait bounds.
So, let's make this easier for users and expose Backend in sync.
Signed-off-by: Lyude Paul <lyude@redhat.com>
---
rust/kernel/sync.rs | 1 +
1 file changed, 1 insertion(+)
diff --git a/rust/kernel/sync.rs b/rust/kernel/sync.rs
index f293bbe13e855..795cbf3fc10f7 100644
--- a/rust/kernel/sync.rs
+++ b/rust/kernel/sync.rs
@@ -29,6 +29,7 @@
pub use lock::spinlock::{
new_spinlock, new_spinlock_irq, SpinLock, SpinLockGuard, SpinLockIrq, SpinLockIrqGuard,
};
+pub use lock::Backend;
pub use locked_by::LockedBy;
pub use refcount::Refcount;
--
2.51.1
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v14 14/16] rust: sync: lock/global: Add Backend parameter to GlobalGuard
2025-11-20 21:45 [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust Lyude Paul
` (12 preceding siblings ...)
2025-11-20 21:46 ` [PATCH v14 13/16] rust: sync: Expose lock::Backend Lyude Paul
@ 2025-11-20 21:46 ` Lyude Paul
2025-11-20 21:46 ` [PATCH v14 15/16] rust: sync: lock/global: Add BackendInContext support to GlobalLock Lyude Paul
` (2 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Lyude Paul @ 2025-11-20 21:46 UTC (permalink / raw)
To: rust-for-linux, linux-kernel, Thomas Gleixner
Cc: Boqun Feng, Daniel Almeida, Miguel Ojeda, Alex Gaynor, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Andrew Morton, Peter Zijlstra,
Ingo Molnar, Will Deacon, Waiman Long
Due to the introduction of sync::lock::Backend::Context, it's now possible
for normal locks to return a Guard with a different Backend than their
respective lock (e.g. Backend::BackendInContext). We want to be able to
support global locks with contexts as well, so add a trait bound to
explicitly specify which Backend is in use for a GlobalGuard.
Signed-off-by: Lyude Paul <lyude@redhat.com>
---
rust/kernel/sync/lock/global.rs | 20 ++++++++++----------
1 file changed, 10 insertions(+), 10 deletions(-)
diff --git a/rust/kernel/sync/lock/global.rs b/rust/kernel/sync/lock/global.rs
index 1d279be1a2eca..31264b68086b3 100644
--- a/rust/kernel/sync/lock/global.rs
+++ b/rust/kernel/sync/lock/global.rs
@@ -77,14 +77,14 @@ pub unsafe fn init(&'static self) {
}
/// Lock this global lock.
- pub fn lock(&'static self) -> GlobalGuard<'static, G> {
+ pub fn lock(&'static self) -> GlobalGuard<'static, G, G::Backend> {
GlobalGuard {
inner: self.inner.lock(),
}
}
/// Try to lock this global lock.
- pub fn try_lock(&'static self) -> Option<GlobalGuard<'static, G>> {
+ pub fn try_lock(&'static self) -> Option<GlobalGuard<'static, G, G::Backend>> {
Some(GlobalGuard {
inner: self.inner.try_lock()?,
})
@@ -94,11 +94,11 @@ pub fn try_lock(&'static self) -> Option<GlobalGuard<'static, G>> {
/// A guard for a [`GlobalLock`].
///
/// See [`global_lock!`] for examples.
-pub struct GlobalGuard<'a, G: GlobalLockBackend> {
- inner: Guard<'a, G::Item, G::Backend>,
+pub struct GlobalGuard<'a, G: GlobalLockBackend, B: Backend> {
+ inner: Guard<'a, G::Item, B>,
}
-impl<'a, G: GlobalLockBackend> core::ops::Deref for GlobalGuard<'a, G> {
+impl<'a, G: GlobalLockBackend, B: Backend> core::ops::Deref for GlobalGuard<'a, G, B> {
type Target = G::Item;
fn deref(&self) -> &Self::Target {
@@ -106,7 +106,7 @@ fn deref(&self) -> &Self::Target {
}
}
-impl<'a, G: GlobalLockBackend> core::ops::DerefMut for GlobalGuard<'a, G> {
+impl<'a, G: GlobalLockBackend, B: Backend> core::ops::DerefMut for GlobalGuard<'a, G, B> {
fn deref_mut(&mut self) -> &mut Self::Target {
&mut self.inner
}
@@ -154,7 +154,7 @@ impl<T: ?Sized, G: GlobalLockBackend> GlobalLockedBy<T, G> {
/// Access the value immutably.
///
/// The caller must prove shared access to the lock.
- pub fn as_ref<'a>(&'a self, _guard: &'a GlobalGuard<'_, G>) -> &'a T {
+ pub fn as_ref<'a, B: Backend>(&'a self, _guard: &'a GlobalGuard<'_, G, B>) -> &'a T {
// SAFETY: The lock is globally unique, so there can only be one guard.
unsafe { &*self.value.get() }
}
@@ -162,7 +162,7 @@ pub fn as_ref<'a>(&'a self, _guard: &'a GlobalGuard<'_, G>) -> &'a T {
/// Access the value mutably.
///
/// The caller must prove shared exclusive to the lock.
- pub fn as_mut<'a>(&'a self, _guard: &'a mut GlobalGuard<'_, G>) -> &'a mut T {
+ pub fn as_mut<'a, B: Backend>(&'a self, _guard: &'a mut GlobalGuard<'_, G, B>) -> &'a mut T {
// SAFETY: The lock is globally unique, so there can only be one guard.
unsafe { &mut *self.value.get() }
}
@@ -216,7 +216,7 @@ pub fn get_mut(&mut self) -> &mut T {
/// ```
/// # mod ex {
/// # use kernel::prelude::*;
-/// use kernel::sync::{GlobalGuard, GlobalLockedBy};
+/// use kernel::sync::{Backend, GlobalGuard, GlobalLockedBy};
///
/// kernel::sync::global_lock! {
/// // SAFETY: Initialized in module initializer before first use.
@@ -232,7 +232,7 @@ pub fn get_mut(&mut self) -> &mut T {
/// /// Increment the counter in this instance.
/// ///
/// /// The caller must hold the `MY_MUTEX` mutex.
-/// fn increment(&self, guard: &mut GlobalGuard<'_, MY_MUTEX>) -> u32 {
+/// fn increment<B: Backend>(&self, guard: &mut GlobalGuard<'_, MY_MUTEX, B>) -> u32 {
/// let my_counter = self.my_counter.as_mut(guard);
/// *my_counter += 1;
/// *my_counter
--
2.51.1
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v14 15/16] rust: sync: lock/global: Add BackendInContext support to GlobalLock
2025-11-20 21:45 [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust Lyude Paul
` (13 preceding siblings ...)
2025-11-20 21:46 ` [PATCH v14 14/16] rust: sync: lock/global: Add Backend parameter to GlobalGuard Lyude Paul
@ 2025-11-20 21:46 ` Lyude Paul
2025-11-20 21:46 ` [PATCH v14 16/16] locking: Switch to _irq_{disable,enable}() variants in cleanup guards Lyude Paul
2025-11-20 23:16 ` [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust John Hubbard
16 siblings, 0 replies; 25+ messages in thread
From: Lyude Paul @ 2025-11-20 21:46 UTC (permalink / raw)
To: rust-for-linux, linux-kernel, Thomas Gleixner
Cc: Boqun Feng, Daniel Almeida, Miguel Ojeda, Alex Gaynor, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Andrew Morton, Peter Zijlstra,
Ingo Molnar, Will Deacon, Waiman Long
Now that we have the ability to provide an explicit lifetime for a
GlobalGuard and an explicit Backend for a GlobalGuard, we can finally
implement lock_with() and try_lock_with().
Signed-off-by: Lyude Paul <lyude@redhat.com>
---
rust/kernel/sync/lock/global.rs | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/rust/kernel/sync/lock/global.rs b/rust/kernel/sync/lock/global.rs
index 31264b68086b3..20fbe11d1a04b 100644
--- a/rust/kernel/sync/lock/global.rs
+++ b/rust/kernel/sync/lock/global.rs
@@ -89,6 +89,34 @@ pub fn try_lock(&'static self) -> Option<GlobalGuard<'static, G, G::Backend>> {
inner: self.inner.try_lock()?,
})
}
+
+ /// Lock this global lock with the provided `context`.
+ pub fn lock_with<'a, B>(
+ &'static self,
+ context: <G::Backend as Backend>::Context<'a>,
+ ) -> GlobalGuard<'a, G, B>
+ where
+ G::Backend: Backend<BackendInContext = B>,
+ B: Backend,
+ {
+ GlobalGuard {
+ inner: self.inner.lock_with(context),
+ }
+ }
+
+ /// Try to lock this global lock with the provided `context`.
+ pub fn try_lock_with<'a, B>(
+ &'static self,
+ context: <G::Backend as Backend>::Context<'a>,
+ ) -> Option<GlobalGuard<'a, G, B>>
+ where
+ G::Backend: Backend<BackendInContext = B>,
+ B: Backend,
+ {
+ Some(GlobalGuard {
+ inner: self.inner.try_lock_with(context)?,
+ })
+ }
}
/// A guard for a [`GlobalLock`].
--
2.51.1
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v14 16/16] locking: Switch to _irq_{disable,enable}() variants in cleanup guards
2025-11-20 21:45 [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust Lyude Paul
` (14 preceding siblings ...)
2025-11-20 21:46 ` [PATCH v14 15/16] rust: sync: lock/global: Add BackendInContext support to GlobalLock Lyude Paul
@ 2025-11-20 21:46 ` Lyude Paul
2025-11-20 23:16 ` [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust John Hubbard
16 siblings, 0 replies; 25+ messages in thread
From: Lyude Paul @ 2025-11-20 21:46 UTC (permalink / raw)
To: rust-for-linux, linux-kernel, Thomas Gleixner
Cc: Boqun Feng, Daniel Almeida, Miguel Ojeda, Alex Gaynor, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Andrew Morton, Peter Zijlstra,
Ingo Molnar, Will Deacon, Waiman Long
From: Boqun Feng <boqun.feng@gmail.com>
The semantics of various irq disabling guards match what
*_irq_{disable,enable}() provide, i.e. the interrupt disabling is
properly nested, therefore it's OK to switch to use
*_irq_{disable,enable}() primitives.
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
---
V10:
* Add PREEMPT_RT build fix from Guangbo Cui
include/linux/spinlock.h | 26 ++++++++++++--------------
1 file changed, 12 insertions(+), 14 deletions(-)
diff --git a/include/linux/spinlock.h b/include/linux/spinlock.h
index bbbee61c6f5df..72bb6ae5319c7 100644
--- a/include/linux/spinlock.h
+++ b/include/linux/spinlock.h
@@ -568,10 +568,10 @@ DEFINE_LOCK_GUARD_1(raw_spinlock_nested, raw_spinlock_t,
raw_spin_unlock(_T->lock))
DEFINE_LOCK_GUARD_1(raw_spinlock_irq, raw_spinlock_t,
- raw_spin_lock_irq(_T->lock),
- raw_spin_unlock_irq(_T->lock))
+ raw_spin_lock_irq_disable(_T->lock),
+ raw_spin_unlock_irq_enable(_T->lock))
-DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irq, _try, raw_spin_trylock_irq(_T->lock))
+DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irq, _try, raw_spin_trylock_irq_disable(_T->lock))
DEFINE_LOCK_GUARD_1(raw_spinlock_bh, raw_spinlock_t,
raw_spin_lock_bh(_T->lock),
@@ -580,12 +580,11 @@ DEFINE_LOCK_GUARD_1(raw_spinlock_bh, raw_spinlock_t,
DEFINE_LOCK_GUARD_1_COND(raw_spinlock_bh, _try, raw_spin_trylock_bh(_T->lock))
DEFINE_LOCK_GUARD_1(raw_spinlock_irqsave, raw_spinlock_t,
- raw_spin_lock_irqsave(_T->lock, _T->flags),
- raw_spin_unlock_irqrestore(_T->lock, _T->flags),
- unsigned long flags)
+ raw_spin_lock_irq_disable(_T->lock),
+ raw_spin_unlock_irq_enable(_T->lock))
DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irqsave, _try,
- raw_spin_trylock_irqsave(_T->lock, _T->flags))
+ raw_spin_trylock_irq_disable(_T->lock))
DEFINE_LOCK_GUARD_1(spinlock, spinlock_t,
spin_lock(_T->lock),
@@ -594,11 +593,11 @@ DEFINE_LOCK_GUARD_1(spinlock, spinlock_t,
DEFINE_LOCK_GUARD_1_COND(spinlock, _try, spin_trylock(_T->lock))
DEFINE_LOCK_GUARD_1(spinlock_irq, spinlock_t,
- spin_lock_irq(_T->lock),
- spin_unlock_irq(_T->lock))
+ spin_lock_irq_disable(_T->lock),
+ spin_unlock_irq_enable(_T->lock))
DEFINE_LOCK_GUARD_1_COND(spinlock_irq, _try,
- spin_trylock_irq(_T->lock))
+ spin_trylock_irq_disable(_T->lock))
DEFINE_LOCK_GUARD_1(spinlock_bh, spinlock_t,
spin_lock_bh(_T->lock),
@@ -608,12 +607,11 @@ DEFINE_LOCK_GUARD_1_COND(spinlock_bh, _try,
spin_trylock_bh(_T->lock))
DEFINE_LOCK_GUARD_1(spinlock_irqsave, spinlock_t,
- spin_lock_irqsave(_T->lock, _T->flags),
- spin_unlock_irqrestore(_T->lock, _T->flags),
- unsigned long flags)
+ spin_lock_irq_disable(_T->lock),
+ spin_unlock_irq_enable(_T->lock))
DEFINE_LOCK_GUARD_1_COND(spinlock_irqsave, _try,
- spin_trylock_irqsave(_T->lock, _T->flags))
+ spin_trylock_irq_disable(_T->lock))
DEFINE_LOCK_GUARD_1(read_lock, rwlock_t,
read_lock(_T->lock),
--
2.51.1
^ permalink raw reply related [flat|nested] 25+ messages in thread* Re: [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust
2025-11-20 21:45 [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust Lyude Paul
` (15 preceding siblings ...)
2025-11-20 21:46 ` [PATCH v14 16/16] locking: Switch to _irq_{disable,enable}() variants in cleanup guards Lyude Paul
@ 2025-11-20 23:16 ` John Hubbard
2025-11-21 17:47 ` Boqun Feng
16 siblings, 1 reply; 25+ messages in thread
From: John Hubbard @ 2025-11-20 23:16 UTC (permalink / raw)
To: Lyude Paul, rust-for-linux, linux-kernel, Thomas Gleixner
Cc: Boqun Feng, Daniel Almeida, Miguel Ojeda, Alex Gaynor, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Andrew Morton, Peter Zijlstra,
Ingo Molnar, Will Deacon, Waiman Long
On 11/20/25 1:45 PM, Lyude Paul wrote:
...
> this new interface is indeed possible, more on this below.
> * Also thank to Joel, we also now have actual benchmarks for how this
> affects performance:
> https://lore.kernel.org/rust-for-linux/20250619175335.2905836-1-joelagnelf@nvidia.com/
> * Also some small changes to the kunit test I added, mainly just making
> sure I don't forget to include a MODULE_DESCRIPTION or MODULE_LICENSE.
Hi,
The above link says that local_interrupt takes 3.6x as as as local_irq.
This is alarming, but is it the final word? In other words, is the Rust
side of this doomed to slower performance forever, or is there some
hope of reaching performance parity with the C part of the kernel?
Do we have to start telling the Rust for Linux story this way: "our
new Rust-based drivers are slower, but memory-safer"?
I'm not able to deduce the answer from a quick scan through the patches
and the cover letter.
thanks,
--
John Hubbard
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust
2025-11-20 23:16 ` [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust John Hubbard
@ 2025-11-21 17:47 ` Boqun Feng
2025-11-22 1:09 ` John Hubbard
0 siblings, 1 reply; 25+ messages in thread
From: Boqun Feng @ 2025-11-21 17:47 UTC (permalink / raw)
To: John Hubbard
Cc: Lyude Paul, rust-for-linux, linux-kernel, Thomas Gleixner,
Daniel Almeida, Miguel Ojeda, Alex Gaynor, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Andrew Morton, Peter Zijlstra,
Ingo Molnar, Will Deacon, Waiman Long
On Thu, Nov 20, 2025 at 03:16:04PM -0800, John Hubbard wrote:
> On 11/20/25 1:45 PM, Lyude Paul wrote:
> ...
> > this new interface is indeed possible, more on this below.
> > * Also thank to Joel, we also now have actual benchmarks for how this
> > affects performance:
> > https://lore.kernel.org/rust-for-linux/20250619175335.2905836-1-joelagnelf@nvidia.com/
> > * Also some small changes to the kunit test I added, mainly just making
> > sure I don't forget to include a MODULE_DESCRIPTION or MODULE_LICENSE.
>
> Hi,
>
> The above link says that local_interrupt takes 3.6x as as as local_irq.
>
> This is alarming, but is it the final word? In other words, is the Rust
> side of this doomed to slower performance forever, or is there some
> hope of reaching performance parity with the C part of the kernel?
>
Note that local_interrupt API is for safe Rust code, you can always
use unsafe local_irq if the interrupt disabling is the performance
bottleneck for you. So language-wise there is no difference between Rust
and C.
> Do we have to start telling the Rust for Linux story this way: "our
> new Rust-based drivers are slower, but memory-safer"?
>
I would not jump into that conclusion at the moment, because 1) as I
mentioned you can always go into unsafe if something begins the
bottleneck, and 2) there is always a gap between micro benchmark results
and the whole system performance, being slow on one operation doesn't
means the whole system will perform observably worse.
Think about a similar thing in C, we recommend people to use existing
locks instead of customized synchronization vi atomics in most cases,
and technically, locks can be slower compared to a special
synchronization based on atomics, but it's more difficult to mess up.
Regards,
Boqun
> I'm not able to deduce the answer from a quick scan through the patches
> and the cover letter.
>
> thanks,
> --
> John Hubbard
>
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust
2025-11-21 17:47 ` Boqun Feng
@ 2025-11-22 1:09 ` John Hubbard
2025-11-22 2:38 ` Boqun Feng
0 siblings, 1 reply; 25+ messages in thread
From: John Hubbard @ 2025-11-22 1:09 UTC (permalink / raw)
To: Boqun Feng
Cc: Lyude Paul, rust-for-linux, linux-kernel, Thomas Gleixner,
Daniel Almeida, Miguel Ojeda, Alex Gaynor, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Andrew Morton, Peter Zijlstra,
Ingo Molnar, Will Deacon, Waiman Long
On 11/21/25 9:47 AM, Boqun Feng wrote:
> On Thu, Nov 20, 2025 at 03:16:04PM -0800, John Hubbard wrote:
>> On 11/20/25 1:45 PM, Lyude Paul wrote:
>> ...
>> This is alarming, but is it the final word? In other words, is the Rust
>> side of this doomed to slower performance forever, or is there some
>> hope of reaching performance parity with the C part of the kernel?
>>
>
> Note that local_interrupt API is for safe Rust code, you can always
> use unsafe local_irq if the interrupt disabling is the performance
> bottleneck for you. So language-wise there is no difference between Rust
> and C.
>
OK, but there *is* a performance difference between Safe Rust (which is
the whole point of this project, after all) and C.
Is 3.6x longer really something we are stuck with? Or is there some other
way forward that could potentially provide higher performance, for Safe
Rust?
>> Do we have to start telling the Rust for Linux story this way: "our
>> new Rust-based drivers are slower, but memory-safer"?
>>
>
> I would not jump into that conclusion at the moment, because 1) as I
> mentioned you can always go into unsafe if something begins the
> bottleneck, and 2) there is always a gap between micro benchmark results
> and the whole system performance, being slow on one operation doesn't
> means the whole system will perform observably worse.
>
> Think about a similar thing in C, we recommend people to use existing
> locks instead of customized synchronization vi atomics in most cases,
> and technically, locks can be slower compared to a special
> synchronization based on atomics, but it's more difficult to mess up.
>
Yes yes, I fully understand that micro benchmarks don't always translate
to a real-world observable effects. But interrupt operations...those can
be on a hot path. So it's prudent to worry about these.
thanks,
--
John Hubbard
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust
2025-11-22 1:09 ` John Hubbard
@ 2025-11-22 2:38 ` Boqun Feng
2025-11-22 2:56 ` John Hubbard
0 siblings, 1 reply; 25+ messages in thread
From: Boqun Feng @ 2025-11-22 2:38 UTC (permalink / raw)
To: John Hubbard
Cc: Lyude Paul, rust-for-linux, linux-kernel, Thomas Gleixner,
Daniel Almeida, Miguel Ojeda, Alex Gaynor, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Andrew Morton, Peter Zijlstra,
Ingo Molnar, Will Deacon, Waiman Long
On Fri, Nov 21, 2025 at 05:09:24PM -0800, John Hubbard wrote:
> On 11/21/25 9:47 AM, Boqun Feng wrote:
> > On Thu, Nov 20, 2025 at 03:16:04PM -0800, John Hubbard wrote:
> >> On 11/20/25 1:45 PM, Lyude Paul wrote:
> >> ...
> >> This is alarming, but is it the final word? In other words, is the Rust
> >> side of this doomed to slower performance forever, or is there some
> >> hope of reaching performance parity with the C part of the kernel?
> >>
> >
> > Note that local_interrupt API is for safe Rust code, you can always
> > use unsafe local_irq if the interrupt disabling is the performance
> > bottleneck for you. So language-wise there is no difference between Rust
> > and C.
> >
>
> OK, but there *is* a performance difference between Safe Rust (which is
> the whole point of this project, after all) and C.
>
Again, this is a premature statement.
First of all, the safe SpinLockIrq API is made to work with other API
like CondVar, there are certain design requirements making it being
implemented in a certain way. In other words, the cost is justified.
Second, one safe API being slow than unsafe code or C doesn't mean Safe
Rust is slow than C in all the cases.
Last but not least, safe Rust is preferred, but it doesn't mean unsafe
code should be avoided completely, if we establish some data that shows
some unsafe code provides better performance and we have clear guideline
for the particular scenarios, then it's definitely OK. Hence I don't
fully agree your saying "Safe Rust is the whole point of this project",
to me understanding how we can utilize the type system and other tools
is more of a realistic goal.
> Is 3.6x longer really something we are stuck with? Or is there some other
> way forward that could potentially provide higher performance, for Safe
> Rust?
>
Well by 3.6x longer, you mean ~1.3ns vs ~4.5ns, right? And in real world
code, the code in the interrupt disabling critical section would be more
than couples of nano seconds, hence the delta will probably be
noise-out. But again, yes if 3ns turns out to be a bottleneck in the
driver, we are happy to look into, but you need to show the data.
>
> >> Do we have to start telling the Rust for Linux story this way: "our
> >> new Rust-based drivers are slower, but memory-safer"?
> >>
> >
> > I would not jump into that conclusion at the moment, because 1) as I
> > mentioned you can always go into unsafe if something begins the
> > bottleneck, and 2) there is always a gap between micro benchmark results
> > and the whole system performance, being slow on one operation doesn't
> > means the whole system will perform observably worse.
> >
> > Think about a similar thing in C, we recommend people to use existing
> > locks instead of customized synchronization vi atomics in most cases,
> > and technically, locks can be slower compared to a special
> > synchronization based on atomics, but it's more difficult to mess up.
> >
>
> Yes yes, I fully understand that micro benchmarks don't always translate
> to a real-world observable effects. But interrupt operations...those can
> be on a hot path. So it's prudent to worry about these.
>
Note that it's the interrupt *disabling* operations, which means the
code could be otherwise interrupted outside the critical section, so yes
it could still be hot path, but there are more things than 3ns to affect
here.
Also one thing to notice is that
local_interrupt_disable();
<some other function>
local_interrupt_disable();
should be cheaper than
local_irq_save();
<some other function>
local_irq_save();
because the latter will access the interrupt disabling register twice.
So it's really hard to say whether the new API is strictly worse than
the existing ones.
Regards,
Boqun
>
> thanks,
> --
> John Hubbard
>
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust
2025-11-22 2:38 ` Boqun Feng
@ 2025-11-22 2:56 ` John Hubbard
2025-11-22 3:35 ` Boqun Feng
0 siblings, 1 reply; 25+ messages in thread
From: John Hubbard @ 2025-11-22 2:56 UTC (permalink / raw)
To: Boqun Feng
Cc: Lyude Paul, rust-for-linux, linux-kernel, Thomas Gleixner,
Daniel Almeida, Miguel Ojeda, Alex Gaynor, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Andrew Morton, Peter Zijlstra,
Ingo Molnar, Will Deacon, Waiman Long
On 11/21/25 6:38 PM, Boqun Feng wrote:
> On Fri, Nov 21, 2025 at 05:09:24PM -0800, John Hubbard wrote:
>> On 11/21/25 9:47 AM, Boqun Feng wrote:
>>> On Thu, Nov 20, 2025 at 03:16:04PM -0800, John Hubbard wrote:
>>>> On 11/20/25 1:45 PM, Lyude Paul wrote:
>>>> ...
>> OK, but there *is* a performance difference between Safe Rust (which is
>> the whole point of this project, after all) and C.
>
> Again, this is a premature statement.
Yes, it is. :)
>
> First of all, the safe SpinLockIrq API is made to work with other API
> like CondVar, there are certain design requirements making it being
> implemented in a certain way. In other words, the cost is justified.
This argument doesn't apply directly, because the response to "Safe Rust
is slower in this case", is "the slowdown is justified".
Perhaps it is, but that doesn't refute the claim. It's still slower.
Maybe that's not a problem at all, but it's not yet clear.
>
> Second, one safe API being slow than unsafe code or C doesn't mean Safe
> Rust is slow than C in all the cases.
Yes, we have already mentioned several times that micro-benchmarks are
not the final word. However, they can be an early indication of a problem.
>
> Last but not least, safe Rust is preferred, but it doesn't mean unsafe
> code should be avoided completely, if we establish some data that shows
Perhaps we need to be slightly more precise. I'm not sure if you are
referring to the usual practice of creating an unsafe block, wrapped
within a safe Rust function, or something else?
> some unsafe code provides better performance and we have clear guideline
> for the particular scenarios, then it's definitely OK. Hence I don't
> fully agree your saying "Safe Rust is the whole point of this project",
> to me understanding how we can utilize the type system and other tools
> is more of a realistic goal.
>
>> Is 3.6x longer really something we are stuck with? Or is there some other
>> way forward that could potentially provide higher performance, for Safe
>> Rust?
>>
>
> Well by 3.6x longer, you mean ~1.3ns vs ~4.5ns, right? And in real world
> code, the code in the interrupt disabling critical section would be more
> than couples of nano seconds, hence the delta will probably be
> noise-out. But again, yes if 3ns turns out to be a bottleneck in the
> driver, we are happy to look into, but you need to show the data.
>
So this is what I'm asking about: given that we *already know* that we
have a performance drop in the micro-benchmark, is there any reasonable
approach that avoids this? Or has a less noticeable impact?
I'm asking early (see above: I agree that this is "premature"), because
we have early data.
It would be nice to explore now, rather than later, after someone shows
up with detailed perf data about their use case.
thanks,
--
John Hubbard
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust
2025-11-22 2:56 ` John Hubbard
@ 2025-11-22 3:35 ` Boqun Feng
2025-11-22 4:14 ` John Hubbard
0 siblings, 1 reply; 25+ messages in thread
From: Boqun Feng @ 2025-11-22 3:35 UTC (permalink / raw)
To: John Hubbard
Cc: Lyude Paul, rust-for-linux, linux-kernel, Thomas Gleixner,
Daniel Almeida, Miguel Ojeda, Alex Gaynor, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Andrew Morton, Peter Zijlstra,
Ingo Molnar, Will Deacon, Waiman Long
On Fri, Nov 21, 2025 at 06:56:28PM -0800, John Hubbard wrote:
> On 11/21/25 6:38 PM, Boqun Feng wrote:
[...]
> >
> > Last but not least, safe Rust is preferred, but it doesn't mean unsafe
> > code should be avoided completely, if we establish some data that shows
>
> Perhaps we need to be slightly more precise. I'm not sure if you are
> referring to the usual practice of creating an unsafe block, wrapped
> within a safe Rust function, or something else?
>
I was referring to providing an unsafe API for core kernel
functionality, for example local_irq_disable(), and then teaching how to
use it correctly.
> > some unsafe code provides better performance and we have clear guideline
> > for the particular scenarios, then it's definitely OK. Hence I don't
> > fully agree your saying "Safe Rust is the whole point of this project",
> > to me understanding how we can utilize the type system and other tools
> > is more of a realistic goal.
> >
> >> Is 3.6x longer really something we are stuck with? Or is there some other
> >> way forward that could potentially provide higher performance, for Safe
> >> Rust?
> >>
> >
> > Well by 3.6x longer, you mean ~1.3ns vs ~4.5ns, right? And in real world
> > code, the code in the interrupt disabling critical section would be more
> > than couples of nano seconds, hence the delta will probably be
> > noise-out. But again, yes if 3ns turns out to be a bottleneck in the
> > driver, we are happy to look into, but you need to show the data.
> >
>
> So this is what I'm asking about: given that we *already know* that we
> have a performance drop in the micro-benchmark, is there any reasonable
> approach that avoids this? Or has a less noticeable impact?
>
Lyude had tried another approach [1], which uses an unsafe public API,
and doesn't work (easily) with CondVar or PREEMPT_RT And that eventually
triggered more discussion about a better API design, and as Thomas
pointed out [2]: "Stop worrying about mostly irrelevant low level
details which are not relevant to the primary audience of rust adoption.
We can worry about them when we replace the scheduler and the low level
interrupt handling code ten years down the road." And I agreed. The
current implementation is actually quite efficient and should even
out-perform the existing API in some cases as I pointed out. More
importantly, it utilizes Rust type system and make it easy to use (or
hard to mis-use).
That being said, if anyone has a better idea, feel free to bring it up.
> I'm asking early (see above: I agree that this is "premature"), because
> we have early data.
>
> It would be nice to explore now, rather than later, after someone shows
> up with detailed perf data about their use case.
>
>
Not sure I fully agree with this, given it's to my knowledge the best
solution at the moment, I feel it's hard to justify the cost of
exploring a better solution without a real usage. But then again, if
anyone has any better idea feel free to bring it up.
[1]: https://lore.kernel.org/rust-for-linux/20240916213025.477225-2-lyude@redhat.com/
[2]: https://lore.kernel.org/rust-for-linux/87iktrahld.ffs@tglx/
Regards,
Boqun
> thanks,
> --
> John Hubbard
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust
2025-11-22 3:35 ` Boqun Feng
@ 2025-11-22 4:14 ` John Hubbard
2025-11-22 4:24 ` Boqun Feng
0 siblings, 1 reply; 25+ messages in thread
From: John Hubbard @ 2025-11-22 4:14 UTC (permalink / raw)
To: Boqun Feng
Cc: Lyude Paul, rust-for-linux, linux-kernel, Thomas Gleixner,
Daniel Almeida, Miguel Ojeda, Alex Gaynor, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Andrew Morton, Peter Zijlstra,
Ingo Molnar, Will Deacon, Waiman Long
On 11/21/25 7:35 PM, Boqun Feng wrote:
> On Fri, Nov 21, 2025 at 06:56:28PM -0800, John Hubbard wrote:
>> On 11/21/25 6:38 PM, Boqun Feng wrote:
> [...]
>>>
>>> Last but not least, safe Rust is preferred, but it doesn't mean unsafe
>>> code should be avoided completely, if we establish some data that shows
>>
>> Perhaps we need to be slightly more precise. I'm not sure if you are
>> referring to the usual practice of creating an unsafe block, wrapped
>> within a safe Rust function, or something else?
>>
>
> I was referring to providing an unsafe API for core kernel
> functionality, for example local_irq_disable(), and then teaching how to
> use it correctly.
Ack.
>
>>> some unsafe code provides better performance and we have clear guideline
>>> for the particular scenarios, then it's definitely OK. Hence I don't
>>> fully agree your saying "Safe Rust is the whole point of this project",
>>> to me understanding how we can utilize the type system and other tools
>>> is more of a realistic goal.
>>>
>>>> Is 3.6x longer really something we are stuck with? Or is there some other
>>>> way forward that could potentially provide higher performance, for Safe
>>>> Rust?
>>>>
>>>
>>> Well by 3.6x longer, you mean ~1.3ns vs ~4.5ns, right? And in real world
>>> code, the code in the interrupt disabling critical section would be more
>>> than couples of nano seconds, hence the delta will probably be
>>> noise-out. But again, yes if 3ns turns out to be a bottleneck in the
>>> driver, we are happy to look into, but you need to show the data.
>>>
>>
>> So this is what I'm asking about: given that we *already know* that we
>> have a performance drop in the micro-benchmark, is there any reasonable
>> approach that avoids this? Or has a less noticeable impact?
>>
>
> Lyude had tried another approach [1], which uses an unsafe public API,
> and doesn't work (easily) with CondVar or PREEMPT_RT And that eventually
> triggered more discussion about a better API design, and as Thomas
> pointed out [2]: "Stop worrying about mostly irrelevant low level
> details which are not relevant to the primary audience of rust adoption.
> We can worry about them when we replace the scheduler and the low level
> interrupt handling code ten years down the road." And I agreed. The
> current implementation is actually quite efficient and should even
> out-perform the existing API in some cases as I pointed out. More
> importantly, it utilizes Rust type system and make it easy to use (or
> hard to mis-use).
>
> That being said, if anyone has a better idea, feel free to bring it up.
>
>> I'm asking early (see above: I agree that this is "premature"), because
>> we have early data.
>>
>> It would be nice to explore now, rather than later, after someone shows
>> up with detailed perf data about their use case.
>>
>>
>
> Not sure I fully agree with this, given it's to my knowledge the best
> solution at the moment, I feel it's hard to justify the cost of
> exploring a better solution without a real usage. But then again, if
> anyone has any better idea feel free to bring it up.
>
> [1]: https://lore.kernel.org/rust-for-linux/20240916213025.477225-2-lyude@redhat.com/
> [2]: https://lore.kernel.org/rust-for-linux/87iktrahld.ffs@tglx/
>
Thanks for this context, I hadn't followed the earlier discussions,
and when looking at this v14, it seemed to gloss over the performance
implications (they were linked to, but not discussed).
I won't further harass you all about this, let's see how it goes. :)
Optionally, it might be helpful to include some top-level notes
that justify the choices made so far.
thanks,
--
John Hubbard
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v14 00/16] Refcounted interrupts, SpinLockIrq for rust
2025-11-22 4:14 ` John Hubbard
@ 2025-11-22 4:24 ` Boqun Feng
0 siblings, 0 replies; 25+ messages in thread
From: Boqun Feng @ 2025-11-22 4:24 UTC (permalink / raw)
To: John Hubbard
Cc: Lyude Paul, rust-for-linux, linux-kernel, Thomas Gleixner,
Daniel Almeida, Miguel Ojeda, Alex Gaynor, Gary Guo,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Andrew Morton, Peter Zijlstra,
Ingo Molnar, Will Deacon, Waiman Long
On Fri, Nov 21, 2025 at 08:14:59PM -0800, John Hubbard wrote:
[...]
> >
> > Lyude had tried another approach [1], which uses an unsafe public API,
> > and doesn't work (easily) with CondVar or PREEMPT_RT And that eventually
> > triggered more discussion about a better API design, and as Thomas
> > pointed out [2]: "Stop worrying about mostly irrelevant low level
> > details which are not relevant to the primary audience of rust adoption.
> > We can worry about them when we replace the scheduler and the low level
> > interrupt handling code ten years down the road." And I agreed. The
> > current implementation is actually quite efficient and should even
> > out-perform the existing API in some cases as I pointed out. More
> > importantly, it utilizes Rust type system and make it easy to use (or
> > hard to mis-use).
> >
> > That being said, if anyone has a better idea, feel free to bring it up.
> >
> > > I'm asking early (see above: I agree that this is "premature"), because
> > > we have early data.
> > >
> > > It would be nice to explore now, rather than later, after someone shows
> > > up with detailed perf data about their use case.
> > >
> > >
> >
> > Not sure I fully agree with this, given it's to my knowledge the best
> > solution at the moment, I feel it's hard to justify the cost of
> > exploring a better solution without a real usage. But then again, if
> > anyone has any better idea feel free to bring it up.
> >
> > [1]: https://lore.kernel.org/rust-for-linux/20240916213025.477225-2-lyude@redhat.com/
> > [2]: https://lore.kernel.org/rust-for-linux/87iktrahld.ffs@tglx/
> >
>
> Thanks for this context, I hadn't followed the earlier discussions,
> and when looking at this v14, it seemed to gloss over the performance
> implications (they were linked to, but not discussed).
>
I could have provided this earlier ;-)
> I won't further harass you all about this, let's see how it goes. :)
>
> Optionally, it might be helpful to include some top-level notes
> that justify the choices made so far.
>
Sure, I will keep a note on that when applying the series. Thank you!
Regards,
Boqun
> thanks,
> --
> John Hubbard
>
^ permalink raw reply [flat|nested] 25+ messages in thread