Re: [RFC RESEND v10 03/14] irq & spin_lock: Add counted interrupt disabling/enabling

[Joel Fernandes <joelagnelf@nvidia.com>]

On Wed, May 28, 2025 at 07:47:09AM -0700, Boqun Feng wrote:
> On Wed, May 28, 2025 at 11:10:23AM +0200, Peter Zijlstra wrote:
> > On Tue, May 27, 2025 at 06:21:44PM -0400, Lyude Paul wrote:
> > > From: Boqun Feng <boqun.feng@gmail.com>
> > > 
> > > Currently the nested interrupt disabling and enabling is present by
> > > _irqsave() and _irqrestore() APIs, which are relatively unsafe, for
> > > example:
> > > 
> > > 	<interrupts are enabled as beginning>
> > > 	spin_lock_irqsave(l1, flag1);
> > > 	spin_lock_irqsave(l2, flag2);
> > > 	spin_unlock_irqrestore(l1, flags1);
> > > 	<l2 is still held but interrupts are enabled>
> > > 	// accesses to interrupt-disable protect data will cause races.
> > > 
> > > This is even easier to triggered with guard facilities:
> > > 
> > > 	unsigned long flag2;
> > > 
> > > 	scoped_guard(spin_lock_irqsave, l1) {
> > > 		spin_lock_irqsave(l2, flag2);
> > > 	}
> > > 	// l2 locked but interrupts are enabled.
> > > 	spin_unlock_irqrestore(l2, flag2);
> > > 
> > > (Hand-to-hand locking critical sections are not uncommon for a
> > > fine-grained lock design)
> > > 
> > > And because this unsafety, Rust cannot easily wrap the
> > > interrupt-disabling locks in a safe API, which complicates the design.
> > > 
> > > To resolve this, introduce a new set of interrupt disabling APIs:
> > > 
> > > *	local_interrupt_disable();
> > > *	local_interrupt_enable();
> > > 
> > > They work like local_irq_save() and local_irq_restore() except that 1)
> > > the outermost local_interrupt_disable() call save the interrupt state
> > > into a percpu variable, so that the outermost local_interrupt_enable()
> > > can restore the state, and 2) a percpu counter is added to record the
> > > nest level of these calls, so that interrupts are not accidentally
> > > enabled inside the outermost critical section.
> > > 
> > > Also add the corresponding spin_lock primitives: spin_lock_irq_disable()
> > > and spin_unlock_irq_enable(), as a result, code as follow:
> > > 
> > > 	spin_lock_irq_disable(l1);
> > > 	spin_lock_irq_disable(l2);
> > > 	spin_unlock_irq_enable(l1);
> > > 	// Interrupts are still disabled.
> > > 	spin_unlock_irq_enable(l2);
> > > 
> > > doesn't have the issue that interrupts are accidentally enabled.
> > > 
> > > This also makes the wrapper of interrupt-disabling locks on Rust easier
> > > to design.
> > > 
> > > Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
> > > 
> > > ---
> > > V10:
> > > * Add missing __raw_spin_lock_irq_disable() definition in spinlock.c
> > > 
> > > Signed-off-by: Lyude Paul <lyude@redhat.com>
> > 
> > Your SOB is placed wrong, should be below Boqun's. This way it gets
> > lost.
> > 
> > Also, is there effort planned to fully remove the save/restore variant?
> > As before, my main objection is adding variants with overlapping
> > functionality while not cleaning up the pre-existing code.
> > 
> 
> My plan is to map local_irq_disable() to local_interrupt_disable() and
> keep local_irq_save() as it is. That is, local_irq_disable() is the
> auto-pilot version and local_irq_save/restore() is the manual version.
> The reason is that I can see more "creative" (i.e. unpaired) usage of
> local_irq_save/restore(), and maybe someone would like to keep them.
> Thoughts?

My thought is it is better to keep them separate at first, let
local_interrupt_disable() stabilize with a few users, then convert the
callers (possibly with deprecation warnings with checkpatch), and then remove
the old API.

That appears lowest risk and easier transition.

thanks,

 - Joel