Re: [PATCH v6 3/3] rust: sync: Add SpinLockIrq

[Boqun Feng <boqun.feng@gmail.com>]

On Wed, Oct 02, 2024 at 10:53:32PM +0200, Thomas Gleixner wrote:
> On Mon, Sep 16 2024 at 17:28, Lyude Paul wrote:
> > A variant of SpinLock that is expected to be used in noirq contexts, and
> > thus requires that the user provide an kernel::irq::IrqDisabled to prove
> > they are in such a context upon lock acquisition. This is the rust
> > equivalent of spin_lock_irqsave()/spin_lock_irqrestore().
> 
> This fundamentally does not work with CONFIG_PREEMPT_RT. See:
> 
>    https://www.kernel.org/doc/html/latest/locking/locktypes.html
> 
> for further information. TLDR:
> 
> On RT enabled kernels spin/rw_lock are substituted by sleeping locks. So
> you _cannot_ disable interrupts before taking the lock on RT enabled
> kernels. That will result in a 'might_sleep()' splat.
> 

One thing I was missing when I suggested Lyude with the current API is
that local_irq_save() disables interrupts even on RT. I was under the
impression that local_irq_save() will only disable preemption per:

	https://lwn.net/Articles/146861/

but seems it's not the case right now: we move the RT vs non-RT games
and hardware interrupt disabling vs preemption/migration disabling to
local_lock_*() I guess?

> There is a reason why the kernel has two distinct spinlock types:
> 
>     raw_spinlock_t and spinlock_t
> 
> raw_spinlock_t is a real spinning lock independent of CONFIG_PREEMPT_RT,
> spinlock_t is mapped to raw_spinlock_t on CONFIG_PREEMPT_RT=n and to a
> rtmutex based implementation for CONFIG_PREEMPT_RT=y.
> 
> As a consequence spin_lock_irq() and spin_lock_irqsave() will _NOT_
> disable interrupts on a CONFIG_PREEMPT_RT=y kernel.
> 
> The proposed rust abstraction is not suitable for that.
> 
> At this phase of rust integration there is no need to wrap
> raw_spinlock_t, so you have two options to solve that:
> 
>    1) Map Rust's SpinLockIrq() to spin_lock_irqsave() and
>       spin_unlock_irqrestore() which does the right thing
> 
>    2) Play all the PREEMPT_RT games in the local irq disable abstraction
> 
> #1 is the right thing to do because no driver should rely on actually
> disabling interrupts on the CPU. If there is a driver which does that,
> then it's not compatible with RT and should use a local lock instead.
> 
> local locks aside of being RT compatible have the benefit that they give
> scope to the protected region/data, while a plain local_irq_disable()
> does not.
> 
> Don't even think about exposing this 'with_irq_disabled' interface
> unless you are trying to move actual core code like the scheduler or low
> level interrupt handling to rust.
> 
> Create explicit interrupt safe interfaces which map to the underlying
> locking primitives instead.
> 

Then we should have a SpinLockIrq<T> type, and a function:

	fn with_locked<U>(&self, cb: impl FnOnce(&mut T) -> U) -> U {
	    <spin_lock_irqsave()>
	    let ret = cb(...);
	    <spin_lock_irqrestore()>
	    ret
	}

FYI, the reason that we cannot have a SpinLockIrq::lock() return a guard
that holds the irq state is because:

	https://github.com/Rust-for-Linux/linux/issues/998

namely:

	// interrupts are enabled here.
	let a = spin_lock_irq1.lock();
	let b = spin_lock_irq2.lock();

	drop(a); // releasing spin_lock_irq1 and restore the irq state.
	// `b` exists with interrupts enabled, which breaks the
	// invariants of b.

(technically we can, but that requires a rework of how nested
irq_save()s are handled, that's another can of worms)

Regards,
Boqun

> Thanks,
> 
>         tglx
> 
>