Re: [patch V2 08/15] Documentation: Add lock ordering and nesting documentation

Paul E. McKenney
2020-03-20T21:02:48Z

On Fri, Mar 20, 2020 at 08:51:44PM +0100, Thomas Gleixner wrote:
> "Paul E. McKenney" <paulmck@kernel.org> writes:
> >
> > - The soft interrupt related suffix (_bh()) still disables softirq
> > handlers. However, unlike non-PREEMPT_RT kernels (which disable
> > preemption to get this effect), PREEMPT_RT kernels use a per-CPU
> > lock to exclude softirq handlers.
>
> I've made that:
>
> - The soft interrupt related suffix (_bh()) still disables softirq
> handlers.
>
> Non-PREEMPT_RT kernels disable preemption to get this effect.
>
> PREEMPT_RT kernels use a per-CPU lock for serialization. The lock
> disables softirq handlers and prevents reentrancy by a preempting
> task.

That works! At the end, I would instead say "prevents reentrancy
due to task preemption", but what you have works.

> On non-RT this is implicit through preemption disable, but it's non
> obvious for RT as preemption stays enabled.
>
> > PREEMPT_RT kernels preserve all other spinlock_t semantics:
> >
> > - Tasks holding a spinlock_t do not migrate. Non-PREEMPT_RT kernels
> > avoid migration by disabling preemption. PREEMPT_RT kernels instead
> > disable migration, which ensures that pointers to per-CPU variables
> > remain valid even if the task is preempted.
> >
> > - Task state is preserved across spinlock acquisition, ensuring that the
> > task-state rules apply to all kernel configurations. In non-PREEMPT_RT
> > kernels leave task state untouched. However, PREEMPT_RT must change
> > task state if the task blocks during acquisition. Therefore, the
> > corresponding lock wakeup restores the task state. Note that regular
> > (not lock related) wakeups do not restore task state.
>
> - Task state is preserved across spinlock acquisition, ensuring that the
> task-state rules apply to all kernel configurations. Non-PREEMPT_RT
> kernels leave task state untouched. However, PREEMPT_RT must change
> task state if the task blocks during acquisition. Therefore, it
> saves the current task state before blocking and the corresponding
> lock wakeup restores it. A regular not lock related wakeup sets the
> task state to RUNNING. If this happens while the task is blocked on
> a spinlock then the saved task state is changed so that correct
> state is restored on lock wakeup.
>
> Hmm?

I of course cannot resist editing the last two sentences:

... Other types of wakeups unconditionally set task state to RUNNING.
If this happens while a task is blocked while acquiring a spinlock,
then the task state is restored to its pre-acquisition value at
lock-wakeup time.

> > But this code failes on PREEMPT_RT kernels because the memory allocator
> > is fully preemptible and therefore cannot be invoked from truly atomic
> > contexts. However, it is perfectly fine to invoke the memory allocator
> > while holding a normal non-raw spinlocks because they do not disable
> > preemption::
> >
> >> + spin_lock(&lock);
> >> + p = kmalloc(sizeof(*p), GFP_ATOMIC);
> >> +
> >> +Most places which use GFP_ATOMIC allocations are safe on PREEMPT_RT as the
> >> +execution is forced into thread context and the lock substitution is
> >> +ensuring preemptibility.
> >
> > Interestingly enough, most uses of GFP_ATOMIC allocations are
> > actually safe on PREEMPT_RT because the the lock substitution ensures
> > preemptibility. Only those GFP_ATOMIC allocations that are invoke
> > while holding a raw spinlock or with preemption otherwise disabled need
> > adjustment to work correctly on PREEMPT_RT.
> >
> > [ I am not as confident of the above as I would like to be... ]
>
> I'd leave that whole paragraph out. This documents the rules and from
> the above code examples it's pretty clear what works and what not :)

Works for me! ;-)

> > And meeting time, will continue later!
>
> Enjoy!

Not bad, actually, as meetings go.

Thanx, Paul