osq_lock on arm64

osq_lock on arm64

[David Laight]

Commit f5bfdc8e3947 change the osq_lock (optimistic spin queue) code to
use smp_cond_load_relaxed() to wait for another cpu to relase the lock.

It waits for 'VAL || need_resched()' with a comment that the IPI that
sets need_resched() will break any monitor/wait code.
However arm64 implements it as (03e3c2b7edbe and 1cfc63b5ae6):

#define smp_cond_load_acquire(ptr, cond_expr)                          \
({                                                                     \
       typeof(ptr) __PTR = (ptr);                                      \
       typeof(*ptr) VAL;                                               \
       for (;;) {                                                      \
               VAL = smp_load_acquire(__PTR);                          \
               if (cond_expr)                                          \
                       break;                                          \
               __cmpwait_relaxed(__PTR, VAL);                          \
       }                                                               \
       VAL;                                                            \
})

#define __CMPWAIT_CASE(w, sz, name)                                    \
static inline void __cmpwait_case_##name(volatile void *ptr,           \
                                        unsigned long val)             \
{                                                                      \
       unsigned long tmp;                                              \
                                                                       \
       asm volatile(                                                   \
       "       sevl\n"                                                 \
       "       wfe\n"                                                  \
       "       ldxr" #sz "\t%" #w "[tmp], %[v]\n"                      \
       "       eor     %" #w "[tmp], %" #w "[tmp], %" #w "[val]\n"     \
       "       cbnz    %" #w "[tmp], 1f\n"                             \
       "       wfe\n"                                                  \
       "1:"                                                            \
       : [tmp] "=&r" (tmp), [v] "+Q" (*(unsigned long *)ptr)           \
       : [val] "r" (val));                                             \
}

The "eor" and "cbnz" seem to be there to skip the "wfe" if the value has
already changed, but this only works if the condition only depends on VAL.
This means it can miss the IPI that makes need_resched() return non-zero.

To work I think it would need the cmp_load_acquire() to be "ldxr" with
a "wfe" instead of __cmpwait_relaxed().
But that wouldn't work if 'cond_expr' issued another "ldx".

The documentation I found for the arm64 instruction set doesn't mention that
losing the exclusive access requested by ldx will wake wfe, I assume that
is an omission from the docs?

	David 

Re: osq_lock on arm64

[Marc Zyngier]

On Sun, 15 Mar 2026 20:11:53 +0000,
David Laight <david.laight.linux@gmail.com> wrote:

[...]

> The documentation I found for the arm64 instruction set doesn't mention that
> losing the exclusive access requested by ldx will wake wfe, I assume that
> is an omission from the docs?

R_XRZRK explicitly calls for "The clearing of the global monitor for
the PE." as a something that sets the event register, and therefore
acts as a wake-up event from WFE.

	M.

-- 
Jazz isn't dead. It just smells funny.