From: "Lorenzo Stoakes (ARM)" <ljs@kernel.org>
To: Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@redhat.com>, Will Deacon <will@kernel.org>,
Boqun Feng <boqun@kernel.org>, Waiman Long <longman@redhat.com>
Cc: linux-kernel@vger.kernel.org,
"David Hildenbrand (Arm)" <david@kernel.org>
Subject: Re: [PATCH] locking/rwsem: don't let a read lock steal from a queued writer
Date: Thu, 19 Nov 2026 14:14:45 +0000 [thread overview]
Message-ID: <av8DKviO1fVURLcJ@gremlin> (raw)
In-Reply-To: <20261105-rwsem-dont-steal-from-queued-writer-v1-1-275e070f426f@kernel.org>
I know the cycle is grim on workload (equally so for us), but ping on this, as
we're relying on this behaviour for a series [0].
I could re-implement as an optional mode and respin [0] if necessary, but Peter
objected [1] so hoisted the rwsem change to here :)
Thanks!
[0]: https://lore.kernel.org/linux-mm/20261110-khugepaged-latency-fix-v3-0-6eea743cf9a3@kernel.org/
[1]: https://lore.kernel.org/all/20261103115159.GK1813469@noisy.programming.kicks-ass.net/
On Thu, Nov 05, 2026 at 08:24:27PM +0000, Lorenzo Stoakes (ARM) wrote:
> Since commit 1a728dff855a ("locking/rwsem: Enable reader optimistic lock
> stealing"), a racing read lock can steal an rwsem from under a queued
> writer whose handoff timeout hasn't expired yet.
>
> This is most problematic when an rwsem read lock is released then
> reacquired in order to relieve lock contention, as instead it will simply
> have the lock handed right back to it ahead of the intended beneficiary.
>
> Naturally, this behaviour rather defeats the object of doing this kind of
> thing, so this is less than ideal.
>
> Fix this by updating the optimistic read lock stealing code in
> rwsem_down_read_slowpath() to explicitly check whether the first queued
> lock is a waiting writer, and to not allow the theft if so.
>
> When a steal is disallowed, keep the existing sem->wait_lock held when
> jumping to the queue code to avoid useless lock acquisition/release.
>
> In order to perform this check, sem->wait_lock is acquired whenever there
> are queued waiters in the steal code, rather than the previous behaviour of
> only doing so when the reader count is 1.
>
> Since failing to steal a read lock unconditionally results in acquisition
> of the sem->wait_lock, this doesn't seem likely to be a significant
> overhead.
>
> Also update the comment describing optimistic read lock acquisition
> conditions to include this newly introduced behaviour.
>
> Care is taken to ensure correct ordering of lock acquisition, lock waking
> and marking the lock reader-owned.
>
> Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
> ---
> kernel/locking/rwsem.c | 30 ++++++++++++++++++------------
> 1 file changed, 18 insertions(+), 12 deletions(-)
>
> diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c
> index b9c180ac1eee..cdb3dd6260bf 100644
> --- a/kernel/locking/rwsem.c
> +++ b/kernel/locking/rwsem.c
> @@ -59,7 +59,8 @@
> * A fast path reader optimistic lock stealing is supported when the rwsem
> * is previously owned by a writer and the following conditions are met:
> * - rwsem is not currently writer owned
> - * - the handoff isn't set.
> + * - the handoff isn't set
> + * - a writer isn't at the head of the wait queue.
> */
> #define RWSEM_READER_OWNED (1UL << 0)
> #define RWSEM_NONSPINNABLE (1UL << 1)
> @@ -1034,31 +1035,36 @@ rwsem_down_read_slowpath(struct rw_semaphore *sem, long count, unsigned int stat
> * Reader optimistic lock stealing.
> */
> if (!(count & (RWSEM_WRITER_LOCKED | RWSEM_FLAG_HANDOFF))) {
> - rwsem_set_reader_owned(sem);
> - lockevent_inc(rwsem_rlock_steal);
> -
> - /*
> - * Wake up other readers in the wait queue if it is
> - * the first reader.
> - */
> - if ((rcnt == 1) && (count & RWSEM_FLAG_WAITERS)) {
> + if (count & RWSEM_FLAG_WAITERS) {
> raw_spin_lock_irq(&sem->wait_lock);
> - if (sem->first_waiter)
> + first = sem->first_waiter;
> + /* A writer at the head of queue takes precedence. */
> + if (first && first->type == RWSEM_WAITING_FOR_WRITE)
> + goto queue_locked;
> + rwsem_set_reader_owned(sem);
> + /* If first reader, wake other readers in queue. */
> + if (first && rcnt == 1)
> rwsem_mark_wake(sem, RWSEM_WAKE_READ_OWNED,
> &wake_q);
> raw_spin_unlock_irq(&sem->wait_lock);
> - wake_up_q(&wake_q);
> + } else {
> + rwsem_set_reader_owned(sem);
> }
> +
> + lockevent_inc(rwsem_rlock_steal);
> + if (!wake_q_empty(&wake_q))
> + wake_up_q(&wake_q);
> return sem;
> }
>
> queue:
> + raw_spin_lock_irq(&sem->wait_lock);
> +queue_locked:
> waiter.task = current;
> waiter.type = RWSEM_WAITING_FOR_READ;
> waiter.timeout = jiffies + RWSEM_WAIT_TIMEOUT;
> waiter.handoff_set = false;
>
> - raw_spin_lock_irq(&sem->wait_lock);
> first = sem->first_waiter;
> if (!first) {
> /*
>
> ---
> base-commit: 05441d697f7f9a927264f4058cbbc6a03dfcac9d
> change-id: 20261105-rwsem-dont-steal-from-queued-writer-04b11d9d406b
>
> Cheers,
> --
> Lorenzo Stoakes (ARM) <ljs@kernel.org>
>
--
Cheers, Lorenzo
next prev parent reply other threads:[~2026-11-19 14:14 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-05 20:24 [PATCH] locking/rwsem: don't let a read lock steal from a queued writer Lorenzo Stoakes (ARM)
2026-11-06 4:55 ` Waiman Long
2026-11-07 19:28 ` Lorenzo Stoakes (ARM)
2026-11-19 14:14 ` Lorenzo Stoakes (ARM) [this message]
2026-11-19 15:56 ` Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=av8DKviO1fVURLcJ@gremlin \
--to=ljs@kernel.org \
--cc=boqun@kernel.org \
--cc=david@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=longman@redhat.com \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.