From: "Lorenzo Stoakes (ARM)" <ljs@kernel.org>
To: Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@redhat.com>, Will Deacon <will@kernel.org>,
Boqun Feng <boqun@kernel.org>, Waiman Long <longman@redhat.com>
Cc: linux-kernel@vger.kernel.org,
"David Hildenbrand (Arm)" <david@kernel.org>
Subject: Re: [PATCH] locking/rwsem: don't let a read lock steal from a queued writer
Date: Thu, 19 Nov 2026 14:14:45 +0000 [thread overview]
Message-ID: <av8DKviO1fVURLcJ@gremlin> (raw)
In-Reply-To: <20261105-rwsem-dont-steal-from-queued-writer-v1-1-275e070f426f@kernel.org>
I know the cycle is grim on workload (equally so for us), but ping on this, as
we're relying on this behaviour for a series [0].
I could re-implement as an optional mode and respin [0] if necessary, but Peter
objected [1] so hoisted the rwsem change to here :)
Thanks!
[0]: https://lore.kernel.org/linux-mm/20261110-khugepaged-latency-fix-v3-0-6eea743cf9a3@kernel.org/
[1]: https://lore.kernel.org/all/20261103115159.GK1813469@noisy.programming.kicks-ass.net/
On Thu, Nov 05, 2026 at 08:24:27PM +0000, Lorenzo Stoakes (ARM) wrote:
> Since commit 1a728dff855a ("locking/rwsem: Enable reader optimistic lock
> stealing"), a racing read lock can steal an rwsem from under a queued
> writer whose handoff timeout hasn't expired yet.
>
> This is most problematic when an rwsem read lock is released then
> reacquired in order to relieve lock contention, as instead it will simply
> have the lock handed right back to it ahead of the intended beneficiary.
>
> Naturally, this behaviour rather defeats the object of doing this kind of
> thing, so this is less than ideal.
>
> Fix this by updating the optimistic read lock stealing code in
> rwsem_down_read_slowpath() to explicitly check whether the first queued
> lock is a waiting writer, and to not allow the theft if so.
>
> When a steal is disallowed, keep the existing sem->wait_lock held when
> jumping to the queue code to avoid useless lock acquisition/release.
>
> In order to perform this check, sem->wait_lock is acquired whenever there
> are queued waiters in the steal code, rather than the previous behaviour of
> only doing so when the reader count is 1.
>
> Since failing to steal a read lock unconditionally results in acquisition
> of the sem->wait_lock, this doesn't seem likely to be a significant
> overhead.
>
> Also update the comment describing optimistic read lock acquisition
> conditions to include this newly introduced behaviour.
>
> Care is taken to ensure correct ordering of lock acquisition, lock waking
> and marking the lock reader-owned.
>
> Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
> ---
> kernel/locking/rwsem.c | 30 ++++++++++++++++++------------
> 1 file changed, 18 insertions(+), 12 deletions(-)
>
> diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c
> index b9c180ac1eee..cdb3dd6260bf 100644
> --- a/kernel/locking/rwsem.c
> +++ b/kernel/locking/rwsem.c
> @@ -59,7 +59,8 @@
> * A fast path reader optimistic lock stealing is supported when the rwsem
> * is previously owned by a writer and the following conditions are met:
> * - rwsem is not currently writer owned
> - * - the handoff isn't set.
> + * - the handoff isn't set
> + * - a writer isn't at the head of the wait queue.
> */
> #define RWSEM_READER_OWNED (1UL << 0)
> #define RWSEM_NONSPINNABLE (1UL << 1)
> @@ -1034,31 +1035,36 @@ rwsem_down_read_slowpath(struct rw_semaphore *sem, long count, unsigned int stat
> * Reader optimistic lock stealing.
> */
> if (!(count & (RWSEM_WRITER_LOCKED | RWSEM_FLAG_HANDOFF))) {
> - rwsem_set_reader_owned(sem);
> - lockevent_inc(rwsem_rlock_steal);
> -
> - /*
> - * Wake up other readers in the wait queue if it is
> - * the first reader.
> - */
> - if ((rcnt == 1) && (count & RWSEM_FLAG_WAITERS)) {
> + if (count & RWSEM_FLAG_WAITERS) {
> raw_spin_lock_irq(&sem->wait_lock);
> - if (sem->first_waiter)
> + first = sem->first_waiter;
> + /* A writer at the head of queue takes precedence. */
> + if (first && first->type == RWSEM_WAITING_FOR_WRITE)
> + goto queue_locked;
> + rwsem_set_reader_owned(sem);
> + /* If first reader, wake other readers in queue. */
> + if (first && rcnt == 1)
> rwsem_mark_wake(sem, RWSEM_WAKE_READ_OWNED,
> &wake_q);
> raw_spin_unlock_irq(&sem->wait_lock);
> - wake_up_q(&wake_q);
> + } else {
> + rwsem_set_reader_owned(sem);
> }
> +
> + lockevent_inc(rwsem_rlock_steal);
> + if (!wake_q_empty(&wake_q))
> + wake_up_q(&wake_q);
> return sem;
> }
>
> queue:
> + raw_spin_lock_irq(&sem->wait_lock);
> +queue_locked:
> waiter.task = current;
> waiter.type = RWSEM_WAITING_FOR_READ;
> waiter.timeout = jiffies + RWSEM_WAIT_TIMEOUT;
> waiter.handoff_set = false;
>
> - raw_spin_lock_irq(&sem->wait_lock);
> first = sem->first_waiter;
> if (!first) {
> /*
>
> ---
> base-commit: 05441d697f7f9a927264f4058cbbc6a03dfcac9d
> change-id: 20261105-rwsem-dont-steal-from-queued-writer-04b11d9d406b
>
> Cheers,
> --
> Lorenzo Stoakes (ARM) <ljs@kernel.org>
>
--
Cheers, Lorenzo
prev parent reply other threads:[~2026-11-19 14:14 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-05 20:24 [PATCH] locking/rwsem: don't let a read lock steal from a queued writer Lorenzo Stoakes (ARM)
2026-11-06 4:55 ` Waiman Long
2026-11-07 19:28 ` Lorenzo Stoakes (ARM)
2026-11-19 14:14 ` Lorenzo Stoakes (ARM) [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=av8DKviO1fVURLcJ@gremlin \
--to=ljs@kernel.org \
--cc=boqun@kernel.org \
--cc=david@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=longman@redhat.com \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox