Re: [PATCH v3] locking/rwsem: reduce spinlock contention in wakeup after up_read/up_write

[Waiman Long <waiman.long@hp.com>]

On 04/28/2015 02:17 PM, Davidlohr Bueso wrote:
> On Mon, 2015-04-27 at 16:25 -0400, Waiman Long wrote:
>>>> +	/*
>>>> +	 * up_write() cleared the owner field before calling this function.
>>>> +	 * If that field is now set, a writer must have stolen the lock and
>>>> +	 * the wakeup operation should be aborted.
>>>> +	 */
>>>> +	if (rwsem_has_active_writer(sem))
>>>> +		goto out;
>>> We currently allow small races between rwsem owner and counter checks.
>>> And __rwsem_do_wake() can be called by checking the former -- and lock
>>> stealing is done with the counter as well. Please see below how we back
>>> out of such cases, as it is very much considered when granting the next
>>> reader. So nack to this as is, sorry.
>> If the first one in the queue is a writer, wake_up_process() may be
>> called directly which can be quite expensive if the lock has already
>> been stolen as the task will have to sleep again.
> But how can this occur? Lock stealing takes form in two places:
>
> 1) fastpath: only if the counter is 0 -- which, since we are discussing
> waking up waiter(s) code, obviously cannot occur.
>
> 2) With the cmpxchg() in rwsem_try_write_lock(), which is serialized
> with the wait_lock, so again this cannot occur.
>
> Which is why this is not considered in __rwsem_do_wake() when waking the
> writer fist in the queue.
>
> Thanks,
> Davidlohr
>

A write lock can also be acquired by a spinning writer in 
rwsem_try_write_lock_unqueued() where wait_lock isn't used. With 
multiple down_read's, it is possible that the first exiting reader wakes 
up a writer who acquires the write lock while the other readers are 
waiting for acquiring the wait_lock.

Cheers,
Longman