Re: [PATCH v3 7/8] Move IO APIC to its own lock.

["Paul E. McKenney" <paulmck@linux.vnet.ibm.com>]

On Thu, Aug 13, 2009 at 01:44:06PM +0300, Avi Kivity wrote:
> On 08/13/2009 01:09 PM, Gleb Natapov wrote:
>>> There's also srcu.
>>>      
>> What are the disadvantages? There should be some, otherwise why not use
>> it all the time.
>
> I think it incurs an atomic op in the read path, but not much overhead 
> otherwise.  Paul?

There are not atomic operations in srcu_read_lock():

	int srcu_read_lock(struct srcu_struct *sp)
	{
		int idx;

		preempt_disable();
		idx = sp->completed & 0x1;
		barrier();  /* ensure compiler looks -once- at sp->completed. */
		per_cpu_ptr(sp->per_cpu_ref, smp_processor_id())->c[idx]++;
		srcu_barrier();  /* ensure compiler won't misorder critical section. */
		preempt_enable();
		return idx;
	}

There is a preempt_disable() and a preempt_enable(), which
non-atomically manipulate a field in the thread_info structure.
There is a barrier() and an srcu_barrier(), which are just compiler
directives (no code generated).  Other than that, simple arithmetic
and array accesses.  Shouldn't even be any cache misses in the common
case (the uncommon case being where synchronize_srcu() executing on
some other CPU).

There is even less in srcu_read_unlock():

	void srcu_read_unlock(struct srcu_struct *sp, int idx)
	{
		preempt_disable();
		srcu_barrier();  /* ensure compiler won't misorder critical section. */
		per_cpu_ptr(sp->per_cpu_ref, smp_processor_id())->c[idx]--;
		preempt_enable();
	}

So SRCU should have pretty low overhead.  And, as with other forms
of RCU, legal use of the read-side primitives cannot possibly
participate in deadlocks.

So, to answer the question above, what are the disadvantages?

o	On the update side, synchronize_srcu() does takes some time,
	mostly blocking in synchronize_sched().  So, like other
	forms of RCU, you would use SRCU in read-mostly situations.

o	Just as with RCU, reads and updates run concurrently, with
	all the good and bad that this implies.  For an example
	of the good, srcu_read_lock() executes deterministically,
	no blocking or spinning.  For an example of the bad, there
	is no way to shut down SRCU readers.  These are opposite
	sides of the same coin.  ;-)

o	Although srcu_read_lock() and srcu_read_unlock() are light
	weight, they are expensive compared to other forms of RCU.

o	In contrast to other forms of RCU, SRCU requires that the
	return value from srcu_read_lock() be passed into
	srcu_read_unlock().  Usually not a problem, but does place
	another constraint on the code.

Please keep in mind that I have no idea about what you are thinking of
using SRCU for, so the above advice is necessarily quite generic.  ;-)

							Thanx, Paul