Re: Q: locking mechanisms

["Paul E. McKenney" <paulmck@us.ibm.com>]

On Wed, Jul 05, 2006 at 11:35:52AM +0200, Urs Thuermann wrote:
> "Paul E. McKenney" <paulmck@us.ibm.com> writes:
> 
> > > I have code that receives network packets by registering with
> > > dev_add_pack().  Each packet received is then delivered to a list
> > > of receivers, where this list can contain quite a lot of items:
> > > 
> > > 	receive_function(struct sk_buff *skb, struct net_device *dev,
> > > 			struct packet_type *pt, struct net_device *orig_dev)
> > > 	{
> > > 		...
> > > 		rcu_read_lock();
> > >             head = find_list(dev);
> > > 		hlist_for_each_entry_rcu(p, n, head, list) {
> > > 			deliver_packet_to_receiver(skb, p);
> > > 		}
> > > 		rcu_read_unlock();
> > > 	}
> > > 
> > > The deliver_packet_to_receiver() function finally ends up in a call to
> > > sock_queue_rcv_skb().
> 
> > "Holding" rcu_read_lock() for long time periods is much less of a
> > concern than holding other types of synchronization mechanisms.
> 
> Why is that?  I thought, if I hold a spinlock (or rw_lock in my case)
> I only block other threads that try to get that same lock.  With
> rcu_read_lock() I disable preemption which I thought affects more
> (all) other parts of the kernel.

In any kernel in which rcu_read_lock() disables preemption, both
spin_lock() and read_lock() (and friends) also disables preemption.
In addition, as you pointed out above, spin_lock(), read_lock(), and
friends are also blocking any other task that is trying to acquire
the lock in a conflicting manner to task(s) already holding it.

For reference, the kernels handle preemption as follows:

o	non-CONFIG_PREEMPT:  preemption is already disabled anyway,
	so any time anywhere in the kernel counts against realtime
	latency.

o	CONFIG_PREEMPT: all spinlock acquisitions, as well as rcu_read_lock(),
	disable preemption.

o	CONFIG_PREEMPT_RT (in Ingo Molnar's -rt patchset): neither "normal"
	spinlock acquisitions nor rcu_read_lock() disable preemption.

> > But I have to ask: roughly how long is "quite long"?
> 
> Depends on how many and what types of sockets are opened and which
> packets they want to receive.  The code is part of a new protocol
> family implementation for CAN (controller area network), which you can
> see at belios.de, project name socket-can.  It implements several
> types of PF_CAN sockets, which register for packets of certain CAN
> IDs, which are then lightly processed (filtered) and eventually
> delivered into a queue using sock_queue_rcv_skb().  Usually, the list
> has one receiver per open PF_CAN sockets.  Typical usage here has
> shown list lengths of 30-100 entries, i.e. 30-100 packets delivered
> with sock_queue_rcv_skb().  But it all depends on what the user space
> does, how many PF_CAN sockets are opened by all processes.
> 
> Is that value of "quite long" also "too long" for doing an
> rcu_read_lock()?

Seems like 30-100 entries would be OK in many cases -- but is it possible
to use a hash table, a tree, or some such if problems arise?

> urs
> 
> BTW, while implementing PF_CAN and reading kernel code and
> documentation I often run into questions on details like this which
> I'd like answered to get a better understanding of kernel internals,
> sometimes not directly related to some concrete implementation
> problem.  Is LKML the right place for these questions?

Yes, though it is almost always worthwhile to google "xxx site:lwn.net"
or "xxx site:lkml.org" for topic "xxx".  And the contents of the
Documentation directory, as well -- sometimes a little grepping in
that directory can be quite helpful.  In addition, there are a number
of books on Linux kernel internals.

						Thanx, Paul