Re: Locking Between User Context and Soft IRQs in 2.4.0

[Jeff Garzik <jgarzik@mandrakesoft.com>]

[-- Attachment #1: Type: text/plain, Size: 2604 bytes --]

"Hen, Shmulik" wrote:
> We are trying to port a network driver from 2.2.x to 2.4.x and have some
> question regarding locks.
> According to the kernel locking HOWTO, we have to take extra care when
> locking between user context threads and BH/tasklet/softIRQ,
> so we learned (the hard way ;-) that when running the ioctl system call from
> an application we should use spin_lock/unlock_bh() and not
> spin_lock/unlock() inside dev->do_ioctl().

That is not necessarily true.  If you have timers or tasklets going,
sure.  I prefer kernel threads for a lot of tasks nowadays, because you
only have two cases for locking -- user and interrupt -- and you can
sleep all you want to in a kernel thread.


> *       What about the other entry points implemented in net_device ?

I wrote the attached doc, after tracing through the code.  It has not
been reviewed yet so it is not canonical, but hopefully it is
informative...


> *       We've got dev->get_stats, dev->set_mac_address,
> dev->set_mutlicast_list and others that are all called from running
> 'ifconfig' which is an application. Are they considered user context too ?

You are inside a spinlock in get_stats, so you cannot sleep.  But you
can sleep in set_multicast_list.  Not sure about set_mac_address.


> *       What about dev->open and dev->stop ?

Sleep all you want, we'll leave the light on for ya.


> *       We figured that dev->hard_start_xmit() and timer callbacks are not
> considered user context, but how can I find out if they are being run as
> SoftIRQ or as tasklets or as Bottom Halves ? (their different definitions
> require different types of protections)

I'm not sure about the context from which hard_start_xmit is called... 
Its inside a spinlock, so you shouldn't be sleeping.  timers are unique
unto themselves... but you lock against them using spin_lock_bh outside
the timer, and spin_lock inside the timer.

> wrap entire operations from top to bottom. For example, our
> dev->hard_start_xmit() will have a spin_lock() at the beginning and a
> spin_unlock() at the end of the function.

Why?  dev->xmit_lock is obtained before dev->hard_start_xmit is called,
and released after it returns.


> *       What about other calls to the kernel ? can the running thread be
> switched out of context when calling kernel entries and not be switched back
> in when they finish ? should I beware of deadlocks in such case ?

You should always beware of deadlocks!

	Jeff


-- 
Jeff Garzik             | Dinner is ready when
Building 1024           | the smoke alarm goes off.
MandrakeSoft            |	-/usr/games/fortune

[-- Attachment #2: netdevices.txt --]
[-- Type: text/plain, Size: 785 bytes --]



struct net_device synchronization rules
=======================================
dev->open:
	Locking: Inside rtnl_lock() semaphore.
	Sleeping: OK

dev->stop:
	Locking: Inside rtnl_lock() semaphore.
	Sleeping: OK

dev->do_ioctl:
	Locking: Inside rtnl_lock() semaphore.
	Sleeping: OK

dev->get_stats:
	Locking: Inside dev_base_lock spinlock.
	Sleeping: NO

dev->hard_start_xmit:
	Locking: Inside dev->xmit_lock spinlock.
	Sleeping: NO[1]

dev->tx_timeout:
	Locking: Inside dev->xmit_lock spinlock.
	Sleeping: NO[1]

dev->set_multicast_list:
	Locking: Inside dev->xmit_lock spinlock.
	Sleeping: NO[1]


NOTE [1]: On principle, you should not sleep when a spinlock is held.
However, since this spinlock is per-net-device, we only block ourselves
if we sleep, so the effect is mitigated.