Re: [RFC] [PATCH] Don't run __qdisc_run() on a stopped TX queue

[Jarek Poplawski <jarkao2@gmail.com>]

On Wed, Jul 29, 2009 at 08:44:28AM +0800, Herbert Xu wrote:
> On Tue, Jul 28, 2009 at 12:59:19PM -0700, David Miller wrote:
> >
> > The premise is that there'd be only one.  The qdisc lock.
> > 
> > If the traffic is distributed, flow wise, the driver XMIT
> > lock would spread due to multiqueue.
> 
> Suppose that we have a single large flow going through that has
> filled up the hardware queue and is now backlogged in the qdisc
> with qdisc_run on CPU A.   Now some other flow comes along and
> sends a packet on CPU B.
> 
> So now CPU A and B will both be processing packets for the first
> flow causing loads of lock contention.
> 
> But worse yet, we have introduced packet reordering.  So are you
> convinced now :)

How about this: instead of the _RUNNING flag we take tx lock while
holding qdisc lock and release qdisc lock just after (before xmit).
This should prevent reordering, and probably could improve cache use:
CPU B which takes qdisc lock only for enqueuing now, would use it for
dequeuing too, plus if accidentally the next xmit goes to a different
tx queue, it could start before CPU A finishes. Otherwise it would
simply wait for CPU A (without tx lock contention). Of course it
needs testing... 

Cheers,
Jarek P.