net_device->queue_lock contention on 32-way box

net_device->queue_lock contention on 32-way box

[Chen, Kenneth W]

I'm hitting a very strange lock contention that I can't explain.  The
lock in question is net_device->queue_lock, some of the holding time
measured by lockmeter are as long as 36ms !! (it's very pathetic if
I may add my feeling to it).

The workload by no mean is exercising any network component on purpose.
In fact it's a db workload.  The network gets involved only because the
db server has to transmit query data back to the client.  The xmit rate
on the server side should be around 120 megabit / sec and we are using
gigabit NIC and gigabit switch.

So far we have tried both tigon3 and e1000 NIC, both has the same amount
of lock contention.  Total system time just in the spin is around 10% on
32-way numa machine.

Looked up the code in dev_queue_xmit and qdisc_restart, this lock is
used to enqueue/dequeue xmit packets.  Don't see how it is possible
that some process hold on to it for 36ms!  Possible a numa related
issue? or something I missed in the kernel network xmit path?

Thoughts, comments?


SPINLOCKS         HOLD            WAIT
  UTIL  CON    MEAN(  MAX )   MEAN(  MAX )(% CPU)     TOTAL NOWAIT SPIN RJECT  NAME
  7.5% 52.0%  1.7us( 257us)  175us(  29ms)( 4.7%)   2666132 48.0% 19.0% 33.1%  net_device->queue_lock
  1.2% 78.5%  0.7us( 125us)    0us                  1124063 21.5%    0% 78.5%    net_tx_action+0x240
  6.2% 32.8%  2.4us( 257us)  175us(  29ms)( 4.7%)   1542069 67.2% 32.8%    0%    qdisc_restart+0x450

 14.3% 26.2%  5.5us( 333us)  268us(  36ms)( 5.7%)   1542071 73.8% 26.2%    0%    dev_queue_xmit+0x1a0

Re: net_device->queue_lock contention on 32-way box

[David S. Miller]

The net_tx_action() --> qdisc_run() --> qdisc_restart() code path
can hold the lock for a long time especially if lots of packets
have been enqueued before net_tx_action() had a chance to run.

For each enqueued packet, we go all the way into the device driver
to give the packet to the device.  Given that PCI PIO accesses are
likely in these paths, along with some memory accesses (to setup
packet descriptors and the like) this could take quite a bit of
time.

We do temporarily release the dev->queue_lock in between each
packet while we go into the driver.  It could be what you're
seeing is the latency to get the device's dev->xmit_lock because
we have to acquire that before we can release the dev->queue_lock

If you bind the device interrupts to one cpu, do things change?

RE: net_device->queue_lock contention on 32-way box

[Chen, Kenneth W]

>>>> David S. Miller wrote on Wednesday, May 26, 2004 1:55 PM
> The net_tx_action() --> qdisc_run() --> qdisc_restart() code path
> can hold the lock for a long time especially if lots of packets
> have been enqueued before net_tx_action() had a chance to run.
>
> For each enqueued packet, we go all the way into the device driver
> to give the packet to the device.  Given that PCI PIO accesses are
> likely in these paths, along with some memory accesses (to setup
> packet descriptors and the like) this could take quite a bit of
> time.
>
> We do temporarily release the dev->queue_lock in between each
> packet while we go into the driver.  It could be what you're
> seeing is the latency to get the device's dev->xmit_lock because
> we have to acquire that before we can release the dev->queue_lock
>

That's where I'm having trouble in interpreting the lockmeter
data.  qdisc_restart() does a trylock on dev->xmit_lock, if it gets
it, unlock queue_lock right away. If it didn't get it, queue the
packet and return, then qdisc_run terminates and the caller to
qdisc_run will unlock queue_lock.  Don't see that as a long operation
at all.  Could netif_schedule() take long time to run?


> If you bind the device interrupts to one cpu, do things change?

We always bind network interrupt to one cpu, haven't tried with
non-bound configuration.

Re: net_device->queue_lock contention on 32-way box

[David S. Miller]

On Wed, 26 May 2004 14:47:15 -0700
"Chen, Kenneth W" <kenneth.w.chen@intel.com> wrote:

> Could netif_schedule() take long time to run?

No, it's a series of a few atomic operations.

1) test_and_set_bit()
2) disable cpu interrupts
3) Link list insertion
4) Set a local-cpu flag
5) re-enable cpu interrupts