Re: Requeues

[John Fastabend <john.r.fastabend@intel.com>]

On 3/2/2012 9:48 PM, John A. Sullivan III wrote:
> Hello, all.  I am seeing a small but significant number of requeues on
> our pfifo_fast qdiscs.  I've not been able to find much on what this
> means but the little I have implies it may be a problem with the
> physical interfaces.  However, these are fairly high end systems with
> Intel e1000 quad port cards.
> 
> We thought it might have something to do with the bonded interfaces so
> we checked some other high end systems without bonded interfaces but the
> same quad port cards and, lo and behold, the same small but significant
> number of requeues.
> 
> Is this normal or does it indicate a problem somewhere? Thanks - John
> 
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

One of two things can happen to cause the requeue counter to increment.

When the qdisc dequeues a packet it then gets enqueued in yet another
queue in the e1000 driver. If the qdisc is dequeueing packets faster
than the hardware can consume them the driver will return a
NETDEV_TX_BUSY error. This causes the qdisc to 'requeue' the packet and
in the process increments this counter. 

The gist is pfifo_fast dequeued a packet tried to give it to e1000 which
pushed back so pfifo_fast put the packet back on the queue.

By the way you can tune the size of the e1000 queues manually by playing
with:

	/sys/class/net/ethx/tx_queue_len

But you likely don't want to make them any larger else you'll hit the
buffer bloat problem. I'm guessing we should add byte queue limit support
here.

The second way to trigger this is multiple cpus contending for a lock.

In short requeue counts as long as its not excessive are just part 
of normal operation. So shouldn't be a problem.

Thanks,
John