Requeues

Requeues

[John A. Sullivan III]

Hello, all.  I am seeing a small but significant number of requeues on
our pfifo_fast qdiscs.  I've not been able to find much on what this
means but the little I have implies it may be a problem with the
physical interfaces.  However, these are fairly high end systems with
Intel e1000 quad port cards.

We thought it might have something to do with the bonded interfaces so
we checked some other high end systems without bonded interfaces but the
same quad port cards and, lo and behold, the same small but significant
number of requeues.

Is this normal or does it indicate a problem somewhere? Thanks - John

Re: Requeues

[John Fastabend]

On 3/2/2012 9:48 PM, John A. Sullivan III wrote:
> Hello, all.  I am seeing a small but significant number of requeues on
> our pfifo_fast qdiscs.  I've not been able to find much on what this
> means but the little I have implies it may be a problem with the
> physical interfaces.  However, these are fairly high end systems with
> Intel e1000 quad port cards.
> 
> We thought it might have something to do with the bonded interfaces so
> we checked some other high end systems without bonded interfaces but the
> same quad port cards and, lo and behold, the same small but significant
> number of requeues.
> 
> Is this normal or does it indicate a problem somewhere? Thanks - John
> 
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

One of two things can happen to cause the requeue counter to increment.

When the qdisc dequeues a packet it then gets enqueued in yet another
queue in the e1000 driver. If the qdisc is dequeueing packets faster
than the hardware can consume them the driver will return a
NETDEV_TX_BUSY error. This causes the qdisc to 'requeue' the packet and
in the process increments this counter. 

The gist is pfifo_fast dequeued a packet tried to give it to e1000 which
pushed back so pfifo_fast put the packet back on the queue.

By the way you can tune the size of the e1000 queues manually by playing
with:

	/sys/class/net/ethx/tx_queue_len

But you likely don't want to make them any larger else you'll hit the
buffer bloat problem. I'm guessing we should add byte queue limit support
here.

The second way to trigger this is multiple cpus contending for a lock.

In short requeue counts as long as its not excessive are just part 
of normal operation. So shouldn't be a problem.

Thanks,
John

Re: Requeues

[John A. Sullivan III]

On Fri, 2012-03-02 at 22:54 -0800, John Fastabend wrote:
> On 3/2/2012 9:48 PM, John A. Sullivan III wrote:
> > Hello, all.  I am seeing a small but significant number of requeues on
> > our pfifo_fast qdiscs.  I've not been able to find much on what this
> > means but the little I have implies it may be a problem with the
> > physical interfaces.  However, these are fairly high end systems with
> > Intel e1000 quad port cards.
> > 
> > We thought it might have something to do with the bonded interfaces so
> > we checked some other high end systems without bonded interfaces but the
> > same quad port cards and, lo and behold, the same small but significant
> > number of requeues.
> > 
> > Is this normal or does it indicate a problem somewhere? Thanks - John
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> One of two things can happen to cause the requeue counter to increment.
> 
> When the qdisc dequeues a packet it then gets enqueued in yet another
> queue in the e1000 driver. If the qdisc is dequeueing packets faster
> than the hardware can consume them the driver will return a
> NETDEV_TX_BUSY error. This causes the qdisc to 'requeue' the packet and
> in the process increments this counter. 
> 
> The gist is pfifo_fast dequeued a packet tried to give it to e1000 which
> pushed back so pfifo_fast put the packet back on the queue.
> 
> By the way you can tune the size of the e1000 queues manually by playing
> with:
> 
> 	/sys/class/net/ethx/tx_queue_len
> 
> But you likely don't want to make them any larger else you'll hit the
> buffer bloat problem. I'm guessing we should add byte queue limit support
> here.
> 
> The second way to trigger this is multiple cpus contending for a lock.
> 
> In short requeue counts as long as its not excessive are just part 
> of normal operation. So shouldn't be a problem.
<snip>
Thank you very much.  I was not aware of CPU contention being a cause.
We did notice the problem primarily on systems with 8 to 16 cores - John

Re: Requeues

[Dave Taht]

On Sat, Mar 3, 2012 at 10:13 AM, John A. Sullivan III
<jsullivan@opensourcedevel.com> wrote:
> On Fri, 2012-03-02 at 22:54 -0800, John Fastabend wrote:
>> On 3/2/2012 9:48 PM, John A. Sullivan III wrote:
>> > Hello, all.  I am seeing a small but significant number of requeues on
>> > our pfifo_fast qdiscs.  I've not been able to find much on what this
>> > means but the little I have implies it may be a problem with the
>> > physical interfaces.  However, these are fairly high end systems with
>> > Intel e1000 quad port cards.


From what you describe it sounds like this is not a 'problem', but a
sign that your environment is capable of saturating e1000 quad port
cards.

Adding a little active queue management (red or sfqred) will reduce
the impact of the extra buffering and lead to more responsive streams
with more accurate estimates of the BDP.

if you are already running bql, you will generally see more requeues.

>> >
>> > We thought it might have something to do with the bonded interfaces so
>> > we checked some other high end systems without bonded interfaces but the
>> > same quad port cards and, lo and behold, the same small but significant
>> > number of requeues.
>> >
>> > Is this normal or does it indicate a problem somewhere? Thanks - John
>> >
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe netdev" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>> One of two things can happen to cause the requeue counter to increment.
>>
>> When the qdisc dequeues a packet it then gets enqueued in yet another
>> queue in the e1000 driver. If the qdisc is dequeueing packets faster
>> than the hardware can consume them the driver will return a
>> NETDEV_TX_BUSY error. This causes the qdisc to 'requeue' the packet and
>> in the process increments this counter.
>>
>> The gist is pfifo_fast dequeued a packet tried to give it to e1000 which
>> pushed back so pfifo_fast put the packet back on the queue.
>>
>> By the way you can tune the size of the e1000 queues manually by playing
>> with:
>>
>>       /sys/class/net/ethx/tx_queue_len
>>
>> But you likely don't want to make them any larger else you'll hit the
>> buffer bloat problem. I'm guessing we should add byte queue limit support
>> here.
>>
>> The second way to trigger this is multiple cpus contending for a lock.
>>
>> In short requeue counts as long as its not excessive are just part
>> of normal operation. So shouldn't be a problem.
> <snip>
> Thank you very much.  I was not aware of CPU contention being a cause.
> We did notice the problem primarily on systems with 8 to 16 cores - John
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net