[RFC] driver adjusts qlen, increases CPU

[RFC] driver adjusts qlen, increases CPU

[Jesse Brandeburg]

So we've recently put a bit of code in our e1000 driver to decrease the 
qlen based on the speed of the link.

On the surface it seems like a great idea.  A driver knows when the link 
speed changed, and having a 1000 packet deep queue (the default for most 
kernels now) on top of a 100Mb/s link (or 10Mb/s worst case for us) makes 
for a *lot* of latency if many packets are queued up in the qdisc.

Problem we've seen is that setting this shorter queue causes a large spike 
in cpu when transmitting using UDP:

100Mb/s link
txqueuelen: 1000 Throughput: 92.44 CPU: 5.00
txqueuelen: 100 Throughput: 93.80 CPU: 61.59

Is this expected? any comments?

Jesse

Re: [RFC] driver adjusts qlen, increases CPU

[Rick Jones]

Jesse Brandeburg wrote:
> So we've recently put a bit of code in our e1000 driver to decrease the 
> qlen based on the speed of the link.
> 
> On the surface it seems like a great idea.  A driver knows when the link 
> speed changed, and having a 1000 packet deep queue (the default for most 
> kernels now) on top of a 100Mb/s link (or 10Mb/s worst case for us) makes 
> for a *lot* of latency if many packets are queued up in the qdisc.
> 
> Problem we've seen is that setting this shorter queue causes a large spike 
> in cpu when transmitting using UDP:
> 
> 100Mb/s link
> txqueuelen: 1000 Throughput: 92.44 CPU: 5.00
> txqueuelen: 100 Throughput: 93.80 CPU: 61.59
> 
> Is this expected? any comments?

Triggering intra-stack flow-control perhaps?  Perhaps 10X more often 
than before if the queue is 1/10th what it was before?

Out of curiousity, how does the UDP socket's SO_SNDBUF compare to the 
queue depth?

rick jones

Re: [RFC] driver adjusts qlen, increases CPU

[Jesse Brandeburg]

On 8/4/06, Rick Jones <rick.jones2@hp.com> wrote:
> > Problem we've seen is that setting this shorter queue causes a large spike
> > in cpu when transmitting using UDP:
> >
> > 100Mb/s link
> > txqueuelen: 1000 Throughput: 92.44 CPU: 5.00
> > txqueuelen: 100 Throughput: 93.80 CPU: 61.59
> >
> > Is this expected? any comments?
>
> Triggering intra-stack flow-control perhaps?  Perhaps 10X more often
> than before if the queue is 1/10th what it was before?

yes, that is my guess too, lock thrash coming into and out of qdisc full?

> Out of curiousity, how does the UDP socket's SO_SNDBUF compare to the
> queue depth?

forgive my ignorance, how do i measure that?

Jesse

Re: [RFC] driver adjusts qlen, increases CPU

[Rick Jones]

>> Out of curiousity, how does the UDP socket's SO_SNDBUF compare to the
>> queue depth?
> 
> 
> forgive my ignorance, how do i measure that?

getsockopt(SO_SNDBUF)?  I was just casting about on the possiblity that 
interactions between the depth of that queue and how much can be kept in 
the SO_SNDBUF might be involved.

rick jones