RE: TxDescriptors -> 1024 default. Please not for every NIC!

RE: TxDescriptors -> 1024 default. Please not for every NIC!

[Marc Herbert]

On Wed, 26 May 2004, Brandeburg, Jesse wrote:

> I'm not sure that you could actually get the problem to occur on 100
> or 10Mb/s hardware however because of TCP window size limitation and
> such. What I'm getting at is that even if you have a device that can
> queue lots of packets, it probably won't unless you're using an
> unreliable protocol like UDP.  Theoretically it's a problem, but I'm
> not convinced that in real world scenarios it actually creates any
> issues.  Do you have a test that demonstrates the problem?

OK, let's go for it.

The following message details a simple experiment demonstrating how a
too big (1000-packet) txqueuelen creates a dreaded latency at the IP
level inside the sender for under-gigabit Ethernet interfaces, and so
finally advocates a default 1000-packet txqueuelen defined _only_
by/for the _GigE_ drivers, leaving the previous (before sep 2003 in
2.4) 100-packet default untouched for slower interfaces. The
experiment should be very quick and easy to reproduce in your lab,
even in your home.

Sorry, this message is way too long because it's... detailed, and
tries to anticipate questions, hopefully avoiding the need to come
back on this issue.  The counter-part is that it's not dense and thus
hopefully quick to read for anyone in the field. And please pardon my
english mistakes.


Detailed Experiment
-------------------

You need at least 2 hosts, but ideally 3. The sender "S", the
receiver "R1", and some witness host "R2". R1 and R2 can probably
be collapsed together if you don't have enough hosts but I am
afraid of unknown nasty side effects in this case.

Host S is using a simple TCP connection to upload an infinite file to
host R. The bottleneck is S's own 100Mb/s (or worst, 10Mb/s) network
interface. This is very important: no packet drop must occur elsewhere
between S and R, else TCP congestion avoidance algorithm will
interpret this as a congestion sign and throttle down, and the txqueue
will stay empty. If your TCP connection is under-performing your
sending wire for any reason, you will obviously never fill your
txqueue. The ACK-clocking property of TCP has for consequence that
only the queue of the bottleneck of the path may fill up (except in
very dynamic environnements where the bottleneck may be fast-changing,
but let's stay simple).

Actually almost everything below is still true when the bottleneck is
elsewhere in some router further on the path instead of local in the
sender. It's still true, just... elsewhere. For instance if you use a
linux box as a router, and if it happens to be the bottleneck, I
suspect this latency issue will appear more or less the same.  But
again, let's stay simple for the moment, forget those further routers
and get back to this _local_ IP bottleneck and its too big
txqueuelen. By the way, forcing your GigE interface to 100 or 10 is
ok.

I used iperf <http://dast.nlanr.net/Projects/Iperf/> to upload the
infinite file, but any equivalent tool should do it.

Since your TCP connection will suffer this artificial txqueue latency,
you also need to increase SND_BUF and RCV_BUF, else the number of TCP
packets sent (and thus the txqueue filling) will be capped (wait below
for more about this).


- So just run:

host_R $ iperf --server --interval 1 --window 1M
host_S $ iperf --client R --time 1000 --window 1M

Check that you get a full 94.1Mb/s (resp. 9.4Mb/s) wire-rate. If not,
investigate why and don't bother going on.


- Now just watch the latency between S some other host "R2". For
instance using mtr:

S$ mtr -l R2

As the txqueue fills up, you will see perceived latency increasing
every round-trip time, up to 120ms (worst with 10Mb/s: up to
1.2s!). When the txqueue is full, TCP detects it and enters congestion
window reduction, providing some temporary relief. Then the artificial latency
quickly ramps up again.

- You can also try to start another simultaneous upload to R2:

host_R2 $ iperf -s -i 1
host_S  $  iperf -c R2 -t 1000

... and watch how the artificial latency harms the start of the other
TCP connection, which need ages to ramp up it's throughput.

I also heard from here: "A Map of the Networking Code in Linux Kernel
 2.4.20", Technical Report DataTAG-2004-1, section 4.4
 http://datatag.web.cern.ch/datatag/publications.html
that you can get interesting qdisc stats using such commands:

# tc qdisc add dev eth1 root pfifo limit 100
# tc -s -d qdisc show dev eth1

But I did not tried them.


Warning: the interface tx_ring size has to be added to the qdisc's
txqueuelen to get the total sender's queue length perceived by TCP.
Some drivers may also set it big.



Solution
--------

Now reduce the txqueuelen to the previous value:

		ifconfig eth1 txqueuelen 100

for 10Mb/s you can even try:

		ifconfig eth1 txqueuelen 10


Now your latency is now back to a sensible value (12ms max), and
everything works fine. It's a simple as that. Throughput is not harmed
at all, you still get the full 94.1 Mb/s wire-rate. If this (previous)
setting was harmful to throughput for 100Mb/s interfaces, people would
have complained since long.



The more complex truth
-----------------------

If there is a real-world, distance-caused latency between S and R,
then having some equivalent amount of buffering in txqueuelen helps
average performance, because the interface has then a backlog of
packets to send while TCP takes time to ramp up its congestion window
again a decrease, the former compensating the latter. (This may be
what the e1000 guys observed in the first place, motivating the
increase to 1000 ? After all, 1.2ms of buffering was small) The
txqueue may smooth the sawtooth evolution of TCP congestion window,
minimizing the interface idle time.  But increased perceived latency
is the price to pay for this nice damper. There is a tradeoff between
latency and TCP throughput _on wide area_ routes to tune here, but
pushing it as far as storing in txqueuelen _multiple_ times any
real-world latency (did I say "1.2s" already?) brings no benefit at
all for throughput; it's just terribly harmful for perceived latency.
No IP router does so much buffering. Besides linux :-> I don't think
IP queues should be sized to cope with moon-earth latency by default.



Conclusion
----------
(aka: let's harass the maintainers)

Of course I just demonstrated here the worst case. In many other
cases, TCP will throttle down for some reason (packet losses, too
small socket buffers,...), it will not fill the pipe nor the txqueue,
and this dreaded latency will not appear. You could argue that my test
case is very seldom in the real world/not representative (and
I would _not_ agree), so the txqueue will never be full in practice,
since there will always be some other reason making TCP
under-performing the sending wire. OK. Even then, why defining it
uselessly so high for every NIC? Why take this risk? Just as a small
convenience for e1000 users? Mmmmm...

So now every interface has this 1000-packet queue (and soon 10,000
because of these upcoming 10Gb/s interfaces). To ensure no one ever
falls in this too big txqueue trap, I suggest the following user
documentation:

  "if you have a 100Mb/s interface, be warned that your txqueuelen is
  too high (it was tuned for real, gigabit men). So please reduce
  it using ifconfig. Alternatively, if you are not root,
  please tune your socket buffers finely. Too small, you will under-perform.
  Too big, you will fill up your txqueue and create artificial
  latency. Good luck. Of course, you can forget all the above when
  your interface is not the bottleneck"

On the other hand, having a default max txqueuelen defined in
_milliseconds_ (just like most other routers do) is quite easy to
implement and covers correctly all cases, without complex tuning
instructions for the end user. The ideal implementation is that every
driver defines txqueuelen by itself, depending on the actual link
speed. It is unrealistic in the short-term, but the incremental
implementation path is very easy: define a "sensible, generic default"
of 100-packet, perfect for the 100Mb/s masses, not too bad for 10Mb/s
masses, and let the (only few until now, let's hurry up) gigabit
drivers override/optimize this to 1000, or whatever else even more
finely tuned, for the cheap price of a few lines of code per driver.


Thanks in advance for agreeing OR proving that I am wrong.
I mean: thanks in advance for anything besides remaining silent.

And thanks for reading all this gossiping, an impressive effort indeed.

RE: TxDescriptors -> 1024 default. Please not for every NIC!

[Cheng Jin]

Marc,

In general, I very much agree with what you have stated about not having
a large txqueuelen.  Txqueuelen should be something that alleviates
the mismatch between CPU speed and NIC transmission speed, temporarily.
As long as the txqueuelne is greater than zero, say 10 just to be safe, 
NIC will be running at full speed (unless there were inefficiencies in 
scheduling) so there is no incentive in setting it to be an excessively 
large value like 1000.

> > I'm not sure that you could actually get the problem to occur on 100
> > or 10Mb/s hardware however because of TCP window size limitation and

With today's CPU, I think you will be able to fill up the txqueuelen
on a 10 or 100 Mbps NIC, assuming there is a large file transfer and
large window size and stuff.  

> If there is a real-world, distance-caused latency between S and R,
> then having some equivalent amount of buffering in txqueuelen helps
> average performance, because the interface has then a backlog of
> packets to send while TCP takes time to ramp up its congestion window
> again a decrease, the former compensating the latter. (This may be
> what the e1000 guys observed in the first place, motivating the
> increase to 1000 ? After all, 1.2ms of buffering was small) The
> txqueue may smooth the sawtooth evolution of TCP congestion window,
> minimizing the interface idle time.  But increased perceived latency
> is the price to pay for this nice damper. There is a tradeoff between
> latency and TCP throughput _on wide area_ routes to tune here, but
> pushing it as far as storing in txqueuelen _multiple_ times any
> real-world latency (did I say "1.2s" already?) brings no benefit at
> all for throughput; it's just terribly harmful for perceived latency.
> No IP router does so much buffering. Besides linux :-> I don't think
> IP queues should be sized to cope with moon-earth latency by default.

Very much agree with this paragraph.  As long as the buffer is more
than one bandwidth delay product, for a single TCP flow, window halving 
after each loss will still sustain a large enough window to maintain 
packets in the buffer to have full utilization.  The downside is exactly
what Marc said, very very large queueing delay for a long time.

Going back to what Marc said in an earlier e-mail about having txqueuelen 
in the unit of bytes rather than packets to provide a fixed queueing 
delay in ms rather than packets.  Maintaining txqueuelen in ms would be
an ideal solution, but probably hard to achieve in practice.

Keeping txqueuelen in bytes may be a problem for senders that 
wants to send many small pacekts.  While the byte count may be small, the 
overhead of sending small packets may introduce large delays.  

Cheng

RE: TxDescriptors -> 1024 default. Please not for every NIC!

[jamal]

On Wed, 2004-06-02 at 15:49, Cheng Jin wrote:
> Marc,
> 
> In general, I very much agree with what you have stated about not having
> a large txqueuelen.  Txqueuelen should be something that alleviates
> the mismatch between CPU speed and NIC transmission speed,

Thats the theory. More interesting of course are bus speeds, arbitration
schemes,  RAM latencies and throughput and other dynamic bottlenecks
like system loads. 

>  temporarily.
> As long as the txqueuelne is greater than zero, say 10 just to be safe, 
> NIC will be running at full speed (unless there were inefficiencies in 
> scheduling) so there is no incentive in setting it to be an excessively 
> large value like 1000.

In theory as well, the only time you even need to queue is when theres
congestion..
In reality, totaly different ballgame. In other words its not a simple
system that you can throw Littles theorems at.

Marc, good email, at least you didnt hand wave and declare the wind was
blowing towards the south today.

My opinion:
I agree that the 1000 qlen is excessive for 10/100 - infact i think the
value should dynamically adjust itself even for gige capable NICs
(example if a gige NIC negotiates a 10Mbps speed with link partner, then
you should adjust the qlen)[1]. That wont be trivial to do - but more
importantly motivation lacks because i dont think the situation we have
right now is devastating. To clarify:
A single TCP flow will fill in any pipe you give it under proper
conditions (proper congestion control algorithms, buffer etc)[2]. 
Most apps using TCP dont care very much about latency; the only
exception would be some scientific clustering technologies using TCP for
control messaging. And for those type of apps, you should be able to 
tune the qlen to your liking using tc or ip utilities (I claim they
shouldnt be using tcp to begin with, but thats another discussion). If
you dont want to take that extra step to tune then you dont care, and
IMO you shouldnt complain.

Having said all that: i still think theres value in maybe issuing a
warning or making the default qlen selection a compile time config.


cheers,
jamal

[1] I think it would make a nice project for someone with time. I can
consult for anyone interested.
[2] Looking at the recent patches on BIC, it does seem pretty agressive
and should have no problem filling a 10Gige pipe with proper processing
power.

Re: [e1000 2.6 10/11] TxDescriptors -> 1024 default

[Ricardo C Gonzalez]

David Miller wrote:

>Back to the main topic, maybe we should set dev->tx_queue_len to
>1000 by default for all ethernet devices.


I definately agree with setting the dev->tx_queue_len to 1000 as a default
for all ethernet adapters. All adapters will benefit from this change.


regards,

----------------------------------------------------------------------------------

***  ALWAYS THINK POSITIVE ***

TxDescriptors -> 1024 default. Please not for every NIC!

[Marc Herbert]

On Sun, 14 Sep 2003, Ricardo C Gonzalez wrote:

> David Miller wrote:
>
> >Back to the main topic, maybe we should set dev->tx_queue_len to
> >1000 by default for all ethernet devices.
>
>
> I definately agree with setting the dev->tx_queue_len to 1000 as a default
> for all ethernet adapters. All adapters will benefit from this change.
>

<http://oss.sgi.com/projects/netdev/archive/2003-09/threads.html#00247>

Sorry to exhume this discussion but I only recently discovered this
change, the hard way.

I carefully read this old thread and did not grasp _every_ detail, but
there is one thing that I am sure of: 1000 packets @ 1 Gb/s looks
good, but on the other hand, 1000 full-size Ethernet packets @ 10 Mb/s
are about 1.2 seconds long!

Too little buffering means not enough dampering effect, which is very
important for performance in asynchronous systems, granted. However,
_too much_ buffering means too big and too variable latencies. When
discussing buffers, duration is very often more important than size.
Applications, TCP's dynamic (and kernel dynamics too?) do not care
much about buffer sizes, they more often care about latencies (and
throughput, of course). Buffers sizes is often "just a small matter of
implementation" :-) For instance people designing routers talk about
buffers in _milliseconds_ much more often than in _bytes_ (despite the
fact that their memories cost more than in hosts, considering the
throughputs involved).

100 packets @ 100 Mb/s was 12 ms. 1000 packets @ 1 Gb/s is still
12 ms. 12 ms is great. It's a "good" latency because it is the
order of magnitude of real-world constants like:  comfortable
interactive applications, operating system sheduler granularity or
propagation time in 2000 km of cable.

But 1000 packets @ 100 Mb/s is 120 ms and is neither very good nor
very useful anymore. 1000 packets @ 10 Mb/s is 1.2 s, which is
ridiculous. It does mean that, when joe user is uploading some big
file through his cheap Ethernet card, and that there are no other
bottleneck/drops further in the network, every concurrent application
will have to wait 1.2 s before accessing the network!
 It this hard to believe for you, just make the test yourself, it's
very easy: force one of you NICs to 10Mb/s full duplex, txqueuelen
1000 and send a continuous flow to a nearby machine. Then try to ping
anything.
 Imagine now that some packet is lost for whatever reason on some
_other_ TCP connection going through this terrible 1.2 s queue. Then
you need one SACK/RTX extra round trip time to recover from it: so
it's now _2.4 s_ to deliver the data sent just after the dropped
packet...  Assuming of course TCP timers do not become confused by
this huge latency and probably huge jitter.

And I don't think you want to make fiddling with "tc" mandatory for
joe user. Or tell him: "oh, please just 'ifconfig txqueuelen 10', or
buy a new Ethernet card".

I am unfortunately not familiar with this part of the linux kernel,
but I really think that, if possible, txqueuelen should be initialized
at some "constant 12 ms" and not at the "1000 packets" highly variable
latency setting. I can imagine there are some corner cases, like for
instance when some GEth NIC is hot-plugged into a 100 Mb/s, or jumbo
frames, but hey, those are corner cases : as a first step, even a
simple constant-per-model txqueuelen initialization would be already
great.

Cheers,

Marc.


PS: one workaround for joe user against this 1.2s latency would be to
keep his SND_BUF and number of sockets small. But this is poor.


-- 
"Je n'ai fait cette lettre-ci plus longue que parce que je n'ai pas eu
le loisir de la faire plus courte." -- Blaise Pascal

Re: TxDescriptors -> 1024 default. Please not for every NIC!

[Marc Herbert]

On Sat, 15 May 2004, Marc Herbert wrote:

> <http://oss.sgi.com/projects/netdev/archive/2003-09/threads.html#00247>
>
> Sorry to exhume this discussion but I only recently discovered this
> change, the hard way.
>

> I am unfortunately not familiar with this part of the linux kernel,
> but I really think that, if possible, txqueuelen should be initialized
> at some "constant 12 ms" and not at the "1000 packets" highly variable
> latency setting. I can imagine there are some corner cases, like for
> instance when some GEth NIC is hot-plugged into a 100 Mb/s, or jumbo
> frames, but hey, those are corner cases : as a first step, even a
> simple constant-per-model txqueuelen initialization would be already
> great.

After some further study, I was glad to discover my suggestion above
both easy and short to implement. See patch below.

Trying to sum-it up:

- Ricardo asks (among others) for a new 1000 packets default
  txqueuelen for Intel's e1000, based on some data (couldn't not find
  this data, please send me the pointer if you have it, thanks).

- Me argues that we all lived happy for ages with this default
  setting of 100 packets @ 100 Mb/s (and lived approximately happy @
  10 Mb/s), but we'll soon see doom and gloom with this new and
  brutal change to 1000 packets for all this _legacy_ 10-100 Mb/s
  hardware. e1000 data only is not enough to justify this radical
  shift.

If you are convinced by _both_ items above, then the patch below
content _both_, and we're done.

If you are not, then... wait for further discussion, including answers
to latest Ricardo's post.


PS: several people seem to think TCP "drops" packets when the qdisc is
full. My analysis of the code _and_ my experiments makes me think they
are wrong: TCP rather "blocks" when the qdisc is full. See explanation
here: <http://oss.sgi.com/archives/netdev/2004-05/msg00151.html>
(Subject: Re: TcpOutSegs way too optimistic (netstat -s))


===== drivers/net/net_init.c 1.11 vs edited =====
--- 1.11/drivers/net/net_init.c	Tue Sep 16 01:12:25 2003
+++ edited/drivers/net/net_init.c	Wed May 19 11:05:34 2004
@@ -420,7 +420,10 @@
 	dev->hard_header_len 	= ETH_HLEN;
 	dev->mtu		= 1500; /* eth_mtu */
 	dev->addr_len		= ETH_ALEN;
-	dev->tx_queue_len	= 1000;	/* Ethernet wants good queues */
+	dev->tx_queue_len	= 100; /* This is a sensible generic default for
+					100 Mb/s: about 12ms with 1500 full size packets.
+					Drivers should tune this depending on interface
+					specificities and settings */

 	memset(dev->broadcast,0xFF, ETH_ALEN);

===== drivers/net/e1000/e1000_main.c 1.56 vs edited =====
--- 1.56/drivers/net/e1000/e1000_main.c	Tue Feb  3 01:43:42 2004
+++ edited/drivers/net/e1000/e1000_main.c	Wed May 19 03:14:32 2004
@@ -400,6 +400,8 @@
 		err = -ENOMEM;
 		goto err_alloc_etherdev;
 	}
+
+	netdev->tx_queue_len = 1000;

 	SET_MODULE_OWNER(netdev);

Re: TxDescriptors -> 1024 default. Please not for every NIC!

[Pekka Pietikainen]

On Wed, May 19, 2004 at 11:30:28AM +0200, Marc Herbert wrote:
> - Me argues that we all lived happy for ages with this default
>   setting of 100 packets @ 100 Mb/s (and lived approximately happy @
>   10 Mb/s), but we'll soon see doom and gloom with this new and
>   brutal change to 1000 packets for all this _legacy_ 10-100 Mb/s
>   hardware. e1000 data only is not enough to justify this radical
>   shift.
> 
> If you are convinced by _both_ items above, then the patch below
> content _both_, and we're done.
> 
> If you are not, then... wait for further discussion, including answers
> to latest Ricardo's post.
Not to mention that not all modern hardware is gigabit, current
2.6 seems to be setting txqueuelen of 1000 for 802.11 devices too (at least
my prism54), which might be causing major problems for me.

Well, I'm still trying to figure out whether it's txqueue or WEP that causes
all traffic to stop (with rx invalid crypt packets showing up in iwconfig
afterwards, AP is a linksys wrt54g in case it makes a difference) every now
and then until a ifdown / ifup. Tried both vanilla 2.6 prism54 and CVS
(which seems to have a reset on tx timeout thing added), but if txqueue is
1000 that won't easily get triggered will it?
 
It's been running for a few days just fine with txqueue = 100 and no WEP, if
it stays like that i'll start tweaking to find what exactly triggers it.

Re: TxDescriptors -> 1024 default. Please not for every NIC!

[Andi Kleen]

Marc Herbert <marc.herbert@free.fr> writes:
>
> PS: several people seem to think TCP "drops" packets when the qdisc is
> full. My analysis of the code _and_ my experiments makes me think they
> are wrong: TCP rather "blocks" when the qdisc is full. See explanation
> here: <http://oss.sgi.com/archives/netdev/2004-05/msg00151.html>
> (Subject: Re: TcpOutSegs way too optimistic (netstat -s))

This behaviour was only added relatively recently (in late 2.3.x timeframe)
I believe all the default queue lengths tunings were done before that.
So it would probably make sense to reevaluate/rebenchmark the default 
queue lengths for various devices with the newer code.

-Andi