16% regression on 10G caused by TCP small queues

16% regression on 10G caused by TCP small queues

[Stephen Hemminger]

In the course of testing routing functionality, I discovered a that the single flow TCP
throughput was much worse than expected. At first, it looked like a router problem,
or maybe because one end was a FreeBSD system (which has noticeably slower TCP performance).
But reducing it down to two systems directly connected over 10G (ixgbe) found the problem.

With a single TCP flow, in 3.5 kernel the performance with iperf is 9.41 Gbit/sec
which is at the link limit for TCP with timestamps etc. But in 3.6 and later the
throughput dropped to 7.9 Gbit/sec which is a regression of 16%.

Doing bisect shows that the commit causing this is:

  commit 46d3ceabd8d98ed0ad10f20c595ca784e34786c5
  Author: Eric Dumazet <eric.dumazet@gmail.com>
  Date:   Wed Jul 11 05:50:31 2012 +0000

    tcp: TCP Small Queues
    
    This introduce TSQ (TCP Small Queues)


There are several options at this point:
  0. Ignore it. Sorry, this is not acceptable.
     People do transfer files over 10G and expect line rate!
 
  1. Rip it out. which adds to the buffer bloat.
     This is a throughput vs latency tradeoff.

  2. Neuter it by making TCP small queues configurable and default off.
     Allows people who are willing to sacrifice performance go ahead and
     enable it.
 
  3. Tweak it. Make the default queue value in kernel big enough that no loss is
     observable.
 
  4. Do something smarter like a dynamic TCP small queue that adapts.

Re: 16% regression on 10G caused by TCP small queues

[Neal Cardwell]

On Wed, Oct 23, 2013 at 10:29 PM, Stephen Hemminger
<stephen@networkplumber.org> wrote:
> In the course of testing routing functionality, I discovered a that the single flow TCP
> throughput was much worse than expected. At first, it looked like a router problem,
> or maybe because one end was a FreeBSD system (which has noticeably slower TCP performance).
> But reducing it down to two systems directly connected over 10G (ixgbe) found the problem.
...
>   4. Do something smarter like a dynamic TCP small queue that adapts.

Yep, Eric made TSQ dynamic a few weeks ago, and mentioned that his
commit helps a single flow on 10Gbps link:

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c9eeec26e32e087359160406f96e0949b3cc6f10

Can you please check the performance in your setup on 3.12-rc4 or newer? :-)

Thanks!

neal

---

commit c9eeec26e32e087359160406f96e0949b3cc6f10
Author: Eric Dumazet <edumazet@google.com>
Date:   Fri Sep 27 03:28:54 2013 -0700

    tcp: TSQ can use a dynamic limit

    When TCP Small Queues was added, we used a sysctl to limit amount of
    packets queues on Qdisc/device queues for a given TCP flow.

    Problem is this limit is either too big for low rates, or too small
    for high rates.

    Now TCP stack has rate estimation in sk->sk_pacing_rate, and TSO
    auto sizing, it can better control number of packets in Qdisc/device
    queues.

    New limit is two packets or at least 1 to 2 ms worth of packets.

    Low rates flows benefit from this patch by having even smaller
    number of packets in queues, allowing for faster recovery,
    better RTT estimations.

    High rates flows benefit from this patch by allowing more than 2 packets
    in flight as we had reports this was a limiting factor to reach line
    rate. [ In particular if TX completion is delayed because of coalescing
    parameters ]

    Example for a single flow on 10Gbp link controlled by FQ/pacing

    14 packets in flight instead of 2
    ...

Re: 16% regression on 10G caused by TCP small queues

[Stephen Hemminger]

On Wed, Oct 23, 2013 at 7:37 PM, Neal Cardwell <ncardwell@google.com> wrote:
> On Wed, Oct 23, 2013 at 10:29 PM, Stephen Hemminger
> <stephen@networkplumber.org> wrote:
>> In the course of testing routing functionality, I discovered a that the single flow TCP
>> throughput was much worse than expected. At first, it looked like a router problem,
>> or maybe because one end was a FreeBSD system (which has noticeably slower TCP performance).
>> But reducing it down to two systems directly connected over 10G (ixgbe) found the problem.
> ...
>>   4. Do something smarter like a dynamic TCP small queue that adapts.
>
> Yep, Eric made TSQ dynamic a few weeks ago, and mentioned that his
> commit helps a single flow on 10Gbps link:
>
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c9eeec26e32e087359160406f96e0949b3cc6f10
>
> Can you please check the performance in your setup on 3.12-rc4 or newer? :-)
>
> Thanks!
>
> neal

I will check 3.12, but what about users on 3.10 which is the LTS
kernel used by most distros?

Re: 16% regression on 10G caused by TCP small queues

[David Miller]

From: Stephen Hemminger <stephen@networkplumber.org>
Date: Wed, 23 Oct 2013 20:09:49 -0700

> I will check 3.12, but what about users on 3.10 which is the LTS
> kernel used by most distros?

The fix will be backported to -stable, relax Stephen.

Re: 16% regression on 10G caused by TCP small queues

[Stephen Hemminger]

On Wed, 23 Oct 2013 23:38:16 -0400 (EDT)
David Miller <davem@davemloft.net> wrote:

> From: Stephen Hemminger <stephen@networkplumber.org>
> Date: Wed, 23 Oct 2013 20:09:49 -0700
> 
> > I will check 3.12, but what about users on 3.10 which is the LTS
> > kernel used by most distros?

3.12-rc6 gets line rate again (9.41 Gbit/sec)

> The fix will be backported to -stable, relax Stephen.

Sorry, thought sk_pacing_rate depended on FQ qdisc but it is other way around.
In which case doing merge of these two was sufficient to fix the problem.
With a minor manual fix up to tcp.h.


commit 95bd09eb27507691520d39ee1044d6ad831c1168
Author: Eric Dumazet <edumazet@google.com>
Date:   Tue Aug 27 05:46:32 2013 -0700

    tcp: TSO packets automatic sizing

commit c9eeec26e32e087359160406f96e0949b3cc6f10
Author: Eric Dumazet <edumazet@google.com>
Date:   Fri Sep 27 03:28:54 2013 -0700

    tcp: TSQ can use a dynamic limit

Re: 16% regression on 10G caused by TCP small queues

[Eric Dumazet]

On Wed, 2013-10-23 at 21:45 -0700, Stephen Hemminger wrote:
> On Wed, 23 Oct 2013 23:38:16 -0400 (EDT)
> David Miller <davem@davemloft.net> wrote:
> 
> > From: Stephen Hemminger <stephen@networkplumber.org>
> > Date: Wed, 23 Oct 2013 20:09:49 -0700
> > 
> > > I will check 3.12, but what about users on 3.10 which is the LTS
> > > kernel used by most distros?
> 
> 3.12-rc6 gets line rate again (9.41 Gbit/sec)
> 
> > The fix will be backported to -stable, relax Stephen.
> 
> Sorry, thought sk_pacing_rate depended on FQ qdisc but it is other way around.
> In which case doing merge of these two was sufficient to fix the problem.
> With a minor manual fix up to tcp.h.
> 
> 
> commit 95bd09eb27507691520d39ee1044d6ad831c1168
> Author: Eric Dumazet <edumazet@google.com>
> Date:   Tue Aug 27 05:46:32 2013 -0700
> 
>     tcp: TSO packets automatic sizing
> 
> commit c9eeec26e32e087359160406f96e0949b3cc6f10
> Author: Eric Dumazet <edumazet@google.com>
> Date:   Fri Sep 27 03:28:54 2013 -0700
> 
>     tcp: TSQ can use a dynamic limit

Perfect, thanks for testing Stephen !

Re: 16% regression on 10G caused by TCP small queues

[Eric Dumazet]

On Wed, 2013-10-23 at 21:45 -0700, Stephen Hemminger wrote:
> On Wed, 23 Oct 2013 23:38:16 -0400 (EDT)
> David Miller <davem@davemloft.net> wrote:
> 
> > From: Stephen Hemminger <stephen@networkplumber.org>
> > Date: Wed, 23 Oct 2013 20:09:49 -0700
> > 
> > > I will check 3.12, but what about users on 3.10 which is the LTS
> > > kernel used by most distros?
> 
> 3.12-rc6 gets line rate again (9.41 Gbit/sec)
> 
> > The fix will be backported to -stable, relax Stephen.
> 
> Sorry, thought sk_pacing_rate depended on FQ qdisc but it is other way around.
> In which case doing merge of these two was sufficient to fix the problem.
> With a minor manual fix up to tcp.h.

Btw what is the NIC you are using ?

I also had to patch mlx4 driver because it had too big coalescing
parameters, it was before TCP Small Queue dynamic sizing, but its worth
noting that for the initial ramp up (when flow sk_pacing_rate is low
because initial cwin is 10), it might make a difference

This was 

commit ecfd2ce1a9d5e6376ff5c00b366345160abdbbb7
Author: Eric Dumazet <edumazet@google.com>
Date:   Mon Nov 5 16:20:42 2012 +0000

    mlx4: change TX coalescing defaults
    
    mlx4 currently uses a too high tx coalescing setting, deferring
    TX completion interrupts by up to 128 us.
    
    With the recent skb_orphan() removal in commit 8112ec3b872,
    performance of a single TCP flow is capped to ~4 Gbps, unless
    we increase tcp_limit_output_bytes.
    
    I suggest using 16 us instead of 128 us, allowing a finer control.
    
    Performance of a single TCP flow is restored to previous levels,
    while keeping TCP small queues fully enabled with default sysctl.
    
    This patch is also a BQL prereq.

Re: 16% regression on 10G caused by TCP small queues

[Eric Dumazet]

On Wed, 2013-10-23 at 23:10 -0700, Eric Dumazet wrote:

> Btw what is the NIC you are using ?

Oh well I read you mail and saw ixgbe was mentioned ;)

Re: 16% regression on 10G caused by TCP small queues

[David Miller]

From: Stephen Hemminger <stephen@networkplumber.org>
Date: Wed, 23 Oct 2013 21:45:57 -0700

> Sorry, thought sk_pacing_rate depended on FQ qdisc but it is other way around.
> In which case doing merge of these two was sufficient to fix the problem.
> With a minor manual fix up to tcp.h.

I know, I already have a half-built tree of -stable submissions
that does exactlty this.

Re: 16% regression on 10G caused by TCP small queues

[Dave Taht]

On Thu, Oct 24, 2013 at 03:01:21AM -0400, David Miller wrote:
> From: Stephen Hemminger <stephen@networkplumber.org>
> Date: Wed, 23 Oct 2013 21:45:57 -0700
> 
> > Sorry, thought sk_pacing_rate depended on FQ qdisc but it is other way around.
> > In which case doing merge of these two was sufficient to fix the problem.
> > With a minor manual fix up to tcp.h.
> 
> I know, I already have a half-built tree of -stable submissions
> that does exactlty this.

I know that the "fq" qdisc is not exactly a -stable thing, but if it's simpler
to include it rather than sort through the patch sets, I'm all for it.

Re: 16% regression on 10G caused by TCP small queues

[David Miller]

From: Dave Taht <dave.taht@bufferbloat.net>
Date: Sat, 26 Oct 2013 21:33:52 -0700

> On Thu, Oct 24, 2013 at 03:01:21AM -0400, David Miller wrote:
>> From: Stephen Hemminger <stephen@networkplumber.org>
>> Date: Wed, 23 Oct 2013 21:45:57 -0700
>> 
>> > Sorry, thought sk_pacing_rate depended on FQ qdisc but it is other way around.
>> > In which case doing merge of these two was sufficient to fix the problem.
>> > With a minor manual fix up to tcp.h.
>> 
>> I know, I already have a half-built tree of -stable submissions
>> that does exactlty this.
> 
> I know that the "fq" qdisc is not exactly a -stable thing, but if it's simpler
> to include it rather than sort through the patch sets, I'm all for it.

I'm not including 'fq' and it's absolutely not necessary to fix this
bug.

'fq' is only incidentary to this bug fix because it just so happens to
make use of sk->sk_pacing_rate.  There is no other connection between
these two things.