From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <netdev-owner@vger.kernel.org>
Received: from mail-pl0-f45.google.com ([209.85.160.45]:40104 "EHLO
        mail-pl0-f45.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1750934AbeBRWYM (ORCPT
        <rfc822;netdev@vger.kernel.org>); Sun, 18 Feb 2018 17:24:12 -0500
Received: by mail-pl0-f45.google.com with SMTP id g18so4639912plo.7
        for <netdev@vger.kernel.org>; Sun, 18 Feb 2018 14:24:11 -0800 (PST)
Message-ID: <1518992649.55655.19.camel@gmail.com>
Subject: Re: TCP and BBR: reproducibly low cwnd and bandwidth
From: Eric Dumazet <eric.dumazet@gmail.com>
To: Oleksandr Natalenko <oleksandr@natalenko.name>
Cc: Eric Dumazet <edumazet@google.com>,
        Neal Cardwell <ncardwell@google.com>,
        "David S. Miller" <davem@davemloft.net>,
        Netdev <netdev@vger.kernel.org>,
        Yuchung Cheng <ycheng@google.com>,
        Soheil Hassas Yeganeh <soheil@google.com>,
        Jerry Chu <hkchu@google.com>, Dave Taht <dave.taht@gmail.com>
Date: Sun, 18 Feb 2018 14:24:09 -0800
In-Reply-To: <2562547.S27nl9fb2E@natalenko.name>
References: <1697118.nv5eASg0nx@natalenko.name>
         <1518893571.55655.12.camel@gmail.com> <1518987867.55655.15.camel@gmail.com>
         <2562547.S27nl9fb2E@natalenko.name>
Content-Type: text/plain; charset="UTF-8"
Mime-Version: 1.0
Content-Transfer-Encoding: 8bit
Sender: netdev-owner@vger.kernel.org
List-ID: <netdev.vger.kernel.org>

On Sun, 2018-02-18 at 22:49 +0100, Oleksandr Natalenko wrote:
> Hi.
> 
> On neděle 18. února 2018 22:04:27 CET Eric Dumazet wrote:
> > I was able to take a look today, and I believe this is the time to
> > switch TCP to GSO being always on.
> > 
> > As a bonus, we get speed boost for cubic as well.
> > 
> > Todays high BDP and recent TCP improvements (rtx queue as rb-tree, sack
> > coalescing, TCP pacing...) all were developed/tested/maintained with
> > GSO/TSO being the norm.
> > 
> > Can you please test the following patch ?
> 
> Yes, results below:
> 
> BBR+fq:
> sg on:  6.02 Gbits/sec
> sg off: 1.33 Gbits/sec
> 
> BBR+pfifo_fast:
> sg on:  4.13 Gbits/sec
> sg off: 1.34 Gbits/sec
> 
> BBR+fq_codel:
> sg on:  4.16 Gbits/sec
> sg off: 1.35 Gbits/sec
> 
> Reno+fq:
> sg on:  6.44 Gbits/sec
> sg off: 1.39 Gbits/sec
> 
> Reno+pfifo_fast:
> sg on:  6.36 Gbits/sec
> sg off: 1.39 Gbits/sec
> 
> Reno+fq_codel:
> sg on:  6.41 Gbits/sec
> sg off: 1.38 Gbits/sec
> 
> While BBR still suffers when fq is not used, disabling sg doesn't bring 
> drastic throughput drop anymore. So, looks good to me, eh?
> 

Indeed :)

Here are my results on 40Gbit link (mlx4) :

BBR+fq:
sg on:  26 Gbits/sec
sg off: 15.7 Gbits/sec   (was 2.3 Gbit before patch)

BBR+pfifo_fast:
sg on:  24.2 Gbits/sec
sg off: 14.9 Gbits/sec  (was 0.66 Gbit before patch !!! )

BBR+fq_codel:
sg on:  24.4 Gbits/sec
sg off: 15 Gbits/sec  (was 0.66 Gbit before patch !!! )

Reno+fq:
sg on:  20 Gbits/sec
sg off: 15.7 Gbits/sec  (was 6 Gbit)

Reno+pfifo_fast:
sg on:  25.7 Gbits/sec
sg off: 15.5 Gbits/sec  (was 7 Gbit)

Reno+fq_codel:
sg on:  25.8 Gbits/sec
sg off: 16 Gbits/sec    (was 7 Gbit)

Definitely worth it ;)

Thanks !