Re: Bad performance when CUBIC HyStart enabled

[Eric Dumazet <eric.dumazet@gmail.com>]

Le jeudi 03 mars 2011 à 15:22 +0100, Lucas Nussbaum a écrit :
> Hi,
> 
> I'm experiencing bad performance transferring data over a large BDP
> network with CUBIC when HyStart is enabled. Disabling HyStart fixes the
> problem.
> 
> Test setup:
> I'm transfering a 2-GB file between two machines using netcat. On both
> machines, /dev/shm is used to store the data.
> On the sending side, I use:
> /dev/shm# while true; do dd if=foo bs=10M | nc -l -p 2222 -q 0; done
> On the receiving side, I use:
> /dev/shm# /usr/bin/time nc 131.254.203.6 2222 > foo
> Both machines are on Gigabit ethernet networks, and the RTT between them
> is 11.3ms.
> The default sysctl values are used.
> I use nload -i 1000000 -o 1000000 -u m eth0 to visualize the transfer
> rate.
> 
> With a 2.6.26 kernel, everything is fine. The bandwidth quickly gets to
> ~940 Mbps, and I get the following in nload:
>       ..................................
>      .##################################
>      ###################################
>      ###################################
>      ###################################
>      ####################################
>      ####################################
>      ####################################
>      ####################################
>      ####################################
>      ####################################
>      ####################################
>      ####################################
>      ####################################
> The transfer time is 18.2s, which is reasonable for a 2-GB file.
> 
> With a 2.6.32 kernel (Debian's) or with a 2.6.37 kernel (on the sending
> side only), with the default HyStart parameters, I get the following:
>                            ..........................
>                           |##########################.
>                          |############################
>                         |#############################
>                        |##############################
>                       |###############################
>                     .#################################
>                    |##################################| 
>                  |##################################### 
>             ..||####################################### 
>     ....|############################################## 
>  .##################################################### 
> The transfer time is 26.8s. (There's one character every 0.5s on the
> graph above)
> 
> Disabling hystart on the sending machine with:
> echo 0 > /sys/module/tcp_cubic/parameters/hystart
> Fixes the problem.
> 
> It seems that waiting for some time (or rebooting) fixes the problem for
> some time, so I'm suspecting that the problem is caused by a cache of
> some sort. However, doing:
> echo 1 > /proc/sys/net/ipv4/route/flush
> doesn't fix the problem.
> 
> Ideas? I could arrange to give access to the testbed if needed.
> (https://www.grid5000.fr/)

eventually, after fresh boot of both sender/receiver, it would be nice
to have "netstat -s" results after one transfert with said problems.