Re: Bad performance when CUBIC HyStart enabled

[Lucas Nussbaum <lucas.nussbaum@loria.fr>]

[-- Attachment #1: Type: text/plain, Size: 3694 bytes --]

On 03/03/11 at 16:03 +0100, Eric Dumazet wrote:
> Le jeudi 03 mars 2011 à 15:22 +0100, Lucas Nussbaum a écrit :
> > Hi,
> > 
> > I'm experiencing bad performance transferring data over a large BDP
> > network with CUBIC when HyStart is enabled. Disabling HyStart fixes the
> > problem.
> > 
> > Test setup:
> > I'm transfering a 2-GB file between two machines using netcat. On both
> > machines, /dev/shm is used to store the data.
> > On the sending side, I use:
> > /dev/shm# while true; do dd if=foo bs=10M | nc -l -p 2222 -q 0; done
> > On the receiving side, I use:
> > /dev/shm# /usr/bin/time nc 131.254.203.6 2222 > foo
> > Both machines are on Gigabit ethernet networks, and the RTT between them
> > is 11.3ms.
> > The default sysctl values are used.
> > I use nload -i 1000000 -o 1000000 -u m eth0 to visualize the transfer
> > rate.
> > 
> > With a 2.6.26 kernel, everything is fine. The bandwidth quickly gets to
> > ~940 Mbps, and I get the following in nload:
> >       ..................................
> >      .##################################
> >      ###################################
> >      ###################################
> >      ###################################
> >      ####################################
> >      ####################################
> >      ####################################
> >      ####################################
> >      ####################################
> >      ####################################
> >      ####################################
> >      ####################################
> >      ####################################
> > The transfer time is 18.2s, which is reasonable for a 2-GB file.
> > 
> > With a 2.6.32 kernel (Debian's) or with a 2.6.37 kernel (on the sending
> > side only), with the default HyStart parameters, I get the following:
> >                            ..........................
> >                           |##########################.
> >                          |############################
> >                         |#############################
> >                        |##############################
> >                       |###############################
> >                     .#################################
> >                    |##################################| 
> >                  |##################################### 
> >             ..||####################################### 
> >     ....|############################################## 
> >  .##################################################### 
> > The transfer time is 26.8s. (There's one character every 0.5s on the
> > graph above)
> > 
> > Disabling hystart on the sending machine with:
> > echo 0 > /sys/module/tcp_cubic/parameters/hystart
> > Fixes the problem.
> > 
> > It seems that waiting for some time (or rebooting) fixes the problem for
> > some time, so I'm suspecting that the problem is caused by a cache of
> > some sort. However, doing:
> > echo 1 > /proc/sys/net/ipv4/route/flush
> > doesn't fix the problem.
> > 
> > Ideas? I could arrange to give access to the testbed if needed.
> > (https://www.grid5000.fr/)
> 
> eventually, after fresh boot of both sender/receiver, it would be nice
> to have "netstat -s" results after one transfert with said problems.

Hi,

Attached.

I had 6 successful transfers. There's the netstat -s output after the 5
first successful transfers. Then the 7th transfer showed the problem,
as well as the 8th and the 9th. There's the netstat -s output after each
of them.
-- 
| Lucas Nussbaum             MCF Université Nancy 2 |
| lucas.nussbaum@loria.fr         LORIA / AlGorille |
| http://www.loria.fr/~lnussbau/  +33 3 54 95 86 19 |

[-- Attachment #2: netstat-s.tgz --]
[-- Type: application/x-gtar, Size: 2275 bytes --]