Bad performance when CUBIC HyStart enabled

Bad performance when CUBIC HyStart enabled

[Lucas Nussbaum]

Hi,

I'm experiencing bad performance transferring data over a large BDP
network with CUBIC when HyStart is enabled. Disabling HyStart fixes the
problem.

Test setup:
I'm transfering a 2-GB file between two machines using netcat. On both
machines, /dev/shm is used to store the data.
On the sending side, I use:
/dev/shm# while true; do dd if=foo bs=10M | nc -l -p 2222 -q 0; done
On the receiving side, I use:
/dev/shm# /usr/bin/time nc 131.254.203.6 2222 > foo
Both machines are on Gigabit ethernet networks, and the RTT between them
is 11.3ms.
The default sysctl values are used.
I use nload -i 1000000 -o 1000000 -u m eth0 to visualize the transfer
rate.

With a 2.6.26 kernel, everything is fine. The bandwidth quickly gets to
~940 Mbps, and I get the following in nload:
      ..................................
     .##################################
     ###################################
     ###################################
     ###################################
     ####################################
     ####################################
     ####################################
     ####################################
     ####################################
     ####################################
     ####################################
     ####################################
     ####################################
The transfer time is 18.2s, which is reasonable for a 2-GB file.

With a 2.6.32 kernel (Debian's) or with a 2.6.37 kernel (on the sending
side only), with the default HyStart parameters, I get the following:
                           ..........................
                          |##########################.
                         |############################
                        |#############################
                       |##############################
                      |###############################
                    .#################################
                   |##################################| 
                 |##################################### 
            ..||####################################### 
    ....|############################################## 
 .##################################################### 
The transfer time is 26.8s. (There's one character every 0.5s on the
graph above)

Disabling hystart on the sending machine with:
echo 0 > /sys/module/tcp_cubic/parameters/hystart
Fixes the problem.

It seems that waiting for some time (or rebooting) fixes the problem for
some time, so I'm suspecting that the problem is caused by a cache of
some sort. However, doing:
echo 1 > /proc/sys/net/ipv4/route/flush
doesn't fix the problem.

Ideas? I could arrange to give access to the testbed if needed.
(https://www.grid5000.fr/)
-- 
| Lucas Nussbaum             MCF Université Nancy 2 |
| lucas.nussbaum@loria.fr         LORIA / AlGorille |
| http://www.loria.fr/~lnussbau/  +33 3 54 95 86 19 |

Re: Bad performance when CUBIC HyStart enabled

[Eric Dumazet]

Le jeudi 03 mars 2011 à 15:22 +0100, Lucas Nussbaum a écrit :
> Hi,
> 
> I'm experiencing bad performance transferring data over a large BDP
> network with CUBIC when HyStart is enabled. Disabling HyStart fixes the
> problem.
> 
> Test setup:
> I'm transfering a 2-GB file between two machines using netcat. On both
> machines, /dev/shm is used to store the data.
> On the sending side, I use:
> /dev/shm# while true; do dd if=foo bs=10M | nc -l -p 2222 -q 0; done
> On the receiving side, I use:
> /dev/shm# /usr/bin/time nc 131.254.203.6 2222 > foo
> Both machines are on Gigabit ethernet networks, and the RTT between them
> is 11.3ms.
> The default sysctl values are used.
> I use nload -i 1000000 -o 1000000 -u m eth0 to visualize the transfer
> rate.
> 
> With a 2.6.26 kernel, everything is fine. The bandwidth quickly gets to
> ~940 Mbps, and I get the following in nload:
>       ..................................
>      .##################################
>      ###################################
>      ###################################
>      ###################################
>      ####################################
>      ####################################
>      ####################################
>      ####################################
>      ####################################
>      ####################################
>      ####################################
>      ####################################
>      ####################################
> The transfer time is 18.2s, which is reasonable for a 2-GB file.
> 
> With a 2.6.32 kernel (Debian's) or with a 2.6.37 kernel (on the sending
> side only), with the default HyStart parameters, I get the following:
>                            ..........................
>                           |##########################.
>                          |############################
>                         |#############################
>                        |##############################
>                       |###############################
>                     .#################################
>                    |##################################| 
>                  |##################################### 
>             ..||####################################### 
>     ....|############################################## 
>  .##################################################### 
> The transfer time is 26.8s. (There's one character every 0.5s on the
> graph above)
> 
> Disabling hystart on the sending machine with:
> echo 0 > /sys/module/tcp_cubic/parameters/hystart
> Fixes the problem.
> 
> It seems that waiting for some time (or rebooting) fixes the problem for
> some time, so I'm suspecting that the problem is caused by a cache of
> some sort. However, doing:
> echo 1 > /proc/sys/net/ipv4/route/flush
> doesn't fix the problem.
> 
> Ideas? I could arrange to give access to the testbed if needed.
> (https://www.grid5000.fr/)

eventually, after fresh boot of both sender/receiver, it would be nice
to have "netstat -s" results after one transfert with said problems.

Re: Bad performance when CUBIC HyStart enabled

[Lucas Nussbaum]

[-- Attachment #1: Type: text/plain, Size: 3694 bytes --]

On 03/03/11 at 16:03 +0100, Eric Dumazet wrote:
> Le jeudi 03 mars 2011 à 15:22 +0100, Lucas Nussbaum a écrit :
> > Hi,
> > 
> > I'm experiencing bad performance transferring data over a large BDP
> > network with CUBIC when HyStart is enabled. Disabling HyStart fixes the
> > problem.
> > 
> > Test setup:
> > I'm transfering a 2-GB file between two machines using netcat. On both
> > machines, /dev/shm is used to store the data.
> > On the sending side, I use:
> > /dev/shm# while true; do dd if=foo bs=10M | nc -l -p 2222 -q 0; done
> > On the receiving side, I use:
> > /dev/shm# /usr/bin/time nc 131.254.203.6 2222 > foo
> > Both machines are on Gigabit ethernet networks, and the RTT between them
> > is 11.3ms.
> > The default sysctl values are used.
> > I use nload -i 1000000 -o 1000000 -u m eth0 to visualize the transfer
> > rate.
> > 
> > With a 2.6.26 kernel, everything is fine. The bandwidth quickly gets to
> > ~940 Mbps, and I get the following in nload:
> >       ..................................
> >      .##################################
> >      ###################################
> >      ###################################
> >      ###################################
> >      ####################################
> >      ####################################
> >      ####################################
> >      ####################################
> >      ####################################
> >      ####################################
> >      ####################################
> >      ####################################
> >      ####################################
> > The transfer time is 18.2s, which is reasonable for a 2-GB file.
> > 
> > With a 2.6.32 kernel (Debian's) or with a 2.6.37 kernel (on the sending
> > side only), with the default HyStart parameters, I get the following:
> >                            ..........................
> >                           |##########################.
> >                          |############################
> >                         |#############################
> >                        |##############################
> >                       |###############################
> >                     .#################################
> >                    |##################################| 
> >                  |##################################### 
> >             ..||####################################### 
> >     ....|############################################## 
> >  .##################################################### 
> > The transfer time is 26.8s. (There's one character every 0.5s on the
> > graph above)
> > 
> > Disabling hystart on the sending machine with:
> > echo 0 > /sys/module/tcp_cubic/parameters/hystart
> > Fixes the problem.
> > 
> > It seems that waiting for some time (or rebooting) fixes the problem for
> > some time, so I'm suspecting that the problem is caused by a cache of
> > some sort. However, doing:
> > echo 1 > /proc/sys/net/ipv4/route/flush
> > doesn't fix the problem.
> > 
> > Ideas? I could arrange to give access to the testbed if needed.
> > (https://www.grid5000.fr/)
> 
> eventually, after fresh boot of both sender/receiver, it would be nice
> to have "netstat -s" results after one transfert with said problems.

Hi,

Attached.

I had 6 successful transfers. There's the netstat -s output after the 5
first successful transfers. Then the 7th transfer showed the problem,
as well as the 8th and the 9th. There's the netstat -s output after each
of them.
-- 
| Lucas Nussbaum             MCF Université Nancy 2 |
| lucas.nussbaum@loria.fr         LORIA / AlGorille |
| http://www.loria.fr/~lnussbau/  +33 3 54 95 86 19 |

[-- Attachment #2: netstat-s.tgz --]
[-- Type: application/x-gtar, Size: 2275 bytes --]

Re: Bad performance when CUBIC HyStart enabled

[Eric Dumazet]

Le jeudi 03 mars 2011 à 18:37 +0100, Lucas Nussbaum a écrit :

> I had 6 successful transfers. There's the netstat -s output after the 5
> first successful transfers. Then the 7th transfer showed the problem,
> as well as the 8th and the 9th. There's the netstat -s output after each
> of them.

Its a bit strange : my 'netstat -s' are more verbose than yours :

 netstat -s
Ip:
    1033944 total packets received
    0 forwarded
    0 incoming packets discarded
    1033705 incoming packets delivered
    701852 requests sent out
    4 dropped because of missing route
    120 reassemblies required
    60 packets reassembled ok
    60 fragments received ok
    120 fragments created
Icmp:
    704 ICMP messages received
    0 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 177
        timeout in transit: 117
        echo requests: 103
        echo replies: 307
    693 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 177
        echo request: 413
        echo replies: 103
IcmpMsg:
        InType0: 307
        InType3: 177
        InType8: 103
        InType11: 117
        OutType0: 103
        OutType3: 177
        OutType8: 413
Tcp:
    10526 active connections openings
    212 passive connection openings
    78 failed connection attempts
    300 connection resets received
    26 connections established
    969786 segments received
    690213 segments send out
    894 segments retransmited
    0 bad segments received.
    305 resets sent
Udp:
    43123 packets received
    4 packets to unknown port received.
    0 packet receive errors
    9570 packets sent
UdpLite:
TcpExt:
    877 TCP sockets finished time wait in fast timer
    23798 delayed acks sent
    10 delayed acks further delayed because of locked socket
    Quick ack mode was activated 598 times
    542574 packets directly queued to recvmsg prequeue.
    140834 bytes directly in process context from backlog
    45033318 bytes directly received in process context from prequeue
    646384 packet headers predicted
    81439 packets header predicted and directly queued to user
    24971 acknowledgments not containing data payload received
    189383 predicted acknowledgments
    2 times recovered from packet loss by selective acknowledgements
    3 congestion windows recovered without slow start by DSACK
    308 congestion windows recovered without slow start after partial
ack
    2 TCP data loss events
    3 timeouts after SACK recovery
    2 fast retransmits
    1 retransmits in slow start
    632 other TCP timeouts
    9 times receiver scheduled too late for direct processing
    587 DSACKs sent for old packets
    2 DSACKs sent for out of order packets
    417 DSACKs received
    130 connections reset due to unexpected data
    72 connections aborted due to timeout
    TCPDSACKIgnoredOld: 176
    TCPDSACKIgnoredNoUndo: 56
    TCPSackMerged: 5
    TCPSackShiftFallback: 4
    TCPDeferAcceptDrop: 106
IpExt:
    InNoRoutes: 5
    InMcastPkts: 35832
    OutMcastPkts: 1403
    InBcastPkts: 19536
    OutBcastPkts: 89
    InOctets: 787953535
    OutOctets: 113825814
    InMcastOctets: 4098542
    OutMcastOctets: 211245
    InBcastOctets: 1906912
    OutBcastOctets: 13625

Re: Bad performance when CUBIC HyStart enabled

[Lucas Nussbaum]

On 03/03/11 at 18:46 +0100, Eric Dumazet wrote:
> Le jeudi 03 mars 2011 à 18:37 +0100, Lucas Nussbaum a écrit :
> 
> > I had 6 successful transfers. There's the netstat -s output after the 5
> > first successful transfers. Then the 7th transfer showed the problem,
> > as well as the 8th and the 9th. There's the netstat -s output after each
> > of them.
> 
> Its a bit strange : my 'netstat -s' are more verbose than yours :

Strange. Mine is from the debian package, and it hasn't been updated
recently, neither in Debian nor at
https://developer.berlios.de/project/showfiles.php?group_id=4537.
Maybe some fields are not displayed when the value is 0?
-- 
| Lucas Nussbaum             MCF Université Nancy 2 |
| lucas.nussbaum@loria.fr         LORIA / AlGorille |
| http://www.loria.fr/~lnussbau/  +33 3 54 95 86 19 |