* Bad performance when CUBIC HyStart enabled
@ 2011-03-03 14:22 Lucas Nussbaum
2011-03-03 15:03 ` Eric Dumazet
0 siblings, 1 reply; 5+ messages in thread
From: Lucas Nussbaum @ 2011-03-03 14:22 UTC (permalink / raw)
To: netdev; +Cc: Sangtae Ha
Hi,
I'm experiencing bad performance transferring data over a large BDP
network with CUBIC when HyStart is enabled. Disabling HyStart fixes the
problem.
Test setup:
I'm transfering a 2-GB file between two machines using netcat. On both
machines, /dev/shm is used to store the data.
On the sending side, I use:
/dev/shm# while true; do dd if=foo bs=10M | nc -l -p 2222 -q 0; done
On the receiving side, I use:
/dev/shm# /usr/bin/time nc 131.254.203.6 2222 > foo
Both machines are on Gigabit ethernet networks, and the RTT between them
is 11.3ms.
The default sysctl values are used.
I use nload -i 1000000 -o 1000000 -u m eth0 to visualize the transfer
rate.
With a 2.6.26 kernel, everything is fine. The bandwidth quickly gets to
~940 Mbps, and I get the following in nload:
..................................
.##################################
###################################
###################################
###################################
####################################
####################################
####################################
####################################
####################################
####################################
####################################
####################################
####################################
The transfer time is 18.2s, which is reasonable for a 2-GB file.
With a 2.6.32 kernel (Debian's) or with a 2.6.37 kernel (on the sending
side only), with the default HyStart parameters, I get the following:
..........................
|##########################.
|############################
|#############################
|##############################
|###############################
.#################################
|##################################|
|#####################################
..||#######################################
....|##############################################
.#####################################################
The transfer time is 26.8s. (There's one character every 0.5s on the
graph above)
Disabling hystart on the sending machine with:
echo 0 > /sys/module/tcp_cubic/parameters/hystart
Fixes the problem.
It seems that waiting for some time (or rebooting) fixes the problem for
some time, so I'm suspecting that the problem is caused by a cache of
some sort. However, doing:
echo 1 > /proc/sys/net/ipv4/route/flush
doesn't fix the problem.
Ideas? I could arrange to give access to the testbed if needed.
(https://www.grid5000.fr/)
--
| Lucas Nussbaum MCF Université Nancy 2 |
| lucas.nussbaum@loria.fr LORIA / AlGorille |
| http://www.loria.fr/~lnussbau/ +33 3 54 95 86 19 |
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: Bad performance when CUBIC HyStart enabled
2011-03-03 14:22 Bad performance when CUBIC HyStart enabled Lucas Nussbaum
@ 2011-03-03 15:03 ` Eric Dumazet
2011-03-03 17:37 ` Lucas Nussbaum
0 siblings, 1 reply; 5+ messages in thread
From: Eric Dumazet @ 2011-03-03 15:03 UTC (permalink / raw)
To: Lucas Nussbaum; +Cc: netdev, Sangtae Ha
Le jeudi 03 mars 2011 à 15:22 +0100, Lucas Nussbaum a écrit :
> Hi,
>
> I'm experiencing bad performance transferring data over a large BDP
> network with CUBIC when HyStart is enabled. Disabling HyStart fixes the
> problem.
>
> Test setup:
> I'm transfering a 2-GB file between two machines using netcat. On both
> machines, /dev/shm is used to store the data.
> On the sending side, I use:
> /dev/shm# while true; do dd if=foo bs=10M | nc -l -p 2222 -q 0; done
> On the receiving side, I use:
> /dev/shm# /usr/bin/time nc 131.254.203.6 2222 > foo
> Both machines are on Gigabit ethernet networks, and the RTT between them
> is 11.3ms.
> The default sysctl values are used.
> I use nload -i 1000000 -o 1000000 -u m eth0 to visualize the transfer
> rate.
>
> With a 2.6.26 kernel, everything is fine. The bandwidth quickly gets to
> ~940 Mbps, and I get the following in nload:
> ..................................
> .##################################
> ###################################
> ###################################
> ###################################
> ####################################
> ####################################
> ####################################
> ####################################
> ####################################
> ####################################
> ####################################
> ####################################
> ####################################
> The transfer time is 18.2s, which is reasonable for a 2-GB file.
>
> With a 2.6.32 kernel (Debian's) or with a 2.6.37 kernel (on the sending
> side only), with the default HyStart parameters, I get the following:
> ..........................
> |##########################.
> |############################
> |#############################
> |##############################
> |###############################
> .#################################
> |##################################|
> |#####################################
> ..||#######################################
> ....|##############################################
> .#####################################################
> The transfer time is 26.8s. (There's one character every 0.5s on the
> graph above)
>
> Disabling hystart on the sending machine with:
> echo 0 > /sys/module/tcp_cubic/parameters/hystart
> Fixes the problem.
>
> It seems that waiting for some time (or rebooting) fixes the problem for
> some time, so I'm suspecting that the problem is caused by a cache of
> some sort. However, doing:
> echo 1 > /proc/sys/net/ipv4/route/flush
> doesn't fix the problem.
>
> Ideas? I could arrange to give access to the testbed if needed.
> (https://www.grid5000.fr/)
eventually, after fresh boot of both sender/receiver, it would be nice
to have "netstat -s" results after one transfert with said problems.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Bad performance when CUBIC HyStart enabled
2011-03-03 15:03 ` Eric Dumazet
@ 2011-03-03 17:37 ` Lucas Nussbaum
2011-03-03 17:46 ` Eric Dumazet
0 siblings, 1 reply; 5+ messages in thread
From: Lucas Nussbaum @ 2011-03-03 17:37 UTC (permalink / raw)
To: Eric Dumazet; +Cc: netdev, Sangtae Ha
[-- Attachment #1: Type: text/plain, Size: 3694 bytes --]
On 03/03/11 at 16:03 +0100, Eric Dumazet wrote:
> Le jeudi 03 mars 2011 à 15:22 +0100, Lucas Nussbaum a écrit :
> > Hi,
> >
> > I'm experiencing bad performance transferring data over a large BDP
> > network with CUBIC when HyStart is enabled. Disabling HyStart fixes the
> > problem.
> >
> > Test setup:
> > I'm transfering a 2-GB file between two machines using netcat. On both
> > machines, /dev/shm is used to store the data.
> > On the sending side, I use:
> > /dev/shm# while true; do dd if=foo bs=10M | nc -l -p 2222 -q 0; done
> > On the receiving side, I use:
> > /dev/shm# /usr/bin/time nc 131.254.203.6 2222 > foo
> > Both machines are on Gigabit ethernet networks, and the RTT between them
> > is 11.3ms.
> > The default sysctl values are used.
> > I use nload -i 1000000 -o 1000000 -u m eth0 to visualize the transfer
> > rate.
> >
> > With a 2.6.26 kernel, everything is fine. The bandwidth quickly gets to
> > ~940 Mbps, and I get the following in nload:
> > ..................................
> > .##################################
> > ###################################
> > ###################################
> > ###################################
> > ####################################
> > ####################################
> > ####################################
> > ####################################
> > ####################################
> > ####################################
> > ####################################
> > ####################################
> > ####################################
> > The transfer time is 18.2s, which is reasonable for a 2-GB file.
> >
> > With a 2.6.32 kernel (Debian's) or with a 2.6.37 kernel (on the sending
> > side only), with the default HyStart parameters, I get the following:
> > ..........................
> > |##########################.
> > |############################
> > |#############################
> > |##############################
> > |###############################
> > .#################################
> > |##################################|
> > |#####################################
> > ..||#######################################
> > ....|##############################################
> > .#####################################################
> > The transfer time is 26.8s. (There's one character every 0.5s on the
> > graph above)
> >
> > Disabling hystart on the sending machine with:
> > echo 0 > /sys/module/tcp_cubic/parameters/hystart
> > Fixes the problem.
> >
> > It seems that waiting for some time (or rebooting) fixes the problem for
> > some time, so I'm suspecting that the problem is caused by a cache of
> > some sort. However, doing:
> > echo 1 > /proc/sys/net/ipv4/route/flush
> > doesn't fix the problem.
> >
> > Ideas? I could arrange to give access to the testbed if needed.
> > (https://www.grid5000.fr/)
>
> eventually, after fresh boot of both sender/receiver, it would be nice
> to have "netstat -s" results after one transfert with said problems.
Hi,
Attached.
I had 6 successful transfers. There's the netstat -s output after the 5
first successful transfers. Then the 7th transfer showed the problem,
as well as the 8th and the 9th. There's the netstat -s output after each
of them.
--
| Lucas Nussbaum MCF Université Nancy 2 |
| lucas.nussbaum@loria.fr LORIA / AlGorille |
| http://www.loria.fr/~lnussbau/ +33 3 54 95 86 19 |
[-- Attachment #2: netstat-s.tgz --]
[-- Type: application/x-gtar, Size: 2275 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Bad performance when CUBIC HyStart enabled
2011-03-03 17:37 ` Lucas Nussbaum
@ 2011-03-03 17:46 ` Eric Dumazet
2011-03-03 18:12 ` Lucas Nussbaum
0 siblings, 1 reply; 5+ messages in thread
From: Eric Dumazet @ 2011-03-03 17:46 UTC (permalink / raw)
To: Lucas Nussbaum; +Cc: netdev, Sangtae Ha
Le jeudi 03 mars 2011 à 18:37 +0100, Lucas Nussbaum a écrit :
> I had 6 successful transfers. There's the netstat -s output after the 5
> first successful transfers. Then the 7th transfer showed the problem,
> as well as the 8th and the 9th. There's the netstat -s output after each
> of them.
Its a bit strange : my 'netstat -s' are more verbose than yours :
netstat -s
Ip:
1033944 total packets received
0 forwarded
0 incoming packets discarded
1033705 incoming packets delivered
701852 requests sent out
4 dropped because of missing route
120 reassemblies required
60 packets reassembled ok
60 fragments received ok
120 fragments created
Icmp:
704 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
destination unreachable: 177
timeout in transit: 117
echo requests: 103
echo replies: 307
693 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 177
echo request: 413
echo replies: 103
IcmpMsg:
InType0: 307
InType3: 177
InType8: 103
InType11: 117
OutType0: 103
OutType3: 177
OutType8: 413
Tcp:
10526 active connections openings
212 passive connection openings
78 failed connection attempts
300 connection resets received
26 connections established
969786 segments received
690213 segments send out
894 segments retransmited
0 bad segments received.
305 resets sent
Udp:
43123 packets received
4 packets to unknown port received.
0 packet receive errors
9570 packets sent
UdpLite:
TcpExt:
877 TCP sockets finished time wait in fast timer
23798 delayed acks sent
10 delayed acks further delayed because of locked socket
Quick ack mode was activated 598 times
542574 packets directly queued to recvmsg prequeue.
140834 bytes directly in process context from backlog
45033318 bytes directly received in process context from prequeue
646384 packet headers predicted
81439 packets header predicted and directly queued to user
24971 acknowledgments not containing data payload received
189383 predicted acknowledgments
2 times recovered from packet loss by selective acknowledgements
3 congestion windows recovered without slow start by DSACK
308 congestion windows recovered without slow start after partial
ack
2 TCP data loss events
3 timeouts after SACK recovery
2 fast retransmits
1 retransmits in slow start
632 other TCP timeouts
9 times receiver scheduled too late for direct processing
587 DSACKs sent for old packets
2 DSACKs sent for out of order packets
417 DSACKs received
130 connections reset due to unexpected data
72 connections aborted due to timeout
TCPDSACKIgnoredOld: 176
TCPDSACKIgnoredNoUndo: 56
TCPSackMerged: 5
TCPSackShiftFallback: 4
TCPDeferAcceptDrop: 106
IpExt:
InNoRoutes: 5
InMcastPkts: 35832
OutMcastPkts: 1403
InBcastPkts: 19536
OutBcastPkts: 89
InOctets: 787953535
OutOctets: 113825814
InMcastOctets: 4098542
OutMcastOctets: 211245
InBcastOctets: 1906912
OutBcastOctets: 13625
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: Bad performance when CUBIC HyStart enabled
2011-03-03 17:46 ` Eric Dumazet
@ 2011-03-03 18:12 ` Lucas Nussbaum
0 siblings, 0 replies; 5+ messages in thread
From: Lucas Nussbaum @ 2011-03-03 18:12 UTC (permalink / raw)
To: Eric Dumazet; +Cc: netdev, Sangtae Ha
On 03/03/11 at 18:46 +0100, Eric Dumazet wrote:
> Le jeudi 03 mars 2011 à 18:37 +0100, Lucas Nussbaum a écrit :
>
> > I had 6 successful transfers. There's the netstat -s output after the 5
> > first successful transfers. Then the 7th transfer showed the problem,
> > as well as the 8th and the 9th. There's the netstat -s output after each
> > of them.
>
> Its a bit strange : my 'netstat -s' are more verbose than yours :
Strange. Mine is from the debian package, and it hasn't been updated
recently, neither in Debian nor at
https://developer.berlios.de/project/showfiles.php?group_id=4537.
Maybe some fields are not displayed when the value is 0?
--
| Lucas Nussbaum MCF Université Nancy 2 |
| lucas.nussbaum@loria.fr LORIA / AlGorille |
| http://www.loria.fr/~lnussbau/ +33 3 54 95 86 19 |
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2011-03-03 18:11 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-03-03 14:22 Bad performance when CUBIC HyStart enabled Lucas Nussbaum
2011-03-03 15:03 ` Eric Dumazet
2011-03-03 17:37 ` Lucas Nussbaum
2011-03-03 17:46 ` Eric Dumazet
2011-03-03 18:12 ` Lucas Nussbaum
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox