TCP performance drop 2.6.6->2.6.7

TCP performance drop 2.6.6->2.6.7

[Baruch Even]

Hello,

I'm trying to port my H-TCP and SACK improvements to 2.6.11, but I seem 
to hit performance problems that are unrelated to what I worked on so far.

My tests show that between 2.6.6 and 2.6.7 the TCP performance dropped 
considerably.

The setup for the tests, vanilla kernels, dummynet with 100 Mbit/s and 
40ms rtt using the BIC protocol. No patches applied whatsoever.

iperf tests for 2.6.6 get about 90Mbit/s while 2.6.7 gets 30Mbit/s.

I was wondering if someone can think of a reason why this happens?

Is there a way to get the different network related patches between 
these two versions? I don't have access to bk to get it myself.

Baruch

Re: TCP performance drop 2.6.6->2.6.7

[David S. Miller]

On Tue, 12 Apr 2005 23:18:35 +0100
Baruch Even <baruch@ev-en.org> wrote:

> I was wondering if someone can think of a reason why this happens?

Probably some bug we fixed in 2.6.8 or later...

> Is there a way to get the different network related patches between 
> these two versions? I don't have access to bk to get it myself.

Get patch-2.6.7.bz2 or whatever from kernel.org, then scan around for
changes to include/*/tcp.h and net/ipv4/tcp*.c

Re: TCP performance drop 2.6.6->2.6.7

[Stephen Hemminger]

On Tue, 12 Apr 2005 23:18:35 +0100
Baruch Even <baruch@ev-en.org> wrote:

> Hello,
> 
> I'm trying to port my H-TCP and SACK improvements to 2.6.11, but I seem 
> to hit performance problems that are unrelated to what I worked on so far.
> 
> My tests show that between 2.6.6 and 2.6.7 the TCP performance dropped 
> considerably.

Remember 2.6.6 had the old hyper aggressive and over correcting version
of BIC.

> The setup for the tests, vanilla kernels, dummynet with 100 Mbit/s and 
> 40ms rtt using the BIC protocol. No patches applied whatsoever.
> 
> iperf tests for 2.6.6 get about 90Mbit/s while 2.6.7 gets 30Mbit/s.

Haven't measured old kernels recently. But haven't seen that with the
newest stuff I tested last night. At 40ms delay  with netem
over 1G to 100M bridge.
	Reno	99.6
	Vegas	61.6
	BIC	  97.8
	Hstcp	 99.4
	Westwood 99.4

What are your sysctl settings. 

> I was wondering if someone can think of a reason why this happens?
> 
> Is there a way to get the different network related patches between 
> these two versions? I don't have access to bk to get it myself.
> 
> Baruch

Re: TCP performance drop 2.6.6->2.6.7

[Baruch Even]

[-- Attachment #1: Type: text/plain, Size: 772 bytes --]

Stephen Hemminger wrote:
> On Tue, 12 Apr 2005 23:18:35 +0100
> Baruch Even <baruch@ev-en.org> wrote:
>>The setup for the tests, vanilla kernels, dummynet with 100 Mbit/s and 
>>40ms rtt using the BIC protocol. No patches applied whatsoever.
>>
>>iperf tests for 2.6.6 get about 90Mbit/s while 2.6.7 gets 30Mbit/s.
> 
> 
> Haven't measured old kernels recently. But haven't seen that with the
> newest stuff I tested last night. At 40ms delay  with netem
> over 1G to 100M bridge.
> 	Reno	99.6
> 	Vegas	61.6
> 	BIC	  97.8
> 	Hstcp	 99.4
> 	Westwood 99.4

I've now tested 2.6.11.7 and I get about 50Mbit/s, it starts at about 
100 and drops to about 45. Tested for about a minute but it didn't show 
signs of recovery.

> What are your sysctl settings. 

Attached.

Baruch

[-- Attachment #2: sysctls --]
[-- Type: text/plain, Size: 1349 bytes --]

icmp_echo_ignore_all 0
icmp_echo_ignore_broadcasts 0
icmp_ignore_bogus_error_responses 0
icmp_ratelimit 1000
icmp_ratemask 6168
igmp_max_memberships 20
igmp_max_msf 10
inet_peer_gc_maxtime 120
inet_peer_gc_mintime 10
inet_peer_maxttl 600
inet_peer_minttl 120
inet_peer_threshold 65664
ip_autoconfig 0
ip_default_ttl 64
ip_dynaddr 0
ip_forward 0
ip_local_port_range 32768 61000
ip_no_pmtu_disc 0
ip_nonlocal_bind 0
ipfrag_high_thresh 262144
ipfrag_low_thresh 196608
ipfrag_secret_interval 600
ipfrag_time 30
tcp_abort_on_overflow 0
tcp_adv_win_scale 2
tcp_app_win 31
tcp_bic 1
tcp_bic_beta 819
tcp_bic_fast_convergence 1
tcp_bic_low_window 14
tcp_dsack 1
tcp_ecn 0
tcp_fack 1
tcp_fin_timeout 60
tcp_frto 0
tcp_keepalive_intvl 75
tcp_keepalive_probes 9
tcp_keepalive_time 7200
tcp_low_latency 0
tcp_max_orphans 8192
tcp_max_syn_backlog 1024
tcp_max_tw_buckets 180000
tcp_mem 8388608 8388608 8388608
tcp_moderate_rcvbuf 1
tcp_no_metrics_save 0
tcp_orphan_retries 0
tcp_reordering 3
tcp_retrans_collapse 1
tcp_retries1 3
tcp_retries2 15
tcp_rfc1337 0
tcp_rmem 4096 87380 8388608
tcp_sack 1
tcp_stdurg 0
tcp_syn_retries 5
tcp_synack_retries 5
tcp_timestamps 1
tcp_tso_win_divisor 8
tcp_tw_recycle 0
tcp_tw_reuse 0
tcp_vegas_alpha 2
tcp_vegas_beta 6
tcp_vegas_cong_avoid 0
tcp_vegas_gamma 2
tcp_westwood 0
tcp_window_scaling 1
tcp_wmem 4096 87380 8388608

Re: TCP performance drop 2.6.6->2.6.7

[Stephen Hemminger]

On Wed, 13 Apr 2005 00:59:25 +0100
Baruch Even <baruch@ev-en.org> wrote:

> Stephen Hemminger wrote:
> > On Tue, 12 Apr 2005 23:18:35 +0100
> > Baruch Even <baruch@ev-en.org> wrote:
> >>The setup for the tests, vanilla kernels, dummynet with 100 Mbit/s and 
> >>40ms rtt using the BIC protocol. No patches applied whatsoever.

What is the queue length? Better yet what is the exact commandline
used to setup dummynet.

> >>
> >>iperf tests for 2.6.6 get about 90Mbit/s while 2.6.7 gets 30Mbit/s.
> > 
> > 
> > Haven't measured old kernels recently. But haven't seen that with the
> > newest stuff I tested last night. At 40ms delay  with netem
> > over 1G to 100M bridge.
> > 	Reno	99.6
> > 	Vegas	61.6
> > 	BIC	  97.8
> > 	Hstcp	 99.4
> > 	Westwood 99.4
> 
> I've now tested 2.6.11.7 and I get about 50Mbit/s, it starts at about 
> 100 and drops to about 45. Tested for about a minute but it didn't show 
> signs of recovery.
> 
> > What are your sysctl settings. 
> 
> Attached.
> 
> Baruch

Re: TCP performance drop 2.6.6->2.6.7

[Baruch Even]

I have no clue *why* it happens, but a hint on when it happens. If I 
remove the ixgb driver from my .config, performance is back to normal. 
Note though that I don't even use the ixgb driver for my tests, it's 
just sitting idle on the machine and connected to nothing (the cisco 
router is turned off to reduce noise polution).

To continue my work I'll disable the ixgb for now.

Baruch

Baruch Even wrote:
> Hello,
> 
> I'm trying to port my H-TCP and SACK improvements to 2.6.11, but I seem 
> to hit performance problems that are unrelated to what I worked on so far.
> 
> My tests show that between 2.6.6 and 2.6.7 the TCP performance dropped 
> considerably.
> 
> The setup for the tests, vanilla kernels, dummynet with 100 Mbit/s and 
> 40ms rtt using the BIC protocol. No patches applied whatsoever.
> 
> iperf tests for 2.6.6 get about 90Mbit/s while 2.6.7 gets 30Mbit/s.
> 
> I was wondering if someone can think of a reason why this happens?
> 
> Is there a way to get the different network related patches between 
> these two versions? I don't have access to bk to get it myself.
> 
> Baruch
> 

Re: TCP performance drop 2.6.6->2.6.7

[Nivedita Singhvi]

Baruch Even wrote:
> I have no clue *why* it happens, but a hint on when it happens. If I 
> remove the ixgb driver from my .config, performance is back to normal. 
> Note though that I don't even use the ixgb driver for my tests, it's 
> just sitting idle on the machine and connected to nothing (the cisco 
> router is turned off to reduce noise polution).

Check your interrupts - it's probably sending out a large number
of interrupts which are clogging the system..

thanks,
Nivedita