TCP_STREAM performance regression on commit b3613118

TCP_STREAM performance regression on commit b3613118

[Alex,Shi]

The tcp_stream loop back performance has about 10% drop on the
commitment on our core2 2 sockets server. This commit has 2
parents(7505afe28, 5983fe), but both of them have no regression. So
guess the impact just happened when this 2 parents joint. That beyond
our capability to dig it more.

Any ideas? 

Regards! 
Alex 

Re: TCP_STREAM performance regression on commit b3613118

[Eric Dumazet]

Le vendredi 17 février 2012 à 16:18 +0800, Alex,Shi a écrit :
> The tcp_stream loop back performance has about 10% drop on the
> commitment on our core2 2 sockets server. This commit has 2
> parents(7505afe28, 5983fe), but both of them have no regression. So
> guess the impact just happened when this 2 parents joint. That beyond
> our capability to dig it more.
> 
> Any ideas? 

Most probably the more accurate truesize determination is responsible of
this tcp regression, since some prior assumptions might be wrong.

Want to give more information on the workload ?
Is it a 32 or 64 bit kernel ?

cat /proc/sys/net/ipv4/tcp_rmem
cat /proc/sys/net/ipv4/tcp_wmem
netstat -s

Re: TCP_STREAM performance regression on commit b3613118

[David Miller]

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Fri, 17 Feb 2012 10:50:39 +0100

> Le vendredi 17 février 2012 à 16:18 +0800, Alex,Shi a écrit :
>> The tcp_stream loop back performance has about 10% drop on the
>> commitment on our core2 2 sockets server. This commit has 2
>> parents(7505afe28, 5983fe), but both of them have no regression. So
>> guess the impact just happened when this 2 parents joint. That beyond
>> our capability to dig it more.
>> 
>> Any ideas? 
> 
> Most probably the more accurate truesize determination is responsible of
> this tcp regression, since some prior assumptions might be wrong.
> 
> Want to give more information on the workload ?
> Is it a 32 or 64 bit kernel ?

And let's start CC:'ing netdev too.

Re: TCP_STREAM performance regression on commit b3613118

[Alex,Shi]

On Fri, 2012-02-17 at 13:33 -0500, David Miller wrote:
> From: Eric Dumazet <eric.dumazet@gmail.com>
> Date: Fri, 17 Feb 2012 10:50:39 +0100
> 
> > Le vendredi 17 février 2012 à 16:18 +0800, Alex,Shi a écrit :
> >> The tcp_stream loop back performance has about 10% drop on the
> >> commitment on our core2 2 sockets server. This commit has 2
> >> parents(7505afe28, 5983fe), but both of them have no regression. So
> >> guess the impact just happened when this 2 parents joint. That beyond
> >> our capability to dig it more.
> >> 
> >> Any ideas? 
> > 
> > Most probably the more accurate truesize determination is responsible of
> > this tcp regression, since some prior assumptions might be wrong.
> > 
> > Want to give more information on the workload ?
> > Is it a 32 or 64 bit kernel ?

It is 64 bit kernel. Currently we only care 64bit. 

Will try to bring more statistic after our lab recovered. 
> 
> And let's start CC:'ing netdev too.

Re: TCP_STREAM performance regression on commit b3613118

[Alex Shi]

On Fri, 2012-02-17 at 13:33 -0500, David Miller wrote:
> From: Eric Dumazet <eric.dumazet@gmail.com>
> Date: Fri, 17 Feb 2012 10:50:39 +0100
> 
> > Le vendredi 17 février 2012 à 16:18 +0800, Alex,Shi a écrit :
> >> The tcp_stream loop back performance has about 10% drop on the
> >> commitment on our core2 2 sockets server. This commit has 2
> >> parents(7505afe28, 5983fe), but both of them have no regression. So
> >> guess the impact just happened when this 2 parents joint. That beyond
> >> our capability to dig it more.
> >> 
> >> Any ideas? 
> > 
> > Most probably the more accurate truesize determination is responsible of
> > this tcp regression, since some prior assumptions might be wrong.
> > 
> > Want to give more information on the workload ?
> > Is it a 32 or 64 bit kernel ?
> 
> And let's start CC:'ing netdev too.

Add CC to tang feng, He is working on this issue. 

Re: TCP_STREAM performance regression on commit b3613118

[David Miller]

From: Alex Shi <alex.shi@intel.com>
Date: Fri, 02 Mar 2012 10:45:17 +0800

> Add CC to tang feng, He is working on this issue. 

Is he?  I'm pretty sure this is due to the TCP receive window growing
issue Eric Dumazet, Neal Cardwell and I are discussing in the thread
starting at:

http://marc.info/?l=linux-netdev&m=132916352815286&w=2

Re: TCP_STREAM performance regression on commit b3613118

[Alex Shi]

On Thu, 2012-03-01 at 22:07 -0500, David Miller wrote:
> From: Alex Shi <alex.shi@intel.com>
> Date: Fri, 02 Mar 2012 10:45:17 +0800
> 
> > Add CC to tang feng, He is working on this issue. 
> 
> Is he?  I'm pretty sure this is due to the TCP receive window growing
> issue Eric Dumazet, Neal Cardwell and I are discussing in the thread
> starting at:
> 
> http://marc.info/?l=linux-netdev&m=132916352815286&w=2

That's great! 
May my language make you confusing. Actually, Feng is doing
investigation on this issue. Since you fund out the reason, it saved his
work. :) 

Re: TCP_STREAM performance regression on commit b3613118

[Eric Dumazet]

Le vendredi 02 mars 2012 à 14:37 +0800, Alex Shi a écrit :
> On Thu, 2012-03-01 at 22:07 -0500, David Miller wrote:
> > From: Alex Shi <alex.shi@intel.com>
> > Date: Fri, 02 Mar 2012 10:45:17 +0800
> > 
> > > Add CC to tang feng, He is working on this issue. 
> > 
> > Is he?  I'm pretty sure this is due to the TCP receive window growing
> > issue Eric Dumazet, Neal Cardwell and I are discussing in the thread
> > starting at:
> > 
> > http://marc.info/?l=linux-netdev&m=132916352815286&w=2
> 
> That's great! 
> May my language make you confusing. Actually, Feng is doing
> investigation on this issue. Since you fund out the reason, it saved his
> work. :) 
> 

Actually it would be great if you or Feng could post the exact tests
done and numbers so that we can make sure this is the tracked issue.

Thanks

Re: TCP_STREAM performance regression on commit b3613118

[Feng Tang]

On Thu, Mar 01, 2012 at 10:07:43PM -0500, David Miller wrote:
> From: Alex Shi <alex.shi@intel.com>
> Date: Fri, 02 Mar 2012 10:45:17 +0800
> 
> > Add CC to tang feng, He is working on this issue. 
> 
> Is he?  I'm pretty sure this is due to the TCP receive window growing
> issue Eric Dumazet, Neal Cardwell and I are discussing in the thread
> starting at:
> 
> http://marc.info/?l=linux-netdev&m=132916352815286&w=2

Yes, probably, as we did find some clue related with the tcp_r/wmem.

Here is the regression we found:
On some machines, we found there is about 10% resgression of netperf
TCP-64K loopback test between 3.2 and 3.3-rc1. The exact test is:
./netperf -t TCP_STREAM -l 60 -H 127.0.0.1 -i 50,3 -I 99,5 -- -s 32768 -S 32768 -m 4096


The test machine is a 2 socket Quad Core Core 2 Duo server(2.66GHz) with
8 GB RAM. Following are the debug info (ifconfig/netstat -s/tcp_rwmem)
before and after the test: 

The most obvious differences I can see are:
1) 311 GB vs 241 GB from ifconfig
2) the difference of the tcp_r/wmem

Thanks,
Feng

----------------------------------------------------------
Before the test (v3.2)
===========================
ifconfig info
eth1      Link encap:Ethernet  HWaddr 00:30:48:7C:CF:72  
          inet addr:192.168.1.58  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::230:48ff:fe7c:cf72/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:947 errors:0 dropped:0 overruns:0 frame:0
          TX packets:253 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:76573 (74.7 KiB)  TX bytes:31034 (30.3 KiB)
          Memory:d8420000-d8440000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:197 errors:0 dropped:0 overruns:0 frame:0
          TX packets:197 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:21827 (21.3 KiB)  TX bytes:21827 (21.3 KiB)

netstat info
Ip:
    593 total packets received
    3 with invalid addresses
    0 forwarded
    0 incoming packets discarded
    590 incoming packets delivered
    431 requests sent out
Icmp:
    7 ICMP messages received
    0 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 7
    7 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 7
IcmpMsg:
        InType3: 7
        OutType3: 7
Tcp:
    7 active connections openings
    10 passive connection openings
    0 failed connection attempts
    0 connection resets received
    3 connections established
    371 segments received
    328 segments send out
    0 segments retransmited
    0 bad segments received.
    3 resets sent
Udp:
    87 packets received
    7 packets to unknown port received.
    0 packet receive errors
    94 packets sent
UdpLite:
tcp_rmem
4096	87380	4194304
tcp_wmem
4096	16384	4194304
=========================


After the test (v3.2)
===========================
ifconfig info
eth1      Link encap:Ethernet  HWaddr 00:30:48:7C:CF:72  
          inet addr:192.168.1.58  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::230:48ff:fe7c:cf72/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2278 errors:0 dropped:0 overruns:0 frame:0
          TX packets:690 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:174366 (170.2 KiB)  TX bytes:88711 (86.6 KiB)
          Memory:d8420000-d8440000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:54845546 errors:0 dropped:0 overruns:0 frame:0
          TX packets:54845546 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:334960558399 (311.9 GiB)  TX bytes:334960558399 (311.9 GiB)

netstat info
Ip:
    54846593 total packets received
    3 with invalid addresses
    0 forwarded
    0 incoming packets discarded
    54846590 incoming packets delivered
    54846212 requests sent out
Icmp:
    7 ICMP messages received
    0 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 7
    7 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 7
IcmpMsg:
        InType3: 7
        OutType3: 7
Tcp:
    21 active connections openings
    22 passive connection openings
    0 failed connection attempts
    0 connection resets received
    3 connections established
    54846182 segments received
    54846067 segments send out
    0 segments retransmited
    0 bad segments received.
    3 resets sent
Udp:
    126 packets received
    7 packets to unknown port received.
    0 packet receive errors
    134 packets sent
UdpLite:
tcp_rmem
4096	87380	4194304
tcp_wmem
4096	16384	4194304
=========================



Before the test (v3.3-rc1)
===========================
ifconfig info
eth1      Link encap:Ethernet  HWaddr 00:30:48:7C:CF:72  
          inet addr:192.168.1.58  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::230:48ff:fe7c:cf72/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:576 errors:0 dropped:0 overruns:0 frame:0
          TX packets:205 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:50045 (48.8 KiB)  TX bytes:27018 (26.3 KiB)
          Memory:d8420000-d8440000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:88 errors:0 dropped:0 overruns:0 frame:0
          TX packets:88 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:9613 (9.3 KiB)  TX bytes:9613 (9.3 KiB)

netstat info
Ip:
    365 total packets received
    3 with invalid addresses
    0 forwarded
    0 incoming packets discarded
    362 incoming packets delivered
    278 requests sent out
Icmp:
    3 ICMP messages received
    0 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 3
    3 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 3
IcmpMsg:
        InType3: 3
        OutType3: 3
Tcp:
    3 active connections openings
    6 passive connection openings
    0 failed connection attempts
    0 connection resets received
    3 connections established
    226 segments received
    206 segments send out
    0 segments retransmited
    0 bad segments received.
    3 resets sent
Udp:
    64 packets received
    3 packets to unknown port received.
    0 packet receive errors
    67 packets sent
UdpLite:
tcp_rmem
4096	87380	87380
tcp_wmem
4096	16384	65536
=========================


After the test (v3.3-rc1)
===========================
ifconfig info
eth1      Link encap:Ethernet  HWaddr 00:30:48:7C:CF:72  
          inet addr:192.168.1.58  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::230:48ff:fe7c:cf72/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1724 errors:0 dropped:0 overruns:0 frame:0
          TX packets:608 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:134666 (131.5 KiB)  TX bytes:79003 (77.1 KiB)
          Memory:d8420000-d8440000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:45143339 errors:0 dropped:0 overruns:0 frame:0
          TX packets:45143339 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:259440765641 (241.6 GiB)  TX bytes:259440765641 (241.6 GiB)

netstat info
Ip:
    45144169 total packets received
    3 with invalid addresses
    0 forwarded
    0 incoming packets discarded
    45144166 incoming packets delivered
    45143928 requests sent out
Icmp:
    3 ICMP messages received
    0 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 3
    3 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 3
IcmpMsg:
        InType3: 3
        OutType3: 3
Tcp:
    16 active connections openings
    17 passive connection openings
    0 failed connection attempts
    0 connection resets received
    3 connections established
    45143859 segments received
    45143814 segments send out
    0 segments retransmited
    0 bad segments received.
    3 resets sent
Udp:
    103 packets received
    3 packets to unknown port received.
    0 packet receive errors
    107 packets sent
UdpLite:
tcp_rmem
4096	87380	87380
tcp_wmem
4096	16384	65536
=========================

Re: TCP_STREAM performance regression on commit b3613118

[Jason Wang]

On 03/06/2012 04:11 PM, Feng Tang wrote:
> On Thu, Mar 01, 2012 at 10:07:43PM -0500, David Miller wrote:
>> >  From: Alex Shi<alex.shi@intel.com>
>> >  Date: Fri, 02 Mar 2012 10:45:17 +0800
>> >  
>>> >  >  Add CC to tang feng, He is working on this issue.
>> >  
>> >  Is he?  I'm pretty sure this is due to the TCP receive window growing
>> >  issue Eric Dumazet, Neal Cardwell and I are discussing in the thread
>> >  starting at:
>> >  
>> >  http://marc.info/?l=linux-netdev&m=132916352815286&w=2
> Yes, probably, as we did find some clue related with the tcp_r/wmem.
>
> Here is the regression we found:
> On some machines, we found there is about 10% resgression of netperf
> TCP-64K loopback test between 3.2 and 3.3-rc1. The exact test is:
> ./netperf -t TCP_STREAM -l 60 -H 127.0.0.1 -i 50,3 -I 99,5 -- -s 32768 -S 32768 -m 4096
>
>
> The test machine is a 2 socket Quad Core Core 2 Duo server(2.66GHz) with
> 8 GB RAM. Following are the debug info (ifconfig/netstat -s/tcp_rwmem)
> before and after the test:
>
> The most obvious differences I can see are:
> 1) 311 GB vs 241 GB from ifconfig
> 2) the difference of the tcp_r/wmem

Hi:

Could you try the newest kernel? Looks like the difference has been 
already fixed by commit c43b874d5d714f271b80d4c3f49e05d0cbf51ed2.

Thanks

Re: TCP_STREAM performance regression on commit b3613118

[Eric Dumazet]

Le mardi 06 mars 2012 à 16:26 +0800, Jason Wang a écrit :
> On 03/06/2012 04:11 PM, Feng Tang wrote:
> > On Thu, Mar 01, 2012 at 10:07:43PM -0500, David Miller wrote:
> >> >  From: Alex Shi<alex.shi@intel.com>
> >> >  Date: Fri, 02 Mar 2012 10:45:17 +0800
> >> >  
> >>> >  >  Add CC to tang feng, He is working on this issue.
> >> >  
> >> >  Is he?  I'm pretty sure this is due to the TCP receive window growing
> >> >  issue Eric Dumazet, Neal Cardwell and I are discussing in the thread
> >> >  starting at:
> >> >  
> >> >  http://marc.info/?l=linux-netdev&m=132916352815286&w=2
> > Yes, probably, as we did find some clue related with the tcp_r/wmem.
> >
> > Here is the regression we found:
> > On some machines, we found there is about 10% resgression of netperf
> > TCP-64K loopback test between 3.2 and 3.3-rc1. The exact test is:
> > ./netperf -t TCP_STREAM -l 60 -H 127.0.0.1 -i 50,3 -I 99,5 -- -s 32768 -S 32768 -m 4096
> >
> >
> > The test machine is a 2 socket Quad Core Core 2 Duo server(2.66GHz) with
> > 8 GB RAM. Following are the debug info (ifconfig/netstat -s/tcp_rwmem)
> > before and after the test:
> >
> > The most obvious differences I can see are:
> > 1) 311 GB vs 241 GB from ifconfig
> > 2) the difference of the tcp_r/wmem
> 
> Hi:
> 
> Could you try the newest kernel? Looks like the difference has been 
> already fixed by commit c43b874d5d714f271b80d4c3f49e05d0cbf51ed2.
> 

Most likely yes. 

tcp_rmem
4096    87380   87380
tcp_wmem
4096    16384   65536

Is way pessimistic :(

Re: TCP_STREAM performance regression on commit b3613118

[Feng Tang]

On Tue, Mar 06, 2012 at 04:26:13PM +0800, Jason Wang wrote:
> On 03/06/2012 04:11 PM, Feng Tang wrote:
> >On Thu, Mar 01, 2012 at 10:07:43PM -0500, David Miller wrote:
> >>>  From: Alex Shi<alex.shi@intel.com>
> >>>  Date: Fri, 02 Mar 2012 10:45:17 +0800
> >>>
> >>>>  >  Add CC to tang feng, He is working on this issue.
> >>>  >  Is he?  I'm pretty sure this is due to the TCP receive
> >>window growing
> >>>  issue Eric Dumazet, Neal Cardwell and I are discussing in the thread
> >>>  starting at:
> >>>  >  http://marc.info/?l=linux-netdev&m=132916352815286&w=2
> >Yes, probably, as we did find some clue related with the tcp_r/wmem.
> >
> >Here is the regression we found:
> >On some machines, we found there is about 10% resgression of netperf
> >TCP-64K loopback test between 3.2 and 3.3-rc1. The exact test is:
> >./netperf -t TCP_STREAM -l 60 -H 127.0.0.1 -i 50,3 -I 99,5 -- -s 32768 -S 32768 -m 4096
> >
> >
> >The test machine is a 2 socket Quad Core Core 2 Duo server(2.66GHz) with
> >8 GB RAM. Following are the debug info (ifconfig/netstat -s/tcp_rwmem)
> >before and after the test:
> >
> >The most obvious differences I can see are:
> >1) 311 GB vs 241 GB from ifconfig
> >2) the difference of the tcp_r/wmem
> 
> Hi:
> 
> Could you try the newest kernel? Looks like the difference has been
> already fixed by commit c43b874d5d714f271b80d4c3f49e05d0cbf51ed2.

Yeah, with the newest kernel, the regression of this simple test is gone,
the performance difference with 3.2 kernel is now only about 1-2%. Thanks
for the info.

- Feng

> 
> Thanks

Re: TCP_STREAM performance regression on commit b3613118

[Rick Jones]

On 03/06/2012 12:11 AM, Feng Tang wrote:
> On some machines, we found there is about 10% resgression of netperf
> TCP-64K loopback test between 3.2 and 3.3-rc1. The exact test is:
> ./netperf -t TCP_STREAM -l 60 -H 127.0.0.1 -i 50,3 -I 99,5 -- -s 32768 -S 32768 -m 4096

Side comment on the netperf command line.  The maximum confidence 
interactions is silently capped at 30, so that might as well be "-i 
30,3" 
http://www.netperf.org/svn/netperf2/tags/netperf-2.5.0/doc/netperf.html#index-g_t_002di_002c-Global-28

Also, for reproducibility, it might be desirable to pin netperf and 
netserver to a specific CPU or CPUs.  That would be with a global -T 
option: 
http://www.netperf.org/svn/netperf2/tags/netperf-2.5.0/doc/netperf.html#index-g_t_002dT_002c-Global-41

happy benchmarking,

rick jones