Re: TCP funny-ness when over-driving a 1Gbps link (and wifi)

[Ben Greear <greearb@candelatech.com>]

On 05/19/2011 08:39 PM, Ben Greear wrote:
> On 05/19/2011 05:46 PM, Rick Jones wrote:
>> On Thu, 2011-05-19 at 17:37 -0700, Ben Greear wrote:
>>> On 05/19/2011 05:24 PM, Rick Jones wrote:
>>>>>>> [root@i7-965-1 igb]# netstat -an|grep tcp|grep 8.1.1
>>>>>>> tcp 0 0 8.1.1.1:33038 0.0.0.0:* LISTEN
>>>>>>> tcp 0 0 8.1.1.1:33040 0.0.0.0:* LISTEN
>>>>>>> tcp 0 0 8.1.1.1:33042 0.0.0.0:* LISTEN
>>>>>>> tcp 0 9328612 8.1.1.2:33039 8.1.1.1:33040 ESTABLISHED
>>>>>>> tcp 0 17083176 8.1.1.1:33038 8.1.1.2:33037 ESTABLISHED
>>>>>>> tcp 0 9437340 8.1.1.2:33037 8.1.1.1:33038 ESTABLISHED
>>>>>>> tcp 0 17024620 8.1.1.1:33040 8.1.1.2:33039 ESTABLISHED
>>>>>>> tcp 0 19557040 8.1.1.1:33042 8.1.1.2:33041 ESTABLISHED
>>>>>>> tcp 0 9416600 8.1.1.2:33041 8.1.1.1:33042 ESTABLISHED
>>>>>>
>>>>>> I take it your system has higher values for the tcp_wmem value:

I tried a different test today:  3 TCP connections between two
wifi station interfaces (using ath9k).  Each connection is
endpoint configured to send 100Mbps of traffic to the peer.

With a single connection, it does OK (maybe 250ms round-trip time max).
With 3 of them running, round-trip user-space to user-space latency
often goes above 3 seconds.

I had set tcp_wmem smaller for this test, and I verified that
the socket SND/RCV buffer setsockopt was not being called.

[root@lec2010-ath9k-1 lanforge]# netstat -an|grep tcp|grep 33
tcp        0      0 12.12.12.4:33040            0.0.0.0:*                   LISTEN
tcp        0      0 12.12.12.4:33042            0.0.0.0:*                   LISTEN
tcp        0      0 12.12.12.4:33044            0.0.0.0:*                   LISTEN
tcp        0 556072 12.12.12.4:33040            12.12.12.3:33039            ESTABLISHED
tcp        0 274916 12.12.12.3:33043            12.12.12.4:33044            ESTABLISHED
tcp        0      0 192.168.100.138:33738       192.168.100.3:2049          ESTABLISHED
tcp        0 205156 12.12.12.4:33042            12.12.12.3:33041            ESTABLISHED
tcp        0 217184 12.12.12.3:33041            12.12.12.4:33042            ESTABLISHED
tcp        0 436552 12.12.12.3:33039            12.12.12.4:33040            ESTABLISHED
tcp        0 288820 12.12.12.4:33044            12.12.12.3:33043            ESTABLISHED

[root@lec2010-ath9k-1 lanforge]# cat /proc/sys/net/ipv4/tcp_wmem
4096	16384	4000000

This is 2.6.39-wl+ kernel.

So, seems a general issue with over-driving links with multiple TCP connections.
Doesn't seem like a regression, and probably not really a bug, but maybe the
buffer-bloat project will help this sort of thing...

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com