Controlling TCP window size

Controlling TCP window size

[Andy Furniss]

I've been doing some testing of my new wan connection and noticed that 
when I specify a window with ip route it still changes after a while.

Using 2.6.16.11 and latest iproute2.

ip ro del default
ip ro add default via 192.168.0.1 window 28000
ip ro ls
192.168.0.0/24 dev eth0  proto kernel  scope link  src 192.168.0.3
default via 192.168.0.1 dev eth0  window 28000

timestamps and window scaling are off using reno.

Downloading a 5 meg file I advertise the correct window - the whole 
multiple of my mss below 28000 = 27322.

After about 1300 lines of the dump the window creeps back up to 32767.

001959 IP 81.31.115.186.80 > 192.168.0.3.42305: . 1051979:1053417(1438) 
ack 123 win 57520
000011 IP 192.168.0.3.42305 > 81.31.115.186.80: . ack 1053417 win 27322
002030 IP 81.31.115.186.80 > 192.168.0.3.42305: . 1053417:1054855(1438) 
ack 123 win 57520
001962 IP 81.31.115.186.80 > 192.168.0.3.42305: . 1054855:1056293(1438) 
ack 123 win 57520
000011 IP 192.168.0.3.42305 > 81.31.115.186.80: . ack 1056293 win 27322
003020 IP 81.31.115.186.80 > 192.168.0.3.42305: . 1056293:1057731(1438) 
ack 123 win 57520
001968 IP 81.31.115.186.80 > 192.168.0.3.42305: . 1057731:1059169(1438) 
ack 123 win 57520
000012 IP 192.168.0.3.42305 > 81.31.115.186.80: . ack 1059169 win 30198
002024 IP 81.31.115.186.80 > 192.168.0.3.42305: . 1059169:1060607(1438) 
ack 123 win 57520
000012 IP 192.168.0.3.42305 > 81.31.115.186.80: . ack 1060607 win 32767
001980 IP 81.31.115.186.80 > 192.168.0.3.42305: . 1060607:1062045(1438) 
ack 123 win 57520
000011 IP 192.168.0.3.42305 > 81.31.115.186.80: . ack 1062045 win 32767

Andy.

Re: Controlling TCP window size

[John Heffner]

Andy Furniss wrote:
> 
> I've been doing some testing of my new wan connection and noticed that 
> when I specify a window with ip route it still changes after a while.

Looks like this is occurring in net/ipv4/tcp_input.c:tcp_rcv_space_adjust().

The problem really is that the window_clamp variable is overloaded. 
It's used as a kind of cache for rcvbuf -> window conversion, but also 
as a user-settable bound in window size.  On examination, it looks like 
this is also broken (in that window size won't increase) if a user does 
a setsockopt(SO_RCVBUF).  Can we make window_clamp a true clamp?  Then 
we can get rid of the hack of raising it in tcp_rcv_space_adjust().  I 
can cook up a patch if interested.

   -John

Re: Controlling TCP window size

[Andy Furniss]

John Heffner wrote:
> Andy Furniss wrote:
> 
>>
>> I've been doing some testing of my new wan connection and noticed that 
>> when I specify a window with ip route it still changes after a while.
> 
> 
> Looks like this is occurring in 
> net/ipv4/tcp_input.c:tcp_rcv_space_adjust().
> 
> The problem really is that the window_clamp variable is overloaded. It's 
> used as a kind of cache for rcvbuf -> window conversion, but also as a 
> user-settable bound in window size.  On examination, it looks like this 
> is also broken (in that window size won't increase) if a user does a 
> setsockopt(SO_RCVBUF).  Can we make window_clamp a true clamp?  Then we 
> can get rid of the hack of raising it in tcp_rcv_space_adjust().  I can 
> cook up a patch if interested.
> 
>   -John
> 

Thanks - I did get an answer off list (probably accidently?) to the 
effect that I shouldn't rely on ip route to limit window as it's only a 
hint and would make more sense if I wanted it to start really big.

Playing with r_mem etc is the way to go to hard limit.

Andy.

Re: Controlling TCP window size

[John Heffner]

Andy Furniss wrote:
> John Heffner wrote:
>> Andy Furniss wrote:
>>
>>>
>>> I've been doing some testing of my new wan connection and noticed 
>>> that when I specify a window with ip route it still changes after a 
>>> while.
>>
>>
>> Looks like this is occurring in 
>> net/ipv4/tcp_input.c:tcp_rcv_space_adjust().
>>
>> The problem really is that the window_clamp variable is overloaded. 
>> It's used as a kind of cache for rcvbuf -> window conversion, but also 
>> as a user-settable bound in window size.  On examination, it looks 
>> like this is also broken (in that window size won't increase) if a 
>> user does a setsockopt(SO_RCVBUF).  Can we make window_clamp a true 
>> clamp?  Then we can get rid of the hack of raising it in 
>> tcp_rcv_space_adjust().  I can cook up a patch if interested.
>>
>>   -John
>>
> 
> Thanks - I did get an answer off list (probably accidently?) to the 
> effect that I shouldn't rely on ip route to limit window as it's only a 
> hint and would make more sense if I wanted it to start really big.

I guess the other problem is that it's not well defined. :)  As it 
stands now though it's nearly useless if you have tcp_moderate_rcvbuf 
turned on.


> Playing with r_mem etc is the way to go to hard limit.

This will work, but is not available per-route or per-socket.  The 
purpose is really different.

   -John

Re: Controlling TCP window size

[David S. Miller]

From: John Heffner <jheffner@psc.edu>
Date: Tue, 16 May 2006 14:28:23 -0400

> The problem really is that the window_clamp variable is overloaded. 
> It's used as a kind of cache for rcvbuf -> window conversion, but also 
> as a user-settable bound in window size.  On examination, it looks like 
> this is also broken (in that window size won't increase) if a user does 
> a setsockopt(SO_RCVBUF).  Can we make window_clamp a true clamp?  Then 
> we can get rid of the hack of raising it in tcp_rcv_space_adjust().  I 
> can cook up a patch if interested.

The clamp gets moved along also to handle dynamic receive window
sizing.  To be honest, a window clamp doesn't have clear semantics
in the presence of dynamic window sizing.

Anyways, I'd like to see your patch.  It might make rcvbuf -->
window conversions a bit more expensive, but we'll see.