Re: TCP connection hang problem with 2.6.16.16, e1000 (found work-around)

[Stephen Hemminger <shemminger@osdl.org>]

On Wed, 31 May 2006 14:45:13 -0700
Ben Greear <greearb@candelatech.com> wrote:

> Ok, I think I found the problem.  It appears to be at least partially
> self-inflicted.  I was modifying the kernel buffer settings via /proc/*
> to improve performance.  I noticed that when I used kernel-defaults,
> the TCP connection hangs stopped (and ran ~20% slower).  My personal opinion
> is that this is still a bug somewhere in TCP, but at least I have a work-around.

If you set bogus values, you can end up consuming all the memory on your system.

> Since these settings appear somewhat delicate, it might be nice if
> the kernel could warn about configurations that could cause TCP deadlocks
> and such.
> 

There is little or no range checking on most all sysctl values. Whether or
not that is a good thing is debatable. 

> My old settings that exhibitted the problem were:
> 
> my $netdev_max_backlog = 5000;
This only matters for non-NAPI devices.

> my $wmem_max = 4096000;
> my $wmem_default = 128000;
> my $rmem_max = 8096000;
> my $rmem_default = 128000;
> 
> my $tcp_rmem_min     = 4096;
> my $tcp_rmem_default = 256000;
Receiver auto-tuning makes setting this unnecessary
in most cases. 

> my $tcp_rmem_max     = 30000000;
> 
> my $tcp_wmem_min     = 4096;
> my $tcp_wmem_default = 256000;

That's too big with current auto-tuning.

> my $tcp_wmem_max     = 30000000;
> 
> my $tcp_mem_lo       = 20000000;
> my $tcp_mem_pressure = 30000000;
> my $tcp_mem_high     = 60000000;
> 
> 
> The new settings that appear to work around the TCP connection hang,
> and still give decent performance are below.  Documentation being thin,
> I used the suggested values in Documentation/networking/ixgb.txt as
> hints:

Settings don't belong in this Documentation file because it a general
thing not specific to that hardware.


> my $netdev_max_backlog = 30000;
> my $wmem_max = 524287;
> my $wmem_default = 128000;
> my $rmem_max = 524287;
> my $rmem_default = 128000;
> 
> my $tcp_rmem_min     = 4096;
> my $tcp_rmem_default = 256000;
> my $tcp_rmem_max     = 5000000;
> 
> my $tcp_wmem_min     = 4096;
> my $tcp_wmem_default = 256000;
> my $tcp_wmem_max     = 5000000;
> 
> my $tcp_mem_lo       = 10000000;
> my $tcp_mem_pressure = 20000000;
> my $tcp_mem_high     = 30000000;
> 
> Thanks,
> Ben
>