Large numbers of TCP resets

Large numbers of TCP resets

[gphat]

Recently, the web-app at the company I work for started having problems load 
balancing.  This was traced back to a large number of tcp-resets being sent 
from the web servers to the clients.

Here's netstat output for one of the boxen:
    437761 active connections openings
    0 passive connection openings
    21 failed connection attempts
    0 connection resets received
    101 connections established
    140497907 segments received
    149967684 segments send out
    116610 segments retransmited
    111 bad segments received.
    1003306 resets sent

This box is running an apache/tomcat/jboss setup serving both HTTP and RMI 
requests.  The reset packets disrupt the load balancing method, as the load 
balancers think the reset means the connection can now be round-robined to the 
next machine.  This can be masked by using cookie-based balancing, but I am 
worried that we would be masking a problem.

Documentation concerning tuning buffers for network purposes say that 2.4 is 
good at auto-tuning, and that only /proc/sys/net/ipv4/tcp_wmem|tcp_rmem might 
need tweaking.

The kernel is 2.4.9-31smp from RedHat 7.2.  The cards are Intel 82557's.  There 
are two interfaces, with 1 for client access and 1 used for communication with 
the backend database.  (The ->client is the one we are worried about).

Any ideas on why the interfaces would be doing this?  The boxes served 1.6 
hits/sec average over last month for a total of about 10Gb.  Is there some 
tuning that needs to take place?

Thanks in advance.

Cory 'G'
Watson

Re: Large numbers of TCP resets

[Andi Kleen]

gphat@cafes.net writes:

>     111 bad segments received.
>     1003306 resets sent

Linux 2.4 sends a reset when the application closes the socket before all
already arrived  data is read. That could be a likely cause.

-Andi

Re: Large numbers of TCP resets

[Willy Tarreau]

On Wed, Jul 03, 2002 at 08:15:53PM +0000, gphat@cafes.net wrote:
> Recently, the web-app at the company I work for started having problems load 
> balancing.  This was traced back to a large number of tcp-resets being sent 
> from the web servers to the clients.

aren't your load balancers Alteon ACE Directors ? If this is the case, I
suspect you use "fastage 0", which ends sessions prematurely. This
particularly happens in case of direct access mode (DAM) because the
switch needs to remap source ports, and when the session expires, it
simply routes packets from server to client without DNATing them.

The client then receives ACKs and/or FINs for closed ports, or for
open ports, but with bad sequence numbers, and then sends RESETS,
which the server doesn't understand. I observed real RST storms
during tens of minutes because of this. They disappeared when I
set "fastage" to something higher than 4 (=keep the session at least
16 seconds, even in final states).

I think this is not specific to Linux 2.4 since I observed a similar
behaviour with Solaris 8.

Cheers,
Willy