[PATCH 2.6.5] Re: Fw: Stack sends SYN/ACKs even though accept queue is full

[PATCH 2.6.5] Re: Fw: Stack sends SYN/ACKs even though accept queue is full

[Nivedita Singhvi]

Andrew Morton wrote:

>Begin forwarded message:
>
>Date: Thu, 29 Apr 2004 14:53:36 -0700
>From: Jan Olderdissen <jan@ixiacom.com>
>To: "'linux-kernel@vger.kernel.org'" <linux-kernel@vger.kernel.org>
>Subject: Stack sends SYN/ACKs even though accept queue is full
>
Attaching a patch which adds a sysctl to turn off this
behaviour.   Could you test this, please?  Patch against
2.6.5 vanilla kernel.  If you need a 2.4 version, let me
know.

>Because newly accepted connections are considered 'young', two such
>connections put on the synq will cause additional SYNs to be dropped until
>young connections age and additional connections are SYN/ACKed , etc. Since
>the initial TCP timeout is three seconds, you would expect two additional
>connections to be accepted every three seconds. However, experiments with
>2.4.25 show that number to be two connections every four seconds for unclear
>reasons.
>
Normally, I think the expected behaviour was that connections
would be short-lived. This is a reasonable expectation for most
web-servers etc.   In which case, the accept queue would free
up frequently, and having the syn request right there would save
a full timeout and round trip over the Internet. i.e. useful in the
common case.

I don't think it is worthwhile for environments where connections
are long-lived and turnover is infrequent, added to a heavily
congested network.   I'd prefer to have this be a tunable option.

thanks,
Nivedita

diff -urN linux-2.6.5/include/linux/sysctl.h
linux-2.6.5synq/include/linux/sysctl.h
--- linux-2.6.5/include/linux/sysctl.h	2004-04-03 19:37:23.000000000
-0800
+++ linux-2.6.5synq/include/linux/sysctl.h	2004-04-30 15:24:12.000000000
-0700
@@ -323,6 +323,7 @@
 	NET_IPV4_IPFRAG_SECRET_INTERVAL=94,
 	NET_TCP_WESTWOOD=95,
 	NET_IPV4_IGMP_MAX_MSF=96,
+	NET_TCP_PRELOAD_SYNQ=97,
 };
 
 enum {
diff -urN linux-2.6.5/include/net/tcp.h
linux-2.6.5synq/include/net/tcp.h
--- linux-2.6.5/include/net/tcp.h	2004-04-03 19:36:18.000000000 -0800
+++ linux-2.6.5synq/include/net/tcp.h	2004-04-30 13:55:18.000000000
-0700
@@ -583,6 +583,7 @@
 extern int sysctl_tcp_frto;
 extern int sysctl_tcp_low_latency;
 extern int sysctl_tcp_westwood;
+extern int sysctl_tcp_preload_synq;
 
 extern atomic_t tcp_memory_allocated;
 extern atomic_t tcp_sockets_allocated;
diff -urN linux-2.6.5/net/ipv4/sysctl_net_ipv4.c
linux-2.6.5synq/net/ipv4/sysctl_net_ipv4.c
--- linux-2.6.5/net/ipv4/sysctl_net_ipv4.c	2004-04-03 19:37:37.000000000
-0800
+++ linux-2.6.5synq/net/ipv4/sysctl_net_ipv4.c	2004-04-30
16:16:57.000000000 -0700
@@ -601,6 +601,14 @@
 		.mode		= 0644,
 		.proc_handler	= &proc_dointvec,
 	},
+	{
+		.ctl_name	= NET_TCP_PRELOAD_SYNQ, 
+		.procname	= "tcp_preload_synq",
+		.data		= &sysctl_tcp_preload_synq,
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.proc_handler	= &proc_dointvec,
+	},
 	{ .ctl_name = 0 }
 };
 
diff -urN linux-2.6.5/net/ipv4/tcp_ipv4.c
linux-2.6.5synq/net/ipv4/tcp_ipv4.c
--- linux-2.6.5/net/ipv4/tcp_ipv4.c	2004-04-03 19:36:55.000000000 -0800
+++ linux-2.6.5synq/net/ipv4/tcp_ipv4.c	2004-04-30 15:51:38.000000000
-0700
@@ -78,6 +78,7 @@
 extern int sysctl_ip_dynaddr;
 int sysctl_tcp_tw_reuse;
 int sysctl_tcp_low_latency;
+int sysctl_tcp_preload_synq = 1;
 
 /* Check TCP sequence numbers in ICMP packets. */
 #define ICMP_MIN_LENGTH 8
@@ -1442,8 +1443,11 @@
 	 * clogging syn queue with openreqs with exponentially increasing
 	 * timeout.
 	 */
-	if (tcp_acceptq_is_full(sk) && tcp_synq_young(sk) > 1)
-		goto drop;
+	if (tcp_acceptq_is_full(sk)) {
+		if (!sysctl_tcp_preload_synq ||
+		   (sysctl_tcp_preload_synq && (tcp_synq_young(sk) > 1)))
+			goto drop;
+	}
 
 	req = tcp_openreq_alloc();
 	if (!req)
@@ -2683,4 +2687,5 @@
 EXPORT_SYMBOL(sysctl_local_port_range);
 EXPORT_SYMBOL(sysctl_max_syn_backlog);
 EXPORT_SYMBOL(sysctl_tcp_low_latency);
+EXPORT_SYMBOL(sysctl_tcp_preload_synq);
 #endif

RE: [PATCH 2.6.5] Re: Fw: Stack sends SYN/ACKs even though accept queue is full

[Jan Olderdissen]

Nivedita Singhvi wrote:

> Attaching a patch which adds a sysctl to turn off this
> behaviour.   Could you test this, please?  Patch against
> 2.6.5 vanilla kernel.  If you need a 2.4 version, let me
> know.

We'll be happy to test it. We would prefer a 2.4 kernel patch though. Ideal
would be 2.4.21 vanilla which we use in our embedded systems, but we can
make other targets work as well.

> Normally, I think the expected behaviour was that connections
> would be short-lived. This is a reasonable expectation for most
> web-servers etc.   In which case, the accept queue would free
> up frequently, and having the syn request right there would save
> a full timeout and round trip over the Internet. i.e. useful in the
> common case.

I have to disagree. 

As the code is implemented right now, you get at most two pending young
connections in the syn queue in addition to the ones in the accept queue -
all others have already had their full timeout. I will argue that in high
load conditions it doesn't really matter whether you have one or two warm
connections sitting around or not because other new connections are going to
come by anyway sooner than you want to have them. If the server is able to
keep up, the accept queue is going to be mostly empty anyway.

The idea of having warm connections is a good one - that is what the accept
queue is for. As long as the accept queue is not full, the syn queue will
have warm bodies for you. If the user wanted to achieve a very similar
effect, he could simply increase the accept queue size by 2.

In the end I don't think the minimal benefit warrants complicating the code
and/or adding an option that few would understand or care to know about.
Also, the concept of suggesting to the client that there is a receiver and
then just dropping all the packets it sends seems a bit rude if not outright
non-compliant.

Your call, of course.

Jan

Re: [PATCH 2.6.5] Re: Fw: Stack sends SYN/ACKs even though accept queue is full

[David S. Miller]

On Fri, 30 Apr 2004 17:09:59 -0700
Nivedita Singhvi <niv@us.ibm.com> wrote:

> Normally, I think the expected behaviour was that connections
> would be short-lived. This is a reasonable expectation for most
> web-servers etc.

I don't know how relevant the "lifespan" of a connection is in
this case.

Servers tend to sip from the accept queue in a seperate thread
or approximate such behavior.  Therefore we have two time-frames,
the incoming connection acceptq one and the one that actually
deals with the established state data transation.  You're talking
about the latter, and the behavior in question is about the
former.

I really haven't seen a convincing argument yet to change the behavior
here, I think it's sane but I'm ready to be convinced otherwise :-)

RE: [PATCH 2.6.5] Re: Fw: Stack sends SYN/ACKs even though accept queue is full

[Jan Olderdissen]

Nivedita,

> Attaching a patch which adds a sysctl to turn off this
> behaviour.   Could you test this, please?  Patch against
> 2.6.5 vanilla kernel.  If you need a 2.4 version, let me
> know.

Thomas tested your patch against 2.6.5 with success. The patch didn't apply
cleanly against the sources he was using, but after some twiddling it did
the job. Internally, we'll use the 2.4.21 patch, of course. Thanks!

Jan

RE: [PATCH 2.6.5] Re: Fw: Stack sends SYN/ACKs even though accept queue is full

[Jan Olderdissen]

David,

> I really haven't seen a convincing argument yet to change the behavior
> here, I think it's sane but I'm ready to be convinced otherwise :-)

Upon further reflection, I've convinced myself that the current algorithm is
more broken than I thought - at least in some situations.

Let's review the status quo in tcp_v4_conn_request(): Incoming SYNs are
dropped if

1. The synq is full, or
2. The accept queue is full AND the number of "young" entries in the synq is
greater than 1.

If the SYN is not dropped, a synq entry is allocated, marked "young", and a
SYN/ACK is sent. Entries one the synq "age" on their first timeout.

In previous posts in this thread I suggested removing the condition dealing
with "young" entries because this allows the synq to grow even though there
is no room in the accept queue. The symptom this causes is a sequence like
this:

C -> S SYN
S -> C SYN/ACK
C -> S ACK # if the acceptq is still full at this point, this ack is dropped
C -> S data1 # this is also dropped
C -> S data2 # this is also dropped
C -> S data3 # this is also dropped
S -> C SYN/ACK
C -> S ACK # if the acceptq is still full at this point, this ack is dropped
C -> S data1 # this is also dropped
C -> S data2 # this is also dropped
C -> S data3 # this is also dropped

and so forth until timeout. The client thinks the connection is up and
bandwidth is wasted.

Nivedita was very helpful in providing a patch for this. It works inasmuch
it fixes the problem for the case where the acceptq is full.

Sadly, I missed the second condition under which SYNs can go on the synq
without room on the acceptq being available. If the acceptq is almost full
and the server receives a burst of SYNs, the server will accept and SYN/ACK
_all_ SYNs that fit in the synq. I've tested this condition using an Ixia
400T chassis with an ALM1000T8 load module. Using three of the eight test
ports (each has a PowerPC 750 processor and gigE interface) to hammer on a
fourth test port configured as a server, the synq quickly outgrew the
acceptq in length. Obviously, most of the entries on the synq were in the
unhappy mode described above as there was no way the server could keep up
with the clients.

Ultimately, I propose that the concept of SYN/ACKing a connection that we
don't have room for in the acceptq is evil and should be replaced by a
mechanism that remembers SYNs but doesn't SYN/ACK right away if there is no
room on the acceptq. This provides the benefit of keeping the packet traces
sane as well as keeping around warm connections to be SYN/ACKed as soon as
the acceptq starts emptying.

I therefore suggest the following algorithm:

Three queues are used: 

synq1: holds connections that have sent a SYN but have not been SYN/ACKed
synq2: holds connections that have been SYN/ACKed
acceptq: holds connections that have completed 3-way handshake

When a SYN comes in, the following pseudo code is executed:

if (len(synq1)+len(synq2) > MAX_SYNQ) goto drop;
if (len(synq2)+len(acceptq) < MAX_ACCEPTQ) {
    send_synack();
    synq2.add (new_connection);
} else {
    synq1.add (new_connection);
}

When an ACK is received, the following code is executed:

synq2.remove (connection);
acceptq.add (connection);

When an entry has been removed from the acceptq by the app, the following
code is executed:

int max_synack = MAX_ACCEPTQ - len(acceptq) - len(synq2);
while (max_synack-- && len(synq1)) {
    # move connection from synq1 to synq2 and send SYN/ACK
    send_synack();
    connection = synq1.pop();
    synq2.add (connection);
}

Notes:

- MAX_ACCEPTQ is shorthand for the socket specific accept queue length.

Jan