Re: connlimit reached - cannot open connections even after I close some

[David Gubler <dg@doodle.com>]

Hi Pascal,

Thanks for your response. Here's exactly what I'm doing and what I get.

# uname -a
Linux 3.2.0-36-generic #57-Ubuntu SMP Tue Jan 8 21:44:52 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux (Ubuntu 12.04)

# iptables -A INPUT -p tcp --syn --dport 8080 -m connlimit --connlimit-above 4 -j REJECT --reject-with tcp-reset

# conntrack -L | grep 8080
conntrack v1.0.0 (conntrack-tools): 62 flow entries have been shown.

Starting four downloads with wget --limit-rate=1:

# conntrack -L | grep 8080
tcp      6 431994 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60279 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60279 [ASSURED] mark=0 use=1
tcp      6 431994 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60278 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60278 [ASSURED] mark=0 use=1
tcp      6 431995 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60284 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60284 [ASSURED] mark=0 use=1
tcp      6 431999 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60285 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60285 [ASSURED] mark=0 use=1

Starting a fifth download fails:

$ wget --limit-rate=1  http://localhost:8080/....
--2013-02-04 11:48:41--  http://localhost:8080/...
Auflösen des Hostnamen »localhost (localhost)«... 127.0.0.1
Verbindungsaufbau zu localhost (localhost)|127.0.0.1|:8080... fehlgeschlagen: Verbindungsaufbau abgelehnt.
(connection rejected)

# conntrack -L | grep 8080
tcp      6 431949 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60279 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60279 [ASSURED] mark=0 use=1
tcp      6 431990 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60278 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60278 [ASSURED] mark=0 use=1
tcp      6 431956 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60284 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60284 [ASSURED] mark=0 use=1
tcp      6 431963 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60285 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60285 [ASSURED] mark=0 use=1
tcp      6 71 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60291 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60291 mark=0 use=1

Note: The rejected connection is in the conntrack table in the state SYN_SENT.

Killing one of the four running downloads with CTRL+C:

# conntrack -L | grep 8080
tcp      6 431950 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60279 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60279 [ASSURED] mark=0 use=1
tcp      6 7 CLOSE src=127.0.0.1 dst=127.0.0.1 sport=60278 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60278 [ASSURED] mark=0 use=1
tcp      6 431958 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60284 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60284 [ASSURED] mark=0 use=1
tcp      6 431965 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60285 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60285 [ASSURED] mark=0 use=1
tcp      6 17 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60291 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60291 mark=0 use=1

Now we have a connection in state CLOSED and one in state SYN_SENT in the table. wget still fails as it did before.

After making a few connect attempts with wget (all failing):

# conntrack -L | grep 8080
tcp      6 431946 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60279 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60279 [ASSURED] mark=0 use=1
tcp      6 87 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60294 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60294 mark=0 use=1
tcp      6 431954 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60284 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60284 [ASSURED] mark=0 use=1
tcp      6 431961 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60285 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60285 [ASSURED] mark=0 use=1
tcp      6 117 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60298 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60298 mark=0 use=2
tcp      6 14 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60292 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60292 mark=0 use=1
tcp      6 117 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60299 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60299 mark=0 use=1
tcp      6 116 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60295 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60295 mark=0 use=1
tcp      6 116 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60296 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60296 mark=0 use=1
tcp      6 117 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60297 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60297 mark=0 use=1
tcp      6 86 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60293 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60293 mark=0 use=2

... waiting for about two minutes ...

# conntrack -L | grep 8080
conntrack v1.0.0 (conntrack-tools): 57 flow entries have been shown.
tcp      6 431983 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60279 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60279 [ASSURED] mark=0 use=1
tcp      6 431991 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60284 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60284 [ASSURED] mark=0 use=1
tcp      6 431998 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60285 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60285 [ASSURED] mark=0 use=1

... and now I can start another wget download.


On 03.02.2013 12:51, Pascal Hambourg wrote:
 >
 > This is not the expected behaviour. AFAIK, when a packet creating a new
 > connection is DROPPed or REJECTed, the conntrack entry should be
 > deleted. This is what I observe on my system.

Ok this is weird. I have made a similar (although not exactly the same) attempt on "Linux 2.6.32-5-amd64 #1 SMP Sun Sep 23 10:07:46 UTC 2012 x86_64 GNU/Linux" (Debian stable with stock kernel). On that kernel it behaves as you describe it! No 
SYN_SENT entries pop up in the conntrack table, instead the denied connections directly go into TIME_WAIT, and the connection limit works fine.

On "Linux 3.2.0-2-amd64 #1 SMP Mon Jun 11 17:24:18 UTC 2012 x86_64 GNU/Linux" (Debian stable with backports kernel), on the other hand, I get the exact same behavior as with 3.2 in Ubuntu (broken). Which makes me guess that this is not caused by some 
weird Ubuntu specific default setting, but rather a general problem in Linux 3.2.

Bug...?

Thanks,

David

-- 
David Gubler
Senior Software & Operations Engineer
MeetMe: http://doodle.com/david
E-Mail: dg@doodle.com