connlimit reached - cannot open connections even after I close some

connlimit reached - cannot open connections even after I close some

[David Gubler]

Hi list,

I want to solve a seemingly simple problem: Limit the number of TCP 
connections coming from a single IP address to a web server on port 
8080. (background: we had some misbehaving clients eating up a lot of 
server resources by opening 400 parallel connections from a single IP - 
that number is unreasonnable even for proxy servers, so I want to 
enforce a limit)

This sounds like a case for connlimit, so this is what I did (max 4 
connections for easier testing):

iptables -A INPUT -p tcp --syn --dport 8080 -m connlimit 
--connlimit-above 4 -j REJECT

Fetching files with wget --limit-rate=1 shows that this works... sort of.

The fifth connection fails as expected, but after I kill some of the 
other connections (verified with netstat -anp | grep ESTABLISHED), I 
*still* cannot open new connections!

To be able to connect again, I have to cease *any* connect attempts for 
about two minutes. If I repeatedly try to connect, I'm not able to 
connect ever again (!), even when there are no more established connections.

As far as I can tell, the problem is the way connlimit works: It looks 
at the conntrack table and considers all entries there, even the 
SYN_WAIT ones (the ones that have been rejected by connlimit end up in 
that state). Or to put it differently: If connlimit denies a connection, 
that connection will *still* create a conntrack entry and thus will also 
count against the connection limit. This can be verified using the 
"conntrack -L" command.

This behaviour makes this simple setup completely unusable for us, 
because if some IP accidentally hits the limit I want that IP to be able 
to connect again as soon as it is back below the limit. Think well 
behaving NATs or proxy servers: They might have a very short burst which 
goes over the limit, so I want to block them for a few seconds, but 
after that things should immediately go back to normal.

After some research I came up with the following:

In addition to the connlimit rule above, I added

iptables -A PREROUTING -t raw -p tcp --dport 8080 -m connlimit 
--connlimit-above 4 -j NOTRACK
iptables -A OUTPUT -t raw -p tcp --sport 8080 -m connlimit 
--connlimit-above 4 -j NOTRACK

This prevents conntrack table entries from being created for connections 
that are over the limit.

But:

* Does this really do what I think it does (i.e. do you think this is 
safe for production)?
* It looks like there is a race condition: Some other connection could 
get closed between the NOTRACK and the REJECT rule. Result: A new 
connection would not get a conntrack entry because it is over the limit 
at first, but later would not be rejected because it is now under the 
limit. Could that be an issue?
* Is this a bug in connlimit?
* Am I missing something else (e.g. an undocumented parameter for 
connlimit)?

Thanks!

David

-- 
David Gubler
Senior Software & Operations Engineer
MeetMe: http://doodle.com/david
E-Mail: dg@doodle.com

Re: connlimit reached - cannot open connections even after I close some

[Pascal Hambourg]

Hello,

David Gubler a écrit :
> 
> iptables -A INPUT -p tcp --syn --dport 8080 -m connlimit 
> --connlimit-above 4 -j REJECT

You should reject with TCP reset instead of (default) ICMP destination
port unreachable.

> The fifth connection fails as expected, but after I kill some of the 
> other connections (verified with netstat -anp | grep ESTABLISHED), I 
> *still* cannot open new connections!
> 
> To be able to connect again, I have to cease *any* connect attempts for 
> about two minutes. If I repeatedly try to connect, I'm not able to 
> connect ever again (!), even when there are no more established connections.
> 
> As far as I can tell, the problem is the way connlimit works: It looks 
> at the conntrack table and considers all entries there, even the 
> SYN_WAIT ones (the ones that have been rejected by connlimit end up in 
> that state). Or to put it differently: If connlimit denies a connection,

The connlimit match just matches packets, it does not deny anything. The
REJECT target does.

> that connection will *still* create a conntrack entry and thus will also 
> count against the connection limit. This can be verified using the 
> "conntrack -L" command.

This is not the expected behaviour. AFAIK, when a packet creating a new
connection is DROPPed or REJECTed, the conntrack entry should be
deleted. This is what I observe on my system. And connlimit does not
even count closed connection still in the TIME_WAIT state.
What is your kernel version ?

Re: connlimit reached - cannot open connections even after I close some

[David Gubler]

Hi Pascal,

Thanks for your response. Here's exactly what I'm doing and what I get.

# uname -a
Linux 3.2.0-36-generic #57-Ubuntu SMP Tue Jan 8 21:44:52 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux (Ubuntu 12.04)

# iptables -A INPUT -p tcp --syn --dport 8080 -m connlimit --connlimit-above 4 -j REJECT --reject-with tcp-reset

# conntrack -L | grep 8080
conntrack v1.0.0 (conntrack-tools): 62 flow entries have been shown.

Starting four downloads with wget --limit-rate=1:

# conntrack -L | grep 8080
tcp      6 431994 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60279 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60279 [ASSURED] mark=0 use=1
tcp      6 431994 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60278 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60278 [ASSURED] mark=0 use=1
tcp      6 431995 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60284 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60284 [ASSURED] mark=0 use=1
tcp      6 431999 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60285 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60285 [ASSURED] mark=0 use=1

Starting a fifth download fails:

$ wget --limit-rate=1  http://localhost:8080/....
--2013-02-04 11:48:41--  http://localhost:8080/...
Auflösen des Hostnamen »localhost (localhost)«... 127.0.0.1
Verbindungsaufbau zu localhost (localhost)|127.0.0.1|:8080... fehlgeschlagen: Verbindungsaufbau abgelehnt.
(connection rejected)

# conntrack -L | grep 8080
tcp      6 431949 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60279 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60279 [ASSURED] mark=0 use=1
tcp      6 431990 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60278 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60278 [ASSURED] mark=0 use=1
tcp      6 431956 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60284 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60284 [ASSURED] mark=0 use=1
tcp      6 431963 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60285 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60285 [ASSURED] mark=0 use=1
tcp      6 71 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60291 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60291 mark=0 use=1

Note: The rejected connection is in the conntrack table in the state SYN_SENT.

Killing one of the four running downloads with CTRL+C:

# conntrack -L | grep 8080
tcp      6 431950 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60279 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60279 [ASSURED] mark=0 use=1
tcp      6 7 CLOSE src=127.0.0.1 dst=127.0.0.1 sport=60278 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60278 [ASSURED] mark=0 use=1
tcp      6 431958 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60284 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60284 [ASSURED] mark=0 use=1
tcp      6 431965 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60285 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60285 [ASSURED] mark=0 use=1
tcp      6 17 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60291 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60291 mark=0 use=1

Now we have a connection in state CLOSED and one in state SYN_SENT in the table. wget still fails as it did before.

After making a few connect attempts with wget (all failing):

# conntrack -L | grep 8080
tcp      6 431946 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60279 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60279 [ASSURED] mark=0 use=1
tcp      6 87 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60294 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60294 mark=0 use=1
tcp      6 431954 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60284 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60284 [ASSURED] mark=0 use=1
tcp      6 431961 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60285 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60285 [ASSURED] mark=0 use=1
tcp      6 117 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60298 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60298 mark=0 use=2
tcp      6 14 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60292 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60292 mark=0 use=1
tcp      6 117 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60299 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60299 mark=0 use=1
tcp      6 116 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60295 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60295 mark=0 use=1
tcp      6 116 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60296 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60296 mark=0 use=1
tcp      6 117 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60297 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60297 mark=0 use=1
tcp      6 86 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60293 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60293 mark=0 use=2

... waiting for about two minutes ...

# conntrack -L | grep 8080
conntrack v1.0.0 (conntrack-tools): 57 flow entries have been shown.
tcp      6 431983 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60279 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60279 [ASSURED] mark=0 use=1
tcp      6 431991 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60284 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60284 [ASSURED] mark=0 use=1
tcp      6 431998 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60285 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60285 [ASSURED] mark=0 use=1

... and now I can start another wget download.


On 03.02.2013 12:51, Pascal Hambourg wrote:
 >
 > This is not the expected behaviour. AFAIK, when a packet creating a new
 > connection is DROPPed or REJECTed, the conntrack entry should be
 > deleted. This is what I observe on my system.

Ok this is weird. I have made a similar (although not exactly the same) attempt on "Linux 2.6.32-5-amd64 #1 SMP Sun Sep 23 10:07:46 UTC 2012 x86_64 GNU/Linux" (Debian stable with stock kernel). On that kernel it behaves as you describe it! No 
SYN_SENT entries pop up in the conntrack table, instead the denied connections directly go into TIME_WAIT, and the connection limit works fine.

On "Linux 3.2.0-2-amd64 #1 SMP Mon Jun 11 17:24:18 UTC 2012 x86_64 GNU/Linux" (Debian stable with backports kernel), on the other hand, I get the exact same behavior as with 3.2 in Ubuntu (broken). Which makes me guess that this is not caused by some 
weird Ubuntu specific default setting, but rather a general problem in Linux 3.2.

Bug...?

Thanks,

David

-- 
David Gubler
Senior Software & Operations Engineer
MeetMe: http://doodle.com/david
E-Mail: dg@doodle.com

Re: connlimit reached - cannot open connections even after I close some

[Pascal Hambourg]

David Gubler a écrit :
> 
> Starting a fifth download fails:
> 
> $ wget --limit-rate=1  http://localhost:8080/....
> --2013-02-04 11:48:41--  http://localhost:8080/...
> Auflösen des Hostnamen »localhost (localhost)«... 127.0.0.1
> Verbindungsaufbau zu localhost (localhost)|127.0.0.1|:8080... fehlgeschlagen: Verbindungsaufbau abgelehnt.
> (connection rejected)
> 
> # conntrack -L | grep 8080
> tcp      6 431949 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60279 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60279 [ASSURED] mark=0 use=1
> tcp      6 431990 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60278 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60278 [ASSURED] mark=0 use=1
> tcp      6 431956 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60284 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60284 [ASSURED] mark=0 use=1
> tcp      6 431963 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60285 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60285 [ASSURED] mark=0 use=1
> tcp      6 71 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60291 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60291 mark=0 use=1
> 
> Note: The rejected connection is in the conntrack table in the state SYN_SENT.

It appears that you're testing over the loopback interface. Note that
the connection tracking behaves a bit differently on this interface :
the connection is confirmed when the first packet reaches the end of
POSTROUTING chains, before entering back the PREROUTING and INPUT
chains. So even though the packet is dropped in the INPUT chain, the
conntrack entry is not new any more and left in the SYN_STATE.

> Killing one of the four running downloads with CTRL+C:
> 
> # conntrack -L | grep 8080
> tcp      6 431950 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60279 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60279 [ASSURED] mark=0 use=1
> tcp      6 7 CLOSE src=127.0.0.1 dst=127.0.0.1 sport=60278 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60278 [ASSURED] mark=0 use=1
> tcp      6 431958 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60284 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60284 [ASSURED] mark=0 use=1
> tcp      6 431965 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=60285 dport=8080 src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60285 [ASSURED] mark=0 use=1
> tcp      6 17 SYN_SENT src=127.0.0.1 dst=127.0.0.1 sport=60291 dport=8080 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=8080 dport=60291 mark=0 use=1
> 
> Now we have a connection in state CLOSED and one in state SYN_SENT in
> the table. wget still fails as it did before.

As expected, as there are already four active (SYN_SENT and ESTABLISHED)
connections.

> On 03.02.2013 12:51, Pascal Hambourg wrote:
>  >
>  > This is not the expected behaviour. AFAIK, when a packet creating a new
>  > connection is DROPPed or REJECTed, the conntrack entry should be
>  > deleted. This is what I observe on my system.

Edit : I observe this if the connections are from a remote host. If the
connections are from the local host over loopback, I observer the same
behaviour as you. I tested both conditions with a Debian Squeeze 2.6.32
kernel. I could not test with other kerne versions.

> Ok this is weird. I have made a similar (although not exactly the
> same) attempt on "Linux 2.6.32-5-amd64 #1 SMP Sun Sep 23 10:07:46 UTC
> 2012 x86_64 GNU/Linux" (Debian stable with stock kernel). On that kernel
> it behaves as you describe it! No SYN_SENT entries pop up in the
> conntrack table, instead the denied connections directly go into
> TIME_WAIT, and the connection limit works fine.
> 
> On "Linux 3.2.0-2-amd64 #1 SMP Mon Jun 11 17:24:18 UTC 2012 x86_64
> GNU/Linux" (Debian stable with backports kernel), on the other hand, I
> get the exact same behavior as with 3.2 in Ubuntu (broken).

Did you test all kernels in the exact same conditions, connecting either
 from a remote host or over loopback ?

Re: connlimit reached - cannot open connections even after I close some

[David Gubler]

Hi Pascal,

Sorry for the delay, I only just got around to test it again.

Indeed my testing methodology was flawed. Whenever I test over loopback 
the weird behaviour shows, but when I test over the network it does 
exactly what it's supposed to, just as you said.

So works as designed, probably, and that's exactly what we need. Thanks 
a lot!

Best regards,

David

Re: connlimit reached - cannot open connections even after I close some

[David Gubler]

Hi Bryan,

 > I would think you could approach the problem by using apache's builtin
 > rate limiting function,

Yes we're using Apache and I researched that as well, but:

First, I'm thinking that an Apache module cannot reliably enforce a 
connection limit - after all, the module can only act *after* the 
connection has been established.

Second, I have not found an apache module that is included in Debian 
(required for automatic security updates) and is able to do that.
* mod_limitipconn is not available for Debian (it seems)
* mod_bw cannot limit connections per IP, only per scope
* mod_evasive counts hits on an object, not parallel connections
However, if you can point me to one I would give it a try anyway.

And after all, it really bugs me that apparently connlimit is supposed 
to do what I want, but shows this erratic behavior...

David

-- 
David Gubler
Senior Software & Operations Engineer
MeetMe: http://doodle.com/david
E-Mail: dg@doodle.com

Re: connlimit reached - cannot open connections even after I close some

[David Gubler]

Hi Nehal,

> I am not expert in this but mod_qos is definitely worth a try.

I have just tried 10.8 (from Debian testing) and 10.13 (from unstable) 
by back-porting them to Debian stable. They seem to work basically, but 
IPv6 support is broken, at least for the directive QS_SrvMaxConnPerIP 
(all IPv6 connections are treated as if they were coming from 
255.255.255.255, thus IPv6 connections wrongly hit the limit all the 
time). Thus not usable :(

david

-- 
David Gubler
Senior Software & Operations Engineer
MeetMe: http://doodle.com/david
E-Mail: dg@doodle.com