connlimit reached - cannot open connections even after I close some

[David Gubler <dg@doodle.com>]

Hi list,

I want to solve a seemingly simple problem: Limit the number of TCP 
connections coming from a single IP address to a web server on port 
8080. (background: we had some misbehaving clients eating up a lot of 
server resources by opening 400 parallel connections from a single IP - 
that number is unreasonnable even for proxy servers, so I want to 
enforce a limit)

This sounds like a case for connlimit, so this is what I did (max 4 
connections for easier testing):

iptables -A INPUT -p tcp --syn --dport 8080 -m connlimit 
--connlimit-above 4 -j REJECT

Fetching files with wget --limit-rate=1 shows that this works... sort of.

The fifth connection fails as expected, but after I kill some of the 
other connections (verified with netstat -anp | grep ESTABLISHED), I 
*still* cannot open new connections!

To be able to connect again, I have to cease *any* connect attempts for 
about two minutes. If I repeatedly try to connect, I'm not able to 
connect ever again (!), even when there are no more established connections.

As far as I can tell, the problem is the way connlimit works: It looks 
at the conntrack table and considers all entries there, even the 
SYN_WAIT ones (the ones that have been rejected by connlimit end up in 
that state). Or to put it differently: If connlimit denies a connection, 
that connection will *still* create a conntrack entry and thus will also 
count against the connection limit. This can be verified using the 
"conntrack -L" command.

This behaviour makes this simple setup completely unusable for us, 
because if some IP accidentally hits the limit I want that IP to be able 
to connect again as soon as it is back below the limit. Think well 
behaving NATs or proxy servers: They might have a very short burst which 
goes over the limit, so I want to block them for a few seconds, but 
after that things should immediately go back to normal.

After some research I came up with the following:

In addition to the connlimit rule above, I added

iptables -A PREROUTING -t raw -p tcp --dport 8080 -m connlimit 
--connlimit-above 4 -j NOTRACK
iptables -A OUTPUT -t raw -p tcp --sport 8080 -m connlimit 
--connlimit-above 4 -j NOTRACK

This prevents conntrack table entries from being created for connections 
that are over the limit.

But:

* Does this really do what I think it does (i.e. do you think this is 
safe for production)?
* It looks like there is a race condition: Some other connection could 
get closed between the NOTRACK and the REJECT rule. Result: A new 
connection would not get a conntrack entry because it is over the limit 
at first, but later would not be rejected because it is now under the 
limit. Could that be an issue?
* Is this a bug in connlimit?
* Am I missing something else (e.g. an undocumented parameter for 
connlimit)?

Thanks!

David

-- 
David Gubler
Senior Software & Operations Engineer
MeetMe: http://doodle.com/david
E-Mail: dg@doodle.com