Conntrack & Unreplied exhausts hashsize

Conntrack & Unreplied exhausts hashsize

[Julien Vehent]

Hi everyone,

I'm analyzing a configuration problem that we are encountering with 
conntrack at work. We have a farm of frontend servers that run apache. Those 
servers run into the classical table full problem:

     Jun 5 09:57:51 web-front1 kernel: [7177214.445925] nf_conntrack: table 
full, dropping packet.

So I started tuning the kernel of one member of the farm. This server has 2 
interfaces: one public and one in the LAN. The problem is on the public 
interface, it seems that connections in the UNREPLIED state continue to grow 
and never get cleaned up by conntrack. Below is a diagram that shows the 
issue:

http://4u.1nw.eu/conntrack_stat3.png

The orange line counts connections on the public IP that are in the 
unreplied state. The script parses /etc/net/ip_conntrack every 10 minutes 
(nothing fancy, see https://gist.github.com/2901349 ).

My questions are:

Should these UNREPLIED connection get removed from conntrack after a certain 
timeout?

What is the parameter that controls this timeout ?
I'm afraid it might be `net.netfilter.nf_conntrack_tcp_timeout_established = 
432000`, which is 5 days. If this is the case, would it be safe to set this 
parameter to 300 seconds instead (5 minutes) ?

Note: apache runs with `KeepAlive On` and `KeepAliveTimeout 3`, in case this 
might be relevant.


Thanks a lot,
Julien


-- 
Julien Vehent - http://1nw.eu/!j

Re: Conntrack & Unreplied exhausts hashsize

[Marco Padovan]

Looks like it's an intendend behaviour:

http://www.netfilter.org/documentation/FAQ/netfilter-faq-3.html#ss3.17

<<UNREPLIED entries are temporary entries, i.e. as soon as we run out of
connection tracking entries (we reach
/proc/sys/net/ipv4/ip_conntrack_max), we delete old UNREPLIED entries.
In other words: instead of having empty conntrack entries, we'd rather
keep some maybe useful information in them until we really need them.>>

when having the conntrack table full "error" did you still have
unreplied entries in the table?

Il 09/06/2012 17:12, Julien Vehent ha scritto:
> Hi everyone,
>
> I'm analyzing a configuration problem that we are encountering with
> conntrack at work. We have a farm of frontend servers that run apache.
> Those servers run into the classical table full problem:
>
>     Jun 5 09:57:51 web-front1 kernel: [7177214.445925] nf_conntrack:
> table full, dropping packet.
>
> So I started tuning the kernel of one member of the farm. This server
> has 2 interfaces: one public and one in the LAN. The problem is on the
> public interface, it seems that connections in the UNREPLIED state
> continue to grow and never get cleaned up by conntrack. Below is a
> diagram that shows the issue:
>
> http://4u.1nw.eu/conntrack_stat3.png
>
> The orange line counts connections on the public IP that are in the
> unreplied state. The script parses /etc/net/ip_conntrack every 10
> minutes (nothing fancy, see https://gist.github.com/2901349 ).
>
> My questions are:
>
> Should these UNREPLIED connection get removed from conntrack after a
> certain timeout?
>
> What is the parameter that controls this timeout ?
> I'm afraid it might be
> `net.netfilter.nf_conntrack_tcp_timeout_established = 432000`, which
> is 5 days. If this is the case, would it be safe to set this parameter
> to 300 seconds instead (5 minutes) ?
>
> Note: apache runs with `KeepAlive On` and `KeepAliveTimeout 3`, in
> case this might be relevant.
>
>
> Thanks a lot,
> Julien
>
>

Fwd: Re: Conntrack & Unreplied exhausts hashsize

[Julien Vehent]

Very interesting find ! So, what that means is that the 5 days timeout will 
not
hurt other connections, because unreplied connections will get removed
automatically if new connections need the spot ?

That would definitely explain why the total of unreplied connection went 
down a
little bit on June 8th at 4am: http://4u.1nw.eu/conntrack_stat3.png

We did fix other parameters earlier, like enabling tcp_tw_reuse and 
directing
localhost traffic to NOTRACK. It would explain why we didn't get more of 
those
`table full, dropping packet`.

- Julien

On 2012-06-09 14:14, Marco Padovan wrote:

> Looks like it's an intendend behaviour:
>
> http://www.netfilter.org/documentation/FAQ/netfilter-faq-3.html#ss3.17 [1]
>
> <
> Jun 5 09:57:51 web-front1 kernel: [7177214.445925] nf_conntrack: table 
> full,
> dropping packet.
>
> So I started tuning the kernel of one member of the farm. This server has 
> 2
> interfaces: one public and one in the LAN. The problem is on the public
> interface, it seems that connections in the UNREPLIED state continue to 
> grow
> and never get cleaned up by conntrack. Below is a diagram that shows the
> issue:
>
> http://4u.1nw.eu/conntrack_stat3.png [2]
>
> The orange line counts connections on the public IP that are in the
> unreplied state. The script parses /etc/net/ip_conntrack every 10 minutes
> (nothing fancy, see https://gist.github.com/2901349 [3] ).
>
> My questions are:
>
> Should these UNREPLIED connection get removed from conntrack after a 
> certain
> timeout?
>
> What is the parameter that controls this timeout ?
> I'm afraid it might be `net.netfilter.nf_conntrack_tcp_timeout_established 
> =
> 432000`, which is 5 days. If this is the case, would it be safe to set 
> this
> parameter to 300 seconds instead (5 minutes) ?
>
> Note: apache runs with `KeepAlive On` and `KeepAliveTimeout 3`, in case 
> this
> might be relevant.
>
> Thanks a lot,
> Julien



Links:
------
[1] http://www.netfilter.org/documentation/FAQ/netfilter-faq-3.html#ss3.17
[2] http://4u.1nw.eu/conntrack_stat3.png
[3] https://gist.github.com/2901349

Re: Conntrack & Unreplied exhausts hashsize

[Julien Vehent]

On 2012-06-09 16:15, Julien Vehent wrote:
> Very interesting find ! So, what that means is that the 5 days timeout 
> will
> not
> hurt other connections, because unreplied connections will get removed
> automatically if new connections need the spot ?
>
> That would definitely explain why the total of unreplied connection went 
> down
> a
> little bit on June 8th at 4am: http://4u.1nw.eu/conntrack_stat3.png
>
> We did fix other parameters earlier, like enabling tcp_tw_reuse and 
> directing
> localhost traffic to NOTRACK. It would explain why we didn't get more of
> those
> `table full, dropping packet`.
>
> - Julien
>
> On 2012-06-09 14:14, Marco Padovan wrote:
>
>> Looks like it's an intendend behaviour:
>>
>> http://www.netfilter.org/documentation/FAQ/netfilter-faq-3.html#ss3.17 
>> [1]
>>

So, I've been monitoring conntrack for a few days now, and I can definitely 
see the UNREPLIED connections get removed when the space is needed for new 
connections. Which is the intended behavior.

http://4u.1nw.eu/conntrack_stat6.png

However, I still get `nf_conntrack: table full, dropping packet` in my logs 
from time to time. Should I be worried about those ? Are they related to 
conntrack removing UNREPLIED connections ?

# grep conntrack /var/log/kern.log
Jun 10 13:23:42 web-forms1 kernel: [8485564.759240] nf_conntrack: table 
full, dropping packet.
Jun 10 14:01:42 web-forms1 kernel: [8487844.952658] nf_conntrack: table 
full, dropping packet.
Jun 11 07:35:49 web-forms1 kernel: [8551092.298203] nf_conntrack: table 
full, dropping packet.
Jun 11 08:10:56 web-forms1 kernel: [8553199.262974] nf_conntrack: table 
full, dropping packet.
Jun 11 08:10:59 web-forms1 kernel: [8553202.270012] nf_conntrack: table 
full, dropping packet.
Jun 11 08:11:05 web-forms1 kernel: [8553208.270012] nf_conntrack: table 
full, dropping packet.
Jun 11 09:36:23 web-forms1 kernel: [8558325.832413] nf_conntrack: table 
full, dropping packet.
Jun 11 10:09:40 web-forms1 kernel: [8560322.716116] nf_conntrack: table 
full, dropping packet.
Jun 11 10:52:30 web-forms1 kernel: [8562892.668218] nf_conntrack: table 
full, dropping packet.
Jun 11 12:14:34 web-forms1 kernel: [8567817.116556] nf_conntrack: table 
full, dropping packet.
Jun 11 12:29:56 web-forms1 kernel: [8568739.067132] nf_conntrack: table 
full, dropping packet.
Jun 11 15:20:50 web-forms1 kernel: [8578993.309849] nf_conntrack: table 
full, dropping packet.
Jun 11 15:29:18 web-forms1 kernel: [8579500.895662] nf_conntrack: table 
full, dropping packet.
Jun 11 15:29:21 web-forms1 kernel: [8579503.901261] nf_conntrack: table 
full, dropping packet.
Jun 11 15:29:21 web-forms1 kernel: [8579504.043856] nf_conntrack: table 
full, dropping packet.
Jun 11 15:54:59 web-forms1 kernel: [8581042.188246] nf_conntrack: table 
full, dropping packet.
Jun 11 15:55:00 web-forms1 kernel: [8581042.695267] nf_conntrack: table 
full, dropping packet.
Jun 11 21:46:37 web-forms1 kernel: [8602139.638368] nf_conntrack: table 
full, dropping packet.

- Julien

Re: Conntrack & Unreplied exhausts hashsize

[Pablo Neira Ayuso]

On Tue, Jun 12, 2012 at 07:22:10AM -0400, Julien Vehent wrote:
[...]
> So, I've been monitoring conntrack for a few days now, and I can
> definitely see the UNREPLIED connections get removed when the space
> is needed for new connections. Which is the intended behavior.
> 
> http://4u.1nw.eu/conntrack_stat6.png

You have lots of entries in ESTABLISHED but UNREPLIED state according
to that figure. Probably someone is sending you forged TCP packets to
enter that state and /proc/sys/net/netfilter/nf_conntrack_tcp_loose is
not set to zero.

portscan tools like nmap can produce this.

> However, I still get `nf_conntrack: table full, dropping packet` in
> my logs from time to time. Should I be worried about those ? Are
> they related to conntrack removing UNREPLIED connections ?

That means it's dropping packets, so you should worry about that, of
course.

The conntrack table can store a limited a number of flow objects. You
probably need to increase that amount.

Still, you should investigate what's going on with your rule-set
configuration and the network traffic that is causing such population.

Wireshark should help.