ip_conntrack table full after upgrade from RHEL3 (2.4/1.2.8) to RHEL4U4 (2.6.9/1.2.11)

ip_conntrack table full after upgrade from RHEL3 (2.4/1.2.8) to RHEL4U4 (2.6.9/1.2.11)

[Alex Tang]

Hi folks.

I've been having a problem on a machine that does a high volume of 
sendmail traffic.  The machine gets approx 50,000 connections per hour 
to port 25. 

The machine was upgraded from a RHEL3 based system (kernel rpm 
2.4.21-47.EL and iptables rpm 1.2.8-12.3) to a RHEL4 based system 
(kernel rpm 2.6.9-55.EL and iptables rpm 1.2.11-3.1.RHEL4).

Since the upgrade has occurred, the conntrack table fills up relatively 
fast (within one day).  The max size is 65536 (as per 
/proc/sys/net/ipv4/netfilter/ip_conntrack_max).

I've been searching through the archives, faq, etc and have found the 
usual standard answer is to increase the ip_conntrack_max.  However, I'm 
concerned for a couple of reasons that this may not be the proper answer.

In particular, i have another machine which is still running the RHEL3 
(kernel 2.4.21-47.EL/iptables 1.2.8-12.3), that gets more connections 
per hour (80,000 vs. 50,000), and there are only about 9000 entries in 
the ip_conntrack table on that machine.

The problem with the conntrack table filling up fast started as soon as 
we did the upgrade.

Also, on the machine that is currently experiencing problems, most (98%) 
of the connections are in the ESTABLISHED state, however the majority of 
these connections are not seen when doing a "netstat".

I admit that I do not fully understand the details of the iptables 
implementation, but it seems that the connection close is not being 
"seen" by the conntrack code and connections that have already gone away 
are still in the ip_conntrack table, and we have to wait for these 
connections to "timeout" before they are expired from the conntrack table.

I could of course, increase the max size of the table, or decrease the 
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established, but 
it seems that would only mask the problem, not actually fix it.

Is this a known issue in iptables? Or am I going down the wrong path?

Thanks for your help.

...alex...

Re: ip_conntrack table full after upgrade from RHEL3 (2.4/1.2.8) to RHEL4U4 (2.6.9/1.2.11)

[Pascal Hambourg]

Hello,

Alex Tang a écrit :
> 
> I've been having a problem on a machine that does a high volume of 
> sendmail traffic.  The machine gets approx 50,000 connections per hour 
> to port 25.
> The machine was upgraded from a RHEL3 based system (kernel rpm 
> 2.4.21-47.EL and iptables rpm 1.2.8-12.3) to a RHEL4 based system 
> (kernel rpm 2.6.9-55.EL and iptables rpm 1.2.11-3.1.RHEL4).

You should really consider upgrading to something more recent, because 
kernel 2.6.9 and iptables 1.2.11 are *badly* outdated.

> Since the upgrade has occurred, the conntrack table fills up relatively 
> fast (within one day).  The max size is 65536 (as per 
> /proc/sys/net/ipv4/netfilter/ip_conntrack_max).
> 
> I've been searching through the archives, faq, etc and have found the 
> usual standard answer is to increase the ip_conntrack_max.  However, I'm 
> concerned for a couple of reasons that this may not be the proper answer.
> 
> In particular, i have another machine which is still running the RHEL3 
> (kernel 2.4.21-47.EL/iptables 1.2.8-12.3), that gets more connections 
> per hour (80,000 vs. 50,000), and there are only about 9000 entries in 
> the ip_conntrack table on that machine.
> 
> The problem with the conntrack table filling up fast started as soon as 
> we did the upgrade.
> 
> Also, on the machine that is currently experiencing problems, most (98%) 
> of the connections are in the ESTABLISHED state, however the majority of 
> these connections are not seen when doing a "netstat".
> 
> I admit that I do not fully understand the details of the iptables 
> implementation, but it seems that the connection close is not being 
> "seen" by the conntrack code and connections that have already gone away 
> are still in the ip_conntrack table, and we have to wait for these 
> connections to "timeout" before they are expired from the conntrack table.

And the default timeout for established TCP connections is 5 days, so it 
takes a looong time.

A noticeable difference between kernel 2.6.9 and earlier versions is the 
TCP window tracking, which was added in 2.6.9. It adds TCP sequence 
number checking to the connection tracking, so any TCP packet with an 
out-of-window sequence number is tagged INVALID. If the TCP window 
tracking considers that the FIN packet sequence numbers are 
out-of-window for whatever reason, this may be the reason why the 2.6.9 
kernel connection tracking keeps old connexions. You may add LOG 
iptables rules to track TCP FIN packets states.

> I could of course, increase the max size of the table, or decrease the 
> /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established, but 
> it seems that would only mask the problem, not actually fix it.

I agree. You could also decrease the value of 
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established to 
something much shorter than 5 days but again this would only mask the 
problem.

Try to set /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal to 
1. This setting makes TCP window tracking more liberal, so only 
out-of-window TCP RST packets are tagged INVALID.

Note that later kernel versions provided some bugfixes for the TCP 
window tracking that may fix this issue.

RE: ip_conntrack table full after upgrade from RHEL3 (2.4/1.2.8) to RHEL4U4 (2.6.9/1.2.11)

[Paul Blondé]

Note that since RHEL is the enterprise Linux version from RedHat, you should
probably keep your updates fairly close to theirs. The focus of enterprise
releases of Linux is stability and security, so you would be better off
updating to RHEL5 than RHEL4 at this time. This will also ensure that you
get at least a reasonably recent version of netfilter/iptables, although it
wouldn't hurt to update that further if it is not the latest.

There is also a good list of PREROUTING filters for a number of purposes at
the iptablesrocks.org site, I have used their "general web server firewall"
configuration virtually unchanged with much success.



~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Paul Blondé


> -----Original Message-----
> From: netfilter-bounces@lists.netfilter.org 
> [mailto:netfilter-bounces@lists.netfilter.org] On Behalf Of 
> Pascal Hambourg
> Sent: Tuesday, May 22, 2007 2:59 AM
> To: netfilter@lists.netfilter.org
> Subject: Re: ip_conntrack table full after upgrade from RHEL3 
> (2.4/1.2.8) to RHEL4U4 (2.6.9/1.2.11)
> 
> 
> Hello,
> 
> Alex Tang a écrit :
> > 
> > I've been having a problem on a machine that does a high volume of 
> > sendmail traffic.  The machine gets approx 50,000 
> connections per hour 
> > to port 25.
> > The machine was upgraded from a RHEL3 based system (kernel rpm 
> > 2.4.21-47.EL and iptables rpm 1.2.8-12.3) to a RHEL4 based system 
> > (kernel rpm 2.6.9-55.EL and iptables rpm 1.2.11-3.1.RHEL4).
> 
> You should really consider upgrading to something more 
> recent, because 
> kernel 2.6.9 and iptables 1.2.11 are *badly* outdated.
> 
> > Since the upgrade has occurred, the conntrack table fills 
> up relatively 
> > fast (within one day).  The max size is 65536 (as per 
> > /proc/sys/net/ipv4/netfilter/ip_conntrack_max).
> > 
> > I've been searching through the archives, faq, etc and have 
> found the 
> > usual standard answer is to increase the ip_conntrack_max.  
> However, I'm 
> > concerned for a couple of reasons that this may not be the 
> proper answer.
> > 
> > In particular, i have another machine which is still 
> running the RHEL3 
> > (kernel 2.4.21-47.EL/iptables 1.2.8-12.3), that gets more 
> connections 
> > per hour (80,000 vs. 50,000), and there are only about 9000 
> entries in 
> > the ip_conntrack table on that machine.
> > 
> > The problem with the conntrack table filling up fast 
> started as soon as 
> > we did the upgrade.
> > 
> > Also, on the machine that is currently experiencing 
> problems, most (98%) 
> > of the connections are in the ESTABLISHED state, however 
> the majority of 
> > these connections are not seen when doing a "netstat".
> > 
> > I admit that I do not fully understand the details of the iptables 
> > implementation, but it seems that the connection close is not being 
> > "seen" by the conntrack code and connections that have 
> already gone away 
> > are still in the ip_conntrack table, and we have to wait for these 
> > connections to "timeout" before they are expired from the 
> conntrack table.
> 
> And the default timeout for established TCP connections is 5 
> days, so it 
> takes a looong time.
> 
> A noticeable difference between kernel 2.6.9 and earlier 
> versions is the 
> TCP window tracking, which was added in 2.6.9. It adds TCP sequence 
> number checking to the connection tracking, so any TCP packet with an 
> out-of-window sequence number is tagged INVALID. If the TCP window 
> tracking considers that the FIN packet sequence numbers are 
> out-of-window for whatever reason, this may be the reason why 
> the 2.6.9 
> kernel connection tracking keeps old connexions. You may add LOG 
> iptables rules to track TCP FIN packets states.
> 
> > I could of course, increase the max size of the table, or 
> decrease the 
> > 
> /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_establis
> hed, but 
> > it seems that would only mask the problem, not actually fix it.
> 
> I agree. You could also decrease the value of 
> /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established to 
> something much shorter than 5 days but again this would only mask the 
> problem.
> 
> Try to set 
> /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal to 
> 1. This setting makes TCP window tracking more liberal, so only 
> out-of-window TCP RST packets are tagged INVALID.
> 
> Note that later kernel versions provided some bugfixes for the TCP 
> window tracking that may fix this issue.
> 
>