nf_conntrack_max

nf_conntrack_max

[John Ratliff]

What are the implications of raising net.ipv4.netfilter.ip_conntrack_max?

I have a pair of firewalls in an active/passive failover setup (using
keepalived and conntrackd) that I want to use to NAT several services
behind. When I added DNS yesterday, I quickly exceeded the default 65536
value. It never appeared to exceed 85000, so I simply doubled it for the
time being.

When I was reading about this online, there were many suggestions for
putting DNS servers outside the firewall. I am ambivalent about this
solution. It will work, but it will require me to duplicate many rules from
my main firewall to the packet filter on the individual DNS servers that I
Would prefer not be duplicated.

Would there be a serious performance penalty to simply raising the
conntrack_max value to 256k, 512k, or 1024k? Is it best to try and avoid
large connection tracking tables like this? I do not know what my average
table would be, but I would expect 100k from the data I have so far.

Thanks.

--John

Re: nf_conntrack_max

[/dev/rob0]

On Wed, Aug 31, 2016 at 06:15:55PM -0400, John Ratliff wrote:
> What are the implications of raising 
> net.ipv4.netfilter.ip_conntrack_max?
> 
> I have a pair of firewalls in an active/passive failover setup 
> (using keepalived and conntrackd) that I want to use to NAT several 
> services behind. When I added DNS yesterday, I quickly exceeded the 
> default 65536 value. It never appeared to exceed 85000, so I simply 
> doubled it for the time being.
> 
> When I was reading about this online, there were many suggestions 
> for putting DNS servers outside the firewall.

If the NAT is not a hard requirement, this is what I would do.  But 
since as you say that would be a lot of trouble, yes, just raise the
net.ipv4.netfilter.ip_conntrack_max.

Each entry costs a relatively small amount of RAM.  Since you are 
talking about numerous high availability measures, I am sure your 
organization has not skimped on the firewall machines.  Increase it, 
don't worry.  Check the RAM use while it's working.  Order more RAM 
if there's any doubt. :)

Perhaps you have already seen this unattributed article at the ISC 
Knowledge Base:

https://kb.isc.org/article/AA-01183/

> I am ambivalent about this solution. It will work, but it will 
> require me to duplicate many rules from my main firewall to the 
> packet filter on the individual DNS servers that I Would prefer not 
> be duplicated.
> 
> Would there be a serious performance penalty to simply raising the 
> conntrack_max value to 256k, 512k, or 1024k? Is it best to try and 
> avoid large connection tracking tables like this? I do not know 
> what my average table would be, but I would expect 100k from the 
> data I have so far.

AFAIK the extra memory use will be the only drawback.

One more thing I can add: I believe it is possible to set different 
conntrack timeouts based on protocol/port.  I don't know exactly how 
to do that, but it would make sense for udp/53 to shorten that to 
something like 15 seconds; just a bit beyond the nameservers' and 
resolver clients' timeout values.
-- 
  http://rob0.nodns4.us/
  Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:

Re: nf_conntrack_max

[Pablo Neira Ayuso]

Hi,

On Wed, Aug 31, 2016 at 05:47:32PM -0500, /dev/rob0 wrote:
[...]
> One more thing I can add: I believe it is possible to set different 
> conntrack timeouts based on protocol/port. 

Right, this is possible.

> I don't know exactly how to do that, but it would make sense for
> udp/53 to shorten that to something like 15 seconds; just a bit
> beyond the nameservers' and resolver clients' timeout values.

Setting custom timeout policies per address/protocol/port (any
selector basically) is possible through -j CT --timeout name from the
raw table.

You have to create the timeout policy in first place through 'nfct'
that comes in the conntrack-tools package. I think there are examples
for this already, otherwise let me know and we can place it on the
manpage.

Now that nft has come, the plan is to unify all these command line
tools such as conntrack and nfct into nft, so we end up using one
single tool to interface with the netfilter subsystem in the future.

Re: nf_conntrack_max

[zrm]

On 09/01/2016 06:59 AM, Pablo Neira Ayuso wrote:

> Setting custom timeout policies per address/protocol/port (any
> selector basically) is possible through -j CT --timeout name from the
> raw table.
>
> You have to create the timeout policy in first place through 'nfct'
> that comes in the conntrack-tools package. I think there are examples
> for this already, otherwise let me know and we can place it on the
> manpage.

There is a TCP example on the nfct manpage but it would be helpful if it 
listed what timeouts are possible to set for each protocol (syn_sent, 
time_wait, etc.) corresponding to ip_conntrack_[proto]_timeout_[state] 
in net.ipv4.netfilter. Especially how to set the timeout corresponding 
to ip_conntrack_udp_timeout since it doesn't follow the pattern.

FWIW it would also be useful (in another context) if there was a stable 
way to set these values for a specific conntrack entry.

Re: nf_conntrack_max

[/dev/rob0]

Hi Pablo, thanks for taking the time to reply.

On Thu, Sep 01, 2016 at 12:59:11PM +0200, Pablo Neira Ayuso wrote:
> On Wed, Aug 31, 2016 at 05:47:32PM -0500, /dev/rob0 wrote:
> [...]
> > One more thing I can add: I believe it is possible to set 
> > different conntrack timeouts based on protocol/port.
> 
> Right, this is possible.
> 
> > I don't know exactly how to do that, but it would make sense for
> > udp/53 to shorten that to something like 15 seconds; just a bit
> > beyond the nameservers' and resolver clients' timeout values.
> 
> Setting custom timeout policies per address/protocol/port (any 
> selector basically) is possible through -j CT --timeout name from 
> the raw table.
> 
> You have to create the timeout policy in first place through 'nfct' 
> that comes in the conntrack-tools package. I think there are 
> examples for this already, otherwise let me know and we can place 
> it on the manpage.

Okay, here's what I'd try for udp/53:

# nfct timeout add dns-udp inet udp established 15 close 1 close_wait 1
# iptables -t raw -A PREROUTING -p udp --dport 53 -j CT --timeout dns-udp
# iptables -t raw -A OUTPUT -p udp --dport 53 -j CT --timeout dns-udp
# iptables -t raw -A PREROUTING -p udp --sport 53 -j CT --timeout dns-udp
# iptables -t raw -A OUTPUT -p udp --sport 53 -j CT --timeout dns-udp

(Understood that not all sites would need both the --dport and the
--sport sets of rules.)

root@tp:~# nfct version
nfct v1.4.2: utility for the Netfilter's Connection Tracking System

Perhaps I am looking at an older version of the man page (Slackware 
14.1, still have not upgraded to 14.2), but indeed more discussion of 
the conntrack states and other arguments would help.

The example for TCP has:

nfct <subsys> <cmd> <args>

where <args> are:

<name> <L3-proto> <L4-proto> <ctstate> <secs> [ <ctstate> <secs> ... ]

The subsys and cmd are documented, but args are not.

It would be ideal to have some information about each argument and 
the various conntrack states which can be set; if possible even some 
words about what default values exist for various protocols, and also 
about what changing these values might do.

What I don't know (can't tell from the man page) is if there is any 
way to clear out the conntrack entry after a reply has been received.  
Again/still we are talking about UDP DNS here, where a complete 
"connection" consists of exactly two packets: the query (--dport 53) 
and the reply (--sport 53).

After the reply is sent (from a nameserver, or received, on a client) 
there is no need for that conntrack entry to remain.

> Now that nft has come, the plan is to unify all these command line 
> tools such as conntrack and nfct into nft, so we end up using one 
> single tool to interface with the netfilter subsystem in the 
> future.

This sounds good.  For the time being, however, I continue to relax 
in the comfort of iptables-land. :)
-- 
  http://rob0.nodns4.us/
  Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:

RE: nf_conntrack_max

[John Ratliff]

Thanks for these suggestions. I will look into upping the connection tables
and shortening the timeouts for udp/dns traffic.

-----Original Message-----
From: netfilter-owner@vger.kernel.org
[mailto:netfilter-owner@vger.kernel.org] On Behalf Of /dev/rob0
Sent: Thursday, September 1, 2016 1:06 PM
To: netfilter@vger.kernel.org
Subject: Re: nf_conntrack_max

Hi Pablo, thanks for taking the time to reply.

On Thu, Sep 01, 2016 at 12:59:11PM +0200, Pablo Neira Ayuso wrote:
> On Wed, Aug 31, 2016 at 05:47:32PM -0500, /dev/rob0 wrote:
> [...]
> > One more thing I can add: I believe it is possible to set different 
> > conntrack timeouts based on protocol/port.
> 
> Right, this is possible.
> 
> > I don't know exactly how to do that, but it would make sense for
> > udp/53 to shorten that to something like 15 seconds; just a bit 
> > beyond the nameservers' and resolver clients' timeout values.
> 
> Setting custom timeout policies per address/protocol/port (any 
> selector basically) is possible through -j CT --timeout name from the 
> raw table.
> 
> You have to create the timeout policy in first place through 'nfct' 
> that comes in the conntrack-tools package. I think there are examples 
> for this already, otherwise let me know and we can place it on the 
> manpage.

Okay, here's what I'd try for udp/53:

# nfct timeout add dns-udp inet udp established 15 close 1 close_wait 1 #
iptables -t raw -A PREROUTING -p udp --dport 53 -j CT --timeout dns-udp #
iptables -t raw -A OUTPUT -p udp --dport 53 -j CT --timeout dns-udp #
iptables -t raw -A PREROUTING -p udp --sport 53 -j CT --timeout dns-udp #
iptables -t raw -A OUTPUT -p udp --sport 53 -j CT --timeout dns-udp

(Understood that not all sites would need both the --dport and the --sport
sets of rules.)

root@tp:~# nfct version
nfct v1.4.2: utility for the Netfilter's Connection Tracking System

Perhaps I am looking at an older version of the man page (Slackware 14.1,
still have not upgraded to 14.2), but indeed more discussion of the
conntrack states and other arguments would help.

The example for TCP has:

nfct <subsys> <cmd> <args>

where <args> are:

<name> <L3-proto> <L4-proto> <ctstate> <secs> [ <ctstate> <secs> ... ]

The subsys and cmd are documented, but args are not.

It would be ideal to have some information about each argument and the
various conntrack states which can be set; if possible even some words about
what default values exist for various protocols, and also about what
changing these values might do.

What I don't know (can't tell from the man page) is if there is any way to
clear out the conntrack entry after a reply has been received.  
Again/still we are talking about UDP DNS here, where a complete "connection"
consists of exactly two packets: the query (--dport 53) and the reply
(--sport 53).

After the reply is sent (from a nameserver, or received, on a client) there
is no need for that conntrack entry to remain.

> Now that nft has come, the plan is to unify all these command line 
> tools such as conntrack and nfct into nft, so we end up using one 
> single tool to interface with the netfilter subsystem in the future.

This sounds good.  For the time being, however, I continue to relax in the
comfort of iptables-land. :)
--
  http://rob0.nodns4.us/
  Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
--
To unsubscribe from this list: send the line "unsubscribe netfilter" in the
body of a message to majordomo@vger.kernel.org More majordomo info at
http://vger.kernel.org/majordomo-info.html

RE: nf_conntrack_max

[John Ratliff]

I have been able to raise the conntrack limits and the hashsize, but I don't
know how to get a udp timeout policy yet. I'm using Debian 8 Jessie with
nfct 1.42.

$ nfct timeout add dns-udp inet udp established 15 close 1 close_wait 1
nfct v1.4.2: Wrong state name: `ESTABLISHED' for protocol `udp'

After looking through conntrack -E -p udp, I saw UNREPLIED which looked
promising. So if I change established to unreplied, it complains about close
and close_wait. This makes sense because udp doesn't have any of these
properties; I'm just not sure what properties nfct is looking for here.
Should it just be 'unreplied'?

I also noticed this in sysctl:

net.netfilter.nf_conntrack_udp_timeout = 30
net.netfilter.nf_conntrack_udp_timeout_stream = 180

This would suggest to me that my global udp timeout is already set to 30
seconds, though I don't know what a timeout_stream value is for. If my
global udp timeout is already at 30 seconds, I doubt dropping it to 15 is
going to alleviate too many problems here.

I don't know If my udp/dns traffic is abnormal or not for our system. They
handle DNS requests for a small ISP of 20,000 customers and are public
facing to the world for our corporate domains along with several customer
domains. We seem to always be bombarded with DNS traffic that is suspect. We
have given thought to separating our public facing DNS from our customer's
DNS to distinguish the two kinds of traffic, but it hasn't seemed important
enough up to now.

Thanks.

-----Original Message-----
From: netfilter-owner@vger.kernel.org
[mailto:netfilter-owner@vger.kernel.org] On Behalf Of John Ratliff
Sent: Friday, September 2, 2016 11:06 AM
To: netfilter@vger.kernel.org
Subject: RE: nf_conntrack_max

Thanks for these suggestions. I will look into upping the connection tables
and shortening the timeouts for udp/dns traffic.

-----Original Message-----
From: netfilter-owner@vger.kernel.org
[mailto:netfilter-owner@vger.kernel.org] On Behalf Of /dev/rob0
Sent: Thursday, September 1, 2016 1:06 PM
To: netfilter@vger.kernel.org
Subject: Re: nf_conntrack_max

Hi Pablo, thanks for taking the time to reply.

On Thu, Sep 01, 2016 at 12:59:11PM +0200, Pablo Neira Ayuso wrote:
> On Wed, Aug 31, 2016 at 05:47:32PM -0500, /dev/rob0 wrote:
> [...]
> > One more thing I can add: I believe it is possible to set different 
> > conntrack timeouts based on protocol/port.
> 
> Right, this is possible.
> 
> > I don't know exactly how to do that, but it would make sense for
> > udp/53 to shorten that to something like 15 seconds; just a bit 
> > beyond the nameservers' and resolver clients' timeout values.
> 
> Setting custom timeout policies per address/protocol/port (any 
> selector basically) is possible through -j CT --timeout name from the 
> raw table.
> 
> You have to create the timeout policy in first place through 'nfct' 
> that comes in the conntrack-tools package. I think there are examples 
> for this already, otherwise let me know and we can place it on the 
> manpage.

Okay, here's what I'd try for udp/53:

# nfct timeout add dns-udp inet udp established 15 close 1 close_wait 1 #
iptables -t raw -A PREROUTING -p udp --dport 53 -j CT --timeout dns-udp #
iptables -t raw -A OUTPUT -p udp --dport 53 -j CT --timeout dns-udp #
iptables -t raw -A PREROUTING -p udp --sport 53 -j CT --timeout dns-udp #
iptables -t raw -A OUTPUT -p udp --sport 53 -j CT --timeout dns-udp

(Understood that not all sites would need both the --dport and the --sport
sets of rules.)

root@tp:~# nfct version
nfct v1.4.2: utility for the Netfilter's Connection Tracking System

Perhaps I am looking at an older version of the man page (Slackware 14.1,
still have not upgraded to 14.2), but indeed more discussion of the
conntrack states and other arguments would help.

The example for TCP has:

nfct <subsys> <cmd> <args>

where <args> are:

<name> <L3-proto> <L4-proto> <ctstate> <secs> [ <ctstate> <secs> ... ]

The subsys and cmd are documented, but args are not.

It would be ideal to have some information about each argument and the
various conntrack states which can be set; if possible even some words about
what default values exist for various protocols, and also about what
changing these values might do.

What I don't know (can't tell from the man page) is if there is any way to
clear out the conntrack entry after a reply has been received.  
Again/still we are talking about UDP DNS here, where a complete "connection"
consists of exactly two packets: the query (--dport 53) and the reply
(--sport 53).

After the reply is sent (from a nameserver, or received, on a client) there
is no need for that conntrack entry to remain.

> Now that nft has come, the plan is to unify all these command line 
> tools such as conntrack and nfct into nft, so we end up using one 
> single tool to interface with the netfilter subsystem in the future.

This sounds good.  For the time being, however, I continue to relax in the
comfort of iptables-land. :)
--
  http://rob0.nodns4.us/
  Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
--
To unsubscribe from this list: send the line "unsubscribe netfilter" in the
body of a message to majordomo@vger.kernel.org More majordomo info at
http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe netfilter" in the
body of a message to majordomo@vger.kernel.org More majordomo info at
http://vger.kernel.org/majordomo-info.html

Re: nf_conntrack_max

[Pablo Neira Ayuso]

On Sat, Sep 03, 2016 at 02:47:08PM -0400, John Ratliff wrote:
> I have been able to raise the conntrack limits and the hashsize, but I don't
> know how to get a udp timeout policy yet. I'm using Debian 8 Jessie with
> nfct 1.42.
> 
> $ nfct timeout add dns-udp inet udp established 15 close 1 close_wait 1
> nfct v1.4.2: Wrong state name: `ESTABLISHED' for protocol `udp'

UDP has two states, replied and unreplied, so this is:

 # nfct add timeout dns-udp inet udp replied 15 unreplied 1

For TCP, states are:

        syn_sent
        syn_recv
        established
        fin_wait
        close_wait
        last_ack
        time_wait
        close
        syn_sent2
        retrans
        unacknowledged

When the generic tracker is in place, you use "timeout", eg.

 # nfct add timeout generic-timeout-policy inet generic timeout 15

For SCTP, states are:

        closed
        cookie_wait
        cookie_eched
        established
        shutdown_sent
        shutdown_recd
        shutdown_ack_sent

For DCCP, states are:

        request
        respond
        partopen
        open
        closereq
        closing
        timewait

For ICMP/v6, there is only one, so you use "timeout".

That's it. If anyone gets some spare cycles, I'd appreciate if you can
contribute a patch to update the manpage so this information is
available there.

You can use this to create your custom timeout policies. You can also
set global default timeouts via nfct:

 # nfct default-set timeout inet tcp established 15

instead of using sysctl, just an alternative.

Thanks!

Re: nf_conntrack_max

[/dev/rob0]

On Mon, Sep 05, 2016 at 11:33:04AM +0200, Pablo Neira Ayuso wrote:
> On Sat, Sep 03, 2016 at 02:47:08PM -0400, John Ratliff wrote:
> > I have been able to raise the conntrack limits and the hashsize, 
> > but I don't know how to get a udp timeout policy yet. I'm using 
> > Debian 8 Jessie with nfct 1.42.
> > 
> > $ nfct timeout add dns-udp inet udp established 15 close 1 close_wait 1
> > nfct v1.4.2: Wrong state name: `ESTABLISHED' for protocol `udp'
> 
> UDP has two states, replied and unreplied, so this is:
> 
>  # nfct add timeout dns-udp inet udp replied 15 unreplied 1

Perhaps I am not understanding, but I think those numbers should 
possibly be reversed? "... replied 1 unreplied 15"?  The goal being 
to hold the conntrack entry up to 15 seconds if we don't hear back 
from the query, but to clear it out quickly after we did.  Would a
"replied 0" be feasible, to get it out of conntrack right away?

[snip]
> That's it. If anyone gets some spare cycles, I'd appreciate if you
> can contribute a patch to update the manpage so this information
> is available there.

I'm at work on it now, if uninterrupted (hah!) I should have it to 
you very soon.  Thank you.
-- 
  http://rob0.nodns4.us/
  Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:

RE: nf_conntrack_max

[André Paulsberg-Csibi (IBM Consultant)]

Hi John ,

You should try to identify WHY your sessions are reaching as much as 80-90k connections ,
as I would say that this is some sort of anomaly ( from my experience ) .

In the meantime there is no real issues increasing you the connection table while you figure that out ,
and you can also GLOBALY fix the UDP timeouts if you so desire as most likely this can be lowered significantly from
the defaults , here you can see one example from a smaller FW then yours ( but still applicable ) :

zotac:~ # grep netfilter /etc/sysctl.conf
net.netfilter.nf_conntrack_max = 10000
net.netfilter.nf_conntrack_tcp_timeout_established = 3600
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 40
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 40
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 40
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 40
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 40
net.netfilter.nf_conntrack_udp_timeout_stream = 60
net.netfilter.nf_conntrack_udp_timeout = 10

You can most likely use lower values depending on your sessions in the FW and how the communicate ,
and you should also look at statistics about WHAT sessions are being left over and adjust any "problem" sessions lower ...


Typically DNS use mostly UDP , so with a default timeout of 30 you would need approx. 3000 UDP sessions
per second without any response to create 90k connections ( which seems excessive for any environment ) .
Having a default of 10 seconds timeout would drop your connections down to 30k at the same session rate ,
but would obviously not fix your underlying issue of why there are so many sessions in the first place .



Best regards
André Paulsberg-Csibi
Senior Network Engineer 
Fault Handling
IBM Services AS
andre.paulsberg-csibi@evry.com
M +47 9070 5988




-----Original Message-----
From: netfilter-owner@vger.kernel.org [mailto:netfilter-owner@vger.kernel.org] On Behalf Of John Ratliff
Sent: 1. september 2016 00:16
To: netfilter@vger.kernel.org
Subject: nf_conntrack_max

What are the implications of raising net.ipv4.netfilter.ip_conntrack_max?

I have a pair of firewalls in an active/passive failover setup (using
keepalived and conntrackd) that I want to use to NAT several services
behind. When I added DNS yesterday, I quickly exceeded the default 65536
value. It never appeared to exceed 85000, so I simply doubled it for the
time being.

When I was reading about this online, there were many suggestions for
putting DNS servers outside the firewall. I am ambivalent about this
solution. It will work, but it will require me to duplicate many rules from
my main firewall to the packet filter on the individual DNS servers that I
Would prefer not be duplicated.

Would there be a serious performance penalty to simply raising the
conntrack_max value to 256k, 512k, or 1024k? Is it best to try and avoid
large connection tracking tables like this? I do not know what my average
table would be, but I would expect 100k from the data I have so far.

Thanks.

--John


--
To unsubscribe from this list: send the line "unsubscribe netfilter" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nf_conntrack_max

[zrm]

On 09/02/2016 02:27 AM, André Paulsberg-Csibi (IBM Consultant) wrote:

> In the meantime there is no real issues increasing you the connection table while you figure that out ,
> and you can also GLOBALY fix the UDP timeouts if you so desire as most likely this can be lowered significantly from
> the defaults , here you can see one example from a smaller FW then yours ( but still applicable ) :
>
> zotac:~ # grep netfilter /etc/sysctl.conf
> net.netfilter.nf_conntrack_max = 10000
> net.netfilter.nf_conntrack_tcp_timeout_established = 3600
> net.netfilter.nf_conntrack_tcp_timeout_close_wait = 40
> net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 40
> net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 40
> net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 40
> net.netfilter.nf_conntrack_tcp_timeout_time_wait = 40
> net.netfilter.nf_conntrack_udp_timeout_stream = 60
> net.netfilter.nf_conntrack_udp_timeout = 10

Global timeouts that short can cause other trouble though. See RFC4787 
Section 4.3 and RFC5382 Section 5. Outside of reducing the timeout only 
for DNS, the best choice is probably just increasing the maximum number 
of conntrack entries (and buying more memory if necessary).

RE: nf_conntrack_max

[André Paulsberg-Csibi (IBM Consultant)]

I have yet to see any implementation of any FW in 2014 - 2016 that has UDP timeouts in the 2-5 minutes range , as described in RFC 4787 .

IF I remember UDP implementations from a lot of todays firewalls is typically set default timeout to about 30-40 seconds ,
and despite RFC4787 I would rather recommend having lower timeouts for UDP and higher ones for just those
protocols that require this than to have 2-5 minutes for UDP which in most cases never last more then 2-4 seconds .

That seems like an unreasonable reverse logic and use of resources , looking at today vs 10-20 years ago's most common UDP traffic patterns .

But yes , I agree on increasing connection table in either case as this was clearly low for such current level of sessions
and find the root cause for the high amounts of sessions and fix that if possible ...



Best regards
André Paulsberg-Csibi
Senior Network Engineer 
Fault Handling
IBM Services AS
andre.paulsberg-csibi@evry.com
M +47 9070 5988




-----Original Message-----
From: netfilter-owner@vger.kernel.org [mailto:netfilter-owner@vger.kernel.org] On Behalf Of zrm
Sent: 2. september 2016 16:48
To: André Paulsberg-Csibi (IBM Consultant) <Andre.Paulsberg-Csibi@evry.com>; John Ratliff <jratliff@bluemarble.net>; netfilter@vger.kernel.org
Subject: Re: nf_conntrack_max



On 09/02/2016 02:27 AM, André Paulsberg-Csibi (IBM Consultant) wrote:

> In the meantime there is no real issues increasing you the connection table while you figure that out ,
> and you can also GLOBALY fix the UDP timeouts if you so desire as most likely this can be lowered significantly from
> the defaults , here you can see one example from a smaller FW then yours ( but still applicable ) :
>
> zotac:~ # grep netfilter /etc/sysctl.conf
> net.netfilter.nf_conntrack_max = 10000
> net.netfilter.nf_conntrack_tcp_timeout_established = 3600
> net.netfilter.nf_conntrack_tcp_timeout_close_wait = 40
> net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 40
> net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 40
> net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 40
> net.netfilter.nf_conntrack_tcp_timeout_time_wait = 40
> net.netfilter.nf_conntrack_udp_timeout_stream = 60
> net.netfilter.nf_conntrack_udp_timeout = 10

Global timeouts that short can cause other trouble though. See RFC4787 
Section 4.3 and RFC5382 Section 5. Outside of reducing the timeout only 
for DNS, the best choice is probably just increasing the maximum number 
of conntrack entries (and buying more memory if necessary).
--
To unsubscribe from this list: send the line "unsubscribe netfilter" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nf_conntrack_max

[zrm]

The default conntrack timeout for UDP streams is 180 seconds, which is 
within that range. The protocols that need that are streams rather than 
transactional protocols like DNS. (Although the other suggestion to 
remove DNS immediately on reply would even cause trouble for DNS, 
because some clients send multiple DNS queries at the same time using 
the same ports, e.g. IPv6 (AAAA) and IPv4 (A) queries, and immediate 
removal would cause the other replies to be lost. One second timeouts 
could do the same.)

The main problem with very short global UDP stream timeouts is for IPsec 
tunnel mode, OpenVPN, or anything else that encapsulates traffic in UDP 
or otherwise needs the UDP entry to continue to exist indefinitely. If 
the app's keepalive interval is shorter than the timeout then packets 
that arrive between the timeout and the keepalive packet go to the bit 
bucket.

Some applications will try to detect that and then shorten the keepalive 
interval (but not all, and generally only after several packets are lost 
with user-visible consequences). And even then the result is more 
deadweight keepalive packets consuming network resources and requiring 
power-expensive radio wakeups on battery-powered user devices.

On 09/05/2016 04:40 AM, André Paulsberg-Csibi (IBM Consultant) wrote:
> I have yet to see any implementation of any FW in 2014 - 2016 that has UDP timeouts in the 2-5 minutes range , as described in RFC 4787 .
>
> IF I remember UDP implementations from a lot of todays firewalls is typically set default timeout to about 30-40 seconds ,
> and despite RFC4787 I would rather recommend having lower timeouts for UDP and higher ones for just those
> protocols that require this than to have 2-5 minutes for UDP which in most cases never last more then 2-4 seconds .
>
> That seems like an unreasonable reverse logic and use of resources , looking at today vs 10-20 years ago's most common UDP traffic patterns .
>
> But yes , I agree on increasing connection table in either case as this was clearly low for such current level of sessions
> and find the root cause for the high amounts of sessions and fix that if possible ...
>
>
>
> Best regards
> André Paulsberg-Csibi
> Senior Network Engineer
> Fault Handling
> IBM Services AS
> andre.paulsberg-csibi@evry.com
> M +47 9070 5988
>
>
>
>
> -----Original Message-----
> From: netfilter-owner@vger.kernel.org [mailto:netfilter-owner@vger.kernel.org] On Behalf Of zrm
> Sent: 2. september 2016 16:48
> To: André Paulsberg-Csibi (IBM Consultant) <Andre.Paulsberg-Csibi@evry.com>; John Ratliff <jratliff@bluemarble.net>; netfilter@vger.kernel.org
> Subject: Re: nf_conntrack_max
>
>
>
> On 09/02/2016 02:27 AM, André Paulsberg-Csibi (IBM Consultant) wrote:
>
>> In the meantime there is no real issues increasing you the connection table while you figure that out ,
>> and you can also GLOBALY fix the UDP timeouts if you so desire as most likely this can be lowered significantly from
>> the defaults , here you can see one example from a smaller FW then yours ( but still applicable ) :
>>
>> zotac:~ # grep netfilter /etc/sysctl.conf
>> net.netfilter.nf_conntrack_max = 10000
>> net.netfilter.nf_conntrack_tcp_timeout_established = 3600
>> net.netfilter.nf_conntrack_tcp_timeout_close_wait = 40
>> net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 40
>> net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 40
>> net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 40
>> net.netfilter.nf_conntrack_tcp_timeout_time_wait = 40
>> net.netfilter.nf_conntrack_udp_timeout_stream = 60
>> net.netfilter.nf_conntrack_udp_timeout = 10
>
> Global timeouts that short can cause other trouble though. See RFC4787
> Section 4.3 and RFC5382 Section 5. Outside of reducing the timeout only
> for DNS, the best choice is probably just increasing the maximum number
> of conntrack entries (and buying more memory if necessary).
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>