netfilter optimization.

netfilter optimization.

[Gary W. Smith]

I'm looking for some firewall tweaking advice.  We have a dedicated
firewall which hit ran out of conntrack slots recently.  We had already
tweaked the number max_conntracks to 131072. That box was an RHEL 4 box.
We are building a new firewall, based on 2.6.22.  Reading some older
docs, they mention that if you can, set conntrack_buckets to the same as
conntack_max, if memory permits.  This box has plenty 512mb.  In the
sample reference doc, it says that you can do about 1048576 at a cost of
about 300mb of ram.  This is fine.

Since this is a dedicated firewall box, with only ssh, cron, smartd and
sysstat running on it, what would you recommend the settings to be?  And
what is the best way to set these (/etc/sysctl.conf)?

Playing around I found that I can set nf_conntrack_max to the value, but
when I set nf_conntrack_buckets to the same I get permission denied.
nf_conntrack_buckets is set to 4096, which if I read the documentation
correctly, would slow down the link list parsing as it would have to
refer to the conntrack list more often.

Re: netfilter optimization.

[Leonardo Rodrigues Magalhães]

[-- Attachment #1: Type: text/plain, Size: 1691 bytes --]



Gary W. Smith escreveu:
> I'm looking for some firewall tweaking advice.  We have a dedicated
> firewall which hit ran out of conntrack slots recently.  We had already
> tweaked the number max_conntracks to 131072. That box was an RHEL 4 box.
> We are building a new firewall, based on 2.6.22.  Reading some older
> docs, they mention that if you can, set conntrack_buckets to the same as
> conntack_max, if memory permits.  This box has plenty 512mb.  In the
> sample reference doc, it says that you can do about 1048576 at a cost of
> about 300mb of ram.  This is fine.
>
> Since this is a dedicated firewall box, with only ssh, cron, smartd and
> sysstat running on it, what would you recommend the settings to be?  And
> what is the best way to set these (/etc/sysctl.conf)?
>
> Playing around I found that I can set nf_conntrack_max to the value, but
> when I set nf_conntrack_buckets to the same I get permission denied.
> nf_conntrack_buckets is set to 4096, which if I read the documentation
> correctly, would slow down the link list parsing as it would have to
> refer to the conntrack list more often.
>
>   

    You missed some important informations. How many machines are behind 
this firewall ? Are they client machines or are they servers ? Are we 
talking of lots of machines that generates low traffic of we're talking 
of some machines that generate LOTS of traffic ??

    Please tell us about your network cenary.

-- 


	Atenciosamente / Sincerily,
	Leonardo Rodrigues
	Solutti Tecnologia
	http://www.solutti.com.br

	Minha armadilha de SPAM, NÃO mandem email
	gertrudes@solutti.com.br
	My SPAMTRAP, do not email it





[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 5672 bytes --]

RE: netfilter optimization.

[Gary W. Smith]

>     You missed some important informations. How many machines are
> behind
> this firewall ? Are they client machines or are they servers ? Are we
> talking of lots of machines that generates low traffic of we're talking
> of some machines that generate LOTS of traffic ??
> 
>     Please tell us about your network cenary.
> 
Leonardo, 

Approximately 50 physical servers running about 300 Xen instances as well as other Windows stuff.  We have three class C IP ranges coming in.  We balance some of this internally with ipvsadm.  So, in most cases, we have one physical external IP coming in per VIP, going to one of several machines on the back end.  

Then we have several other machines that are NAT'd for very specific ports.  In particular, we have a set of web servers that get an extreme number of connections, usually in the afternoon, PST, which will exhaust the pool.

We have tried to balance direct routing and NAT'ing when we can.  That is, if we can direct route, we do it.  In some cases, we simply don't have the IP's for it.

So, this leads us to solving the connection pooling issue.  We have two 1.8ghz machine with 512MB, one is the active firewall, the other one would be the failover.  Each one has 4 nics, two onboard 100MB and a dual 1GB.  Here is the config:

eth0 -> INET (100MB)
eth1 -> Private, heartbeat for linux-HA (100MB) -- Future implementation
eth2 -> DMZ (1GB)
eth3 -> Internal (1GB)

DMZ has public IP's and private IP's.

x.x.51.0/24, x.x.52.0/24, x.x.53.0/24, 10.0.64.0/21.

IP, such that x.x.51.128/25 goes to an ipvs instance, which then forwards it to something on the 10.0.64.0/24 range.
IP, such that x.x.52.0/24 goes to an ipvs instance, which then forwards it to something on the 10.0.68.0/22 range.
IP, such that x.x.53.0/24 gets direct routed in most cases.
IP, such that x.x.51.0/25 gets routed in a couple different ways (both direct and ivps).

As for client workstations behind it, zero, but we do have both direct and NAT'd traffic originating from inside the network (email, Windows update requests, etc).

I used to have a good png file from Visio that showed this but it's so outdated...  I need to update it.  

Anyway, this is one of the reasons we are rebuilding the firewalls.  The other reason being a spinlock but in that kernel version.  So, we wanted to go with something fresher.

This is the overall high level layout.

Gary

Re: netfilter optimization.

[Thomas Jacob]

[-- Attachment #1: Type: text/plain, Size: 1520 bytes --]

> So, this leads us to solving the connection pooling issue.  We have two 1.8ghz machine with 512MB, one is the active firewall, the other one would be the failover.  Each one has 4 nics, two onboard 100MB and a dual 1GB.  Here is the config:
> 
> eth0 -> INET (100MB)
> eth1 -> Private, heartbeat for linux-HA (100MB) -- Future implementation
> eth2 -> DMZ (1GB)
> eth3 -> Internal (1GB)

Unless you have a lot of traffic between the dmz and the internal network, and assuming
100MB means 100Mbps, and that you have some decent NICs (maybe with NAPI/interrupt
throttling, Intel's work nicely) you should probably be fine. We're running something similar 
with about 400mbps peak traffic and a P4 3Ghz and it's maybe at 30-40% capacity in peak hours.

Good NICs, good buses (PCI-Express), high memory transfer rates & large
cache sizes all make a difference though.

Harald Welte gave a talk once about selecting hardware for netfilter firewalls,
the notes are available online, maybe it's helpful to you:

http://www.heinlein-support.de/upload/slac/network_performance.pdf

> Anyway, this is one of the reasons we are rebuilding the firewalls.  The other reason being a spinlock but in that kernel version.  So, we wanted to go with something fresher.

In kernel 2.4 there are some "nice" effects under various load levels and attacks, 2.6
kernels is much more robust there. We've added a packet rate limiter
(using hash limit) for good measure and since then never had any troubles
again....

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 191 bytes --]

Re: netfilter optimization.

[Thomas Jacob]

[-- Attachment #1: Type: text/plain, Size: 680 bytes --]

Ehm, you're original question was about
setting conntrack values :-)

Originally you couldn't change the hash bucket 
number after the ipt_conntrack module had been loaded,
you needed to do that at load time (for
instance through /etc/sysctl.cnf), but
I gather that current netfilter versions
allow changing the number of hash buckets
at runtime through:

/sys/module/ip_conntrack/parameters/hashsize

Setting #hash buckets=conntrack max should be
fine that's what we do as well.

Maybe you want to carefully reduce some of
the /proc/sys/net/ipv4/netfilter/ip_conntrack_*timeout* parameters
to reduce the number of entries in the connection
tracking hash.



[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 191 bytes --]

RE: netfilter optimization.

[Gary W. Smith]

> Ehm, you're original question was about
> setting conntrack values :-)
> 
> Originally you couldn't change the hash bucket number after the
> ipt_conntrack module had been loaded, you needed to do that at load
> time (for instance through /etc/sysctl.cnf), but I gather that current
> netfilter versions allow changing the number of hash buckets at
runtime
> through:
> 
> /sys/module/ip_conntrack/parameters/hashsize
> 
> Setting #hash buckets=conntrack max should be fine that's what we do
as
> well.
> 
> Maybe you want to carefully reduce some of the
> /proc/sys/net/ipv4/netfilter/ip_conntrack_*timeout* parameters to
> reduce the number of entries in the connection tracking hash.
> 

This is what I was looking for.  I'll play around with it.  I just 1)
needed to know where to start and 2) wanted to know what people who
handle this type of traffic currently do.

Thanks, 

Gary

RE: netfilter optimization.

[Gary W. Smith]

> Originally you couldn't change the hash bucket number after the
> ipt_conntrack module had been loaded, you needed to do that at load
> time (for instance through /etc/sysctl.cnf), but I gather that current
> netfilter versions allow changing the number of hash buckets at
runtime
> through:

So this *should* work:

net.netfilter.nf_conntrack_max=1048576
net.netfilter.nf_conntrack_buckets=1048576

But it only does for nf_conntrack_max.  I did overwrite it by going to
/sys/modules/nf_conntrack/parameters/hashsize and it did take it on the
second try.  The first time it complained about file descriptors.  The
second time it seemed to set it, which I verified by looking at
/proc/sys/net/netfilter/nf_conntrack_buckets.

Is there a way to set this on startup?  

Re: netfilter optimization.

[Thomas Jacob]

[-- Attachment #1: Type: text/plain, Size: 897 bytes --]

> net.netfilter.nf_conntrack_max=1048576
> net.netfilter.nf_conntrack_buckets=1048576
> 
> But it only does for nf_conntrack_max.  I did overwrite it by going to
> /sys/modules/nf_conntrack/parameters/hashsize and it did take it on the
> second try.  The first time it complained about file descriptors.  The
> second time it seemed to set it, which I verified by looking at
> /proc/sys/net/netfilter/nf_conntrack_buckets.
> 
> Is there a way to set this on startup?  

Oh yes sorry,  you can't set it in sysctl.conf then, since the
module must probably already be loaded if you can use that. Try
the module load parameters instead (options ip_conntrack hashsize=XXXX
in /etc/modprobe.d/somefile worked in older kernels).

I am actually just patching the numbers in to the kernel version myself,
since I don't want to have a module-based kernel on my firewall box.

    Thomas

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 191 bytes --]

RE: netfilter optimization.

[Gary W. Smith]

> Oh yes sorry,  you can't set it in sysctl.conf then, since the module
> must probably already be loaded if you can use that. Try the module
> load parameters instead (options ip_conntrack hashsize=XXXX in
> /etc/modprobe.d/somefile worked in older kernels).
> 
> I am actually just patching the numbers in to the kernel version
> myself, since I don't want to have a module-based kernel on my
firewall
> box.

I've also found the same setting in a couple of list groups.  Here's
what I have, but it doesn't seem to work:

[root@localhost etc]# cat /etc/modprobe.conf
options ip_conntrack hashsize=1048576

alias eth0 eepro100
...

I'm going to play around with it for a while.  If I can't get it to work
through modprobe I'll just tweak /etc/init.d/iptables to populate
/sys/...  since that seemed to work.  I was just looking for a more
elegant solution.

Thanks for all of the help.  This solve my problem of hitting the limits
in any event.

Gary

Re: netfilter optimization.

[David Lang]

On Sun, 26 Aug 2007, Thomas Jacob wrote:

>> net.netfilter.nf_conntrack_max=1048576
>> net.netfilter.nf_conntrack_buckets=1048576
>>
>> But it only does for nf_conntrack_max.  I did overwrite it by going to
>> /sys/modules/nf_conntrack/parameters/hashsize and it did take it on the
>> second try.  The first time it complained about file descriptors.  The
>> second time it seemed to set it, which I verified by looking at
>> /proc/sys/net/netfilter/nf_conntrack_buckets.
>>
>> Is there a way to set this on startup?
>
> Oh yes sorry,  you can't set it in sysctl.conf then, since the
> module must probably already be loaded if you can use that. Try
> the module load parameters instead (options ip_conntrack hashsize=XXXX
> in /etc/modprobe.d/somefile worked in older kernels).

how can you do this if you don't use modules?

David Lang

RE: netfilter optimization.

[Gary W. Smith]

> Unless you have a lot of traffic between the dmz and the internal
> network, and assuming 100MB means 100Mbps, and that you have some

Thomas,

The internal network pulls backups from the DMZ.  DMZ and Internal
network are on a dual 1GB nic (Intel).  That works fine since it's
pretty much direct router.  

> Harald Welte gave a talk once about selecting hardware for netfilter
> firewalls, the notes are available online, maybe it's helpful to you:

We're not really seeing any hardware problems per say, but rather
limitations to the number of active connections tracked through
netfilter.  We've bumped up the value, but as per the original email, I
see recommendations that we should also increase the buckets for
performance reasons.  This is a problem as NF_CONNTRACK_BUCKETS is read
only.

So the question is should I just increase the NF_CONNTRACK_MAX to
something like 1M since this is a dedicated machine, OR will I run into
some other gotcha.

Overall everything else runs pretty nicely and we're happy with the
performance, we just don't want to lose connections because of a full
conntrack.

Gary

Re: netfilter optimization.

[G.W. Haywood]

Hi there,

On Sun, 26 Aug 2007 Thomas Jacob wrote:

> In kernel 2.4 there are some "nice" effects under various load
> levels and attacks, 2.6 kernels is much more robust there. We've
> added a packet rate limiter (using hash limit) for good measure and
> since then never had any troubles again....

Would you ming letting me have a couple of pointers?

--

73,
Ged.