Performance isues related to a large number of iptables rules

Performance isues related to a large number of iptables rules

[henry]

I am curious what is the maximum number of iptable rules that can be 
installed in a config before performance starts to be a problem. I have 
looked into the possibility of using firewall rules to block "bad" 
networks, but I have been told by most people that I have asked that 
this is bad idea.

Here are my thoughts. If a packet matches the 3rd rule, does it matter 
if there are 100,000 rules below it? 1,000,000 rules below it? If it 
doesn't, then does the number of allowable rules really have to do with 
how intelligently the rules are written, and more specifically, in what 
order?

For ban lists, if for example we wanted to allow everyone access to port 
80, except a list of bad networks, we would obviously have to put a rule 
to allow all connections on port 80 bellow our reject rules, otherwise 
iptbales would never get to the ban rules. So lets say that we have 
50,000 rules, 99% of them being reject rules, with a rule to allow port 
80 to all hosts at the bottom, and a rule to allow RELATED,ESTABLISHED 
packets at the top. When a host connects to us on port 80, iptables will 
have to go though all 50,000 rules (assuming this host doesn't match one 
of the reject rules) until it gets to the last one and decides to allow 
the packet. But then, subsequent packets will have a state of 
ESTABLISHED, and so they will match the first rule. In this case, only 
the first packet of most sessions will the firewall have to do a lot of 
work. Does this make any sense?

It seems to me, that for a reasonably powerful box, processing a large 
number of rules on what would become a small total percentage of packets 
shouldn't be a problem. Does anyone know what the real numbers are, and 
what numbers are feasible and what numbers aren't?

Re: Performance isues related to a large number of iptables rules

[Jan Du Caju]

Hi,

On Thu, Jan 13, 2005 at 05:45:34PM -0500, henry wrote:
> I am curious what is the maximum number of iptable rules that can be 
> installed in a config before performance starts to be a problem. I have 
> looked into the possibility of using firewall rules to block "bad" 
> networks, but I have been told by most people that I have asked that 
> this is bad idea.
> 
> Here are my thoughts. If a packet matches the 3rd rule, does it matter 
> if there are 100,000 rules below it? 1,000,000 rules below it? If it 
> doesn't, then does the number of allowable rules really have to do with 
> how intelligently the rules are written, and more specifically, in what 
> order?

If you want to deal with large rule sets and/or high bandwidth you should
definitely consider hipac (http://www.hipac.org)

Quoted from the hipac web-site:
 iptables, like most packet filters, uses a simple packet classification
 algorithm which traverses the rules in a chain linearly per packet
 until a matching rule is found (or not). Clearly, this approach lacks
 efficiency. As networks grow more and more complex and offer a wider
 bandwidth linear packet filtering is no longer an option if many rules
 have to be matched per packet. Higher bandwidth means more packets per
 second which leads to shorter process times per packet.

 With nf-HiPAC we offer a novel framework for packet classification
 which uses an advanced algorithm to reduce the number of memory lookups
 per packet. It is ideal for environments where large rulesets and/or
 high bandwidth networks are involved. Thereby, the iptables' semantics
 of the rules is preserved, i.e. you can construct your rules like
 you're used to. From a user's point of view there is no need to
 understand anything about the HiPAC algorithm.

At the hipac site you will find a comparison with iptables. The central
firewall of our university ( http://www.kuleuven.ac.be/english/ ) uses 
hipac and we are very pleased with it. At the moment there is only an 
implementation for a 2.4 kernel but the developers are working on a 
2.6 version :-)

Hope this helps,
Jan
--------------------------------------------------- KULeuvenNet -------
Jan.DuCaju@kuleuven.net         http://www.kuleuven.net/e_index.html
K.U.Leuven                      http://www.kuleuven.ac.be/english/
LUDIT - KULeuvenNet             http://ludit.kuleuven.be/index_en.html
de Croylaan 52A                 3001 Leuven                     Belgium
-----------------------------------------------------------------------

Re: Performance isues related to a large number of iptables rules

[R. DuFresne]

Conventional firewalling, places rules in a number of places, not just the
firewall, for one, most folks seem to have forgotten the term 'screening
router' these days.  This is the place to put rules that do things like
place anti-spoofing rules, rules for traffic that should always not be
allowed to pass <i.e. filtering out windows related protocols and ports,
etc>.  ingress and egress filters can also be placed here.

There's another problem with large rulesets beisde preformance issues,
they get to be overtly complex and hard to follow and change.  Breaking up
the rules upon a screening router<s>.  Additionally, there's no reason a
site needs merely one firewall, the DMZ can have a seperate firewall for
those servers and services open to the public that should not hit the
internal network <or these systems can have specific host based firewalls
to address their specific needs and services>.  behind the screening
router and or main firewall <which can be a sub-screening router> subnet
or switch specific  firewalls can deal with traffic specific to those
systems housed therein.

Having worked at a few places that managed firewalls for the corporate 500
world, we found that overtly large rulesets tended to turn a firewall into
merely an open routed env...and often was the result of a failure to
properly assess the sites security policy structurally.  In otherwords the
policy was often far too fluid to get a handle on ahd seemed to change
weekly or bi-weekly.

Thanks,

Ron DuFresne

On Thu, 13 Jan 2005, henry wrote:

> I am curious what is the maximum number of iptable rules that can be 
> installed in a config before performance starts to be a problem. I have 
> looked into the possibility of using firewall rules to block "bad" 
> networks, but I have been told by most people that I have asked that 
> this is bad idea.
> 
> Here are my thoughts. If a packet matches the 3rd rule, does it matter 
> if there are 100,000 rules below it? 1,000,000 rules below it? If it 
> doesn't, then does the number of allowable rules really have to do with 
> how intelligently the rules are written, and more specifically, in what 
> order?
> 
> For ban lists, if for example we wanted to allow everyone access to port 
> 80, except a list of bad networks, we would obviously have to put a rule 
> to allow all connections on port 80 bellow our reject rules, otherwise 
> iptbales would never get to the ban rules. So lets say that we have 
> 50,000 rules, 99% of them being reject rules, with a rule to allow port 
> 80 to all hosts at the bottom, and a rule to allow RELATED,ESTABLISHED 
> packets at the top. When a host connects to us on port 80, iptables will 
> have to go though all 50,000 rules (assuming this host doesn't match one 
> of the reject rules) until it gets to the last one and decides to allow 
> the packet. But then, subsequent packets will have a state of 
> ESTABLISHED, and so they will match the first rule. In this case, only 
> the first packet of most sessions will the firewall have to do a lot of 
> work. Does this make any sense?
> 
> It seems to me, that for a reasonably powerful box, processing a large 
> number of rules on what would become a small total percentage of packets 
> shouldn't be a problem. Does anyone know what the real numbers are, and 
> what numbers are feasible and what numbers aren't?
> 

-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        admin & senior security consultant:  sysinfo.com
                        http://sysinfo.com

...Love is the ultimate outlaw.  It just won't adhere to rules.
The most any of us can do is sign on as it's accomplice.  Instead
of vowing to honor and obey, maybe we should swear to aid and abet.
That would mean that security is out of the question.  The words
"make" and "stay" become inappropriate.  My love for you has no
strings attached.  I love you for free...
                        -Tom Robins <Still Life With Woodpecker>

Re: Performance isues related to a large number of iptables rules

[Samuel Jean]

henry wrote:
> Does anyone know what the real numbers are, and 
> what numbers are feasible and what numbers aren't?
> 

Not me. But if blocking large amount of subnets and host addresses is
your issue here. Only one rule is sufficient. See ipset 2.0

http://people.netfilter.org/kadlec/ipset/

Sorry if am out of topic.

Samuel