adding rules - slow

adding rules - slow

[Július Bemš]

Hi,

When I need to add som rules (in my case 100) to some chain it takes various
time, depending on the number of existing rules in the chain. So when I add
100 rules to empty chain, it takes 1.8sec. If tehere is 1000 rules int the
chain, it takes 4.5sec and if there is 10000 rules it takes 21sec. 

My problem is, that i need to add this rules to chain which contains 20000
rules in short time. I think, that the way of adding rules into chain is
very innefective. 

Could someone tell me how the adding works internally? What data structures
are used? Because I need solve this and find the way how to add new rules
quickly.

Thanks for replies

Re: adding rules - slow

[John A. Sullivan III]

On Fri, 2007-07-27 at 23:30 +0200, Július Bemš wrote:
> Hi,
> 
> When I need to add som rules (in my case 100) to some chain it takes various
> time, depending on the number of existing rules in the chain. So when I add
> 100 rules to empty chain, it takes 1.8sec. If tehere is 1000 rules int the
> chain, it takes 4.5sec and if there is 10000 rules it takes 21sec. 
> 
> My problem is, that i need to add this rules to chain which contains 20000
> rules in short time. I think, that the way of adding rules into chain is
> very innefective. 
> 
> Could someone tell me how the adding works internally? What data structures
> are used? Because I need solve this and find the way how to add new rules
> quickly.
> 
> Thanks for replies
> 
> 
We face the same problem on the ISCS project
(http://iscs.sourceforge.net).  There, do to micro-perimeter network
security, we frequently generate thousands or tens of thousands of rules
with the click of a mouse.

We handle it by adding rules via iptables-restore rather than iptables.
The load time difference is remarkable.  You write your rules into files
with very similar syntax to iptables and then direct them into
iptables-restore, e.g., 

iptables-restore < /etc/PEP/rules.txt

or, if you do not want to overwrite existing rules,

iptables-restore -n < /etc/PEP/runtimerules.txt

Hope this helps - John
-- 
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com

Financially sustainable open source development
http://www.opensourcedevel.com

Re: adding rules - slow

[Tom Eastep]

[-- Attachment #1: Type: text/plain, Size: 648 bytes --]

John A. Sullivan III wrote:

> 
> We handle it by adding rules via iptables-restore rather than iptables.
> The load time difference is remarkable.  You write your rules into files
> with very similar syntax to iptables and then direct them into
> iptables-restore, e.g., 

The new Perl-based Shorewall rules compiler has adopted this same strategy.
As John says, the difference in performance is remarkable.

-Tom
-- 
Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ teastep@shorewall.net
PGP Public Key   \ https://lists.shorewall.net/teastep.pgp.key


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

RE: adding rules - slow

[Július Bemš]

Thank you very much ... good idea.
But do you know why is iptables so slow? Because I think about writing my own utility for adding new rules.

-----Original Message-----
From: netfilter-bounces@lists.netfilter.org [mailto:netfilter-bounces@lists.netfilter.org] On Behalf Of John A. Sullivan III
Sent: Friday, July 27, 2007 11:38 PM
To: Július Bemš
Cc: netfilter@lists.netfilter.org
Subject: Re: adding rules - slow

On Fri, 2007-07-27 at 23:30 +0200, Július Bemš wrote:
> Hi,
> 
> When I need to add som rules (in my case 100) to some chain it takes various
> time, depending on the number of existing rules in the chain. So when I add
> 100 rules to empty chain, it takes 1.8sec. If tehere is 1000 rules int the
> chain, it takes 4.5sec and if there is 10000 rules it takes 21sec. 
> 
> My problem is, that i need to add this rules to chain which contains 20000
> rules in short time. I think, that the way of adding rules into chain is
> very innefective. 
> 
> Could someone tell me how the adding works internally? What data structures
> are used? Because I need solve this and find the way how to add new rules
> quickly.
> 
> Thanks for replies
> 
> 
We face the same problem on the ISCS project
(http://iscs.sourceforge.net).  There, do to micro-perimeter network
security, we frequently generate thousands or tens of thousands of rules
with the click of a mouse.

We handle it by adding rules via iptables-restore rather than iptables.
The load time difference is remarkable.  You write your rules into files
with very similar syntax to iptables and then direct them into
iptables-restore, e.g., 

iptables-restore < /etc/PEP/rules.txt

or, if you do not want to overwrite existing rules,

iptables-restore -n < /etc/PEP/runtimerules.txt

Hope this helps - John
-- 
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com

Financially sustainable open source development
http://www.opensourcedevel.com

RE: adding rules - slow

[Eric Leblond]

[-- Attachment #1: Type: text/plain, Size: 538 bytes --]

Hi,

Le mercredi 01 août 2007 à 09:24 +0200, Július Bemš a écrit :
> Thank you very much ... good idea.
> But do you know why is iptables so slow? Because I think about writing my own utility for adding new rules.

It's a design problem: when you add a new rule, the whole table id
dumped in userspace, modified and sent back to kernel space.

And this take long when there is a large number of rules !

BR,
-- 
Éric Leblond, eleblond@inl.fr
Téléphone : 01 44 89 46 39, Fax : 01 44 89 45 01
INL, http://www.inl.fr

[-- Attachment #2: Ceci est une partie de message numériquement signée --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

RE: adding rules - slow

[Покотиленко Костик]

В Срд, 01/08/2007 в 09:24 +0200, Július Bemš пишет:
> Thank you very much ... good idea.
> But do you know why is iptables so slow? Because I think about writing my own utility for adding new rules.

date; I=0; while [ $I -lt 1000 ]; do I=`expr $I + 1`; iptables
>/dev/null 2>&1; done; date
Wed Aug  1 10:36:32 EEST 2007
Wed Aug  1 10:36:39 EEST 2007

As you can see just invoking iptables 1000 time from a shell takes 7
seconds on my machine. Shell is slow. Moreover when there are many rules
each iptables invocation does same checks that many times.
iptables-restore do much of the work once for all rules.

> -----Original Message-----
> From: netfilter-bounces@lists.netfilter.org [mailto:netfilter-bounces@lists.netfilter.org] On Behalf Of John A. Sullivan III
> Sent: Friday, July 27, 2007 11:38 PM
> To: Július Bemš
> Cc: netfilter@lists.netfilter.org
> Subject: Re: adding rules - slow
> 
> On Fri, 2007-07-27 at 23:30 +0200, Július Bemš wrote:
> > Hi,
> > 
> > When I need to add som rules (in my case 100) to some chain it takes various
> > time, depending on the number of existing rules in the chain. So when I add
> > 100 rules to empty chain, it takes 1.8sec. If tehere is 1000 rules int the
> > chain, it takes 4.5sec and if there is 10000 rules it takes 21sec. 
> > 
> > My problem is, that i need to add this rules to chain which contains 20000
> > rules in short time. I think, that the way of adding rules into chain is
> > very innefective. 
> > 
> > Could someone tell me how the adding works internally? What data structures
> > are used? Because I need solve this and find the way how to add new rules
> > quickly.
> > 
> > Thanks for replies
> > 
> > 
> We face the same problem on the ISCS project
> (http://iscs.sourceforge.net).  There, do to micro-perimeter network
> security, we frequently generate thousands or tens of thousands of rules
> with the click of a mouse.
> 
> We handle it by adding rules via iptables-restore rather than iptables.
> The load time difference is remarkable.  You write your rules into files
> with very similar syntax to iptables and then direct them into
> iptables-restore, e.g., 
> 
> iptables-restore < /etc/PEP/rules.txt
> 
> or, if you do not want to overwrite existing rules,
> 
> iptables-restore -n < /etc/PEP/runtimerules.txt
> 
> Hope this helps - John
-- 
Покотиленко Костик <casper@meteor.dp.ua>