iptables rules in comparable form

iptables rules in comparable form

[Radek Kanovsky]

Hello,

is there some way to get iptables rules in some normalized form?
What I mean is normalized or canonical form that is convenient
for rule comparison. For example following rules are internaly
equivalent although writen differently and I not aware of any
utility that could told me that they are the same:

    iptables -A SSH -s 1.2.3.4 -p tcp --dport ssh
    iptables -A SSH -s 1.2.3.4/32 -p tcp --dport ssh
    iptables -A SSH -s w1.something.com -p tcp --dport ssh
    iptables -A SSH -s w1.something.com -p tcp --dport 22
    iptables -A SSH -s w1.something.com/32 -p tcp -m tcp --dport 22
    iptables -A SSH -s 1.2.3.4 -d 0.0.0.0/0 -p tcp -m tcp --dport 22

When I want update rules on firewall or router with thousands of rules
I want to do it incrementaly. Reloading whole iptables on small change
every ten minutes is not acceptable because it takes very long time
and resets counters defined for accounting purposes. But incremental
solution requires some comparable rule form so machine can decide
which rule already exists, which is new and which should be deleted.

I have prototype in python that does such normalization and is able
to output "patch" for existing rules according to given new rules.
Script generates -N/-X/-F/-P/-I/-D/-A rules via standard python difflib
and is pretty effective and simle (90 lines) but requires rules in
comparable form and this is the harder part of my problem (2662 lines).

I am not sure if I go right way because this concept is a bit fragile.
Iptables can have plugins not known to this script, every host can
have different /etc/services, /etc/protocols used by ipatables, there
are bugs in iptables stdout, etc. How would you solve this?

Regards

Radek Kanovsky

Re: iptables rules in comparable form

[Jan Engelhardt]

On Tuesday 2010-06-01 10:10, Radek Kanovsky wrote:

>Hello,
>
>is there some way to get iptables rules in some normalized form?
>What I mean is normalized or canonical form that is convenient
>for rule comparison.

Load em up, then use iptables-save and/or iptables -S.

>When I want update rules on firewall or router with thousands of rules
>I want to do it incrementaly. Reloading whole iptables on small change
>every ten minutes is not acceptable because it takes very long time

That's because you are not using iptables-restore to do an O(n) reload. 

Calling iptables again and again is going to replace tables over and 
over.

>and resets counters defined for accounting purposes. But incremental
>solution requires some comparable rule form so machine can decide
>which rule already exists, which is new and which should be deleted.
>
>I have prototype in python that does such normalization and is able
>to output "patch" for existing rules according to given new rules.
>Script generates -N/-X/-F/-P/-I/-D/-A rules via standard python difflib
>and is pretty effective and simle (90 lines) but requires rules in
>comparable form and this is the harder part of my problem (2662 lines).
>
>I am not sure if I go right way because this concept is a bit fragile.
>Iptables can have plugins not known to this script, every host can
>have different /etc/services, /etc/protocols used by ipatables, there
>are bugs in iptables stdout, etc. How would you solve this?

I'd analyze why I would even need 1000 rules, and if so, why they should 
change behind my back.

Re: iptables rules in comparable form

[Mart Frauenlob]

On 01.06.2010 10:50, jengelh@medozas.de wrote:
> On Tuesday 2010-06-01 10:10, Radek Kanovsky wrote:
> 
>> Hello,
>>
>> is there some way to get iptables rules in some normalized form?
>> What I mean is normalized or canonical form that is convenient
>> for rule comparison.
> 
> Load em up, then use iptables-save and/or iptables -S.
> 
>> When I want update rules on firewall or router with thousands of rules
>> I want to do it incrementaly. Reloading whole iptables on small change
>> every ten minutes is not acceptable because it takes very long time
> 
> That's because you are not using iptables-restore to do an O(n) reload. 
> 
> Calling iptables again and again is going to replace tables over and 
> over.
> 
>> and resets counters defined for accounting purposes. But incremental
>> solution requires some comparable rule form so machine can decide
>> which rule already exists, which is new and which should be deleted.


Besides there's also iptables-xml (not sure when it came up), which can
give you a unified output. Reading man iptables-xml the way back is
possible like:
Conversion from XML to iptables-save format may be done using the
iptables.xslt script and xsltproc, or a custom program using libxsltproc
or similar;  in this fashion:

       xsltproc iptables.xslt my-iptables.xml | iptables-restore

>>
>> I have prototype in python that does such normalization and is able
>> to output "patch" for existing rules according to given new rules.
>> Script generates -N/-X/-F/-P/-I/-D/-A rules via standard python difflib
>> and is pretty effective and simle (90 lines) but requires rules in
>> comparable form and this is the harder part of my problem (2662 lines).

I bet python has loads of xml libs.

[...]

Best regards

Mart

Re: iptables rules in comparable form

[Radek Kanovsky]

On Tue, Jun 01, 2010 at 11:18:55AM +0200, Mart Frauenlob wrote:

> Besides there's also iptables-xml (not sure when it came up), which can
> give you a unified output. Reading man iptables-xml the way back is
> possible like:
> Conversion from XML to iptables-save format may be done using the
> iptables.xslt script and xsltproc, or a custom program using libxsltproc
> or similar;  in this fashion:
> 
>        xsltproc iptables.xslt my-iptables.xml | iptables-restore
> 
> >>
> >> I have prototype in python that does such normalization and is able
> >> to output "patch" for existing rules according to given new rules.
> >> Script generates -N/-X/-F/-P/-I/-D/-A rules via standard python difflib
> >> and is pretty effective and simle (90 lines) but requires rules in
> >> comparable form and this is the harder part of my problem (2662 lines).
> 
> I bet python has loads of xml libs.

I should clarify some facts about my conditions.

Whole iptables ruleset is represented by few files in /etc. Some of them
are generated, some of them are hand written. I am able to feed /etc
rules to iptables-restore or execute them as shell script. This is
trivial. Although iptables-restore is faster than executing iptables in
shell script, it is still very slow sometimes. Changes in /etc ruleset
are small but frequent. But primarily both solutions reset couters if
used and it is not good for me now. So I ended with script that does
incremental updates.

My script takes iptables-save output as first argument and desired
ruleset declared in /etc files as second argument (but generally any two
inputs). In first step it converts both inputs to comparable form and
builds ruleset in internal representation. Then comes iptables_diff
method, that is able to generate two rules

    iptables -D SSH 10
    iptables -I SSH 10 -j REJECT -s 2.3.2.3

if rule 10 in chain SSH differs in first and second ruleset. It mostly
works but it needs constant maintenance because of changes in iptables
itself so I am not satisfied. I believe that it works as I intended
because this utility dramaticaly lower reload times of firewalls on our
busy routers. As I said, changes are small. Restoring 10000 rules via
iptables-restore is often more slow than my python processing and
executing two iptables commands.

I know iptables-xml. It is mostly wrapper around iptables-save output.
It doesn't help much, because it doesnt convert '--dport ssh' 
to '--dport 22' during processing /etc/ files. First form I have
in /etc (readability of hand written files should be preserved),
second form I get from kernel via iptables-save. How can I compare them?
I hope that it is more clear now.


Thanks
Radek Kanovsky

Re: iptables rules in comparable form

[Jan Engelhardt]

On Tuesday 2010-06-01 13:25, Radek Kanovsky wrote:
>
>Whole iptables ruleset is represented by few files in /etc. Some of them
>are generated, some of them are hand written. I am able to feed /etc
>rules to iptables-restore or execute them as shell script. This is
>trivial. Although iptables-restore is faster than executing iptables in
>shell script, it is still very slow sometimes.
>
>Changes in /etc ruleset are small but frequent. But primarily both
>solutions reset couters if used and it is not good for me now. So I
>ended with script that does incremental updates.

How slow are we talking about? restore is never slower than
iptables - ever, because, like iptables, it does one table replace
operation per invocation of either binary. Your "incremental update"
is in fact none, because tables are always replaced wholesome.

Re: iptables rules in comparable form

[Radek Kanovsky]

On Tue, Jun 01, 2010 at 01:56:37PM +0200, Jan Engelhardt wrote:

> >Whole iptables ruleset is represented by few files in /etc. Some of them
> >are generated, some of them are hand written. I am able to feed /etc
> >rules to iptables-restore or execute them as shell script. This is
> >trivial. Although iptables-restore is faster than executing iptables in
> >shell script, it is still very slow sometimes.
> >
> >Changes in /etc ruleset are small but frequent. But primarily both
> >solutions reset couters if used and it is not good for me now. So I
> >ended with script that does incremental updates.
> 
> How slow are we talking about? restore is never slower than
> iptables - ever, because, like iptables, it does one table replace
> operation per invocation of either binary. Your "incremental update"
> is in fact none, because tables are always replaced wholesome.

I know that iptables-restore with 10000 rules on input is faster than
10000 sequential "iptables -A ..." rules. It is obvious. But anyway it
is sometimes slower than my one "iptables -D" command followed by one
"iptables -A" command that both together reflect one particular change
in ruleset that I am able to recognize with rule comparison. Especially
under higher load. Reason is not obvious. I was forced also implement my
own locking and memoization around iptables-save that prevents its
concurrent invocation from accounting process. The accounting is simple -
it transfers output to accounting machinge. That caused troubles in the
past when I used to see hanging iptables-save/-restore processes. Maybe
some locking issues? It is hard to reproduce or debug for me. We use
Debian Etch and Lenny mainly without any kernel or netfilter customization.

I wanted to discuss this theme before I will start to prepare incremental
updates for tc configuration. It is even worse as restart takes routinely
1 or 2 minutes and there is no tc-restore mechanism and tc output
is totally different from its input.

Also note that my problem is not performance of packet filtering.
It is OK.

Regards

Radek Kanovsky

Re: iptables rules in comparable form

[Jan Engelhardt]

On Tuesday 2010-06-01 18:03, Radek Kanovsky wrote:
>
>I know that iptables-restore with 10000 rules on input is faster than
>10000 sequential "iptables -A ..." rules. It is obvious. But anyway it
>is sometimes slower than my one "iptables -D" command followed by one
>"iptables -A" command that both together reflect one particular change
>in ruleset that I am able to recognize with rule comparison. Especially
>under higher load. Reason is not obvious.

Look obvious to me. Under load, the execution time of a process can
arbitrarily be delayed by other processes. That isn't really a
surprise. The delay itself is even a function of that factor.

>I was forced also implement my
>own locking and memoization around iptables-save that prevents its
>concurrent invocation from accounting process.

Indeed, if iptables has to race with another instance of itself,
when replacing a table, it has to retry whole the operation.

Re: iptables rules in comparable form

[Radek Kanovsky]

On Tue, Jun 01, 2010 at 08:19:35PM +0200, Jan Engelhardt wrote:

> >I know that iptables-restore with 10000 rules on input is faster than
> >10000 sequential "iptables -A ..." rules. It is obvious. But anyway it
> >is sometimes slower than my one "iptables -D" command followed by one
> >"iptables -A" command that both together reflect one particular change
> >in ruleset that I am able to recognize with rule comparison. Especially
> >under higher load. Reason is not obvious.
> 
> Look obvious to me. Under load, the execution time of a process can
> arbitrarily be delayed by other processes. That isn't really a
> surprise. The delay itself is even a function of that factor.

I didn't want to say it but it would be better described as some
kind of deadlock if system is in good health but iptables-save that
started hour before doesn't mean to finish.

Radek Kanovsky

Re: iptables rules in comparable form

[Radek Kanovsky]

On Tue, Jun 01, 2010 at 01:56:37PM +0200, Jan Engelhardt wrote:

> >Changes in /etc ruleset are small but frequent. But primarily both
> >solutions reset couters if used and it is not good for me now. So I
> >ended with script that does incremental updates.
> 
> How slow are we talking about? restore is never slower than
> iptables - ever, because, like iptables, it does one table replace
> operation per invocation of either binary. Your "incremental update"
> is in fact none, because tables are always replaced wholesome.

I take counters snapshot every minute for accounting. I can modify my
system a such way that changes are made immediately after this snapshot
phase via iptables-restore with reseting all counters in time very close
to the last read minimizing outage in accounting. But I can't rely on it
if restore phase takes from 1 seconds to 2 minutes. It would lead to
totally unreliable accounting data and more complicated system. Thats
why I came with incremental updates that doesn't touch unchanged rules.

Regards

Radek Kanovsky

Re: iptables rules in comparable form

[Jan Engelhardt]

On Tuesday 2010-06-01 20:01, Radek Kanovsky wrote:
>On Tue, Jun 01, 2010 at 01:56:37PM +0200, Jan Engelhardt wrote:
>
>> >Changes in /etc ruleset are small but frequent. But primarily both
>> >solutions reset couters if used and it is not good for me now. So I
>> >ended with script that does incremental updates.
>> 
>> How slow are we talking about? restore is never slower than
>> iptables - ever, because, like iptables, it does one table replace
>> operation per invocation of either binary. Your "incremental update"
>> is in fact none, because tables are always replaced wholesome.
>
>I take counters snapshot every minute for accounting. I can modify my
>system a such way that changes are made immediately after this snapshot
>phase via iptables-restore with reseting all counters in time very close
>to the last read minimizing outage in accounting. But I can't rely on it
>if restore phase takes from 1 seconds to 2 minutes. It would lead to
>totally unreliable accounting data and more complicated system.

Sounds like you need xt_quota2. As its counters are independent of
rules when given names, they can never get set back to a value
less than what they were.

>why I came with incremental updates that doesn't touch unchanged rules.

As I said before, there is no concept of unchanged rules.

When you iptables -A, the entire ruleset is fetched from the kernel,
then modified, and finally reinserted - even when having only
added a single rule.

Re: iptables rules in comparable form

[Radek Kanovsky]

On Tue, Jun 01, 2010 at 08:26:30PM +0200, Jan Engelhardt wrote:

> Sounds like you need xt_quota2. As its counters are independent of
> rules when given names, they can never get set back to a value
> less than what they were.

I wanted to avoid any nonstandard packages but this looks promissing.
I will take a look. Thanks.

> As I said before, there is no concept of unchanged rules.
> 
> When you iptables -A, the entire ruleset is fetched from the kernel,
> then modified, and finally reinserted - even when having only
> added a single rule.

But I have scalability problems even if there is declared O(N)
complexity of iptables-restore. There is a really big difference
if counters are reset at 9:14:01 or at 9:14:53. I am not sure
what COMMIT during restoration exactly do but can't it be 
used for tuning in such cases?

Radek Kanovsky

Re: iptables rules in comparable form

[Pieter Smit]

Radek, your problem is getting the rules into a standard format, where
you can match them to existing rules.

Would a solution/workaround not be to
1. create a dummy test chain/table not being used.
2. add new rules to the chain/table  (have to modify them slightly) 1 by 1
3. retrieve them in a standard form as provided by iptables-save -t test
4. spend time in python routine to match and decide what to remove/update.
5. make single insertion/replacement if needed.
6. clear test table and start with next rule.


On Tue, Jun 1, 2010 at 9:36 PM, Radek Kanovsky <rk@dat.cz> wrote:
> On Tue, Jun 01, 2010 at 08:26:30PM +0200, Jan Engelhardt wrote:
>
>> Sounds like you need xt_quota2. As its counters are independent of
>> rules when given names, they can never get set back to a value
>> less than what they were.
>
> I wanted to avoid any nonstandard packages but this looks promissing.
> I will take a look. Thanks.
>
>> As I said before, there is no concept of unchanged rules.
>>
>> When you iptables -A, the entire ruleset is fetched from the kernel,
>> then modified, and finally reinserted - even when having only
>> added a single rule.
>
> But I have scalability problems even if there is declared O(N)
> complexity of iptables-restore. There is a really big difference
> if counters are reset at 9:14:01 or at 9:14:53. I am not sure
> what COMMIT during restoration exactly do but can't it be
> used for tuning in such cases?
>
> Radek Kanovsky
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: iptables rules in comparable form

[Radek Kanovsky]

On Tue, Jun 01, 2010 at 10:29:10PM +0200, Pieter Smit wrote:

> Radek, your problem is getting the rules into a standard format, where
> you can match them to existing rules.
> 
> Would a solution/workaround not be to
> 1. create a dummy test chain/table not being used.
> 2. add new rules to the chain/table  (have to modify them slightly) 1 by 1
> 3. retrieve them in a standard form as provided by iptables-save -t test
> 4. spend time in python routine to match and decide what to remove/update.
> 5. make single insertion/replacement if needed.
> 6. clear test table and start with next rule.

Its not viable because I need also minimize the number of iptables operations.

Another solution could be to dynamicaly load iptables extensions
modules into my script and use somehow their parameter processing
code if possible.

Regards

Radek Kanovsky

Re: iptables rules in comparable form

[Mart Frauenlob]

On 01.06.2010 13:26, rk@dat.cz wrote:

> I should clarify some facts about my conditions.
> 
> Whole iptables ruleset is represented by few files in /etc. Some of them
> are generated, some of them are hand written. I am able to feed /etc
> rules to iptables-restore or execute them as shell script. This is
> trivial. Although iptables-restore is faster than executing iptables in
> shell script, it is still very slow sometimes. Changes in /etc ruleset
> are small but frequent. But primarily both solutions reset couters if
> used and it is not good for me now. So I ended with script that does
> incremental updates.

iptables[-save/restore] have a -c switch to save/restore counters.


> 
> My script takes iptables-save output as first argument and desired
> ruleset declared in /etc files as second argument (but generally any two
> inputs). In first step it converts both inputs to comparable form and
> builds ruleset in internal representation. Then comes iptables_diff
> method, that is able to generate two rules
> 
>     iptables -D SSH 10
>     iptables -I SSH 10 -j REJECT -s 2.3.2.3
> 

you could use -R here.
if many rules look like that, use ipset?

> if rule 10 in chain SSH differs in first and second ruleset. It mostly
> works but it needs constant maintenance because of changes in iptables
> itself so I am not satisfied. I believe that it works as I intended
> because this utility dramaticaly lower reload times of firewalls on our
> busy routers. As I said, changes are small. Restoring 10000 rules via
> iptables-restore is often more slow than my python processing and
> executing two iptables commands.

using the -n switch of iptables-restore you might be able to create
'smaller' changes.

Best regards

Mart

Re: iptables rules in comparable form

[Radek Kanovsky]

On Tue, Jun 01, 2010 at 03:27:58PM +0200, Mart Frauenlob wrote:

> > are small but frequent. But primarily both solutions reset couters if
> > used and it is not good for me now. So I ended with script that does
> > incremental updates.
> 
> iptables[-save/restore] have a -c switch to save/restore counters.

There is some ISP DB that produces XML config for router. XML config
is transformed to iptables-restore rules and stored in some /etc file.
There are also some hand writen rules in /etc files controlled by admins.
I take all these files and prepare one big file which can be feed to
iptables-restore. Obviously without counters. So I take snapshot
of current ruleset via "iptables-save -c" command. Now I have
two ruleset but I am not able to compare them because some writes
destination ports with service names some with port numbers.
Some versions of iptables-save produces "-j MARK 0x2f" some
"-j MARK --set-mark 0x2f". So "iptables-save -c" is useles for me
unless I have some normalization utility that transforms all rules
to some common comparable form. I have it and I am doing it right this
way. I am asking if someone doesn't do this already or if there is some
more clever solution.

Regards

Radek Kanovsky