[NEW SOFTWARE] FIRO - Iptables optimization

[NEW SOFTWARE] FIRO - Iptables optimization

[Tihomir Katic]

Hi

I developed some firewall optimizer for iptables optimization.

FIRO is parsing output of iptables-save command and removes redundant
rules from it. Rules in each chain and table are optimized separately.
Optimization procedure continues until there are no more rules to
remove or to modify. As a result, FIRO generates new file with new set
of rules for every successful optimization procedure. Also, it logs
all actions and changes in separated files.

List of optimization procedures:
 -  Remove irrelevant rules
 -  Remove redundant "shadowed after" rules
 -  Remove redundant "shadowed before" rules
 -  Remove last rules with same action as chain
 -  Merge rules
 -  Remove redundant parameters from rules
 -  Remove redundant elements from parameters
 -  Reposition of "logging" rules in chain

This is free software, 1st published version, and I would appreciate
every download, every test, every reported bug, suggestion, etc.

Link to FIRO:
http://valeria.zesoi.fer.hr/~tkatic

Br
Tihomir

Re: [NEW SOFTWARE] FIRO - Iptables optimization

[Patrick McHardy]

On 09.06.2011 15:18, Tihomir Katic wrote:
> Hi
> 
> I developed some firewall optimizer for iptables optimization.
> 
> FIRO is parsing output of iptables-save command and removes redundant
> rules from it. Rules in each chain and table are optimized separately.
> Optimization procedure continues until there are no more rules to
> remove or to modify. As a result, FIRO generates new file with new set
> of rules for every successful optimization procedure. Also, it logs
> all actions and changes in separated files.
> 
> List of optimization procedures:
>  -  Remove irrelevant rules
>  -  Remove redundant "shadowed after" rules
>  -  Remove redundant "shadowed before" rules
>  -  Remove last rules with same action as chain
>  -  Merge rules
>  -  Remove redundant parameters from rules
>  -  Remove redundant elements from parameters
>  -  Reposition of "logging" rules in chain
> 
> This is free software, 1st published version, and I would appreciate
> every download, every test, every reported bug, suggestion, etc.
> 
> Link to FIRO:
> http://valeria.zesoi.fer.hr/~tkatic

Sounds interesting, but I can't unpack that rar, every file
fails. Please provide a tar.gz/bz2 or something similar.

Re: [NEW SOFTWARE] FIRO - Iptables optimization

[Pierre Chifflier]

On Thu, Jun 09, 2011 at 03:18:38PM +0200, Tihomir Katic wrote:
> Hi
> 
> I developed some firewall optimizer for iptables optimization.
> 
> FIRO is parsing output of iptables-save command and removes redundant
> rules from it. Rules in each chain and table are optimized separately.
> Optimization procedure continues until there are no more rules to
> remove or to modify. As a result, FIRO generates new file with new set
> of rules for every successful optimization procedure. Also, it logs
> all actions and changes in separated files.
> 
> List of optimization procedures:
>  -  Remove irrelevant rules
>  -  Remove redundant "shadowed after" rules
>  -  Remove redundant "shadowed before" rules
>  -  Remove last rules with same action as chain
>  -  Merge rules
>  -  Remove redundant parameters from rules
>  -  Remove redundant elements from parameters
>  -  Reposition of "logging" rules in chain
> 
> This is free software, 1st published version, and I would appreciate
> every download, every test, every reported bug, suggestion, etc.
> 
> Link to FIRO:
> http://valeria.zesoi.fer.hr/~tkatic

Hi,

There are some problems with the download link:
- as Patrick said, please do not use a rar file, tar.gz is better
- please package it as a directory (and not unpack all files in current
  directory)
- please provide a build system (a Makefile)

There are many build errors:
CharField.cpp: In destructor ‘cCharField::~cCharField()’:
CharField.cpp:37:12: error: ‘free’ was not declared in this scope
..
Configuration.cpp: In member function ‘void cConfiguration::LogDebugData(int, std::string)’:
Configuration.cpp:160:38: error: ‘fopen’ was not declared in this scope.

Pierre
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [NEW SOFTWARE] FIRO - Iptables optimization

[Tihomir Katic]

Hi Pierre

Thank you

You are right, my mistake with .rar (please use unrar until I change
it on web server)

Makefile should have only
# g++ -o FIRO FIRO.cpp
g++ (GCC) 3.4.6
Copyright (C) 2006

which is pretty old version, I didn't try it with newer...

Br
Tihomir



2011/6/9 Pierre Chifflier <chifflier@wzdftpd.net>:
> On Thu, Jun 09, 2011 at 03:18:38PM +0200, Tihomir Katic wrote:
>> Hi
>>
>> I developed some firewall optimizer for iptables optimization.
>>
>> FIRO is parsing output of iptables-save command and removes redundant
>> rules from it. Rules in each chain and table are optimized separately.
>> Optimization procedure continues until there are no more rules to
>> remove or to modify. As a result, FIRO generates new file with new set
>> of rules for every successful optimization procedure. Also, it logs
>> all actions and changes in separated files.
>>
>> List of optimization procedures:
>>  -  Remove irrelevant rules
>>  -  Remove redundant "shadowed after" rules
>>  -  Remove redundant "shadowed before" rules
>>  -  Remove last rules with same action as chain
>>  -  Merge rules
>>  -  Remove redundant parameters from rules
>>  -  Remove redundant elements from parameters
>>  -  Reposition of "logging" rules in chain
>>
>> This is free software, 1st published version, and I would appreciate
>> every download, every test, every reported bug, suggestion, etc.
>>
>> Link to FIRO:
>> http://valeria.zesoi.fer.hr/~tkatic
>
> Hi,
>
> There are some problems with the download link:
> - as Patrick said, please do not use a rar file, tar.gz is better
> - please package it as a directory (and not unpack all files in current
>  directory)
> - please provide a build system (a Makefile)
>
> There are many build errors:
> CharField.cpp: In destructor ‘cCharField::~cCharField()’:
> CharField.cpp:37:12: error: ‘free’ was not declared in this scope
> ..
> Configuration.cpp: In member function ‘void cConfiguration::LogDebugData(int, std::string)’:
> Configuration.cpp:160:38: error: ‘fopen’ was not declared in this scope.
>
> Pierre
>
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [NEW SOFTWARE] FIRO - Iptables optimization

[Jan Engelhardt]

On Thursday 2011-06-09 15:18, Tihomir Katic wrote:
>
>I developed some firewall optimizer for iptables optimization.
>
>FIRO is parsing output of iptables-save command and removes redundant
>rules from it. Rules in each chain and table are optimized separately.
>Optimization procedure continues until there are no more rules to
>remove or to modify. As a result, FIRO generates new file with new set
>of rules for every successful optimization procedure. Also, it logs
>all actions and changes in separated files.
>
>List of optimization procedures:
> -  Remove irrelevant rules
> -  Remove redundant "shadowed after" rules
> -  Remove redundant "shadowed before" rules
> -  Remove last rules with same action as chain
> -  Merge rules
> -  Remove redundant parameters from rules
> -  Remove redundant elements from parameters
> -  Reposition of "logging" rules in chain
>
>This is free software, 1st published version, and I would appreciate
>every download, every test, every reported bug, suggestion, etc.
>
>Link to FIRO:
>http://valeria.zesoi.fer.hr/~tkatic

Great idea. Consider 
http://jengelh.medozas.de/documents/Perfect_Ruleset.pdf for more things 
to do in rulesets.

Some of the suggestions are not a 1:1 conversion, such as replacing -s 
127.0.0.0/8 by -i lo. In this case, a warning is the most one can do in 
an automated fashion, and actually sufficient. -- in the spirit of 
checkpatch.pl.


Inbetween, I have created a git repo with Makefiles. If I feel lucky, 
I'll also fix the build in a minute.

	git://dev.medozas.de/firo

Re: [NEW SOFTWARE] FIRO - Iptables optimization

[Tihomir Katic]

Hi Jan

Thank you, I downloaded it, but I have to check how to work with GIT.
Now I am sorry I didn't upload this SW sooner.

Some explanation of it can be found in this work
http://biblio.irb.hr/prikazi-rad?&rad=310019

But it is pretty old. I am preparing document with 100 pages of
explanation, for now it is in Croatian:-( But code is commented in
English

Basic idea behind this work is to have "static optimizator of iptables
rules" or even some other with modifications. It takes only result of
iptables-save, removes redundant rules, removes unnecessary elements
e.g from multiport array, or even whole matching parameter,  merge
rules, reposition LOG rules later in file. This is not perfect,
because network traffic is probably more important.

I will try to generate some test input files, so you could see how
does it work. All optimization actions are documented in generated
files.


Also, I have been doing some tests, and in config.txt you will see:
## Optimal size of multiport - port array
port_array_size_optimal = 10

It means, it will merge 2 rules for example --dport 1:5 and --dport
21:25 into -m multiport --dports 1,2,3,4,5,21,22,23,24,25
but it will not merge them if they have more than 10 elements in array

But, based on my recent tests, it should be
port_array_size_optimal = 15

Because
rule with  --dport 1:5  takes e.g. ~0.2 us
and rule with 15 elements in multiport array lasts ~0.4us, so it is
pretty much the same

Br
Tihomir

2011/6/9 Jan Engelhardt <jengelh@medozas.de>:
> On Thursday 2011-06-09 15:18, Tihomir Katic wrote:
>>
>>I developed some firewall optimizer for iptables optimization.
>>
>>FIRO is parsing output of iptables-save command and removes redundant
>>rules from it. Rules in each chain and table are optimized separately.
>>Optimization procedure continues until there are no more rules to
>>remove or to modify. As a result, FIRO generates new file with new set
>>of rules for every successful optimization procedure. Also, it logs
>>all actions and changes in separated files.
>>
>>List of optimization procedures:
>> -  Remove irrelevant rules
>> -  Remove redundant "shadowed after" rules
>> -  Remove redundant "shadowed before" rules
>> -  Remove last rules with same action as chain
>> -  Merge rules
>> -  Remove redundant parameters from rules
>> -  Remove redundant elements from parameters
>> -  Reposition of "logging" rules in chain
>>
>>This is free software, 1st published version, and I would appreciate
>>every download, every test, every reported bug, suggestion, etc.
>>
>>Link to FIRO:
>>http://valeria.zesoi.fer.hr/~tkatic
>
> Great idea. Consider
> http://jengelh.medozas.de/documents/Perfect_Ruleset.pdf for more things
> to do in rulesets.
>
> Some of the suggestions are not a 1:1 conversion, such as replacing -s
> 127.0.0.0/8 by -i lo. In this case, a warning is the most one can do in
> an automated fashion, and actually sufficient. -- in the spirit of
> checkpatch.pl.
>
>
> Inbetween, I have created a git repo with Makefiles. If I feel lucky,
> I'll also fix the build in a minute.
>
>        git://dev.medozas.de/firo
>
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [NEW SOFTWARE] FIRO - Iptables optimization

[Jan Engelhardt]

On Thursday 2011-06-09 16:07, Tihomir Katic wrote:
>
>Also, I have been doing some tests, and in config.txt you will see:
>## Optimal size of multiport - port array
>port_array_size_optimal = 10
>
>It means, it will merge 2 rules for example --dport 1:5 and --dport
>21:25 into -m multiport --dports 1,2,3,4,5,21,22,23,24,25

This should be -m multiport --dports 1:5,21:25

>But, based on my recent tests, it should be
>port_array_size_optimal = 15

Yes, multiport can hold 15 "things".

>rule with  --dport 1:5  takes e.g. ~0.2 us
>and rule with 15 elements in multiport array lasts ~0.4us, so it is
>pretty much the same

That is purely noise. You need a lot more rules (10000 and up) to 
measure an effect.

Re: [NEW SOFTWARE] FIRO - Iptables optimization

[Tihomir Katic]

>That is purely noise. You need a lot more rules (10000 and up) to
>measure an effect.

I've been testing, list with 1000 rules, list with 10000 rules, list
with 50000 rules.
Searching for minimum time, in 100 tests, etc.

1 MIN ( 1000 single):  0.206000 us
1 MIN (1000 array):  0.264000 us

1 MIN (10000 single):  0.081400 us
1 MIN (10000 array):  0.156900 us

I couldn't restore 50000 array command (memory issue) on Iptables 1.4.4

But it can be restored on 1.2.9 (don't have right now results for that)

Br


2011/6/9 Jan Engelhardt <jengelh@medozas.de>:
> On Thursday 2011-06-09 16:07, Tihomir Katic wrote:
>>
>>Also, I have been doing some tests, and in config.txt you will see:
>>## Optimal size of multiport - port array
>>port_array_size_optimal = 10
>>
>>It means, it will merge 2 rules for example --dport 1:5 and --dport
>>21:25 into -m multiport --dports 1,2,3,4,5,21,22,23,24,25
>
> This should be -m multiport --dports 1:5,21:25
>
>>But, based on my recent tests, it should be
>>port_array_size_optimal = 15
>
> Yes, multiport can hold 15 "things".
>
>>rule with  --dport 1:5  takes e.g. ~0.2 us
>>and rule with 15 elements in multiport array lasts ~0.4us, so it is
>>pretty much the same
>
> That is purely noise. You need a lot more rules (10000 and up) to
> measure an effect.
>
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [NEW SOFTWARE] FIRO - Iptables optimization

[Tihomir Katic]

>> This should be -m multiport --dports 1:5,21:25
Yes, you are right about this, but I didn't know Iptables can work
like this (have range in "array" list), I noticed it recently, so this
in plan to add

http://valeria.zesoi.fer.hr/~tkatic/index.php?appl=fea

Plans for future development:
 -  Allow multiport parameter to contain arrays together with single numbers


2011/6/9 Tihomir Katic <tihomir.katic@gmail.com>:
>>That is purely noise. You need a lot more rules (10000 and up) to
>>measure an effect.
>
> I've been testing, list with 1000 rules, list with 10000 rules, list
> with 50000 rules.
> Searching for minimum time, in 100 tests, etc.
>
> 1 MIN ( 1000 single):  0.206000 us
> 1 MIN (1000 array):  0.264000 us
>
> 1 MIN (10000 single):  0.081400 us
> 1 MIN (10000 array):  0.156900 us
>
> I couldn't restore 50000 array command (memory issue) on Iptables 1.4.4
>
> But it can be restored on 1.2.9 (don't have right now results for that)
>
> Br
>
>
> 2011/6/9 Jan Engelhardt <jengelh@medozas.de>:
>> On Thursday 2011-06-09 16:07, Tihomir Katic wrote:
>>>
>>>Also, I have been doing some tests, and in config.txt you will see:
>>>## Optimal size of multiport - port array
>>>port_array_size_optimal = 10
>>>
>>>It means, it will merge 2 rules for example --dport 1:5 and --dport
>>>21:25 into -m multiport --dports 1,2,3,4,5,21,22,23,24,25
>>
>> This should be -m multiport --dports 1:5,21:25
>>
>>>But, based on my recent tests, it should be
>>>port_array_size_optimal = 15
>>
>> Yes, multiport can hold 15 "things".
>>
>>>rule with  --dport 1:5  takes e.g. ~0.2 us
>>>and rule with 15 elements in multiport array lasts ~0.4us, so it is
>>>pretty much the same
>>
>> That is purely noise. You need a lot more rules (10000 and up) to
>> measure an effect.
>>
>
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [NEW SOFTWARE] FIRO - Iptables optimization

[Jan Engelhardt]

On Thursday 2011-06-09 16:23, Tihomir Katic wrote:

>>That is purely noise. You need a lot more rules (10000 and up) to
>>measure an effect.
>
>I've been testing, list with 1000 rules, list with 10000 rules, list
>with 50000 rules.
>Searching for minimum time, in 100 tests, etc.
>
>1 MIN ( 1000 single):  0.206000 us
>1 MIN (1000 array):  0.264000 us
>
>1 MIN (10000 single):  0.081400 us
>1 MIN (10000 array):  0.156900 us

It seems you are not executing all rules. How else could 10k rules be 
faster than 1k? You must not use any -j.

What you need is something like:

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -p tcp -m multiport --dport 1:5,20:25
-A INPUT -p tcp -m multiport --dport 1:5,20:25
-A INPUT -p tcp -m multiport --dport 1:5,20:25
-A INPUT -p tcp -m multiport --dport 1:5,20:25
-A INPUT -p tcp -m multiport --dport 1:5,20:25
-A INPUT -p tcp -m multiport --dport 1:5,20:25
-A INPUT -p tcp -m multiport --dport 1:5,20:25
-A INPUT -p tcp -m multiport --dport 1:5,20:25
COMMIT
# Completed on Thu Jun  9 16:33:15 2011

(of course, replicating this to 1000 rules), and then sending yourself 
some packet and measure the RTT.

Re: [NEW SOFTWARE] FIRO - Iptables optimization

[Tihomir Katic]

This times are per 1 command.So total time is

 {
1 MIN ( 1000 single):  0.206000 us
1 MIN ( 1000 array):  0.264000 us
1 MIN ( 1000 range):  0.182000 us
} x 1000
=
(10000 single):  206 us
(10000 array):  264 us
(10000 range):  182 us

and
{
1 MIN (10000 single):  0.081400 us
1 MIN (10000 array):  0.156900 us
1 MIN (10000 range):  0.081900 us
}  x 10000
=
(10000 single):  814 us
(10000 array):  1569 us
(10000 range):  819 us


Of course there is some noise ..., but in general 1 rule with array
list (15 single numbers) is in worst case same as 2 rules (1 single
number).
So with current implementation, 15 single ports can be generated from
2 rules (e.g. --dport 80 and --dport 101:114)

But, I must allow ranges in multiport parameter, and than this
discussion will not be needed.

Br


2011/6/9 Jan Engelhardt <jengelh@medozas.de>:
> On Thursday 2011-06-09 16:23, Tihomir Katic wrote:
>
>>>That is purely noise. You need a lot more rules (10000 and up) to
>>>measure an effect.
>>
>>I've been testing, list with 1000 rules, list with 10000 rules, list
>>with 50000 rules.
>>Searching for minimum time, in 100 tests, etc.
>>
>>1 MIN ( 1000 single):  0.206000 us
>>1 MIN (1000 array):  0.264000 us
>>
>>1 MIN (10000 single):  0.081400 us
>>1 MIN (10000 array):  0.156900 us
>
> It seems you are not executing all rules. How else could 10k rules be
> faster than 1k? You must not use any -j.
>
> What you need is something like:
>
> *filter
> :INPUT ACCEPT [0:0]
> :FORWARD ACCEPT [0:0]
> :OUTPUT ACCEPT [0:0]
> -A INPUT -p tcp -m multiport --dport 1:5,20:25
> -A INPUT -p tcp -m multiport --dport 1:5,20:25
> -A INPUT -p tcp -m multiport --dport 1:5,20:25
> -A INPUT -p tcp -m multiport --dport 1:5,20:25
> -A INPUT -p tcp -m multiport --dport 1:5,20:25
> -A INPUT -p tcp -m multiport --dport 1:5,20:25
> -A INPUT -p tcp -m multiport --dport 1:5,20:25
> -A INPUT -p tcp -m multiport --dport 1:5,20:25
> COMMIT
> # Completed on Thu Jun  9 16:33:15 2011
>
> (of course, replicating this to 1000 rules), and then sending yourself
> some packet and measure the RTT.
>
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html