Re: RV: optimizations for large rule sets

[Grant Taylor <gtaylor@riverviewtech.net>]

On 3/6/2008 10:06 AM, Alberto Díez wrote:
> I want per flow (orig addr, dst addr, orig port, dst port, proto) 
> filtering thats why i don´t think i can use ipsets (or can i?) I also 
> would like to have the nice iptables features like  mangle table and 
> counters ..

Ok...

> I dont really understand what the conntrack does, or if it can 
> somehow helpme (where is the nice documentation about this??)

I can't tell you exactly what it does but conntrack is used in 
maintaining state of connections.  So if you want to use the state match 
you will need conntrack.

> What is the netfilter preferred way to have a large set of rules and 
> still do packet filtering?  are HiPAC, iptables with classifiers or 
> any other solution actual?

I don't know what, or if there is, the netfilter preferred way is. 
However I think the best thing you can do is to intelligently write your 
rules.  Quite a few people just use a long list of rules that is 
processed linearly until a match is found.  I think you will have better 
luck with fewer rules that make one decision and then jump to a 
sub-chain where there are a few more rules to make another decision and 
then repeat the process.  I.e. if you are wanting to have a set of rules 
for a bunch of different clients, don't do this:

-d a.b.c.1 -p TCP --dport 22 -j ACCEPT
-d a.b.c.1 -p TCP --dport 25 -j ACCEPT
-d a.b.c.1 -p TCP --dport 80 -j ACCEPT
-d a.b.c.1 -p TCP --dport 110 -j ACCEPT
-d a.b.c.2 -p TCP --dport 22 -j ACCEPT
-d a.b.c.2 -p TCP --dport 25 -j ACCEPT
-d a.b.c.2 -p TCP --dport 80 -j ACCEPT
-d a.b.c.2 -p TCP --dport 110 -j ACCEPT
-d a.b.c.3 -p TCP --dport 22 -j ACCEPT
-d a.b.c.3 -p TCP --dport 25 -j ACCEPT
-d a.b.c.3 -p TCP --dport 80 -j ACCEPT
-d a.b.c.3 -p TCP --dport 110 -j ACCEPT

Packets for POP3 traffic to a.b.c.3 will have to traverse a lot of rules 
that it will never match before it gets to the one that it will. 
Instead write your rules like this.

-d a.b.c.1 -j IP_a.b.c.1
-d a.b.c.2 -j IP_a.b.c.2
-d a.b.c.3 -j IP_a.b.c.3

-A IP_a.b.c.1 -p TCP -j IP_a.b.c.1_TCP
-A IP_a.b.c.1 -p UDP -j IP_a.b.c.1_UDP
-A IP_a.b.c.1 -p ICMP -j IP_a.b.c.1_ICMP

-A IP_a.b.c.2 -p TCP -j IP_a.b.c.2_TCP
-A IP_a.b.c.2 -p UDP -j IP_a.b.c.2_UDP
-A IP_a.b.c.2 -p ICMP -j IP_a.b.c.2_ICMP

-A IP_a.b.c.3 -p TCP -j IP_a.b.c.3_TCP
-A IP_a.b.c.3 -p UDP -j IP_a.b.c.3_UDP
-A IP_a.b.c.3 -p ICMP -j IP_a.b.c.3_ICMP

-A IP_a.b.c.1_TCP -p TCP --dport 22 -j ACCEPT
-A IP_a.b.c.1_TCP -p TCP --dport 25 -j ACCEPT
-A IP_a.b.c.1_TCP -p TCP --dport 80 -j ACCEPT
-A IP_a.b.c.1_TCP -p TCP --dport 110 -j ACCEPT

-A IP_a.b.c.2_TCP -p TCP --dport 22 -j ACCEPT
-A IP_a.b.c.2_TCP -p TCP --dport 25 -j ACCEPT
-A IP_a.b.c.2_TCP -p TCP --dport 80 -j ACCEPT
-A IP_a.b.c.2_TCP -p TCP --dport 110 -j ACCEPT

-A IP_a.b.c.3_TCP -p TCP --dport 22 -j ACCEPT
-A IP_a.b.c.3_TCP -p TCP --dport 25 -j ACCEPT
-A IP_a.b.c.3_TCP -p TCP --dport 80 -j ACCEPT
-A IP_a.b.c.3_TCP -p TCP --dport 110 -j ACCEPT

For the same POP3 traffic to a.b.c.3 the packets now traverse 3 rules 
and jump and traverse 1 rule and then jump and then traverse 4 rules. 
As such, rules set up like this will require processing fewer rules (8) 
verses the other linear method with more rules(12).

This may be more complex on the surface, however most of this is 
repetition.  You can also have just about any rule set per stage per 
destination you want.  If you have a lot of destinations like you are 
alluding to, you can even make your earlier decisions based on 
destination IP and netmask to group a bunch of IPs in one jump and then 
subdivide after the jump while having fewer rules overall that the 
packets have to traverse.

Something else to keep in mind is that you don't want your rules to be 
overly complex.  The only reason I'm checking protocol at the final 
stage is that I have to to be able to check ports.  If it were not for 
that requirement I would not do the protocol check because the previous 
stage checked the protocol already and I know that the only way to get 
to the particular sub-chain at any given stage is to have passed through 
previous stages and rules.

> is there a howto,manual,some kind of documentation, all that I find 
> about this are quite old (3 years?) material in the mailing list ... 
> Is this problem already solved? what was the solution taken?

I'm sure there are other approaches.  However I've found that if you are 
smarter about how you go about what you want to do you will have better 
overall performance.  This same type of logic applies to IPTables and 
HiPAC and many other things too.

> well if you could answer any of this questions i would be very 
> thankful

Can we see an example of some of your rules (sanitized as need be) so 
that we can see where they might be optimized as well as what and how 
you are trying to filter.



Grant. . . .