RV: optimizations for large rule sets

RV: optimizations for large rule sets

[Alberto Díez]

hi!
 
 I am trying to make use of a large number of rules
 with iptables. 
 
 I have seen there are some optimizations referenced
 like nf-HiPAC (www.hipac.org) , iptables with
 classifiers (www.geocities.com/hamidreza_jm) which
 appearently can deal with thousands of rules (thats
 what i need).
 
 I want per flow (orig addr,dst addr, orig port, dst
 port, proto) filtering thats why i don´t think i can
 use ipsets (or can i?)
 I also would like to have the nice iptables features
 like  mangle table and counters ..
 
 I dont really understand what the conntrack does, or
 if it can somehow helpme (where is the nice
 documentation about this??)
 
 What is the netfilter preferred way to have a large
 set of rules and still do packet filtering?  are
 HiPAC, iptables with classifiers or any other
 solution
 actual?
 
 is there a howto,manual,some kind of
 documentation, all that I find about this are quite
 old (3 years?) material in the mailing list ... Is
 this problem already solved? what was the solution
 taken?
 
 
 well if you could answer any of this questions i
 would
 be very thankful
 
 Alberto Diez
 
 
      



      ______________________________________________ 
Enviado desde Correo Yahoo!
Disfruta de una bandeja de entrada más inteligente. http://es.docs.yahoo.com/mail/overview/index.html

Re: RV: optimizations for large rule sets

[Grant Taylor]

On 3/6/2008 10:06 AM, Alberto Díez wrote:
> I want per flow (orig addr, dst addr, orig port, dst port, proto) 
> filtering thats why i don´t think i can use ipsets (or can i?) I also 
> would like to have the nice iptables features like  mangle table and 
> counters ..

Ok...

> I dont really understand what the conntrack does, or if it can 
> somehow helpme (where is the nice documentation about this??)

I can't tell you exactly what it does but conntrack is used in 
maintaining state of connections.  So if you want to use the state match 
you will need conntrack.

> What is the netfilter preferred way to have a large set of rules and 
> still do packet filtering?  are HiPAC, iptables with classifiers or 
> any other solution actual?

I don't know what, or if there is, the netfilter preferred way is. 
However I think the best thing you can do is to intelligently write your 
rules.  Quite a few people just use a long list of rules that is 
processed linearly until a match is found.  I think you will have better 
luck with fewer rules that make one decision and then jump to a 
sub-chain where there are a few more rules to make another decision and 
then repeat the process.  I.e. if you are wanting to have a set of rules 
for a bunch of different clients, don't do this:

-d a.b.c.1 -p TCP --dport 22 -j ACCEPT
-d a.b.c.1 -p TCP --dport 25 -j ACCEPT
-d a.b.c.1 -p TCP --dport 80 -j ACCEPT
-d a.b.c.1 -p TCP --dport 110 -j ACCEPT
-d a.b.c.2 -p TCP --dport 22 -j ACCEPT
-d a.b.c.2 -p TCP --dport 25 -j ACCEPT
-d a.b.c.2 -p TCP --dport 80 -j ACCEPT
-d a.b.c.2 -p TCP --dport 110 -j ACCEPT
-d a.b.c.3 -p TCP --dport 22 -j ACCEPT
-d a.b.c.3 -p TCP --dport 25 -j ACCEPT
-d a.b.c.3 -p TCP --dport 80 -j ACCEPT
-d a.b.c.3 -p TCP --dport 110 -j ACCEPT

Packets for POP3 traffic to a.b.c.3 will have to traverse a lot of rules 
that it will never match before it gets to the one that it will. 
Instead write your rules like this.

-d a.b.c.1 -j IP_a.b.c.1
-d a.b.c.2 -j IP_a.b.c.2
-d a.b.c.3 -j IP_a.b.c.3

-A IP_a.b.c.1 -p TCP -j IP_a.b.c.1_TCP
-A IP_a.b.c.1 -p UDP -j IP_a.b.c.1_UDP
-A IP_a.b.c.1 -p ICMP -j IP_a.b.c.1_ICMP

-A IP_a.b.c.2 -p TCP -j IP_a.b.c.2_TCP
-A IP_a.b.c.2 -p UDP -j IP_a.b.c.2_UDP
-A IP_a.b.c.2 -p ICMP -j IP_a.b.c.2_ICMP

-A IP_a.b.c.3 -p TCP -j IP_a.b.c.3_TCP
-A IP_a.b.c.3 -p UDP -j IP_a.b.c.3_UDP
-A IP_a.b.c.3 -p ICMP -j IP_a.b.c.3_ICMP

-A IP_a.b.c.1_TCP -p TCP --dport 22 -j ACCEPT
-A IP_a.b.c.1_TCP -p TCP --dport 25 -j ACCEPT
-A IP_a.b.c.1_TCP -p TCP --dport 80 -j ACCEPT
-A IP_a.b.c.1_TCP -p TCP --dport 110 -j ACCEPT

-A IP_a.b.c.2_TCP -p TCP --dport 22 -j ACCEPT
-A IP_a.b.c.2_TCP -p TCP --dport 25 -j ACCEPT
-A IP_a.b.c.2_TCP -p TCP --dport 80 -j ACCEPT
-A IP_a.b.c.2_TCP -p TCP --dport 110 -j ACCEPT

-A IP_a.b.c.3_TCP -p TCP --dport 22 -j ACCEPT
-A IP_a.b.c.3_TCP -p TCP --dport 25 -j ACCEPT
-A IP_a.b.c.3_TCP -p TCP --dport 80 -j ACCEPT
-A IP_a.b.c.3_TCP -p TCP --dport 110 -j ACCEPT

For the same POP3 traffic to a.b.c.3 the packets now traverse 3 rules 
and jump and traverse 1 rule and then jump and then traverse 4 rules. 
As such, rules set up like this will require processing fewer rules (8) 
verses the other linear method with more rules(12).

This may be more complex on the surface, however most of this is 
repetition.  You can also have just about any rule set per stage per 
destination you want.  If you have a lot of destinations like you are 
alluding to, you can even make your earlier decisions based on 
destination IP and netmask to group a bunch of IPs in one jump and then 
subdivide after the jump while having fewer rules overall that the 
packets have to traverse.

Something else to keep in mind is that you don't want your rules to be 
overly complex.  The only reason I'm checking protocol at the final 
stage is that I have to to be able to check ports.  If it were not for 
that requirement I would not do the protocol check because the previous 
stage checked the protocol already and I know that the only way to get 
to the particular sub-chain at any given stage is to have passed through 
previous stages and rules.

> is there a howto,manual,some kind of documentation, all that I find 
> about this are quite old (3 years?) material in the mailing list ... 
> Is this problem already solved? what was the solution taken?

I'm sure there are other approaches.  However I've found that if you are 
smarter about how you go about what you want to do you will have better 
overall performance.  This same type of logic applies to IPTables and 
HiPAC and many other things too.

> well if you could answer any of this questions i would be very 
> thankful

Can we see an example of some of your rules (sanitized as need be) so 
that we can see where they might be optimized as well as what and how 
you are trying to filter.



Grant. . . .

Re: RV: optimizations for large rule sets

[G.W. Haywood]

Hi there,

On 3/6/2008 10:06 AM, Alberto Díez wrote:

> What is the netfilter preferred way to have a large set of rules and
> still do packet filtering?

I don't know if there's a "netfilter preferred way" and I don't know
what you mean by "a large set of rules".

Our iptables rules typically number about 200.  We use ipsets for
about 40,000 rules in about 30 sets, on fairly modest hardware.

People seem to run into performance issues with anything on the order
of a thousand iptables rules - obviously it will depend on the rules
and how they interact, and on the hardware, the processor load, etc.

--

73,
Ged.