IP Accounting and performance

IP Accounting and performance

[Gerald Galster]

Hi all,

Perhaps you can give me some hints on a performance problem that I'm
currently experiencing with iptables.

The situation is as follows:

I have a firewall currently running kernel 2.4.20, Celeron 1 GHz and 
512 MB of RAM
that should do traffic accounting based on single IP addresses. I 
thought it would be more
efficient to use iptables than writing a standalone application using 
pcap or the like.

I need to add filtering rules like

/sbin/iptables -A FORWARD -o eth0 -s ip_address/32
/sbin/iptables -A FORWARD -i eth0 -d ip_address/32

for about six class-C networks (this means about 3000 iptables rules).

The average throughput is around 3 Mbits / second.

After I've added those rules, the latency in ping times to a machine 
behind
the firewall increases from 30 ms to over 200 ms ...

Now my question is if I can speed those things up ... do you have any 
ideas?

Thanks in advance.

Regards,
Gerald

Re: IP Accounting and performance

[Michael]

Gerald Galster wrote:
> Hi all,
> 
> Perhaps you can give me some hints on a performance problem that I'm
> currently experiencing with iptables.
> 
> The situation is as follows:
> 
> I have a firewall currently running kernel 2.4.20, Celeron 1 GHz and 512 
> MB of RAM
> that should do traffic accounting based on single IP addresses. I 
> thought it would be more
> efficient to use iptables than writing a standalone application using 
> pcap or the like.
> 
> I need to add filtering rules like
> 
> /sbin/iptables -A FORWARD -o eth0 -s ip_address/32
> /sbin/iptables -A FORWARD -i eth0 -d ip_address/32
> 
> for about six class-C networks (this means about 3000 iptables rules).
> 
> The average throughput is around 3 Mbits / second.
> 
> After I've added those rules, the latency in ping times to a machine behind
> the firewall increases from 30 ms to over 200 ms ...
> 
> Now my question is if I can speed those things up ... do you have any 
> ideas?
> 
> Thanks in advance.
> 
> Regards,
> Gerald
> 
> 
> 

Hmm. Exactly what this site says will happen: http://www.hipac.org/ 
(See the performance tests)

As far as I know, you will not be able to overcome the limitations 
that iptables has with large rulesets. You can minimise the effect 
with carefull design , but once you have that many rules, iptables 
inevitably grinds to a halt.

nf-hipac does not currently have byte and packet counters 
unfortunately. It has occured to me many times that the most likely 
situation in which large rulesets are needed is when per IP accounting 
is being done, yet nf-hipac does not yet have counters..

Cheers,
Michael

Re: IP Accounting and performance

[Leonardo Rodrigues ( listas )]

    I'm sure more efficient designed rules can help you. Just in case, you
can try to create new rules for your C-classes, one for inbound and one for
outbound. like:

iptables -N C_class_1_inbound
iptables -A C_class_1_inbound -s 10.0.0.1
iptables -A C_class_1_inbound -s 10.0.0.2

iptables -N C_class_2_inbound
iptables -A C_class_2_inbound -s 10.0.1.1
iptables -A C_class_2_inbound -s 10.0.1.2

iptables -N C_class_1_outbound
iptables -A C_class_1_outbound -s 10.0.0.1
iptables -A C_class_1_outbound -s 10.0.0.2

iptables -N C_class_2_outbound
iptables -A C_class_2_outbound -s 10.0.1.1
iptables -A C_class_2_outbound -s 10.0.1.2


    and then using them

/sbin/iptables -A FORWARD -i eth0 -s 10.0.0.0/24 -j C_class_1_inbound
/sbin/iptables -A FORWARD -o eth0 -s 10.0.0.0/24 -j C_class_1_outbound
/sbin/iptables -A FORWARD -i eth0 -s 10.0.1.0/24 -j C_class_2_inbound
/sbin/iptables -A FORWARD -o eth0 -s 10.0.1.0/24 -j C_class_2_outbound

    I'm sure this kind of approach will make things go faster ..... you can
even have some simple bash script with some FORs doing the job of creating
all those rules for you, so you can easily adjust them.

    Sincerily,
    Leonardo Rodrigues

----- Original Message -----
From: "Gerald Galster" <Gerry@gerry.de>
To: <netfilter@lists.netfilter.org>
Cc: "Gerald Galster" <Gerry@gerry.de>
Sent: Sunday, December 08, 2002 6:18 PM
Subject: IP Accounting and performance


> Hi all,
>
> Perhaps you can give me some hints on a performance problem that I'm
> currently experiencing with iptables.
>
> The situation is as follows:
>
> I have a firewall currently running kernel 2.4.20, Celeron 1 GHz and
> 512 MB of RAM
> that should do traffic accounting based on single IP addresses. I
> thought it would be more
> efficient to use iptables than writing a standalone application using
> pcap or the like.
>
> I need to add filtering rules like
>
> /sbin/iptables -A FORWARD -o eth0 -s ip_address/32
> /sbin/iptables -A FORWARD -i eth0 -d ip_address/32
>
> for about six class-C networks (this means about 3000 iptables rules).
>
> The average throughput is around 3 Mbits / second.
>
> After I've added those rules, the latency in ping times to a machine
> behind
> the firewall increases from 30 ms to over 200 ms ...
>
> Now my question is if I can speed those things up ... do you have any
> ideas?
>
> Thanks in advance.
>
> Regards,
> Gerald
>
>

Re: IP Accounting and performance

[Joel Newkirk]

On Sunday 08 December 2002 04:18 pm, Gerald Galster wrote:
> Hi all,
>
> Perhaps you can give me some hints on a performance problem that I'm
> currently experiencing with iptables.
>
> The situation is as follows:
>
> I have a firewall currently running kernel 2.4.20, Celeron 1 GHz and
> 512 MB of RAM
> that should do traffic accounting based on single IP addresses. I
> thought it would be more
> efficient to use iptables than writing a standalone application using
> pcap or the like.

You will probably get your best speed (and usability of the resulting 
data) if you just use one of the accounting daemons that can be paired 
with the ULOG target, and use the two rules below.  Google/linux search 
on "Netfilter ULOG accounting"  
http://www.google.com/linux?num=100&restrict=linux&hl=en&lr=&ie=ISO-8859-1&q=netfilter+ULOG+accounting&btnG=Google+Search  
turns up some useful links...  That said, if you still want to do it 
directly in your ruleset, read on! :^)

> I need to add filtering rules like
>
> /sbin/iptables -A FORWARD -o eth0 -s ip_address/32
> /sbin/iptables -A FORWARD -i eth0 -d ip_address/32

Is this the complete rule?  If you don't have a target it will count and 
pass to the next rule.  Since all you said you are doing is counting 
that may seem OK, but it means that each packet is tested against each 
and every rule...  

If you can use a -j ACCEPT target for them it will count, and prevent 
testing against following rules.  This will seriously speed up testing 
for IP's that appear early in the list, but won't help later ones at 
all.  Best is to break the address range down into smaller chunks.

Something expanded from the following might work well:

/sbin/iptables -N ip01to31in
/sbin/iptables -N ip32to63out
/sbin/iptables -A ip01to31in -d 10.11.12.13 -j RETURN
/sbin/iptables -A FORWARD -i eth0 -d 10.11.12.0/27 -j ip01to31in
/sbin/iptables -A ip32to63out -s 10.11.12.33 -j RETURN
/sbin/iptables -A FORWARD -o eth0 -s 10.11.12.32/27 -j ip32to63out

This creates a user-defined chain (two in this sample fragment) to test a 
range of addresses, which can be broken down in whatever fashion gives 
you either an even distribution of rules among the userdef chains, or 
simply two chains and two FORWARD rules for each subnet.  (Obviously the 
breakdown must occur in some fashion that is easy to test, IE based on 
masking the IPs)  The latter would allow you to use the FORWARD chain 
counts to see overall numbers for each subnet, but if one subnet is 5 
machines and another is 1000 this will obviously be seriously 
unbalanced.  You can also further break down from a user defined chain 
to several more sub-chains using the same technique if this is useful.

> for about six class-C networks (this means about 3000 iptables rules).

Your end result can be twelve user-defined chains, one in/one out for 
each subnet, and twelve rules in FORWARD.

Using the RETURN target in the user-defined chains simply passes the 
packet back to the originating chain (IE, FORWARD) and skips testing 
against the remaining rules in the chain.  You can substitute ACCEPT (or 
DROP or REJECT) with no problem here as well.  RETURN shouldn't incur 
much more processing time, since the packet should only match and be 
passed to ONE userdef chain, and the main FORWARD chain should be fairly 
short this way, so it will only test against a few more rules after it 
RETURNS to FORWARD.  Using RETURN in a sub-chain will also allow you to 
perform 'real' FORWARD filtering after the counting has taken place.

Also, as someone else suggested, you can have a script that creates the 
rules in a loop, or perhaps six (one per subnet).  Fanciest and most 
efficient traversal-wise would be a balanced tree of rules, (which could 
bring you down to a max of 2-3 dozen tests for each packet, instead of 
thousands right now) the construction of which is beyond the scope of 
this reply... :^P

Whatever you do, if you handle this with iptables rules make sure you use 
the most restrictive rules possible for your situation at the upper 
levels (IE in FORWARD itself) to prevent traffic being tested that 
doesn't need to be.  If all you're interested in is HTTP to the 
internet, for example, test that in the FORWARD rules before reaching 
the subchains.

j

> The average throughput is around 3 Mbits / second.
>
> After I've added those rules, the latency in ping times to a machine
> behind
> the firewall increases from 30 ms to over 200 ms ...
>
> Now my question is if I can speed those things up ... do you have any
> ideas?
>
> Thanks in advance.
>
> Regards,
> Gerald

Re: IP Accounting and performance

[Gerald Galster]

>> I have a firewall currently running kernel 2.4.20, Celeron 1 GHz and
>> 512 MB of RAM
>> that should do traffic accounting based on single IP addresses. I
>> thought it would be more
>> efficient to use iptables than writing a standalone application using
>> pcap or the like.
>
> You will probably get your best speed (and usability of the resulting
> data) if you just use one of the accounting daemons that can be paired
> with the ULOG target, and use the two rules below.  Google/linux search
> on "Netfilter ULOG accounting"
> http://www.google.com/linux?num=100&restrict=linux&hl=en&lr=&ie=ISO- 
> 8859-1&q=netfilter+ULOG+accounting&btnG=Google+Search
> turns up some useful links...  That said, if you still want to do it
> directly in your ruleset, read on! :^)

This is exactly what I've been looking for, thanks a lot :)
I really searched the net but didn't find anything suiting my needs ...
so I ended up with a perl-Skript parsing my iptables -L -v -x -n output.

>> I need to add filtering rules like
>>
>> /sbin/iptables -A FORWARD -o eth0 -s ip_address/32
>> /sbin/iptables -A FORWARD -i eth0 -d ip_address/32
>
> Is this the complete rule?  If you don't have a target it will count  
> and
> pass to the next rule.  Since all you said you are doing is counting
> that may seem OK, but it means that each packet is tested against each
> and every rule...

I have -j ACCEPT as my target.

> Your end result can be twelve user-defined chains, one in/one out for
> each subnet, and twelve rules in FORWARD.

[...]

> Also, as someone else suggested, you can have a script that creates the
> rules in a loop, or perhaps six (one per subnet).  Fanciest and most
> efficient traversal-wise would be a balanced tree of rules, (which  
> could
> bring you down to a max of 2-3 dozen tests for each packet, instead of
> thousands right now) the construction of which is beyond the scope of
> this reply... :^P

As I suppose that configuring a ULOG-Accounting may take some time
I will try to split up the load using those b-trees in the meantime   
because
I need to get this done real soon.

Thanks to all for your quick help.

Gerald