to solve the performance problem of netfilter

to solve the performance problem of netfilter

[zhengchuanbo]

I noticed that the netfilter module has a big influnce to the performance. 
I tested the throughput of our linux firewall. the result is as follows,
	linux(no netfilter)					580kpps
	with netfilter(no ip_conntrack)	    450kpps
	with ip_conntrack					295kpps
So the throughput dropped about 40% when with ip_conntrack.

I tried NOTRACK module, but the performance is not very good.
	
On our linux firewall, most of the traffic are from a trusted host on the 
DMZ server, which need not to be filtered. So I wish there could be a solution
to open a fast path to the certain server, with no conntrack nor filter.   	
Somebody suggested to install a netfilter-module that gets the packets before 
conntrack and steal the packets, and bypass the rest of iptables as well. Is there
any ideas on that?
	
thanks.
	
regards,
Jack Zheng
zhengcb@netpower.com.cn

Re: to solve the performance problem of netfilter

[tady]

Hi there,

zhengchuanbo wrote:


> I noticed that the netfilter module has a big influnce to the performance.

> I tested the throughput of our linux firewall. the result is as follows,
> 	linux(no netfilter)					580kpps
> 	with netfilter(no ip_conntrack)	    450kpps
> 	with ip_conntrack					295kpps
> So the throughput dropped about 40% when with ip_conntrack.


I can _not_ approve your results. I'm currently running a firewall
using conntrack with much more throughput than you mentioned above. I
did an (udp only for the moment) investigation on the latency
introduced by a netfilter firewall but could not find any significant
throughput decrease.  If someone is interessted have a look at

http://rnvs.informatik.uni-leipzig.de/ipp2p/

at links and latency investigation. Currently I'm using my match at a
campus link for shaping P2P traffic but could not find any drop in
throughput of the not classified traffic.

Kind regards,
Eicke.


-- 
+++ GMX - die erste Adresse für Mail, Message, More +++
Neu: Preissenkung für MMS und FreeMMS! http://www.gmx.net

Re: to solve the performance problem of netfilter

[Roberto Nibali]

Hi,

>>I tested the throughput of our linux firewall. the result is as follows,
>>	linux(no netfilter)					580kpps
>>	with netfilter(no ip_conntrack)	    450kpps
>>	with ip_conntrack					295kpps
>>So the throughput dropped about 40% when with ip_conntrack.
>  
> I can _not_ approve your results. I'm currently running a firewall
> using conntrack with much more throughput than you mentioned above. I
> did an (udp only for the moment) investigation on the latency
> introduced by a netfilter firewall but could not find any significant
> throughput decrease.  If someone is interessted have a look at
> 
> http://rnvs.informatik.uni-leipzig.de/ipp2p/

I fail to see how you had more throughput than he had. Unless I'm completely 
mistaken, he's referring to kilo packets per second (kpps) which if you would 
translate it to your measurements means that your UDP sender maxed out at 65kpps 
with 64Bytes UDP packets. This is not comparable.

So assuming my understanding is correct I'd say he's done tests on GBit Ethernet 
  _and_ I think he's using TCP too as I have myself seen those numbers (at least 
the relative performance drop with conntrack and TCP running switched GBit 
networks).

We (I) need more information from his part as there are solutions to solve his 
problems. At least I would need the average packet size, the NIC and driver 
used, the exact kernel (as there are for example TSO issues with regard to 
netfilter).

Best regards,
Roberto Nibali, ratz
-- 
echo '[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq' | dc

Re: to solve the performance problem of netfilter

[KOVACS Krisztian]

  Hi,

2003-12-17, sze keltezéssel 16:27-kor Roberto Nibali ezt írta:
> >>I tested the throughput of our linux firewall. the result is as follows,
> >>	linux(no netfilter)					580kpps
> >>	with netfilter(no ip_conntrack)	    450kpps
> >>	with ip_conntrack					295kpps
> >>So the throughput dropped about 40% when with ip_conntrack.
> >  
> > I can _not_ approve your results. I'm currently running a firewall
> > using conntrack with much more throughput than you mentioned above. I
> > did an (udp only for the moment) investigation on the latency
> > introduced by a netfilter firewall but could not find any significant
> > throughput decrease.  If someone is interessted have a look at
> > 
> > http://rnvs.informatik.uni-leipzig.de/ipp2p/
> 
> I fail to see how you had more throughput than he had. Unless I'm completely 
> mistaken, he's referring to kilo packets per second (kpps) which if you would 
> translate it to your measurements means that your UDP sender maxed out at 65kpps 
> with 64Bytes UDP packets. This is not comparable.

  Jozsef Kadlecsik has also done some benchmarking of ip_conntrack in a
Gigabit environment. The slides of his presentation prepared for the
Netfilter Workshop held in Budapest can be found at 

http://www.netfilter.org/~kadlec/workshop-2003-budapest/benchmark.sxi

  It contains tuning tips also, and is very interesting indeed. However,
I'd like to see NAT benchmarks as well. Jozsef mentioned that NAT
performance is horrible compared to conntrack, and it would be really
useful if bottlenecks could be identified. Would an OProfile benchmark
be useful?

-- 
 Regards,
   Krisztian KOVACS

Re: to solve the performance problem of netfilter

[Roberto Nibali]

Hello,

>   Jozsef Kadlecsik has also done some benchmarking of ip_conntrack in a
> Gigabit environment. The slides of his presentation prepared for the
> Netfilter Workshop held in Budapest can be found at 
> 
> http://www.netfilter.org/~kadlec/workshop-2003-budapest/benchmark.sxi

Thank you very much for the pointer. Highly interesting, indeed. I was going to 
check out what happened during that workshop one of these days ;).

>   It contains tuning tips also, and is very interesting indeed.

 From a quick skimming over the slides I get the impression that due to 
excessive tweaking of certain variables in the proc-fs a possible discussion of 
his results is highly difficult. Some of the graphs looked extremely suspicious 
due to their sharp and steep curves. I'll have to invest some time into his 
findings. But his tests are indeed extremely interesting.

> However,
> I'd like to see NAT benchmarks as well. Jozsef mentioned that NAT
> performance is horrible compared to conntrack, 

Aehhm, cough, NAT compared to conntrack??

> and it would be really
> useful if bottlenecks could be identified. Would an OProfile benchmark
> be useful? 

When I did my high speed packet filtering tests with huge numbers of rules I 
used OProfile to measure L1 -> L2 transitions and TLB miss rates dependent on 
the amount of rules netfilter has to match against.

I am not sure OProfile helps in identifying the issues which are at work when 
using conntrack, NAT and Gbit networks. I suspect certain performance penalties 
due to the size of the conntrack (-slabs), the bucket size, the functions which 
parse the table and bad hardware ;).

My first try would be to readprofile the packet filter kernel and get some 
information from therein and then continue to isolate the culprit(s).

BTW, if you need fast NAT, use iproute2: ip rule add nat ...

Best regards,
Roberto Nibali, ratz
-- 
echo '[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq' | dc

Re: to solve the performance problem of netfilter

[Henrik Nordstrom]

On Wed, 17 Dec 2003, KOVACS Krisztian wrote:

>   It contains tuning tips also, and is very interesting indeed. However,
> I'd like to see NAT benchmarks as well. Jozsef mentioned that NAT
> performance is horrible compared to conntrack, and it would be really
> useful if bottlenecks could be identified.

The bottlenecks of NAT is pretty much known I think.. the NAT operation is 
very complex as it needs to support a wide variety of different traffic 
patterns. Because of this there is some hefty locking going on on each new 
session and several "conntrack" hash entries to be able to keep track of 
the various packet variants seen for the session.

But still having benchmarks showing how much of a penalty NAT really is
would inedeed be interesting. To give interesting results wrt where the 
bottlenecks of NAT are the following set of benchmarks would be good I 
thinkg:

  - No netfilter
  - Just iptable_filter loaded (no rules)
  - Just ip_conntrack loaded
  - Just ip_conntrack + iptable_nat loaded
  - Some simple NAT rules added

Using each of the following profiles  

  * Lots of small TCP sessions such as a syn flood to a closed port always 
responding with RST (and no RST rate filtering on the targeted host)

  * A few sessions (TCP or UDP) pushing large packets

  * A few sessions (TCP or UDP) pushing small packets

  * Lots of concurrent sessions forcing the NAT to resolve addressing
overlaps, for example testing with a fixed source port from a wide range
of source IP address and using SNAT to a single fixed address.


The above will/should give a pretty clear picture of what of NAT is 
causing the performance drop.

and then ofcouse a benchmark using a "typical" pattern is also interesting 
to get a feeling of how much of an impact NAT is in the typical case not 
caring why.


Regards
Henrik

Re: to solve the performance problem of netfilter

[Jozsef Kadlecsik]

On Wed, 17 Dec 2003, Henrik Nordstrom wrote:

> But still having benchmarks showing how much of a penalty NAT really is
> would inedeed be interesting. To give interesting results wrt where the
> bottlenecks of NAT are the following set of benchmarks would be good I
> thinkg:
>
>   - No netfilter
>   - Just iptable_filter loaded (no rules)
>   - Just ip_conntrack loaded
>   - Just ip_conntrack + iptable_nat loaded
>   - Some simple NAT rules added
>
> Using each of the following profiles
>
>   * Lots of small TCP sessions such as a syn flood to a closed port always
> responding with RST (and no RST rate filtering on the targeted host)
>
>   * A few sessions (TCP or UDP) pushing large packets
>
>   * A few sessions (TCP or UDP) pushing small packets
>
>   * Lots of concurrent sessions forcing the NAT to resolve addressing
> overlaps, for example testing with a fixed source port from a wide range
> of source IP address and using SNAT to a single fixed address.

Next year January we'll repeat our testings with the same targets as you
listed. But I'd like to add architectural testings as well: sparc64
besides i386 at least.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary