netfilter conntrack performance problems

netfilter conntrack performance problems

[Horvath Szabolcs]

Hi!

We have a firewalling-only machine, called natbox. Traffic is around
20-40 MByte/s, ~400 clients snatted to 4 public IPs, approx. 10000-40000
parallel connections.

You can see the traffic here:
http://mrtg.sth.sze.hu/14all.cgi?log=193.224.129.230&cfg=uplink.cfg

When the traffic grows above 30 MByte/sec, the sysinterrupts is around
90%.

vmstat's output at 20 MByte/sec:

gw:~# vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 3  0      0 844720   5936  23476    0    0    12    16 7887  2364  4 57 39  0
 2  0      0 844656   5936  23476    0    0     0     0 30336  3263  5 76 19  0
 0  0      0 844592   5936  23476    0    0     0     0 30102  3314  5 72 23  0
 1  0      0 844656   5936  23476    0    0     0     0 28954  4219  5 66 29  0
 0  0      0 844656   5936  23476    0    0     0     0 29902  3428  6 71 23  0
 1  0      0 844656   5944  23476    0    0     0    64 29250  4071  5 71 24  0

When the sysinterrupt is near to 100%, the machine is natting further,
but we can't manage via ssh. The interactive tasks don't work.

sysctl parameters: http://193.224.129.230/log/sysctl.txt
dmesg info: http://193.224.129.230/log/dmesg.txt
kernel configuration: http://193.224.129.230/log/config.txt
firewall conf: http://193.224.129.230/log/firewall.txt
(If I missed any importation information, please let me know!)

munin: http://193.224.129.230/munin/

from the munin graphics, I see the nic's interrupts generate the machine
load. What can we tuning to provide better performance? 

It is a P4 3.0GHz with 1 GB ram, is this computer enough to do this task?


Thanks for your reply.

Szabolcs Horvath

Re: netfilter conntrack performance problems

[Stephen J. Smoogen]

On 9/19/05, Horvath Szabolcs <hsz@sth.sze.hu> wrote:
> Hi!
> 
> We have a firewalling-only machine, called natbox. Traffic is around
> 20-40 MByte/s, ~400 clients snatted to 4 public IPs, approx. 10000-40000
> parallel connections.
>
> 
> from the munin graphics, I see the nic's interrupts generate the machine
> load. What can we tuning to provide better performance?
> 
> It is a P4 3.0GHz with 1 GB ram, is this computer enough to do this task?
> 
>

This is more dependant on what kind of network cards are on the box,
if they can use NAPI... are they PCI, PCI-X, PCI-Express, and how well
they work. there is also a dependency on the network switches and how
they interact with the network cards. [The SNAT also has an overhead
which probably generates irq's.. not sure how much though.]

A couple of parameters I have seen improve things:

1) use the same network card on both interfaces. and use a network
card that has a good NAPI history. Harald Welt had a couple listed in
his blog a while back.. I think the e1000 came out ok.

2) I think that having the cards on the same PCI-X bus can help... but
could be wrong here.. major allergies and my head isnt too clear. If
you can find a set of cards/motherboard with 2 PCI-Express slots..
that would be best.

3) Make sure that the switches are able to handle the load. We had a
problem where we thought a firewall was crap but it turned out to be
that the switch was the problem causing a lot of resends.. this
generated a lot of load.

4) Try out jumbo frames. I think we found this decreased load.. but
was dependant on the switches/routers handling it correctly.

5) Finally.. does changing this have any effect


irq moderation:  disabled

have to take more allergy medicine.. hope this helped.

-- 
Stephen J Smoogen.
CSIRT/Linux System Administrator

Re: netfilter conntrack performance problems

[KOVACS Krisztian]

  Hi,

On Monday 19 September 2005 22.34, Horvath Szabolcs wrote:
> from the munin graphics, I see the nic's interrupts generate the
> machine load. What can we tuning to provide better performance?

  As someone already recommended, try using network cards which have 
good drivers. e1000 is probably one of those cards. The keyword is NAPI 
(or some kind of driver-dependant interrupt mitigation). For example 
the sk98lin driver you're using for eth1 has some kind of interrupt 
mitigation, however it's not enabled by default - take a look at the 
driver documentation to find out how you could set it up to lower the 
interrupt rate the card is generating.

-- 
 Regards,
  Krisztian Kovacs