performance issues (nat / conntrack)

["Jean-Michel Hemstedt" <jean-michel.hemstedt@alcatel.be>]

dear netdevels,

I'm doing some tcp benches on a netfilter enabled box and noticed
huge and surprising perf decrease when loading iptable_nat module. 

- ip_conntrack is of course also loading the system, but with huge memory
and a large bucket size, the problem can be solved. The big issue with
ip_conntrack are the state timeouts: it simply kill the system and drops
all the traffic with the default ones, because the ip_conntrack table
becomes quickly full, and it seems that there is no way to recover from
that  situation... Keeping unused entries (time_close) even 1 minute in
the cache is really not suitable for configurations handling (relatively)
large number of connections/s. 
o The cumulative effect should be reconsidered.
o Are there ways/plans to tune the timeouts dynamically? and what are
  the valid/invalid ranges of timeouts?
o looking at the code, it seems that one timer is started by tuple...
  wouldn't it be more efficient to have a unique periodic callback
  scanning the whole or part of the table for aged entries?

- The annoying point is iptable_nat: normally the number of entries in
the nat table is much lower than the number of entries in the conntrack
table. So even if the hash function itself could be less efficient than
the ip_conntrack one (because it takes less arguments: src+dst+proto),
the load of nat, should be much lower than the load of conntrack.
o So... why is it the opposite??
o Are there ways to tune the nat performances?

- Another (old) question: why are conntrack or nat active when there are
no rules configured (using them or not)? If not fixed it should be at
least documented... Somebody doing "iptables -t nat -L" takes the risk
of killing its system if it's already under load... In the same spirit,
iptables -F should unload all unused modules (the ip_tables modules 
doesn't hurt). Just one quick fix: replace the 'iptables' executable by
one 'iptables' script calling the exe (located somewhere else) and 
doing an rmmod at the end...

comments are welcome;


here is my test bed:

tested target:
 -kernel 2.4.18 + non_local_bind + small conntrack timeouts...
 -PIII~500MHz, RAM=256MB
 -2*100Mb/s NIC

The target acts as a forwarding gateway between a load generator client
running httperf, and an apache proxy serving cached pages. 100Mb/s NICs
and requests/response sizes insure that BW and packet collisions is not
an issue.

Since in my test, each connection is ephemeral (<10ms), i recompiled the 
kernel with very short conntrack timeouts (i.e: 1 sec for close_wait, 
and about 60 sec for established!) This was also the only way to restrict
the conntrack hash table size (given my RAM) and avoid exagerated hash
collisions. Another limitation comes from my load generator creating traffic
from one source to one destination ipa, with only source port variation 
(but given my configured hash table size and the hash function itself
it shouldn't have been an issue).

results are averages from procinfo -n10 [d]

test results:

1) target = forwarding only (no iptables module or rule)
 -  rate          : 100        conn/s (=request-response/s)
 -> CPU load      : 0%         system
 -> context       : 7          context/s
 -> irq(eth0/eth1): 0.9 / 0.9  kpps   (# of packet/sec = #irq/s)

 -  rate          : 500        conn/s
 -> CPU load      : 10%        system
 -> context       : 18->100    context/s (varying!)
 -> irq(eth0/eth1): 4.4 / 4.4  kpps

 -  rate (max)    : 1050       conn/s (max from my load generator)
 -> CPU load      : 25%        system
 -> context       : 1000       context/s
 -> irq(eth0/eth1): 10 / 10    kpps

2) (1) + insmod ip_conntrack 16384 (no rules)

 -  rate          : 100        conn/s
 -> CPU load      : 0.8%       system
 -> context       : 7          context/s
 -> irq(eth0/eth1): 0.9 / 0.9  kpps
 -> conntrack size: 970        concurrent entries

 -  rate          : 250        conn/s
 -> CPU load      : 10%        system
 -> context       : 12         context/s
 -> irq(eth0/eth1): 2.2 / 2.2  kpps
 -> conntrack size: 2390       concurrent entries

 -  rate          : 500        conn/s
 -> CPU load      : 30-70%     system  (varying)
 -> context       : 45-90      context/s
 -> irq(eth0/eth1): 4 / 4      kpps
 -> conntrack size: 4770       concurrent entries

3) (2) + iptables -t nat -L  [=iptable_nat] (no rules)
 -  rate          : 100        conn/s
 -> CPU load      : 1%         system
 -> context       : 8          context/s
 -> irq(eth0/eth1): 0.9 / 0.9  kpps
 -> conntrack size: 970        concurrent entries

 -  rate          : 250        conn/s
 -> CPU load      : 40%        system
 -> context       : 20         context/s
 -> irq(eth0/eth1): 2.2 / 2.2  kpps
 -> conntrack size: 2390       concurrent entries

 -  rate  (max)   : 420        conn/s (all failed)
 -> CPU load      : 97%        system
 -> context       : 28         context/s
 -> irq(eth0/eth1): 3.1 / 4.1  kpps
 -> conntrack size: 4050       concurrent entries

 -  rate (killing): [500]->0   conn/s (all failed)
 -> CPU load      : 100%       system (no response)
 -> context       : ?          context/s
 -> irq(eth0/eth1): ?          kpps
 -> conntrack size: 10500???   concurrent entries

other results with active rules (i.e. REDIRECT) are dependent
of the load generated by the local process handling the traffic,
and are thus not relevant (FYI: max conn/s < 200 with one process
handling the REDIRECTed traffic)

kr,
_______________________________________________________________________

-jmhe-