need advice for high traffic network

[Konstantin Svist <kostya@relevad.com>]

Hi,

I have a network (LAN) consisting of (mostly) gigabit ethernet on a few 
switches. Most of the traffic is taken up by small HTTP reqests. All 
computers are running Fedora (all are core 4 through 7).

I've been having some problems with servers not being accessible and 
just last night noticed that the problems disappear when I turn off the 
firewall.
What happens is that there are lots of small HTTP requests and 
apparently at some point the firewall starts dropping or disallowing new 
connections. This has been verified with both ab (apache benchmark) and 
plain SSH - a lot of times the connections time out or take a long time 
to get established.
There are ~25 rules total (as listed by 'iptables -L')

As a temporary measure, I've turned off firewalls on more of the servers 
until I can figure out a better solution - I'd like to have a firewall 
on each server, but performance is more important.

I'l looking at nf-HiPAC right now - will probably try it some time soon. 
Beyond that, I'm out of ideas for the moment.

Is there anything else I can do?
Any other firewalls? Tricks with rearranging the rules?
etc...


Thanks!



Notes:
* Problems do not seem to be limited to any specific Fedora version or 
hardware.
* external firewalls are out of the question, unless they're really 
small & cheap: there are >40 servers in the internal network and the 
number is growing