iptables and SMP performance

iptables and SMP performance

[Patrick Higgins]

My employer has a very complex dynamic bridging firewall that is
pegging a 3.2 GHz Xeon (100% of CPU0 is running in softirq). We want
to try to squeeze more performance out of our existing iptables
firewall structure, so we've been testing using the second CPU and/or
hyperthreading. Unfortunately, I've tried several different kernels
(2.4.28, 2.6.10, and a stock RHEL 3.0 update 4 system) and none put
the additional CPUs to work. This seems like it should be on a FAQ
somewhere, but I've been looking for a few days and haven't found
anything decisive yet. All I've found is some comments that using
multiple CPUs to handle the interrupts for a single interface can
cause performance-killing packet re-ordering.

I've compiled all the options for APIC IRQ balancing, but here's what
I see in /proc/interrupts (CPU0 & CPU2 are real, CPU1 & CPU3 are
hyperthreads):


           CPU0       CPU1       CPU2       CPU3
  0:    6102133    6096092    6096091    6096094    IO-APIC-edge  timer
  1:        134       1286        445       1196    IO-APIC-edge  keyboard
  2:          0          0          0          0          XT-PIC  cascade
  8:          1          0          0          0    IO-APIC-edge  rtc
 12:         41          0          0          0    IO-APIC-edge  PS/2 Mouse
 15:          2          0          0          0    IO-APIC-edge  ide1
 16:          0          0          0          0   IO-APIC-level  usb-uhci
 19:          0          0          0          0   IO-APIC-level  usb-uhci
 24:       8547      19134       6490      19697   IO-APIC-level  megaraid
 27:          2          0     121858          0   IO-APIC-level  eth5
 28:         74          0     167116          0   IO-APIC-level  eth3, eth4
 29:          2          0          0     121858   IO-APIC-level  eth2
 30:          2          0         35     121823   IO-APIC-level  eth1
 31:     448223          0          0          0   IO-APIC-level  eth0
 49:         30          0          0          0   IO-APIC-level  aic79xx
 50:         30          0          0          0   IO-APIC-level  aic79xx
NMI:          0          0          0          0
LOC:   24390999   24391009   24391009   24390987
ERR:          0
MIS:          0


The rules are being applied to eth0 in our simplified test, and it
looks like the interrupts are only being serviced by CPU0. It appears
the iptables rules are also being tested on CPU0.

Could we get better performance by balancing the eth0 interrupts
across CPUs? If not, how about balancing the testing of iptables
rules?

Note that we have extra interfaces that we could potentially use to
divide the load physically if there's some clever way to put multiple
interfaces on the same side of a bridge.

Any suggestions?

Re: iptables and SMP performance

[Jose Maria Lopez]

El lun, 24 de 01 de 2005 a las 18:42, Patrick Higgins escribió:
> My employer has a very complex dynamic bridging firewall that is
> pegging a 3.2 GHz Xeon (100% of CPU0 is running in softirq). We want
> to try to squeeze more performance out of our existing iptables
> firewall structure, so we've been testing using the second CPU and/or
> hyperthreading. Unfortunately, I've tried several different kernels
> (2.4.28, 2.6.10, and a stock RHEL 3.0 update 4 system) and none put
> the additional CPUs to work. This seems like it should be on a FAQ
> somewhere, but I've been looking for a few days and haven't found
> anything decisive yet. All I've found is some comments that using
> multiple CPUs to handle the interrupts for a single interface can
> cause performance-killing packet re-ordering.
> 
> I've compiled all the options for APIC IRQ balancing, but here's what
> I see in /proc/interrupts (CPU0 & CPU2 are real, CPU1 & CPU3 are
> hyperthreads):
> 
> 
>            CPU0       CPU1       CPU2       CPU3
>   0:    6102133    6096092    6096091    6096094    IO-APIC-edge  timer
>   1:        134       1286        445       1196    IO-APIC-edge  keyboard
>   2:          0          0          0          0          XT-PIC  cascade
>   8:          1          0          0          0    IO-APIC-edge  rtc
>  12:         41          0          0          0    IO-APIC-edge  PS/2 Mouse
>  15:          2          0          0          0    IO-APIC-edge  ide1
>  16:          0          0          0          0   IO-APIC-level  usb-uhci
>  19:          0          0          0          0   IO-APIC-level  usb-uhci
>  24:       8547      19134       6490      19697   IO-APIC-level  megaraid
>  27:          2          0     121858          0   IO-APIC-level  eth5
>  28:         74          0     167116          0   IO-APIC-level  eth3, eth4
>  29:          2          0          0     121858   IO-APIC-level  eth2
>  30:          2          0         35     121823   IO-APIC-level  eth1
>  31:     448223          0          0          0   IO-APIC-level  eth0
>  49:         30          0          0          0   IO-APIC-level  aic79xx
>  50:         30          0          0          0   IO-APIC-level  aic79xx
> NMI:          0          0          0          0
> LOC:   24390999   24391009   24391009   24390987
> ERR:          0
> MIS:          0
> 
> 
> The rules are being applied to eth0 in our simplified test, and it
> looks like the interrupts are only being serviced by CPU0. It appears
> the iptables rules are also being tested on CPU0.
> 
> Could we get better performance by balancing the eth0 interrupts
> across CPUs? If not, how about balancing the testing of iptables
> rules?
> 
> Note that we have extra interfaces that we could potentially use to
> divide the load physically if there's some clever way to put multiple
> interfaces on the same side of a bridge.
> 
> Any suggestions?

First thing you could do it's to optimize the traverse of
packets through the chains. Use rules to differentiate traffic
from different protocols (TCP/UDP/ICMP) and services. That
will improve the number of rules a packet have to traverse to
get accepted or denied. Maybe you have done this already.

Regards.

-- 
Jose Maria Lopez Hernandez
Director Tecnico de bgSEC
jkerouac@bgsec.com
bgSEC Seguridad y Consultoria de Sistemas Informaticos
http://www.bgsec.com
ESPAÑA

The only people for me are the mad ones -- the ones who are mad to live,
mad to talk, mad to be saved, desirous of everything at the same time,
the ones who never yawn or say a commonplace thing, but burn, burn, burn
like fabulous yellow Roman candles.
                -- Jack Kerouac, "On the Road"

Re: iptables and SMP performance

[Patrick Higgins]

We know that there are ways to optimize the rules themselves, but they
will mostly require new netfilter modules or at least revive some of the
nf-hipac work. The fact is, our firewall is inherently complex and will
probably always be our bottleneck. We're just looking for generic ways
to leverage hardware for short-term speed gains, and are running into a
wall.

On Mon, 2005-01-24 at 20:22 +0100, Jose Maria Lopez wrote:
> El lun, 24 de 01 de 2005 a las 18:42, Patrick Higgins escribió:
> > My employer has a very complex dynamic bridging firewall that is
> > pegging a 3.2 GHz Xeon (100% of CPU0 is running in softirq). We want
> > to try to squeeze more performance out of our existing iptables
> > firewall structure, so we've been testing using the second CPU and/or
> > hyperthreading. Unfortunately, I've tried several different kernels
> > (2.4.28, 2.6.10, and a stock RHEL 3.0 update 4 system) and none put
> > the additional CPUs to work. This seems like it should be on a FAQ
> > somewhere, but I've been looking for a few days and haven't found
> > anything decisive yet. All I've found is some comments that using
> > multiple CPUs to handle the interrupts for a single interface can
> > cause performance-killing packet re-ordering.
> > 
> > I've compiled all the options for APIC IRQ balancing, but here's what
> > I see in /proc/interrupts (CPU0 & CPU2 are real, CPU1 & CPU3 are
> > hyperthreads):
> > 
> > 
> >            CPU0       CPU1       CPU2       CPU3
> >   0:    6102133    6096092    6096091    6096094    IO-APIC-edge  timer
> >   1:        134       1286        445       1196    IO-APIC-edge  keyboard
> >   2:          0          0          0          0          XT-PIC  cascade
> >   8:          1          0          0          0    IO-APIC-edge  rtc
> >  12:         41          0          0          0    IO-APIC-edge  PS/2 Mouse
> >  15:          2          0          0          0    IO-APIC-edge  ide1
> >  16:          0          0          0          0   IO-APIC-level  usb-uhci
> >  19:          0          0          0          0   IO-APIC-level  usb-uhci
> >  24:       8547      19134       6490      19697   IO-APIC-level  megaraid
> >  27:          2          0     121858          0   IO-APIC-level  eth5
> >  28:         74          0     167116          0   IO-APIC-level  eth3, eth4
> >  29:          2          0          0     121858   IO-APIC-level  eth2
> >  30:          2          0         35     121823   IO-APIC-level  eth1
> >  31:     448223          0          0          0   IO-APIC-level  eth0
> >  49:         30          0          0          0   IO-APIC-level  aic79xx
> >  50:         30          0          0          0   IO-APIC-level  aic79xx
> > NMI:          0          0          0          0
> > LOC:   24390999   24391009   24391009   24390987
> > ERR:          0
> > MIS:          0
> > 
> > 
> > The rules are being applied to eth0 in our simplified test, and it
> > looks like the interrupts are only being serviced by CPU0. It appears
> > the iptables rules are also being tested on CPU0.
> > 
> > Could we get better performance by balancing the eth0 interrupts
> > across CPUs? If not, how about balancing the testing of iptables
> > rules?
> > 
> > Note that we have extra interfaces that we could potentially use to
> > divide the load physically if there's some clever way to put multiple
> > interfaces on the same side of a bridge.
> > 
> > Any suggestions?
> 
> First thing you could do it's to optimize the traverse of
> packets through the chains. Use rules to differentiate traffic
> from different protocols (TCP/UDP/ICMP) and services. That
> will improve the number of rules a packet have to traverse to
> get accepted or denied. Maybe you have done this already.
> 
> Regards.
> 

Re: iptables and SMP performance

[Jose Maria Lopez]

El lun, 24 de 01 de 2005 a las 22:53, Patrick Higgins escribió:
> We know that there are ways to optimize the rules themselves, but they
> will mostly require new netfilter modules or at least revive some of the
> nf-hipac work. The fact is, our firewall is inherently complex and will
> probably always be our bottleneck. We're just looking for generic ways
> to leverage hardware for short-term speed gains, and are running into a
> wall.

Reading my own post I have realize that maybe I have not explained
myself well. What I propose it's to separate the traffic in multiple
chains, so the traverse of the packets through the chains makes the
Netfilter system have to test the minimum number of rules. We've
did some test while we were designing our bastion-firewall GPL software
and it sure improves the performance of the firewall a lot.

Probably you had understand me, but I wanted to make it clear.

Regards.

-- 
Jose Maria Lopez Hernandez
Director Tecnico de bgSEC
jkerouac@bgsec.com
bgSEC Seguridad y Consultoria de Sistemas Informaticos
http://www.bgsec.com
ESPAÑA

The only people for me are the mad ones -- the ones who are mad to live,
mad to talk, mad to be saved, desirous of everything at the same time,
the ones who never yawn or say a commonplace thing, but burn, burn, burn
like fabulous yellow Roman candles.
                -- Jack Kerouac, "On the Road"

Re: iptables and SMP performance

[Jason Opperisano]

On Mon, 2005-01-24 at 12:42, Patrick Higgins wrote:
> Any suggestions?

have you considered (or are you already using) better NICs?  this is
mostly hearsay gleaned from listening to the developers on various lists
(netfilter-dev and openbsd-misc), but certain cards seem to be more
"interrupt heavy" than others.  the ones that seemed to get bashed the
most are the broadcom cards, and the ones that seem to receive the most
praise are the intel gigabit server adapters (i assume you are using
gigE adapters regardless of the speed of your links, i further assume
that these gigE cards are plugged into independent 64-bit PCI-X buses). 
also--if you're already using the intel gigE cards--are you using the
less "free-as-in-speech" but less buggy and better performing driver
from intel vs. the one that RH ships with their kernel?

just a few thoughts outside the "we need more CPU" box...as i have never
been under the impression that something like an in-kernel packet filter
could benefit from multiple CPUs (other than the fact that you can bind
all non-kernel processes to other CPU, for a minimal 10% or so
performance gain).

-j

--
"If something is to hard to do, then it's not worth doing."
	--The Simpsons

Re: iptables and SMP performance

[Patrick Higgins]

Thanks for the response. We've also found the Intel NICs are the best,
and that's what we're using. I believe the server I'm testing on has two
onboard and a quad-port expansion card (fiber, I think). Here's my
lspci:

00:00.0 Host bridge: Intel Corp. E7501 Memory Controller Hub (rev 01)
00:00.1 Class ff00: Intel Corp. E7500/E7501 Host RASUM Controller (rev 01)
00:03.0 PCI bridge: Intel Corp. E7500/E7501 Hub Interface C PCI-to-PCI Bridge (rev 01)
00:03.1 Class ff00: Intel Corp. E7500/E7501 Hub Interface C RASUM Controller (rev 01)
00:1d.0 USB Controller: Intel Corp. 82801CA/CAM USB (Hub #1) (rev 02)
00:1d.1 USB Controller: Intel Corp. 82801CA/CAM USB (Hub #2) (rev 02)
00:1e.0 PCI bridge: Intel Corp. 82801 PCI Bridge (rev 42)
00:1f.0 ISA bridge: Intel Corp. 82801CA LPC Interface Controller (rev 02)
00:1f.1 IDE interface: Intel Corp. 82801CA Ultra ATA Storage Controller (rev 02)00:1f.3 SMBus: Intel Corp. 82801CA/CAM SMBus Controller (rev 02)
01:0c.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
02:1c.0 PIC: Intel Corp. 82870P2 P64H2 I/OxAPIC (rev 04)
02:1d.0 PCI bridge: Intel Corp. 82870P2 P64H2 Hub PCI Bridge (rev 04)
02:1e.0 PIC: Intel Corp. 82870P2 P64H2 I/OxAPIC (rev 04)
02:1f.0 PCI bridge: Intel Corp. 82870P2 P64H2 Hub PCI Bridge (rev 04)
03:07.0 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (Copper) (rev 01)
03:07.1 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (Copper) (rev 01)
03:08.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID (rev 01)
03:09.0 Ethernet controller: Intel Corp. 82546GB Gigabit Ethernet Controller (rev 03)
03:09.1 Ethernet controller: Intel Corp. 82546GB Gigabit Ethernet Controller (rev 03)
03:0a.0 Ethernet controller: Intel Corp. 82546GB Gigabit Ethernet Controller (rev 03)
03:0a.1 Ethernet controller: Intel Corp. 82546GB Gigabit Ethernet Controller (rev 03)
04:07.0 SCSI storage controller: Adaptec AIC-7902 U320 (rev 03)
04:07.1 SCSI storage controller: Adaptec AIC-7902 U320 (rev 03)

Is the PCI-X bus the first number (03 in the case of the ethernet
controllers)? If so, it looks like they're all on the same bus. Can this
be changed in software? I'm not sure if I can physically move the
expansion card (2U case).

We currently ship the Intel drivers, but I've been using the stock
2.4.28 and 2.6.10 and Red Hat EL 3 kernels for my testing. My goal has
been to involve multiple CPUs somehow, and it doesn't sound like the
Intel driver will help with that. Correct?

I'm currently looking into using ethernet bonding and IRQ smp_affinity
settings to distribute the load across two interfaces and CPUs. It looks
promising--I'll gladly send the results to anyone who is interested.

On Mon, 2005-01-24 at 18:58 -0500, Jason Opperisano wrote:
> On Mon, 2005-01-24 at 12:42, Patrick Higgins wrote:
> > Any suggestions?
> 
> have you considered (or are you already using) better NICs?  this is
> mostly hearsay gleaned from listening to the developers on various lists
> (netfilter-dev and openbsd-misc), but certain cards seem to be more
> "interrupt heavy" than others.  the ones that seemed to get bashed the
> most are the broadcom cards, and the ones that seem to receive the most
> praise are the intel gigabit server adapters (i assume you are using
> gigE adapters regardless of the speed of your links, i further assume
> that these gigE cards are plugged into independent 64-bit PCI-X buses). 
> also--if you're already using the intel gigE cards--are you using the
> less "free-as-in-speech" but less buggy and better performing driver
> from intel vs. the one that RH ships with their kernel?
> 
> just a few thoughts outside the "we need more CPU" box...as i have never
> been under the impression that something like an in-kernel packet filter
> could benefit from multiple CPUs (other than the fact that you can bind
> all non-kernel processes to other CPU, for a minimal 10% or so
> performance gain).
> 
> -j
> 
> --
> "If something is to hard to do, then it's not worth doing."
> 	--The Simpsons
> 
> 

Re: iptables and SMP performance

[Donato Diaz]

I have a similar problem 100% softirq. My system have one  CPU Pentim IV 
2.4 Ghz,  five NICs, four Gig cards Broadcom and one 10/100 3Com 3C95x.
It has 800 subnetworks (alias) mixing private and public networks. The 
system does NAT and Traffic Control (HTB) only for the private networks. 
And only routing packets for the public networks. I'm not doing 
bridging, packets are forwarding through layer 3.
I'm trying to find a way to improve the performace. I find IRQ-Affinity 
and NAPI. I would like to know if you gain any performance improvement 
using smp_affinity. Could you send me the results of your test.

Thanks
Donato.


Patrick Higgins wrote:

>Thanks for the response. We've also found the Intel NICs are the best,
>and that's what we're using. I believe the server I'm testing on has two
>onboard and a quad-port expansion card (fiber, I think). Here's my
>lspci:
>
>00:00.0 Host bridge: Intel Corp. E7501 Memory Controller Hub (rev 01)
>00:00.1 Class ff00: Intel Corp. E7500/E7501 Host RASUM Controller (rev 01)
>00:03.0 PCI bridge: Intel Corp. E7500/E7501 Hub Interface C PCI-to-PCI Bridge (rev 01)
>00:03.1 Class ff00: Intel Corp. E7500/E7501 Hub Interface C RASUM Controller (rev 01)
>00:1d.0 USB Controller: Intel Corp. 82801CA/CAM USB (Hub #1) (rev 02)
>00:1d.1 USB Controller: Intel Corp. 82801CA/CAM USB (Hub #2) (rev 02)
>00:1e.0 PCI bridge: Intel Corp. 82801 PCI Bridge (rev 42)
>00:1f.0 ISA bridge: Intel Corp. 82801CA LPC Interface Controller (rev 02)
>00:1f.1 IDE interface: Intel Corp. 82801CA Ultra ATA Storage Controller (rev 02)00:1f.3 SMBus: Intel Corp. 82801CA/CAM SMBus Controller (rev 02)
>01:0c.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
>02:1c.0 PIC: Intel Corp. 82870P2 P64H2 I/OxAPIC (rev 04)
>02:1d.0 PCI bridge: Intel Corp. 82870P2 P64H2 Hub PCI Bridge (rev 04)
>02:1e.0 PIC: Intel Corp. 82870P2 P64H2 I/OxAPIC (rev 04)
>02:1f.0 PCI bridge: Intel Corp. 82870P2 P64H2 Hub PCI Bridge (rev 04)
>03:07.0 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (Copper) (rev 01)
>03:07.1 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (Copper) (rev 01)
>03:08.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID (rev 01)
>03:09.0 Ethernet controller: Intel Corp. 82546GB Gigabit Ethernet Controller (rev 03)
>03:09.1 Ethernet controller: Intel Corp. 82546GB Gigabit Ethernet Controller (rev 03)
>03:0a.0 Ethernet controller: Intel Corp. 82546GB Gigabit Ethernet Controller (rev 03)
>03:0a.1 Ethernet controller: Intel Corp. 82546GB Gigabit Ethernet Controller (rev 03)
>04:07.0 SCSI storage controller: Adaptec AIC-7902 U320 (rev 03)
>04:07.1 SCSI storage controller: Adaptec AIC-7902 U320 (rev 03)
>
>Is the PCI-X bus the first number (03 in the case of the ethernet
>controllers)? If so, it looks like they're all on the same bus. Can this
>be changed in software? I'm not sure if I can physically move the
>expansion card (2U case).
>
>We currently ship the Intel drivers, but I've been using the stock
>2.4.28 and 2.6.10 and Red Hat EL 3 kernels for my testing. My goal has
>been to involve multiple CPUs somehow, and it doesn't sound like the
>Intel driver will help with that. Correct?
>
>I'm currently looking into using ethernet bonding and IRQ smp_affinity
>settings to distribute the load across two interfaces and CPUs. It looks
>promising--I'll gladly send the results to anyone who is interested.
>
>On Mon, 2005-01-24 at 18:58 -0500, Jason Opperisano wrote:
>  
>
>>On Mon, 2005-01-24 at 12:42, Patrick Higgins wrote:
>>    
>>
>>>Any suggestions?
>>>      
>>>
>>have you considered (or are you already using) better NICs?  this is
>>mostly hearsay gleaned from listening to the developers on various lists
>>(netfilter-dev and openbsd-misc), but certain cards seem to be more
>>"interrupt heavy" than others.  the ones that seemed to get bashed the
>>most are the broadcom cards, and the ones that seem to receive the most
>>praise are the intel gigabit server adapters (i assume you are using
>>gigE adapters regardless of the speed of your links, i further assume
>>that these gigE cards are plugged into independent 64-bit PCI-X buses). 
>>also--if you're already using the intel gigE cards--are you using the
>>less "free-as-in-speech" but less buggy and better performing driver
>>from intel vs. the one that RH ships with their kernel?
>>
>>just a few thoughts outside the "we need more CPU" box...as i have never
>>been under the impression that something like an in-kernel packet filter
>>could benefit from multiple CPUs (other than the fact that you can bind
>>all non-kernel processes to other CPU, for a minimal 10% or so
>>performance gain).
>>
>>-j
>>
>>--
>>"If something is to hard to do, then it's not worth doing."
>>	--The Simpsons
>>
>>
>>    
>>
>
>
>  
>

Re: iptables and SMP performance

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 962 bytes --]

On Mon, Jan 24, 2005 at 10:42:33AM -0700, Patrick Higgins wrote:

> The rules are being applied to eth0 in our simplified test, and it
> looks like the interrupts are only being serviced by CPU0. It appears
> the iptables rules are also being tested on CPU0.

Yes, the whole network stack is traversed on the CPU that has received
the innterrupt from the nic.

> Could we get better performance by balancing the eth0 interrupts
> across CPUs? If not, how about balancing the testing of iptables
> rules?

You should set irq affinity so that e.g. eth0->cpu0, eth1->cpu1, etc.

-- 
- Harald Welte <laforge@netfilter.org>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

iptables and SMP performance

[Patrick Higgins]

My employer has a very complex dynamic bridging firewall that is
pegging a 3.2 GHz Xeon (100% of CPU0 is running in softirq). We want
to try to squeeze more performance out of our existing iptables
firewall structure, so we've been testing using the second CPU and/or
hyperthreading. Unfortunately, I've tried several different kernels
(2.4.28, 2.6.10, and a stock RHEL 3.0 update 4 system) and none put
the additional CPUs to work. This seems like it should be on a FAQ
somewhere, but I've been looking for a few days and haven't found
anything decisive yet. All I've found is some comments that using
multiple CPUs to handle the interrupts for a single interface can
cause performance-killing packet re-ordering.

I've compiled all the options for APIC IRQ balancing, but here's what
I see in /proc/interrupts (CPU0 & CPU2 are real, CPU1 & CPU3 are
hyperthreads):


           CPU0       CPU1       CPU2       CPU3
  0:    6102133    6096092    6096091    6096094    IO-APIC-edge  timer
  1:        134       1286        445       1196    IO-APIC-edge  keyboard
  2:          0          0          0          0          XT-PIC  cascade
  8:          1          0          0          0    IO-APIC-edge  rtc
 12:         41          0          0          0    IO-APIC-edge  PS/2 Mouse
 15:          2          0          0          0    IO-APIC-edge  ide1
 16:          0          0          0          0   IO-APIC-level  usb-uhci
 19:          0          0          0          0   IO-APIC-level  usb-uhci
 24:       8547      19134       6490      19697   IO-APIC-level  megaraid
 27:          2          0     121858          0   IO-APIC-level  eth5
 28:         74          0     167116          0   IO-APIC-level  eth3, eth4
 29:          2          0          0     121858   IO-APIC-level  eth2
 30:          2          0         35     121823   IO-APIC-level  eth1
 31:     448223          0          0          0   IO-APIC-level  eth0
 49:         30          0          0          0   IO-APIC-level  aic79xx
 50:         30          0          0          0   IO-APIC-level  aic79xx
NMI:          0          0          0          0
LOC:   24390999   24391009   24391009   24390987
ERR:          0
MIS:          0


The rules are being applied to eth0 in our simplified test, and it
looks like the interrupts are only being serviced by CPU0. It appears
the iptables rules are also being tested on CPU0.

Could we get better performance by balancing the eth0 interrupts
across CPUs? If not, how about balancing the testing of iptables
rules?

Note that we have extra interfaces that we could potentially use to
divide the load physically if there's some clever way to put multiple
interfaces on the same side of a bridge.

Any suggestions?