SMP load balancing of softirqs

[Tore Anderson <tore.anderson@redpill-linpro.com>]

Hello,

I've got an "router on a stick" with about 6000 iptables rules.
Connection tracking is in use, including a few protocol helpers.  The
hardware is a SunFire X4100 with 4x e1000 NICs and two AMD 275 CPUs
(dual-core).  It was running a 2.6 kernel (early .20ies).

A while back I noticed a performance problem, the process ksoftirqd/1
was using 100% of its respective CPU core (#1), and there was severe
packet loss.  The forwarding rate was around 600 Mbps / 110 Kpps, so
nothing that the NIC shouldn't be able to handle.  The other CPU cores
were mostly idle.  I found out that I could move the problem around to
ksoftirqd/{0,2,3} by changing the smp_affinity parameter for eth0's IRQ,
so that the interrupts was handled by a different CPU core.  I found no
way to make the softirqs to be balanced across all four CPU cores.

The workaround I ended up with was to simply connect all four NICs and
join them together in a bonded ethernet device (LAG), making sure the
switch load-balanced incoming packets equally amongst all four LAG
members, and also use smp_affinity to make sure the intterupts for each
NIC is handled by separate CPUs.  It works well enouch - I assume I've
roughly quadrupled the maximum capacity of the router compared to using
a single NIC, even though I'm wasting switch ports since I can at most
utilise half of the interfaces' max bandwith.

Anyway, now I'm considering getting a 10G aggregation switch and connect
the router to it.  The high port cost of 10 GbE interfaces/switch ports
rules out using the same trick, so I was wondering if anyone else has
had a problem with this behaviour and found another way to deal with it,
that enables the full utilisation of a SMP system even if the router has
only one network interface?

Best regards,
-- 
Tore Anderson
Redpill Linpro AS - http://www.redpill-linpro.com/