Re: [LARTC] Performance problem on a linux bridge used for shaping.

[Simon Lodal <simonl@parknet.dk>]

I have similar hardware, load and trouble.

Interrupts are only sent to one cpu, instead of all of them, because that was 
only overhead. I think the default was changed somewhere around 2.6.10 
or .12, but I have forgotten the url.

There is a CONFIG_IRQBALANCE option in the kernel, but last time I checked 
(2.6.16) it did not work very well; almost never does anything. So I have 
turned it off. I use the userspace irqbalance daemon that periodically sets 
the smp_affinity's, with the effect that ingoing and outgoing traffic are 
handled by each their cpu (assuming that no other interrupts or processes are 
significant). That helps some. But we only shape in one direction, and it can 
not help spread the shaping load between the CPU's.

There is also an acpi_irq_balance kernel parameter (not related to the kernel 
irq balancer), which apparently uses the APIC to do interrupt round-robin. It 
worked surprisingly well (perfect, actually) on an old dual celeron that I 
tested; the network interrupts are spread nicely and evenly across the cpu's. 
It is probably very chipset dependent, and I have not yet tested it on the 
firewalls.

But as I understand from this:
http://vger.kernel.org/~davem/cgi-bin/blog.cgi/2006/09/14#netconf2006_day2
it might not even be an advantage, since the current shaping code would just 
make the cpu's step on each others toes.


Regards,
Simon


On Saturday 16 September 2006 20:39, Alexandru Dragoi wrote:
> Hello,
>
> Here is the situation. There is a machine with 3 intel gigabit card, 2
> of them on PCI-X and in bridge, the 3rd is used only for management
> access. The machine is a dual Xeon 2.8GHz with HT. With 2.6.8 kernel
> from debian (testing) and htb with u32 on, i usually get about 30-40%
> software interrupts on CPU0 and CPU2, and without htb and u32, 10% less.
> Now, if I boot with 2.6.17.9 kernel, first all irqs are on same CPU. I
> managed with smp_afinity to "move" irq of one card to a different CPU.
> In these circumstances, I have about 20% or a little less soft
> interrupts on each CPU without shaping, but about 60-70% os soft
> interrupts with shapping, and sometimes there is packet loss, also
> dropped packets are shown on ifconfig. The htb script is same. I have
> u32 performance counters enabled on u32 in 2.6.17.9. I also have NAPI,
> of course. I can't think on anything else that can cause the problem,
> but seem to be something in the kernel. Here is the output of lspci:
>
> # lspci
> 0000:00:00.0 Host bridge: Intel Corporation E7320 Memory Controller Hub
> (rev 0a)
> 0000:00:00.1 ff00: Intel Corporation E7320 Error Reporting Registers
> (rev 0a)
> 0000:00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express
> Port A (rev 0a)
> 0000:00:03.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express
> Port A1 (rev 0a)
> 0000:00:1c.0 PCI bridge: Intel Corporation 6300ESB 64-bit PCI-X Bridge
> (rev 02)
> 0000:00:1d.0 USB Controller: Intel Corporation 6300ESB USB Universal
> Host Controller (rev 02)
> 0000:00:1d.1 USB Controller: Intel Corporation 6300ESB USB Universal
> Host Controller (rev 02)
> 0000:00:1d.4 System peripheral: Intel Corporation 6300ESB Watchdog Timer
> (rev 02)
> 0000:00:1d.5 PIC: Intel Corporation 6300ESB I/O Advanced Programmable
> Interrupt Controller (rev 02)
> 0000:00:1d.7 USB Controller: Intel Corporation 6300ESB USB2 Enhanced
> Host Controller (rev 02)
> 0000:00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 0a)
> 0000:00:1f.0 ISA bridge: Intel Corporation 6300ESB LPC Interface
> Controller (rev 02)
> 0000:00:1f.2 IDE interface: Intel Corporation 6300ESB SATA Storage
> Controller (rev 02)
> 0000:00:1f.3 SMBus: Intel Corporation 6300ESB SMBus Controller (rev 02)
> 0000:03:02.0 Ethernet controller: Intel Corporation 82544EI Gigabit
> Ethernet Controller (Copper) (rev 02)
> 0000:03:03.0 Ethernet controller: Intel Corporation 82544EI Gigabit
> Ethernet Controller (Copper) (rev 02)
> 0000:04:02.0 VGA compatible controller: ATI Technologies Inc Rage XL
> (rev 27)
> 0000:04:03.0 Ethernet controller: Intel Corporation 82541GI Gigabit
> Ethernet Controller (rev 05)
>
> The traffic is somewhere at 40kpps of traffic and 120mbit up,
> 120mbit/down. Suggestions about better hardware and kernel, or links
> with docs about these are really welcomed.
>
> Bye
>
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc