* Re: [LARTC] Performance problem on a linux bridge used for shaping.
2006-09-16 18:39 [LARTC] Performance problem on a linux bridge used for shaping Alexandru Dragoi
@ 2006-09-16 21:17 ` Simon Lodal
2006-09-17 16:33 ` Alexandru Dragoi
1 sibling, 0 replies; 3+ messages in thread
From: Simon Lodal @ 2006-09-16 21:17 UTC (permalink / raw)
To: lartc
I have similar hardware, load and trouble.
Interrupts are only sent to one cpu, instead of all of them, because that was
only overhead. I think the default was changed somewhere around 2.6.10
or .12, but I have forgotten the url.
There is a CONFIG_IRQBALANCE option in the kernel, but last time I checked
(2.6.16) it did not work very well; almost never does anything. So I have
turned it off. I use the userspace irqbalance daemon that periodically sets
the smp_affinity's, with the effect that ingoing and outgoing traffic are
handled by each their cpu (assuming that no other interrupts or processes are
significant). That helps some. But we only shape in one direction, and it can
not help spread the shaping load between the CPU's.
There is also an acpi_irq_balance kernel parameter (not related to the kernel
irq balancer), which apparently uses the APIC to do interrupt round-robin. It
worked surprisingly well (perfect, actually) on an old dual celeron that I
tested; the network interrupts are spread nicely and evenly across the cpu's.
It is probably very chipset dependent, and I have not yet tested it on the
firewalls.
But as I understand from this:
http://vger.kernel.org/~davem/cgi-bin/blog.cgi/2006/09/14#netconf2006_day2
it might not even be an advantage, since the current shaping code would just
make the cpu's step on each others toes.
Regards,
Simon
On Saturday 16 September 2006 20:39, Alexandru Dragoi wrote:
> Hello,
>
> Here is the situation. There is a machine with 3 intel gigabit card, 2
> of them on PCI-X and in bridge, the 3rd is used only for management
> access. The machine is a dual Xeon 2.8GHz with HT. With 2.6.8 kernel
> from debian (testing) and htb with u32 on, i usually get about 30-40%
> software interrupts on CPU0 and CPU2, and without htb and u32, 10% less.
> Now, if I boot with 2.6.17.9 kernel, first all irqs are on same CPU. I
> managed with smp_afinity to "move" irq of one card to a different CPU.
> In these circumstances, I have about 20% or a little less soft
> interrupts on each CPU without shaping, but about 60-70% os soft
> interrupts with shapping, and sometimes there is packet loss, also
> dropped packets are shown on ifconfig. The htb script is same. I have
> u32 performance counters enabled on u32 in 2.6.17.9. I also have NAPI,
> of course. I can't think on anything else that can cause the problem,
> but seem to be something in the kernel. Here is the output of lspci:
>
> # lspci
> 0000:00:00.0 Host bridge: Intel Corporation E7320 Memory Controller Hub
> (rev 0a)
> 0000:00:00.1 ff00: Intel Corporation E7320 Error Reporting Registers
> (rev 0a)
> 0000:00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express
> Port A (rev 0a)
> 0000:00:03.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express
> Port A1 (rev 0a)
> 0000:00:1c.0 PCI bridge: Intel Corporation 6300ESB 64-bit PCI-X Bridge
> (rev 02)
> 0000:00:1d.0 USB Controller: Intel Corporation 6300ESB USB Universal
> Host Controller (rev 02)
> 0000:00:1d.1 USB Controller: Intel Corporation 6300ESB USB Universal
> Host Controller (rev 02)
> 0000:00:1d.4 System peripheral: Intel Corporation 6300ESB Watchdog Timer
> (rev 02)
> 0000:00:1d.5 PIC: Intel Corporation 6300ESB I/O Advanced Programmable
> Interrupt Controller (rev 02)
> 0000:00:1d.7 USB Controller: Intel Corporation 6300ESB USB2 Enhanced
> Host Controller (rev 02)
> 0000:00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 0a)
> 0000:00:1f.0 ISA bridge: Intel Corporation 6300ESB LPC Interface
> Controller (rev 02)
> 0000:00:1f.2 IDE interface: Intel Corporation 6300ESB SATA Storage
> Controller (rev 02)
> 0000:00:1f.3 SMBus: Intel Corporation 6300ESB SMBus Controller (rev 02)
> 0000:03:02.0 Ethernet controller: Intel Corporation 82544EI Gigabit
> Ethernet Controller (Copper) (rev 02)
> 0000:03:03.0 Ethernet controller: Intel Corporation 82544EI Gigabit
> Ethernet Controller (Copper) (rev 02)
> 0000:04:02.0 VGA compatible controller: ATI Technologies Inc Rage XL
> (rev 27)
> 0000:04:03.0 Ethernet controller: Intel Corporation 82541GI Gigabit
> Ethernet Controller (rev 05)
>
> The traffic is somewhere at 40kpps of traffic and 120mbit up,
> 120mbit/down. Suggestions about better hardware and kernel, or links
> with docs about these are really welcomed.
>
> Bye
>
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [LARTC] Performance problem on a linux bridge used for shaping.
2006-09-16 18:39 [LARTC] Performance problem on a linux bridge used for shaping Alexandru Dragoi
2006-09-16 21:17 ` Simon Lodal
@ 2006-09-17 16:33 ` Alexandru Dragoi
1 sibling, 0 replies; 3+ messages in thread
From: Alexandru Dragoi @ 2006-09-17 16:33 UTC (permalink / raw)
To: lartc
Simon Lodal wrote:
>I have similar hardware, load and trouble.
>
>Interrupts are only sent to one cpu, instead of all of them, because that was
>only overhead. I think the default was changed somewhere around 2.6.10
>or .12, but I have forgotten the url.
>
>There is a CONFIG_IRQBALANCE option in the kernel, but last time I checked
>(2.6.16) it did not work very well; almost never does anything. So I have
>turned it off. I use the userspace irqbalance daemon that periodically sets
>the smp_affinity's, with the effect that ingoing and outgoing traffic are
>handled by each their cpu (assuming that no other interrupts or processes are
>significant). That helps some. But we only shape in one direction, and it can
>not help spread the shaping load between the CPU's.
>
>There is also an acpi_irq_balance kernel parameter (not related to the kernel
>irq balancer), which apparently uses the APIC to do interrupt round-robin. It
>worked surprisingly well (perfect, actually) on an old dual celeron that I
>tested; the network interrupts are spread nicely and evenly across the cpu's.
>It is probably very chipset dependent, and I have not yet tested it on the
>firewalls.
>
>But as I understand from this:
>http://vger.kernel.org/~davem/cgi-bin/blog.cgi/2006/09/14#netconf2006_day2
>it might not even be an advantage, since the current shaping code would just
>make the cpu's step on each others toes.
>
>
>Regards,
>Simon
>
>
>On Saturday 16 September 2006 20:39, Alexandru Dragoi wrote:
>
>
>>Hello,
>>
>>Here is the situation. There is a machine with 3 intel gigabit card, 2
>>of them on PCI-X and in bridge, the 3rd is used only for management
>>access. The machine is a dual Xeon 2.8GHz with HT. With 2.6.8 kernel
>>from debian (testing) and htb with u32 on, i usually get about 30-40%
>>software interrupts on CPU0 and CPU2, and without htb and u32, 10% less.
>>Now, if I boot with 2.6.17.9 kernel, first all irqs are on same CPU. I
>>managed with smp_afinity to "move" irq of one card to a different CPU.
>>In these circumstances, I have about 20% or a little less soft
>>interrupts on each CPU without shaping, but about 60-70% os soft
>>interrupts with shapping, and sometimes there is packet loss, also
>>dropped packets are shown on ifconfig. The htb script is same. I have
>>u32 performance counters enabled on u32 in 2.6.17.9. I also have NAPI,
>>of course. I can't think on anything else that can cause the problem,
>>but seem to be something in the kernel. Here is the output of lspci:
>>
>># lspci
>>0000:00:00.0 Host bridge: Intel Corporation E7320 Memory Controller Hub
>>(rev 0a)
>>0000:00:00.1 ff00: Intel Corporation E7320 Error Reporting Registers
>>(rev 0a)
>>0000:00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express
>>Port A (rev 0a)
>>0000:00:03.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express
>>Port A1 (rev 0a)
>>0000:00:1c.0 PCI bridge: Intel Corporation 6300ESB 64-bit PCI-X Bridge
>>(rev 02)
>>0000:00:1d.0 USB Controller: Intel Corporation 6300ESB USB Universal
>>Host Controller (rev 02)
>>0000:00:1d.1 USB Controller: Intel Corporation 6300ESB USB Universal
>>Host Controller (rev 02)
>>0000:00:1d.4 System peripheral: Intel Corporation 6300ESB Watchdog Timer
>>(rev 02)
>>0000:00:1d.5 PIC: Intel Corporation 6300ESB I/O Advanced Programmable
>>Interrupt Controller (rev 02)
>>0000:00:1d.7 USB Controller: Intel Corporation 6300ESB USB2 Enhanced
>>Host Controller (rev 02)
>>0000:00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 0a)
>>0000:00:1f.0 ISA bridge: Intel Corporation 6300ESB LPC Interface
>>Controller (rev 02)
>>0000:00:1f.2 IDE interface: Intel Corporation 6300ESB SATA Storage
>>Controller (rev 02)
>>0000:00:1f.3 SMBus: Intel Corporation 6300ESB SMBus Controller (rev 02)
>>0000:03:02.0 Ethernet controller: Intel Corporation 82544EI Gigabit
>>Ethernet Controller (Copper) (rev 02)
>>0000:03:03.0 Ethernet controller: Intel Corporation 82544EI Gigabit
>>Ethernet Controller (Copper) (rev 02)
>>0000:04:02.0 VGA compatible controller: ATI Technologies Inc Rage XL
>>(rev 27)
>>0000:04:03.0 Ethernet controller: Intel Corporation 82541GI Gigabit
>>Ethernet Controller (rev 05)
>>
>>The traffic is somewhere at 40kpps of traffic and 120mbit up,
>>120mbit/down. Suggestions about better hardware and kernel, or links
>>with docs about these are really welcomed.
>>
>>Bye
>>
>>_______________________________________________
>>LARTC mailing list
>>LARTC@mailman.ds9a.nl
>>http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>>
>>
Somebody recomended me to use Gigabit Intel dual port and kernel 2.6.9,
because some distrubutions stopped at 2.6.9. Now, if things changed on
SMP since 2.6.10 or 2.6.12, I believe those distributions (perhaps some
Redhat and CentOS) know what are they doing.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
^ permalink raw reply [flat|nested] 3+ messages in thread