Re: Gigabit/SMP performance problem

["Martin J. Bligh" <mbligh@aracnet.com>]

> Dual Pentium 4 Xeon at 2.4 Ghz. I believe I am using irq load balancing as 
> shown below (seems to be applied to Red Hat's kernel). Here's 
> /proc/interrupts:

Is in 2.4.20-ac2 at least. See if arch/i386/kernel/io_apic.c
has a function called balance_irq.

>            CPU0       CPU1 
>   0:     179670     182501    IO-APIC-edge  timer
>   1:        386        388    IO-APIC-edge  keyboard
>   2:          0          0          XT-PIC  cascade
>   8:          1          0    IO-APIC-edge  rtc
>  12:          9          9    IO-APIC-edge  PS/2 Mouse
>  14:       1698       1511    IO-APIC-edge  ide0
>  24:    1300174    1298071   IO-APIC-level  eth2
>  25:    1935085    1935625   IO-APIC-level  eth3
>  28:    1162013    1162734   IO-APIC-level  eth4
>  29:    1971246    1967758   IO-APIC-level  eth5
>  48:    2753990    2753821   IO-APIC-level  eth0
>  49:    2047386    2043894   IO-APIC-level  eth1
>  72:     838987     841143   IO-APIC-level  eth6
>  73:    2767885    2768307   IO-APIC-level  eth7
> NMI:          0          0 
> LOC:     362009     362008 
> ERR:          0
> MIS:          0
> 
> I started traffic at different times on the various interfaces so the 
> number of interrupts per interface aren't uniform.
> 
> I modified RxIntDelay, TxIntDelay, RxAbsIntDelay, TxAbsIntDelay, 
> FlowControl, RxDescriptors, TxDescriptors. Increasing the various 
> IntDelays seemed to improve performance slightly.

Makes sense, increasing the delays should reduce the interrupt load.

> I'm using 3 Intel PRO/1000 MT Dual Port Server adapters as well as 2 
> onboard Intel PRO/1000 ports. The adapters use the 82546EB chips. I 
> believe that the onboard ports use the same although I'm not sure.
>
> Should I get rid of IRQ load balancing? And what do you mean 
> "Intel broke the P4's interrupt routing"?

P3's distributed interrupts round-robin amongst cpus. P4's send 
everything to CPU 0. If you put irq_balance on, it'll spread
them around, but any given interrupt is still only handled by
one CPU (as far as I understand the code). If you hammer one
adaptor, does that generate more interrupts than 1 cpu can handle?
(turn irq balance off by sticking a return at the top of balance_irq,
and hammer one link, see how much CPU power that burns).

M.