[RFC] irqbalance: Mark in-kernel irqbalance as obsolete, set to N by default

[RFC] irqbalance: Mark in-kernel irqbalance as obsolete, set to N by default

[Auke Kok]

[-- Attachment #1: Type: text/plain, Size: 931 bytes --]


We've recently seen a number of user bug reports against e1000 that the 
in-kernel irqbalance code is detrimental to network latency. The algorithm 
keeps swapping irq's for NICs from cpu to cpu causing extremely high network 
latency (>1000ms). Another NIC driver (cxgb) already has severe warnings in 
their documentation file against using CONFIG_IRQBALANCE, but this is a 
general problem for all NIC drivers and other subsystems. This is especially 
so with cpufreq scaling where the system is slowed down and the migrations 
take much longer.

I suggest that the in-kernel irqbalance is phased out, by marking it OBSOLETE 
first and (perhaps) removing the code later. The userspace irqbalance daemon 
written by Arjan van de Ven does a wonderful job and should be used instead.

Signed-off-by: Auke Kok <auke-jan.h.kok@intel.com>

---

  Kconfig |   15 +++++++++++----
  1 file changed, 11 insertions(+), 4 deletions(-)
---

[-- Attachment #2: mark_CONFIG_IRQBALANCE_as_obsolete.patch --]
[-- Type: text/x-patch, Size: 1013 bytes --]

diff --git a/arch/i386/Kconfig b/arch/i386/Kconfig
index daa75ce..5a40cfe 100644
--- a/arch/i386/Kconfig
+++ b/arch/i386/Kconfig
@@ -690,12 +690,19 @@ config EFI
 	kernel should continue to boot on existing non-EFI platforms.
 
 config IRQBALANCE
- 	bool "Enable kernel irq balancing"
+	bool "Enable kernel irq balancing (obsolete)"
 	depends on SMP && X86_IO_APIC
-	default y
+	default n
 	help
- 	  The default yes will allow the kernel to do irq load balancing.
-	  Saying no will keep the kernel from doing irq load balancing.
+	  The kernel irq balance will migrate interrupts between cpu's
+	  constantly, which may help reduce load in some cases. It is not
+	  beneficial for latency however, and a user-space daemon is available
+	  that does a much better job.
+
+	  The default no will keep the kernel from doing irq load balancing.
+	  Say yes will allow the kernel to do irq load balancing.
+
+	  If unsure, say N.
 
 # turning this on wastes a bunch of space.
 # Summit needs it only when NUMA is on

Re: [RFC] irqbalance: Mark in-kernel irqbalance as obsolete, set to N by default

[Andrew Morton]

On Mon, 31 Jul 2006 10:35:26 -0700
Auke Kok <auke-jan.h.kok@intel.com> wrote:

> We've recently seen a number of user bug reports against e1000 that the 
> in-kernel irqbalance code is detrimental to network latency. The algorithm 
> keeps swapping irq's for NICs from cpu to cpu causing extremely high network 
> latency (>1000ms).

What kernel versions?  Some IRQ balancer fixes went in shortly after 2.6.17.

It would be better if poss to fix the balancer rather than deprecating it.

Re: [RFC] irqbalance: Mark in-kernel irqbalance as obsolete, set to N by default

[Auke Kok]

Andrew Morton wrote:
> On Mon, 31 Jul 2006 10:35:26 -0700
> Auke Kok <auke-jan.h.kok@intel.com> wrote:
> 
>> We've recently seen a number of user bug reports against e1000 that the 
>> in-kernel irqbalance code is detrimental to network latency. The algorithm 
>> keeps swapping irq's for NICs from cpu to cpu causing extremely high network 
>> latency (>1000ms).
> 
> What kernel versions?  Some IRQ balancer fixes went in shortly after 2.6.17.

user reports show 2.6.17.1 having the problem, I'm trying to get more details 
information, and will ask if 2.6.18rc3 or so does better for them.

Cheers,

Auke

Re: [RFC] irqbalance: Mark in-kernel irqbalance as obsolete, set to N by default

[Arjan van de Ven]

Andrew Morton wrote:
> On Mon, 31 Jul 2006 10:35:26 -0700
> Auke Kok <auke-jan.h.kok@intel.com> wrote:
> 
>> We've recently seen a number of user bug reports against e1000 that the 
>> in-kernel irqbalance code is detrimental to network latency. The algorithm 
>> keeps swapping irq's for NICs from cpu to cpu causing extremely high network 
>> latency (>1000ms).
> 
> What kernel versions?  Some IRQ balancer fixes went in shortly after 2.6.17.
> 
> It would be better if poss to fix the balancer rather than deprecating it.

to some degree the in kernel balancer cannot really make the level of decisions that a
userspace balancer can make, at least not without making all kernel developers vomit ;)
(for example the userspace balancer looks in /proc/interrupts and parses that to see
which interrupts are used by networking versus which by storage etc, and has different
balancing policies for those and other classes; the networking policy basically comes down to
"pin the interrupt unless some higher networking interrupt really gets in the way")

Re: [RFC] irqbalance: Mark in-kernel irqbalance as obsolete, set to N by default

[Bodo Eggert]

Arjan van de Ven <arjan@linux.intel.com> wrote:

> to some degree the in kernel balancer cannot really make the level of
> decisions that a userspace balancer can make, at least not without making all
> kernel developers vomit ;)

If you make the drivers set a flag if they know their interrupts should remain
mostly on one CPU, you can avoid the bad cases, and the rest you could gain
from using more clever algorithms should be(*) usurally less than what parsing
/proc/interrupts costs.

*) as in: I guess
-- 
Ich danke GMX dafür, die Verwendung meiner Adressen mittels per SPF
verbreiteten Lügen zu sabotieren.

http://david.woodhou.se/why-not-spf.html