Re: irqbalance mandatory on SMP kernels?

[Stephen Hemminger <shemminger@osdl.org>]

On Tue, 18 Apr 2006 12:35:39 -0400
"Theodore Ts'o" <tytso@mit.edu> wrote:

> On Mon, Apr 17, 2006 at 11:01:33AM -0400, Lee Revell wrote:
> > > There is an in-kernel IRQ balancer. Redhat just choose to turn it
> > > off, and do it in userspace instead. You can re-enable it if you
> > > compile your own kernel.
> > 
> > Round-robin IRQ balancing is inefficient anyway.  You'd get better cache
> > utilization letting one CPU take them all.
> 
> IIRC, Van Jacobsen at his Linux.conf.au presentation made a pretty
> strong argument that irq balancing was never a good idea, describing
> them as a George Bush-like policy.  "Ooh, interrupts are hurting one
> CPU --- let's hurt them **all** and trash everybody's cache!"
> 
> Which brings up an interesting question --- why do we have an IRQ
> balancer in the kernel at all?  Maybe the scheduler's load balancer
> should take this into account so that processes that have the
> misfortune of getting assigned to the wrong CPU don't get hurt too
> badly (or maybe if we have enough cores/CPU's we can afford to
> dedicate one or two CPU's to doing nothing but handling interrupts);
> but spreading IRQ's across all of the CPU's doesn't seem like it's
> ever the right answer.
> 
> 						- Ted

There are two problems.  First the scheduler probably doesn't account
for the reduced capacity of a CPU getting hammer with interrupts. Second,
it does make sense to balance different device's interrupts to different
CPU's. A longer term user mode IRQ balancer can make those decisions.

For the networking case, there is a real win if the application code runs
on the same CPU as the interrupt. Otherwise, you end up cache thrashing
control block structures and headers.