Re: irqbalance mandatory on SMP kernels?

[Martin Bligh <mbligh@google.com>]

Theodore Ts'o wrote:
> On Mon, Apr 17, 2006 at 11:01:33AM -0400, Lee Revell wrote:
> 
>>>There is an in-kernel IRQ balancer. Redhat just choose to turn it
>>>off, and do it in userspace instead. You can re-enable it if you
>>>compile your own kernel.
>>
>>Round-robin IRQ balancing is inefficient anyway.  You'd get better cache
>>utilization letting one CPU take them all.
> 
> 
> IIRC, Van Jacobsen at his Linux.conf.au presentation made a pretty
> strong argument that irq balancing was never a good idea, describing
> them as a George Bush-like policy.  "Ooh, interrupts are hurting one
> CPU --- let's hurt them **all** and trash everybody's cache!"

Nothing nowadays does round-robin of interrupts, either the in-kernel
or userspace balancers ... but we do migrate them occasionally (in the
order of 1s or so)

> Which brings up an interesting question --- why do we have an IRQ
> balancer in the kernel at all?  Maybe the scheduler's load balancer
> should take this into account so that processes that have the
> misfortune of getting assigned to the wrong CPU don't get hurt too
> badly (or maybe if we have enough cores/CPU's we can afford to
> dedicate one or two CPU's to doing nothing but handling interrupts);
> but spreading IRQ's across all of the CPU's doesn't seem like it's
> ever the right answer.

Because *something* has to be balanced, and moving processes around
is expensive too. Personally I find the process model cleaner, but
maybe it's less efficient - you'd also add extra overhead for accounting
to each interrupt, which we don't do now.

I'm not claiming that moving irqs is worse or better than moving
processes - just that it's a tradeoff, both suck. Perhaps the
real answer is that we shouldn't be getting that many interrupts
anyway - technologies like NAPI and simpler device interrupt collation
should reduce the load, and most of the work should be done in the
back-ends anyhow (though those are often locally bonded to the CPU
the irq arrived on).

M.