irqbalance mandatory on SMP kernels?

irqbalance mandatory on SMP kernels?

[Robert M. Stockmann]

Hi,

I noticed that the latest editions of RedHat EL 4.3 and direct
descendants today need a program called irqbalance to activate
true SMP IRQ load balancing for your machine's hardware.

If one boots a SMP kernel (2.4.xx or 2.6.xx) kernel on a machine
which either has 2 or more physical CPU's (also dual-core CPU's) 
, and one does not start up the irqbalance util from the
kernel-utils package ( see e.g. 

ftp://ftp.nluug.nl/pub/os/Linux/distr/RedHat/ftp/redhat/linux/updates/enterprise/4Desktop/en/os/SRPMS/kernel-utils-2.4-13.1.80.src.rpm

) Then basicly your SMP kernel falls back to a ordinary MP kernel and
we see this happening :

[jackson:stock]:(/usr/src/linux)$ cat /proc/interrupts 
           CPU0       CPU1       
  0:    3139568          0    IO-APIC-edge  timer
  1:       8923          0    IO-APIC-edge  i8042
  3:         10          0    IO-APIC-edge  serial
  4:         37          0    IO-APIC-edge  serial
  8:          0          0    IO-APIC-edge  rtc
  9:        240          0   IO-APIC-level  acpi
 12:      75316          0    IO-APIC-edge  i8042
 14:      64291          0    IO-APIC-edge  ide0
 15:      64291          0    IO-APIC-edge  ide1
 16:     235408          0   IO-APIC-level  HiSax, nvidia
 17:      15823          0   IO-APIC-level  libata, AMD AMD8111
 19:        241          0   IO-APIC-level  ohci_hcd, ohci_hcd, ohci1394
 24:      50761          0   IO-APIC-level  eth0
NMI:         89         28 
LOC:    3139042    3139125 
ERR:          0
MIS:          0
[jackson:stock]:(/usr/src/linux)$ 

Only when firing up the irqbalance util at boot time will activate
true SMP, distributing IRQ's across CPU's. Is this on purpose?
Because afaik a Linux SMP kernel, 2.4.xx or 2.6.xx should always
result in distributed IRQ loads across CPU's.

Regards,

Robert
-- 
Robert M. Stockmann - RHCE
Network Engineer - UNIX/Linux Specialist
crashrecovery.org  stock@stokkie.net

Re: irqbalance mandatory on SMP kernels?

[Arjan van de Ven]

> [jackson:stock]:(/usr/src/linux)$ cat /proc/interrupts 
>            CPU0       CPU1       
>   0:    3139568          0    IO-APIC-edge  timer
>   1:       8923          0    IO-APIC-edge  i8042
>   3:         10          0    IO-APIC-edge  serial
>   4:         37          0    IO-APIC-edge  serial
>   8:          0          0    IO-APIC-edge  rtc
>   9:        240          0   IO-APIC-level  acpi
>  12:      75316          0    IO-APIC-edge  i8042
>  14:      64291          0    IO-APIC-edge  ide0
>  15:      64291          0    IO-APIC-edge  ide1
>  16:     235408          0   IO-APIC-level  HiSax, nvidia
>  17:      15823          0   IO-APIC-level  libata, AMD AMD8111
>  19:        241          0   IO-APIC-level  ohci_hcd, ohci_hcd, ohci1394
>  24:      50761          0   IO-APIC-level  eth0
> NMI:         89         28 
> LOC:    3139042    3139125 
> ERR:          0
> MIS:          0
> [jackson:stock]:(/usr/src/linux)$ 

this may or may not be a problem depending on how long a time you used
to collect this. Based on your timer tick count it really looks like
your irq rates are so low that it really doesn't matter much

> 
> Only when firing up the irqbalance util at boot time will activate
> true SMP, distributing IRQ's across CPU's. Is this on purpose?
> Because afaik a Linux SMP kernel, 2.4.xx or 2.6.xx should always
> result in distributed IRQ loads across CPU's.

that is a chipset feature if it happens; not all chipsets do this (and
most that do, do it badly). 

I'm not sure what your actual problem is btw, the irqbalance tool is
supposed to automatically start at boot, did it not do that ?
(and no the kernel doesn't need to do everything, something like this
can perfectly well be done in userspace as irqbalance shows)

Re: irqbalance mandatory on SMP kernels?

[Robert M. Stockmann]

On Mon, 17 Apr 2006, Arjan van de Ven wrote:

> Date: Mon, 17 Apr 2006 15:10:22 +0200
> 
> > [jackson:stock]:(/usr/src/linux)$ cat /proc/interrupts 
> >            CPU0       CPU1       
> >   0:    3139568          0    IO-APIC-edge  timer
> >   1:       8923          0    IO-APIC-edge  i8042
> >   3:         10          0    IO-APIC-edge  serial
> >   4:         37          0    IO-APIC-edge  serial
> >   8:          0          0    IO-APIC-edge  rtc
> >   9:        240          0   IO-APIC-level  acpi
> >  12:      75316          0    IO-APIC-edge  i8042
> >  14:      64291          0    IO-APIC-edge  ide0
> >  15:      64291          0    IO-APIC-edge  ide1
> >  16:     235408          0   IO-APIC-level  HiSax, nvidia
> >  17:      15823          0   IO-APIC-level  libata, AMD AMD8111
> >  19:        241          0   IO-APIC-level  ohci_hcd, ohci_hcd, ohci1394
> >  24:      50761          0   IO-APIC-level  eth0
> > NMI:         89         28 
> > LOC:    3139042    3139125 
> > ERR:          0
> > MIS:          0
> > [jackson:stock]:(/usr/src/linux)$ 
> 
> this may or may not be a problem depending on how long a time you used
> to collect this. Based on your timer tick count it really looks like
> your irq rates are so low that it really doesn't matter much
> 
> > 
> > Only when firing up the irqbalance util at boot time will activate
> > true SMP, distributing IRQ's across CPU's. Is this on purpose?
> > Because afaik a Linux SMP kernel, 2.4.xx or 2.6.xx should always
> > result in distributed IRQ loads across CPU's.
> 
> that is a chipset feature if it happens; not all chipsets do this (and
> most that do, do it badly). 
> 
> I'm not sure what your actual problem is btw, the irqbalance tool is
> supposed to automatically start at boot, did it not do that ?
> (and no the kernel doesn't need to do everything, something like this
> can perfectly well be done in userspace as irqbalance shows)

My question is if the irqbalance util is really needed to activate IRQ 
balancing these days. Which kernel versions and higher (2.4xx and 
2.6.xx) do need this tool? 

To my understanding can a Linux/UNIX kernel not be called SMP if it 
does not activate Symmetric IRQ balancing out of the box. Why was 
irqbalance introduced as a tool inside kernel-utils in the 1st place?  
In other words: What happened to the Linux kernel that we today now 
need a tool called irqbalance ?

Robert
-- 
Robert M. Stockmann - RHCE
Network Engineer - UNIX/Linux Specialist
crashrecovery.org  stock@stokkie.net

Re: irqbalance mandatory on SMP kernels?

[Arjan van de Ven]

> > I'm not sure what your actual problem is btw, the irqbalance tool is
> > supposed to automatically start at boot, did it not do that ?
> > (and no the kernel doesn't need to do everything, something like this
> > can perfectly well be done in userspace as irqbalance shows)
> 
> My question is if the irqbalance util is really needed to activate IRQ 
> balancing these days. Which kernel versions and higher (2.4xx and 
> 2.6.xx) do need this tool? 

again this isn't kernel specific this is chipset specific. 2.4 NEVER
balanced irqs manually, and 2.6 has an optional in-kernel balancer which
is very conservative and which RHEL disables. Again only SOME, old,
hardware does balancing in hardware.

> 
> To my understanding can a Linux/UNIX kernel not be called SMP if it 
> does not activate Symmetric IRQ balancing out of the box.

I think I highly disagree with your statement.
1) I think you are too narrow focusing on "kernel" rather than OS.
2) Why would an OS NOT be SMP even without any IRQ balancing at all?
   (note, on just about any system irq load is < 1% even of 1 cpu)


>  Why was 
> irqbalance introduced as a tool inside kernel-utils in the 1st place?  

because hardware stopped doing the balancing and it turned out that a
userspace balancer could do much better than the hardware anyway

> In other words: What happened to the Linux kernel that we today now 
> need a tool called irqbalance ?

nothing happened to the kernel. Hardware changed.

Re: irqbalance mandatory on SMP kernels?

[Martin J. Bligh]

Robert M. Stockmann wrote:
> Hi,
> 
> I noticed that the latest editions of RedHat EL 4.3 and direct
> descendants today need a program called irqbalance to activate
> true SMP IRQ load balancing for your machine's hardware.
> 
> If one boots a SMP kernel (2.4.xx or 2.6.xx) kernel on a machine
> which either has 2 or more physical CPU's (also dual-core CPU's) 
> , and one does not start up the irqbalance util from the
> kernel-utils package ( see e.g. 

There is an in-kernel IRQ balancer. Redhat just choose to turn it
off, and do it in userspace instead. You can re-enable it if you
compile your own kernel.

M.

Re: irqbalance mandatory on SMP kernels?

[Lee Revell]

On Mon, 2006-04-17 at 07:31 -0700, Martin J. Bligh wrote:
> Robert M. Stockmann wrote:
> > Hi,
> > 
> > I noticed that the latest editions of RedHat EL 4.3 and direct
> > descendants today need a program called irqbalance to activate
> > true SMP IRQ load balancing for your machine's hardware.
> > 
> > If one boots a SMP kernel (2.4.xx or 2.6.xx) kernel on a machine
> > which either has 2 or more physical CPU's (also dual-core CPU's) 
> > , and one does not start up the irqbalance util from the
> > kernel-utils package ( see e.g. 
> 
> There is an in-kernel IRQ balancer. Redhat just choose to turn it
> off, and do it in userspace instead. You can re-enable it if you
> compile your own kernel.

Round-robin IRQ balancing is inefficient anyway.  You'd get better cache
utilization letting one CPU take them all.

Lee

Re: irqbalance mandatory on SMP kernels?

[Theodore Ts'o]

On Mon, Apr 17, 2006 at 11:01:33AM -0400, Lee Revell wrote:
> > There is an in-kernel IRQ balancer. Redhat just choose to turn it
> > off, and do it in userspace instead. You can re-enable it if you
> > compile your own kernel.
> 
> Round-robin IRQ balancing is inefficient anyway.  You'd get better cache
> utilization letting one CPU take them all.

IIRC, Van Jacobsen at his Linux.conf.au presentation made a pretty
strong argument that irq balancing was never a good idea, describing
them as a George Bush-like policy.  "Ooh, interrupts are hurting one
CPU --- let's hurt them **all** and trash everybody's cache!"

Which brings up an interesting question --- why do we have an IRQ
balancer in the kernel at all?  Maybe the scheduler's load balancer
should take this into account so that processes that have the
misfortune of getting assigned to the wrong CPU don't get hurt too
badly (or maybe if we have enough cores/CPU's we can afford to
dedicate one or two CPU's to doing nothing but handling interrupts);
but spreading IRQ's across all of the CPU's doesn't seem like it's
ever the right answer.

						- Ted

Re: irqbalance mandatory on SMP kernels?

[Stephen Hemminger]

On Tue, 18 Apr 2006 12:35:39 -0400
"Theodore Ts'o" <tytso@mit.edu> wrote:

> On Mon, Apr 17, 2006 at 11:01:33AM -0400, Lee Revell wrote:
> > > There is an in-kernel IRQ balancer. Redhat just choose to turn it
> > > off, and do it in userspace instead. You can re-enable it if you
> > > compile your own kernel.
> > 
> > Round-robin IRQ balancing is inefficient anyway.  You'd get better cache
> > utilization letting one CPU take them all.
> 
> IIRC, Van Jacobsen at his Linux.conf.au presentation made a pretty
> strong argument that irq balancing was never a good idea, describing
> them as a George Bush-like policy.  "Ooh, interrupts are hurting one
> CPU --- let's hurt them **all** and trash everybody's cache!"
> 
> Which brings up an interesting question --- why do we have an IRQ
> balancer in the kernel at all?  Maybe the scheduler's load balancer
> should take this into account so that processes that have the
> misfortune of getting assigned to the wrong CPU don't get hurt too
> badly (or maybe if we have enough cores/CPU's we can afford to
> dedicate one or two CPU's to doing nothing but handling interrupts);
> but spreading IRQ's across all of the CPU's doesn't seem like it's
> ever the right answer.
> 
> 						- Ted

There are two problems.  First the scheduler probably doesn't account
for the reduced capacity of a CPU getting hammer with interrupts. Second,
it does make sense to balance different device's interrupts to different
CPU's. A longer term user mode IRQ balancer can make those decisions.

For the networking case, there is a real win if the application code runs
on the same CPU as the interrupt. Otherwise, you end up cache thrashing
control block structures and headers.

Re: irqbalance mandatory on SMP kernels?

[Martin Bligh]

Theodore Ts'o wrote:
> On Mon, Apr 17, 2006 at 11:01:33AM -0400, Lee Revell wrote:
> 
>>>There is an in-kernel IRQ balancer. Redhat just choose to turn it
>>>off, and do it in userspace instead. You can re-enable it if you
>>>compile your own kernel.
>>
>>Round-robin IRQ balancing is inefficient anyway.  You'd get better cache
>>utilization letting one CPU take them all.
> 
> 
> IIRC, Van Jacobsen at his Linux.conf.au presentation made a pretty
> strong argument that irq balancing was never a good idea, describing
> them as a George Bush-like policy.  "Ooh, interrupts are hurting one
> CPU --- let's hurt them **all** and trash everybody's cache!"

Nothing nowadays does round-robin of interrupts, either the in-kernel
or userspace balancers ... but we do migrate them occasionally (in the
order of 1s or so)

> Which brings up an interesting question --- why do we have an IRQ
> balancer in the kernel at all?  Maybe the scheduler's load balancer
> should take this into account so that processes that have the
> misfortune of getting assigned to the wrong CPU don't get hurt too
> badly (or maybe if we have enough cores/CPU's we can afford to
> dedicate one or two CPU's to doing nothing but handling interrupts);
> but spreading IRQ's across all of the CPU's doesn't seem like it's
> ever the right answer.

Because *something* has to be balanced, and moving processes around
is expensive too. Personally I find the process model cleaner, but
maybe it's less efficient - you'd also add extra overhead for accounting
to each interrupt, which we don't do now.

I'm not claiming that moving irqs is worse or better than moving
processes - just that it's a tradeoff, both suck. Perhaps the
real answer is that we shouldn't be getting that many interrupts
anyway - technologies like NAPI and simpler device interrupt collation
should reduce the load, and most of the work should be done in the
back-ends anyhow (though those are often locally bonded to the CPU
the irq arrived on).

M.

Re: irqbalance mandatory on SMP kernels?

[Arjan van de Ven]

> Which brings up an interesting question --- why do we have an IRQ
> balancer in the kernel at all?  

good question; at least RHEL disables it and uses the userspace one
(which only decides to change the balance at most every 10 seconds, but
has a strong tendency to leave the irqs as they are)

> but spreading IRQ's across all of the CPU's doesn't seem like it's
> ever the right answer.

well it is in some cases, imagine having 2 cpus and 2 gige nics that are
very busy doing webserving. That's an obvious case where 1-nic-per-cpu
ends up doing the right thing... the way it ends up is that each nic has
a full cpu for itself and it's own apaches... almost fully independent
of the other one. Now if you moved both irqs to the same cpu, the
apaches would follow, because if they didn't then you'd be bouncing
their data *all the time*. And at that point the other cpu will become
bored ;)

This is what the userspace irqbalance has in its policy more or less:
spread the irqs within the same class to different processors. And as
secondary policy it tries to spread it out a bit, but that's only
secondary.

Re: irqbalance mandatory on SMP kernels?

[Erik Mouw]

On Tue, Apr 18, 2006 at 08:19:17PM +0200, Arjan van de Ven wrote:
> > but spreading IRQ's across all of the CPU's doesn't seem like it's
> > ever the right answer.
> 
> well it is in some cases, imagine having 2 cpus and 2 gige nics that are
> very busy doing webserving. That's an obvious case where 1-nic-per-cpu
> ends up doing the right thing... the way it ends up is that each nic has
> a full cpu for itself and it's own apaches... almost fully independent
> of the other one. Now if you moved both irqs to the same cpu, the
> apaches would follow, because if they didn't then you'd be bouncing
> their data *all the time*. And at that point the other cpu will become
> bored ;)

So what happens with a dual amd64 system where each CPU has its "own"
NIC? Something like this:


MEM0 <--> CPU0 <--- HT ---> CPU1 <--> MEM1
           ^                 ^
           |                 |
           HT                HT
           |                 |
           v                 v
      PCI bridge0      PCI1 bridge
           ^                 ^
           |                 |
          PCI               PCI
           |                 |
           v                 v
       GigE NIC0         GigE NIC1

The "best" approach would be to run an Apache on each CPU using local
memory and NIC and having the IRQs handled by the local CPU. Does the
irq balancer allow such a configuraion, or would it be hamperd by the
process scheduler deciding to run both Apaches on a single CPU?


Erik

-- 
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands

Re: irqbalance mandatory on SMP kernels?

[Arjan van de Ven]

On Wed, 2006-04-19 at 14:42 +0200, Erik Mouw wrote:
> On Tue, Apr 18, 2006 at 08:19:17PM +0200, Arjan van de Ven wrote:
> > > but spreading IRQ's across all of the CPU's doesn't seem like it's
> > > ever the right answer.
> > 
> > well it is in some cases, imagine having 2 cpus and 2 gige nics that are
> > very busy doing webserving. That's an obvious case where 1-nic-per-cpu
> > ends up doing the right thing... the way it ends up is that each nic has
> > a full cpu for itself and it's own apaches... almost fully independent
> > of the other one. Now if you moved both irqs to the same cpu, the
> > apaches would follow, because if they didn't then you'd be bouncing
> > their data *all the time*. And at that point the other cpu will become
> > bored ;)
> 
> So what happens with a dual amd64 system where each CPU has its "own"
> NIC? Something like this:

as long as the irqs are spread the apaches will (on average) follow your
irq to the right cpu. Only if you put both irqs on the same cpu you have
an issue

Re: irqbalance mandatory on SMP kernels?

[Theodore Ts'o]

On Wed, Apr 19, 2006 at 04:23:14PM +0200, Arjan van de Ven wrote:
> as long as the irqs are spread the apaches will (on average) follow your
> irq to the right cpu. Only if you put both irqs on the same cpu you have
> an issue

Maybe I'm being stupid but I don't see how the Apache's will follow
the IRQ's to the right CPU.  I agree this would be a good thing to do,
but how does the scheduler accomplish this?

						- Ted

Re: irqbalance mandatory on SMP kernels?

[Arjan van de Ven]

On Wed, 2006-04-19 at 10:38 -0400, Theodore Ts'o wrote:
> On Wed, Apr 19, 2006 at 04:23:14PM +0200, Arjan van de Ven wrote:
> > as long as the irqs are spread the apaches will (on average) follow your
> > irq to the right cpu. Only if you put both irqs on the same cpu you have
> > an issue
> 
> Maybe I'm being stupid but I don't see how the Apache's will follow
> the IRQ's to the right CPU.  I agree this would be a good thing to do,
> but how does the scheduler accomplish this?

iirc this part of the kernel uses wake_up_sync() and such, which tend to
pull the apache to the cpu (if it's idle) in the long term
(or it ought to; at one point it did)

Re: irqbalance mandatory on SMP kernels?

[Nick Piggin]

Arjan van de Ven wrote:
> On Wed, 2006-04-19 at 10:38 -0400, Theodore Ts'o wrote:
> 
>>On Wed, Apr 19, 2006 at 04:23:14PM +0200, Arjan van de Ven wrote:
>>
>>>as long as the irqs are spread the apaches will (on average) follow your
>>>irq to the right cpu. Only if you put both irqs on the same cpu you have
>>>an issue
>>
>>Maybe I'm being stupid but I don't see how the Apache's will follow
>>the IRQ's to the right CPU.  I agree this would be a good thing to do,
>>but how does the scheduler accomplish this?
> 
> 
> iirc this part of the kernel uses wake_up_sync() and such, which tend to
> pull the apache to the cpu (if it's idle) in the long term
> (or it ought to; at one point it did)

Yeah it has "affine wakeups" now, which will do that for all
types of wakeups, and not just to idle CPUs either (sync
wakeups just get pulled a little more strongly).

IIRC SGI reported something like a factor 8 improvement in
CPU efficiency on a database IO simulation on a smallish
system (16-way maybe).

-- 
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 

Re: irqbalance mandatory on SMP kernels?

[Martin J. Bligh]

Erik Mouw wrote:
> On Tue, Apr 18, 2006 at 08:19:17PM +0200, Arjan van de Ven wrote:
> 
>>>but spreading IRQ's across all of the CPU's doesn't seem like it's
>>>ever the right answer.
>>
>>well it is in some cases, imagine having 2 cpus and 2 gige nics that are
>>very busy doing webserving. That's an obvious case where 1-nic-per-cpu
>>ends up doing the right thing... the way it ends up is that each nic has
>>a full cpu for itself and it's own apaches... almost fully independent
>>of the other one. Now if you moved both irqs to the same cpu, the
>>apaches would follow, because if they didn't then you'd be bouncing
>>their data *all the time*. And at that point the other cpu will become
>>bored ;)
> 
> 
> So what happens with a dual amd64 system where each CPU has its "own"
> NIC? Something like this:
> 
> 
> MEM0 <--> CPU0 <--- HT ---> CPU1 <--> MEM1
>            ^                 ^
>            |                 |
>            HT                HT
>            |                 |
>            v                 v
>       PCI bridge0      PCI1 bridge
>            ^                 ^
>            |                 |
>           PCI               PCI
>            |                 |
>            v                 v
>        GigE NIC0         GigE NIC1
> 
> The "best" approach would be to run an Apache on each CPU using local
> memory and NIC and having the IRQs handled by the local CPU. Does the
> irq balancer allow such a configuraion, or would it be hamperd by the
> process scheduler deciding to run both Apaches on a single CPU?

The trouble is that we're not smart enough to redirect receiving traffic
back to the correct CPU, I think. You'd need to configure different IP
addrs on the same subnet to each NIC, and have intelligent bonding for
outbound traffic.

Then you'd need NUMA locality of IRQs, by bonding them to their closest
CPUs, which is something that's easily done inside the kernel, but is
harder in userspace. but I'm not getting into that silly pissing contest
again. Either mechanism *could* do it. It's just about creating sensbile
APIs and defaults, both of which we're not good at doing as a community.

M.