Although CONFIG_IRQBALANCE is enabled IRQ's don't seem to be balanced very well

Although CONFIG_IRQBALANCE is enabled IRQ's don't seem to be balanced very well

[Jesper Juhl]

I enabled CONFIG_IRQBALANCE with 2.6.15 and 2.6.15-mm2 (which the
numbers below are from), and had expected that to evenly (or at least
close to evenly) balance IRQ's across the two CPU cores of my Athlon
X2 4400+. But as you can see below, CPU0 seems to be heavily favoured
- what's the reason for that and is it something I can improve upon?
Is it due to this being a Dual Core CPU and not two physically sepperate CPU's?
Do I need any userspace tools in addition to CONFIG_IRQBALANCE?

juhl@dragon:~$ date && cat /proc/interrupts
Tue Jan 10 11:20:33 CET 2006
           CPU0       CPU1
  0:    3818901      18920    IO-APIC-edge  timer
  1:       8398          7    IO-APIC-edge  i8042
  8:          0          1    IO-APIC-edge  rtc
  9:          0          0   IO-APIC-level  acpi
 12:     210259         11    IO-APIC-edge  i8042
 18:      12883          1   IO-APIC-level  eth0
 19:      15906        291   IO-APIC-level  aic7xxx
 20:       1284          1   IO-APIC-level  EMU10K1
NMI:          0          0
LOC:    3838143    3837641
ERR:          0
MIS:          0

juhl@dragon:~$ date && cat /proc/interrupts
Tue Jan 10 12:12:38 CET 2006
           CPU0       CPU1
  0:    6944246      18920    IO-APIC-edge  timer
  1:      11291          7    IO-APIC-edge  i8042
  8:          0          1    IO-APIC-edge  rtc
  9:          0          0   IO-APIC-level  acpi
 12:     393217         11    IO-APIC-edge  i8042
 18:      38690          1   IO-APIC-level  eth0
 19:     204122        291   IO-APIC-level  aic7xxx
 20:      10877          1   IO-APIC-level  EMU10K1
NMI:          0          0
LOC:    6963608    6963106
ERR:          0
MIS:          0


--
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: Although CONFIG_IRQBALANCE is enabled IRQ's don't seem to be balanced very well

[Josef Sipek]

On Tue, Jan 10, 2006 at 12:14:42PM +0100, Jesper Juhl wrote:
> Do I need any userspace tools in addition to CONFIG_IRQBALANCE?

Last I checked, yes you do need "irqbalance" (at least that's what
the package is called in debian.

Jeff.

Re: Although CONFIG_IRQBALANCE is enabled IRQ's don't seem to be balanced very well

[Martin Bligh]

Josef Sipek wrote:
> On Tue, Jan 10, 2006 at 12:14:42PM +0100, Jesper Juhl wrote:
> 
>>Do I need any userspace tools in addition to CONFIG_IRQBALANCE?
> 
> 
> Last I checked, yes you do need "irqbalance" (at least that's what
> the package is called in debian.

Nope - you need the kernel option turned on OR the userspace daemon,
not both.

If you're not generating interrupts at a high enough rate, it won't
rotate. That's deliberate.

M.

Re: Although CONFIG_IRQBALANCE is enabled IRQ's don't seem to be balanced very well

[Jesper Juhl]

On 1/10/06, Martin Bligh <mbligh@mbligh.org> wrote:
> Josef Sipek wrote:
> > On Tue, Jan 10, 2006 at 12:14:42PM +0100, Jesper Juhl wrote:
> >
> >>Do I need any userspace tools in addition to CONFIG_IRQBALANCE?
> >
> >
> > Last I checked, yes you do need "irqbalance" (at least that's what
> > the package is called in debian.
>
> Nope - you need the kernel option turned on OR the userspace daemon,
> not both.
>
Ok, good to know.

> If you're not generating interrupts at a high enough rate, it won't
> rotate. That's deliberate.
>

Hmm, and what would count as "a high enough rate"?

I just did a small test with thousands of  ping -f's through my NIC
while at the same time giving the disk a good workout with tons of
find's, sync's & updatedb's - that sure did drive up the number of
interrupts and my load average went sky high (amazingly the box was
still fairly responsive):

root@dragon:/home/juhl# uptime
 22:59:58 up 12:43,  1 user,  load average: 1015.48, 715.93, 429.07

but, not a single interrupt was handled by CPU1, they all went to CPU0.

Do you have a good way to drive up the nr of interrupts above the
treshhold for balancing?

--
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: Although CONFIG_IRQBALANCE is enabled IRQ's don't seem to be balanced very well

[Martin Bligh]

Jesper Juhl wrote:
> On 1/10/06, Martin Bligh <mbligh@mbligh.org> wrote:
> 
>>Josef Sipek wrote:
>>
>>>On Tue, Jan 10, 2006 at 12:14:42PM +0100, Jesper Juhl wrote:
>>>
>>>
>>>>Do I need any userspace tools in addition to CONFIG_IRQBALANCE?
>>>
>>>
>>>Last I checked, yes you do need "irqbalance" (at least that's what
>>>the package is called in debian.
>>
>>Nope - you need the kernel option turned on OR the userspace daemon,
>>not both.
>>
> 
> Ok, good to know.
> 
> 
>>If you're not generating interrupts at a high enough rate, it won't
>>rotate. That's deliberate.
>>
> 
> 
> Hmm, and what would count as "a high enough rate"?
> 
> I just did a small test with thousands of  ping -f's through my NIC
> while at the same time giving the disk a good workout with tons of
> find's, sync's & updatedb's - that sure did drive up the number of
> interrupts and my load average went sky high (amazingly the box was
> still fairly responsive):
> 
> root@dragon:/home/juhl# uptime
>  22:59:58 up 12:43,  1 user,  load average: 1015.48, 715.93, 429.07
> 
> but, not a single interrupt was handled by CPU1, they all went to CPU0.
> 
> Do you have a good way to drive up the nr of interrupts above the
> treshhold for balancing?

Is it HT? ISTR it was intelligent enough to ignore that. But you'd
have to look at the code to be sure.

M.

Re: Although CONFIG_IRQBALANCE is enabled IRQ's don't seem to be balanced very well

[Jesper Juhl]

On 1/10/06, Martin Bligh <mbligh@mbligh.org> wrote:
> Jesper Juhl wrote:
> > On 1/10/06, Martin Bligh <mbligh@mbligh.org> wrote:
> >
> >>Josef Sipek wrote:
> >>
> >>>On Tue, Jan 10, 2006 at 12:14:42PM +0100, Jesper Juhl wrote:
> >>>
> >>>
> >>>>Do I need any userspace tools in addition to CONFIG_IRQBALANCE?
> >>>
> >>>
> >>>Last I checked, yes you do need "irqbalance" (at least that's what
> >>>the package is called in debian.
> >>
> >>Nope - you need the kernel option turned on OR the userspace daemon,
> >>not both.
> >>
> >
> > Ok, good to know.
> >
> >
> >>If you're not generating interrupts at a high enough rate, it won't
> >>rotate. That's deliberate.
> >>
> >
> >
> > Hmm, and what would count as "a high enough rate"?
> >
> > I just did a small test with thousands of  ping -f's through my NIC
> > while at the same time giving the disk a good workout with tons of
> > find's, sync's & updatedb's - that sure did drive up the number of
> > interrupts and my load average went sky high (amazingly the box was
> > still fairly responsive):
> >
> > root@dragon:/home/juhl# uptime
> >  22:59:58 up 12:43,  1 user,  load average: 1015.48, 715.93, 429.07
> >
> > but, not a single interrupt was handled by CPU1, they all went to CPU0.
> >
> > Do you have a good way to drive up the nr of interrupts above the
> > treshhold for balancing?
>
> Is it HT? ISTR it was intelligent enough to ignore that. But you'd
> have to look at the code to be sure.
>
Dual Core Athlon 64 X2 4400+

--
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: Although CONFIG_IRQBALANCE is enabled IRQ's don't seem to be balanced very well

[Nauman Tahir]

On 1/10/06, Jesper Juhl <jesper.juhl@gmail.com> wrote:
> On 1/10/06, Martin Bligh <mbligh@mbligh.org> wrote:
> > Jesper Juhl wrote:
> > > On 1/10/06, Martin Bligh <mbligh@mbligh.org> wrote:
> > >
> > >>Josef Sipek wrote:
> > >>
> > >>>On Tue, Jan 10, 2006 at 12:14:42PM +0100, Jesper Juhl wrote:
> > >>>
> > >>>
> > >>>>Do I need any userspace tools in addition to CONFIG_IRQBALANCE?
> > >>>
> > >>>
> > >>>Last I checked, yes you do need "irqbalance" (at least that's what
> > >>>the package is called in debian.
> > >>
> > >>Nope - you need the kernel option turned on OR the userspace daemon,
> > >>not both.
> > >>
> > >
> > > Ok, good to know.
> > >
> > >
> > >>If you're not generating interrupts at a high enough rate, it won't
> > >>rotate. That's deliberate.
> > >>

What I have read is that first CPU is used more for interrupts to use
the concept of maximizing cache locality. Probably kernel is
optimizing this even with CONFIG option enabled.

Nauman

> > >
> > >
> > > Hmm, and what would count as "a high enough rate"?
> > >
> > > I just did a small test with thousands of  ping -f's through my NIC
> > > while at the same time giving the disk a good workout with tons of
> > > find's, sync's & updatedb's - that sure did drive up the number of
> > > interrupts and my load average went sky high (amazingly the box was
> > > still fairly responsive):
> > >
> > > root@dragon:/home/juhl# uptime
> > >  22:59:58 up 12:43,  1 user,  load average: 1015.48, 715.93, 429.07
> > >
> > > but, not a single interrupt was handled by CPU1, they all went to CPU0.
> > >
> > > Do you have a good way to drive up the nr of interrupts above the
> > > treshhold for balancing?
> >
> > Is it HT? ISTR it was intelligent enough to ignore that. But you'd
> > have to look at the code to be sure.
> >
> Dual Core Athlon 64 X2 4400+
>
> --
> Jesper Juhl <jesper.juhl@gmail.com>
> Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
> Plain text mails only, please      http://www.expita.com/nomime.html
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: Although CONFIG_IRQBALANCE is enabled IRQ's don't seem to be balanced very well

[Stijn Eeckhaut]

Nauman Tahir wrote:
> On 1/10/06, Jesper Juhl <jesper.juhl@gmail.com> wrote:
>>On 1/10/06, Martin Bligh <mbligh@mbligh.org> wrote:
>>>Jesper Juhl wrote:
>>>>On 1/10/06, Martin Bligh <mbligh@mbligh.org> wrote:
>>>>>Josef Sipek wrote:
>>>>>>On Tue, Jan 10, 2006 at 12:14:42PM +0100, Jesper Juhl wrote:
>>>>>>
>>>>>>>Do I need any userspace tools in addition to CONFIG_IRQBALANCE?
>>>>>>
>>>>>>Last I checked, yes you do need "irqbalance" (at least that's what
>>>>>>the package is called in debian.
>>>>>
>>>>>Nope - you need the kernel option turned on OR the userspace daemon,
>>>>>not both.
>>>>
>>>>Ok, good to know.
>>>>
>>>>>If you're not generating interrupts at a high enough rate, it won't
>>>>>rotate. That's deliberate.
> 
> What I have read is that first CPU is used more for interrupts to use
> the concept of maximizing cache locality. Probably kernel is
> optimizing this even with CONFIG option enabled.
> 
>>>>
>>>>Hmm, and what would count as "a high enough rate"?

This is what I tested a few months ago:

Test system: 2 dual Pentium3 systems
- with 2.6.11 kernel and kernel IRQ balancing;
- each with an Intel dual port E1000 NIC (e1000 driver 6.0.54);
- both systems connected back-to-back to each other with 2 links.

Test 1:
- I started 1 UDP flow (< 23 Mbps) on the first link with the Iperf 
network performance measurement tool. For a UDP bandwidth lower than 23 
Mbps the interrupt rate at the receiver interface was lower than 2000 
interrupts per second. In this case all interrupts were distributed to 
CPU 0. 2000 interrupts per second seemed to be the threshold for the 
interrupts to be distributed to 1 CPU.

Test 2:
- Then I started 1 UDP flow of 600 Mbps on the first link. 8000 
interrupts per second were generated by the receiver interface. 
Approximately half of the interrupts were distributed to CPU 0, the 
other half to CPU 1.

Test 3:
- Then I did a test with 2 UDP flows of 600 Mbps, each over their own 
link. 8000 interrupts per second were generated by both receiver 
interfaces. All interrupts generated by the 1st interface were 
distributed to CPU 0, all interrupts generated by the 2nd interface were 
  distributed to CPU 1.


>>>>
>>>>I just did a small test with thousands of  ping -f's through my NIC
>>>>while at the same time giving the disk a good workout with tons of
>>>>find's, sync's & updatedb's - that sure did drive up the number of
>>>>interrupts and my load average went sky high (amazingly the box was
>>>>still fairly responsive):
>>>>
>>>>root@dragon:/home/juhl# uptime
>>>> 22:59:58 up 12:43,  1 user,  load average: 1015.48, 715.93, 429.07
>>>>but, not a single interrupt was handled by CPU1, they all went to CPU0.
>>>>
>>>>Do you have a good way to drive up the nr of interrupts above the
>>>>treshhold for balancing?
>>>
>>>Is it HT? ISTR it was intelligent enough to ignore that. But you'd
>>>have to look at the code to be sure.
>>>
>>
>>Dual Core Athlon 64 X2 4400+
>>