RE: CONFIG_IRQBALANCE for AMD64?

RE: CONFIG_IRQBALANCE for AMD64?

[Nakajima, Jun]

>From: Arjan van de Ven [mailto:arjanv@redhat.com]
>Sent: Friday, May 28, 2004 10:57 AM
>To: Martin J. Bligh
>Cc: Jeff Garzik; Nakajima, Jun; Andrew Morton; Anton Blanchard; linux-
>kernel@vger.kernel.org
>Subject: Re: CONFIG_IRQBALANCE for AMD64?
>
>On Fri, May 28, 2004 at 10:46:18AM -0700, Martin J. Bligh wrote:
>>
>> Personally, I find the argument that it's hardware-specific control
code
>> a much better reason for it to belong in the kernel.
>
>Is it really hardware specific ??

I think automatic IRQ binding business should belong to the user-level;
it can use generic statistics, application, or platform configuration
knowledge.

The kernel-level should have some simple chipset model, such as lowest
priority delivery mode with finer granularity of control. The kirqd at
this point, is doing automatic IRQ binding business as well today,
although it does not literally bind them. So I think we need to remove
that part of code from kirqd. 

Jun

RE: CONFIG_IRQBALANCE for AMD64?

[Martin J. Bligh]

>> On Fri, May 28, 2004 at 10:46:18AM -0700, Martin J. Bligh wrote:
>>> 
>>> Personally, I find the argument that it's hardware-specific control
> code
>>> a much better reason for it to belong in the kernel.
>> 
>> Is it really hardware specific ??
> 
> I think automatic IRQ binding business should belong to the user-level;
> it can use generic statistics, application, or platform configuration
> knowledge.
> 
> The kernel-level should have some simple chipset model, such as lowest
> priority delivery mode with finer granularity of control. The kirqd at
> this point, is doing automatic IRQ binding business as well today,
> although it does not literally bind them. So I think we need to remove
> that part of code from kirqd. 

My personal feeling is that we can't get to happen from userspace what 
really needs to happen .... but we're going about this ass-backwards.
Instead of starting with a solution, and seeing what we can wedge into
it ... what do we actually want to do?

Here's my start at a list ... I'm sure it's woefully incomplete.

1. Utilize all CPUs roughly evenly for IRQ processing load (anything that's
not measured by the scheduler at least, because it's unfair to other 
processes). Also, we may well have more than 1 CPU's worth of traffic to
process in a large network server.

2. Provide some sort of cache-affinity for network interrupt processing,
which also helps us not get into out-of-order packet situations.

3. Utilize idle CPUs where possible to shoulder the load.

4. Provide such a solution for all architectures.

What else? I think we started doing this because of (1 & 2) mainline, 
especially as the P4 is moronically stupid by default but I know Dave 
Olien looked at 3 as well at some point past. 

ISTR one of the objections to the in-kernel stuff was that the way it
calculated costs was to look only at the in_interrupt() part of the
processing ... does the backend stuff get accounted currently to the
poor sucker who is currently running, in the same was as the interrupt?

Whatever we do ... all arches are going to need to provide a way to direct
interrupts to a certain CPU, or group thereof. Can they all do that already?
I'll confess to not having looked at non-i386 arches. And are others as
brain damaged as the P4? or do they do something round-robin by default?

M.

Re: CONFIG_IRQBALANCE for AMD64?

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 1066 bytes --]

On Fri, May 28, 2004 at 11:33:32AM -0700, Martin J. Bligh wrote:
> Here's my start at a list ... I'm sure it's woefully incomplete.
> 
> 1. Utilize all CPUs roughly evenly for IRQ processing load (anything that's
> not measured by the scheduler at least, because it's unfair to other 
> processes).

yep; irqbalance approximates irq processing load by irq count, which seems
to be ok-ish so far.

> Also, we may well have more than 1 CPU's worth of traffic to
> process in a large network server.

One NIC? I've yet to see that ;)

> 2. Provide some sort of cache-affinity for network interrupt processing,
> which also helps us not get into out-of-order packet situations.

yep; irqbalance does that 

> 3. Utilize idle CPUs where possible to shoulder the load.

this is in direct conflict with 2; esp since cpus are idle very short times
all the time in busy scenarios (and non-busy scenarios are boring wrt irq
loadf ;)

 
> 4. Provide such a solution for all architectures.

irqbalanced in principle arch independent since the /proc interface is quite
generic..

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: CONFIG_IRQBALANCE for AMD64?

[Martin J. Bligh]

>> Here's my start at a list ... I'm sure it's woefully incomplete.
>> 
>> 1. Utilize all CPUs roughly evenly for IRQ processing load (anything that's
>> not measured by the scheduler at least, because it's unfair to other 
>> processes).
> 
> yep; irqbalance approximates irq processing load by irq count, which seems
> to be ok-ish so far.

Isn't that exactly what the in-kernel one does though? which people were
complaining about (wrt network backend (softirq?) processing)? And some interrupts are much heavier than others, surely?
 
>> Also, we may well have more than 1 CPU's worth of traffic to
>> process in a large network server.
> 
> One NIC? I've yet to see that ;)

No, multiple NICs. but if I shove 8 gigabit adaptors in a machine, we need
more than 1 cpu to process it ... 
 
>> 2. Provide some sort of cache-affinity for network interrupt processing,
>> which also helps us not get into out-of-order packet situations.
> 
> yep; irqbalance does that 
> 
>> 3. Utilize idle CPUs where possible to shoulder the load.
> 
> this is in direct conflict with 2; esp since cpus are idle very short times
> all the time in busy scenarios (and non-busy scenarios are boring wrt irq
> loadf ;)

Mmmm. for benchmarking scenarios, maybe ... but I do believe than machines
aren't maxed out all the time. definitely a problem to determine how long
the idle interval is though. Past history might be a clue, but ... yes, not
easy.

>> 4. Provide such a solution for all architectures.
> 
> irqbalanced in principle arch independent since the /proc interface is quite
> generic..

In principle either way *could* be arch independant ... though the in-kernel
one certainly isn't right now. 

Is there actually any algorithmic difference between the user-space and
in-kernel ones? or is this just a philosophical debate about user vs kernel
placement of code? ;-)

M.

Re: CONFIG_IRQBALANCE for AMD64?

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 1585 bytes --]

On Fri, May 28, 2004 at 11:57:24AM -0700, Martin J. Bligh wrote:
> >> Here's my start at a list ... I'm sure it's woefully incomplete.
> >> 
> >> 1. Utilize all CPUs roughly evenly for IRQ processing load (anything that's
> >> not measured by the scheduler at least, because it's unfair to other 
> >> processes).
> > 
> > yep; irqbalance approximates irq processing load by irq count, which seems
> > to be ok-ish so far.
> 

> Isn't that exactly what the in-kernel one does though? which people were
> complaining about (wrt network backend (softirq?) processing)? And some
> interrupts are much heavier than others, surely?

irqbalance uses class based balancing to try to balance the "some are
heavier than others" thing. 

>  
> >> Also, we may well have more than 1 CPU's worth of traffic to
> >> process in a large network server.
> > 
> > One NIC? I've yet to see that ;)
> 
> No, multiple NICs. but if I shove 8 gigabit adaptors in a machine, we need
> more than 1 cpu to process it ... 

yeah, and if you have more than 1 cpu, irqbalanced *will* spread them.

> Is there actually any algorithmic difference between the user-space and
> in-kernel ones? or is this just a philosophical debate about user vs kernel
> placement of code? ;-)

the userspace one is quite different and uses different level of information
(for example it makes class groups of irq's, eg it groups all ethernet
irq's, all storage irq's etc in separate classes and uses the class info in
the balancing algorithm). That surely is beyond what you'd want to do inside
the kernel, but it works great ;)



[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: CONFIG_IRQBALANCE for AMD64?

[michael]

Arjan van de Ven <arjanv@redhat.com> writes:
> On Fri, May 28, 2004 at 11:33:32AM -0700, Martin J. Bligh wrote:
> > Here's my start at a list ... I'm sure it's woefully incomplete.
> > 
> > 1. Utilize all CPUs roughly evenly for IRQ processing load (anything that's
> > not measured by the scheduler at least, because it's unfair to other 
> > processes).
> 
> yep; irqbalance approximates irq processing load by irq count, which seems
> to be ok-ish so far.
> 
> > Also, we may well have more than 1 CPU's worth of traffic to
> > process in a large network server.
> 
> One NIC? I've yet to see that ;)

Not too hard. Take a gig-e adaptor, add a bunch of iptables
rules, add some NAT, and you'll max out the CPU a long way
short of a gigabit...

Michael.

Re: CONFIG_IRQBALANCE for AMD64?

[michael]

Arjan van de Ven <arjanv@redhat.com> writes:
> On Fri, May 28, 2004 at 11:33:32AM -0700, Martin J. Bligh wrote:
[...]
> > Also, we may well have more than 1 CPU's worth of traffic to
> > process in a large network server.
> 
> One NIC? I've yet to see that ;)

Oh, and another corner case. 

Say you have a cpu-bound process on an SMP box.
Say you're also using a large chunk of a CPU processing
interrupts from a single IRQ.

What stops the cpu-bound process being scheduled onto
the same CPU as the interrupt handlers?

Now you've got one idle CPU, and one seriously overloaded
CPU.

Michael.

Re: CONFIG_IRQBALANCE for AMD64?

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 906 bytes --]

On Sat, May 29, 2004 at 06:41:46PM +1000, michael@optusnet.com.au wrote:
> Arjan van de Ven <arjanv@redhat.com> writes:
> > On Fri, May 28, 2004 at 11:33:32AM -0700, Martin J. Bligh wrote:
> [...]
> > > Also, we may well have more than 1 CPU's worth of traffic to
> > > process in a large network server.
> > 
> > One NIC? I've yet to see that ;)
> 
> Oh, and another corner case. 
> 
> Say you have a cpu-bound process on an SMP box.
> Say you're also using a large chunk of a CPU processing
> interrupts from a single IRQ.
> 
> What stops the cpu-bound process being scheduled onto
> the same CPU as the interrupt handlers?
> 
> Now you've got one idle CPU, and one seriously overloaded
> CPU.

yes and the only real answer here is to make the scheduler move the process.
"balancing" the irq (say every other irq) will actually use BOTH cpus 100%
(yes balancing is that expensive due to cache misses :)


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

RE: CONFIG_IRQBALANCE for AMD64?

[Nakajima, Jun]

>From: Andi Kleen [mailto:ak@muc.de]
>Sent: Friday, May 28, 2004 3:54 PM
>To: Nakajima, Jun
>Cc: Andi Kleen; Martin J. Bligh; linux-kernel@vger.kernel.org
>Subject: Re: CONFIG_IRQBALANCE for AMD64?
>
>On Fri, May 28, 2004 at 03:05:48PM -0700, Nakajima, Jun wrote:
>> >At least the AMD chipsets found in most Opteron boxes need software
>> >balancing too.
>>
>> Actually lowest priority delivery works on P4 and AMD (I did not
tested
>> it on AMD, though), if we _update_ TPR. But I don't recommend that,
>
>True. I remember there was even a patch for that from James C. long
ago.
>I considered at one point to add it to x86-64, but ended up
>not doing anything and just recommending irqbalanced.
>
>[I didn't test it neither, so no guarantee TPR really works on AMD]
>
>> instead we should implement the similar or optimized behavior in
>> software because "soft TPR" can be more efficient and scalable. And I
>> think this is something in my mind, and I think the kernel should do
it.
>
>I'm not convinced of that. At least the current i386 implemention
>is basically a kernel thread that wakes up regularly, reads some
>statistics and then updates the APIC.

That's true, and the kirqd code basically fixed the initial solution
from Ingo, i.e. round-robin. However, during the development, we found
such simple things (e.g. round-robin) worked better in _some cases_ than
the statistics-based irq balancing. I think SuSE has some improved
round-robin for x86. 

The problems with the initial round-robin were basically:
1. Too frequent -- it hurts cache, as you know
2. Interrupts throng to idle CPUs -- everybody jumps to idle CPU(s) if
any
3. IRQs move together in sync -- they bomb the same CPU together

Instead of digging those more, we implemented the current code in the
kernel simply because we got that idea. So based on those experiences,
there are a couple things we should explore more, and "soft TPR" will be
one of them. Anyway, we need to show the code and data, to convince
people.

Jun

>
>There is not much reason this cannot be done in user space, and
>user space has the advantage that more advanced heuristics (which
>I'm sure will be there) can be more easily implemented.
>
>And handling all interrupts at CPU #0 during early boot up is
>not really an issue.
>
>An kernel implementation may make sense when you're doing something
>really dynamic: e.g. not just a timer, but dynamically redirecting
>network interrupts to the CPU the process who will read from the
>socket runs on. Obviously it would need kernel support for that, since
>user space could not keep up with such a high sampling rate. But that's
>future research work (if it can be even done generically at all)
>and I don't see it on the radar screen anytime soon. We first need to
solve
>the NUMA scheduling problem, which is already hard enough ;-)
>
>And for the simpler heuristics that don't need real time updates
>user space is probably better.
>
>-Andi
>

RE: CONFIG_IRQBALANCE for AMD64?

[Nakajima, Jun]

>From: Andi Kleen [mailto:ak@muc.de]
>Sent: Friday, May 28, 2004 2:45 PM
>To: Martin J. Bligh
>Cc: linux-kernel@vger.kernel.org; Nakajima, Jun
>Subject: Re: CONFIG_IRQBALANCE for AMD64?
>
>"Martin J. Bligh" <mbligh@aracnet.com> writes:
>
>> Whatever we do ... all arches are going to need to provide a way to
>direct
>> interrupts to a certain CPU, or group thereof. Can they all do that
>already?
>> I'll confess to not having looked at non-i386 arches. And are others
as
>> brain damaged as the P4? or do they do something round-robin by
default?
>
>I wouldn't really blame the the P4, it's the IO-APICs in the chipsets
>that balance or not balance.
>
>At least the AMD chipsets found in most Opteron boxes need software
>balancing too.

Actually lowest priority delivery works on P4 and AMD (I did not tested
it on AMD, though), if we _update_ TPR. But I don't recommend that,
instead we should implement the similar or optimized behavior in
software because "soft TPR" can be more efficient and scalable. And I
think this is something in my mind, and I think the kernel should do it.

Jun

>
>-Andi
>

Re: CONFIG_IRQBALANCE for AMD64?

[Andi Kleen]

On Fri, May 28, 2004 at 03:05:48PM -0700, Nakajima, Jun wrote:
> >At least the AMD chipsets found in most Opteron boxes need software
> >balancing too.
> 
> Actually lowest priority delivery works on P4 and AMD (I did not tested
> it on AMD, though), if we _update_ TPR. But I don't recommend that,

True. I remember there was even a patch for that from James C. long ago.
I considered at one point to add it to x86-64, but ended up 
not doing anything and just recommending irqbalanced.

[I didn't test it neither, so no guarantee TPR really works on AMD]

> instead we should implement the similar or optimized behavior in
> software because "soft TPR" can be more efficient and scalable. And I
> think this is something in my mind, and I think the kernel should do it.

I'm not convinced of that. At least the current i386 implemention
is basically a kernel thread that wakes up regularly, reads some
statistics and then updates the APIC.

There is not much reason this cannot be done in user space, and
user space has the advantage that more advanced heuristics (which
I'm sure will be there) can be more easily implemented.

And handling all interrupts at CPU #0 during early boot up is 
not really an issue.

An kernel implementation may make sense when you're doing something
really dynamic: e.g. not just a timer, but dynamically redirecting
network interrupts to the CPU the process who will read from the
socket runs on. Obviously it would need kernel support for that, since
user space could not keep up with such a high sampling rate. But that's
future research work (if it can be even done generically at all)
and I don't see it on the radar screen anytime soon. We first need to solve
the NUMA scheduling problem, which is already hard enough ;-) 

And for the simpler heuristics that don't need real time updates
user space is probably better.

-Andi

Re: CONFIG_IRQBALANCE for AMD64?

[Nick Piggin]

Andi Kleen wrote:

> And handling all interrupts at CPU #0 during early boot up is 
> not really an issue.
> 
> An kernel implementation may make sense when you're doing something
> really dynamic: e.g. not just a timer, but dynamically redirecting
> network interrupts to the CPU the process who will read from the
> socket runs on. Obviously it would need kernel support for that, since
> user space could not keep up with such a high sampling rate. But that's
> future research work (if it can be even done generically at all)
> and I don't see it on the radar screen anytime soon. We first need to solve
> the NUMA scheduling problem, which is already hard enough ;-) 
> 

We're actually doing the converse of this in the sched-domain
scheduler. Processes have a tendancy to follow the interrupts
(ie. try to get onto the same CPU as them).

This makes good interrupt balancing important.

I have a feeling it might be best to keep the interrupts on
the closest CPUs, and move processes to match. Or possibly a
mix of both approaches.

Re: CONFIG_IRQBALANCE for AMD64?

[Andi Kleen]

> We're actually doing the converse of this in the sched-domain
> scheduler. Processes have a tendancy to follow the interrupts
> (ie. try to get onto the same CPU as them).

Hmm? I don't see any code in sched-domain that would care about
interrupts. What I'm missing?

-Andi

Re: CONFIG_IRQBALANCE for AMD64?

[Nick Piggin]

Andi Kleen wrote:
>>We're actually doing the converse of this in the sched-domain
>>scheduler. Processes have a tendancy to follow the interrupts
>>(ie. try to get onto the same CPU as them).
> 
> 
> Hmm? I don't see any code in sched-domain that would care about
> interrupts. What I'm missing?
> 

Well no, what I should have said is wakeups. We do the same
for all wakeups, so there is nothing interrupt specific you
are right.

Re: CONFIG_IRQBALANCE for AMD64?

[Andi Kleen]

On Sat, May 29, 2004 at 08:10:39PM +1000, Nick Piggin wrote:
> Andi Kleen wrote:
> >>We're actually doing the converse of this in the sched-domain
> >>scheduler. Processes have a tendancy to follow the interrupts
> >>(ie. try to get onto the same CPU as them).
> >
> >
> >Hmm? I don't see any code in sched-domain that would care about
> >interrupts. What I'm missing?
> >
> 
> Well no, what I should have said is wakeups. We do the same
> for all wakeups, so there is nothing interrupt specific you
> are right.

wakeups are too late e.g. for network processing.

-Andi

Re: CONFIG_IRQBALANCE for AMD64?

[Andi Kleen]

"Martin J. Bligh" <mbligh@aracnet.com> writes:

> Whatever we do ... all arches are going to need to provide a way to direct
> interrupts to a certain CPU, or group thereof. Can they all do that already?
> I'll confess to not having looked at non-i386 arches. And are others as
> brain damaged as the P4? or do they do something round-robin by default?

I wouldn't really blame the the P4, it's the IO-APICs in the chipsets
that balance or not balance.

At least the AMD chipsets found in most Opteron boxes need software 
balancing too.

-Andi

RE: CONFIG_IRQBALANCE for AMD64?

[Nakajima, Jun]

>From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
>owner@vger.kernel.org] On Behalf Of Chris Wedgwood
>Sent: Thursday, May 27, 2004 2:38 PM
>To: Arjan van de Ven; Anton Blanchard
>Cc: Thomas Zehetbauer; 'linux-kernel@vger.kernel.org'
>Subject: Re: CONFIG_IRQBALANCE for AMD64?
>
>On Thu, May 27, 2004 at 06:50:25PM +0200, Arjan van de Ven wrote:
>
>> irqbalanced has NOT been obsoleted by CONFIG_IRQBALANCE.
>
>On Fri, May 28, 2004 at 03:03:34AM +1000, Anton Blanchard wrote:
>
>> > Seems to work, just like the i386 irqbalanced before it has been
>> > obsoleted by CONFIG_IRQBALANCE
>>
>> No, CONFIG_IRQBALANCE is an x86 specific hack.
>
The issue is a xAPIC thing, and the both kernel-level and user-level are
applicable to x86_64 as well. 

The kernel does the default IRQ balancing, without assuming a user-level
irq balancing (because it's a distribution issue). If the user-level has
better knowledge, it just does a write to /proc/irq/N/smp_affinity to
bind that IRQ to a particular CPU, as Arjan's program is doing. In other
words, the kernel-level does _not_ move the ones bound by the
user-level.

>
>
>Why do we have CONFIG_IRQBALANCE at all then?
>
Today Linux is used for various configurations, including the ones that
substantially limit the set of user commands, libraries, etc. So we want
to keep it.

Jun

>
>
>  --cw
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel"
in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/

Re: CONFIG_IRQBALANCE for AMD64?

[Jeff Garzik]

Nakajima, Jun wrote:
> Today Linux is used for various configurations, including the ones that
> substantially limit the set of user commands, libraries, etc. So we want
> to keep it.


With all due respect, "it might be an embedded box" is not normally a 
reason why we keep stuff in the kernel.  With initramfs et. al., we are 
actively moving in the opposite direction.

If this was the only reason for having kirqd in the kernel, it would be 
long gone.

The reason why kirqd hasn't been removed is simply because nobody has 
stepped up to do a apples-to-apples comparison to prove that userland 
irqbalanced has any performance advantages, or disadvantages, over 
kirqd.  From a hard-numbers perspective, compared to kirqd, the userland 
solution is still largely an unknown quantity.

irqbalanced makes a lot of sense from a flexibility and policy 
perspective, and it works on multiple arches, so it has a lot more going 
for it.

"We like it in the kernel so we don't have to ship a userland component" 
is not a valid reason.

	Jeff

Re: CONFIG_IRQBALANCE for AMD64?

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 340 bytes --]

On Fri, May 28, 2004 at 01:40:15PM -0400, Jeff Garzik wrote:
> kirqd.  From a hard-numbers perspective, compared to kirqd, the userland 
> solution is still largely an unknown quantity.

unfortionately I have no publishable benchmarks ;( Doesn't mean it's an
unknown; all specweb and TPC benchmarks done with RHEL3 are with it for
example.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: CONFIG_IRQBALANCE for AMD64?

[Martin J. Bligh]

> With all due respect, "it might be an embedded box" is not normally a reason why we keep stuff in the kernel.  With initramfs et. al., we are actively moving in the opposite direction.
> 
> If this was the only reason for having kirqd in the kernel, it would be long gone.
> 
> The reason why kirqd hasn't been removed is simply because nobody has stepped up to do a apples-to-apples comparison to prove that userland irqbalanced has any performance advantages, or disadvantages, over kirqd.  From a hard-numbers perspective, compared to kirqd, the userland solution is still largely an unknown quantity.
> 
> irqbalanced makes a lot of sense from a flexibility and policy perspective, and it works on multiple arches, so it has a lot more going for it.
> 
> "We like it in the kernel so we don't have to ship a userland component" is not a valid reason.

Personally, I find the argument that it's hardware-specific control code
a much better reason for it to belong in the kernel. But now it's a config
option, everyone can do what they like ... so there's no need to argue
back and forth any more. Not to say the in-kernel one doesn't need some
fixing up, but we're planning on doing that later this year.

M.

Re: CONFIG_IRQBALANCE for AMD64?

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 234 bytes --]

On Fri, May 28, 2004 at 10:46:18AM -0700, Martin J. Bligh wrote:
> 
> Personally, I find the argument that it's hardware-specific control code
> a much better reason for it to belong in the kernel. 

Is it really hardware specific ??

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: CONFIG_IRQBALANCE for AMD64?

[Nivedita Singhvi]

Arjan van de Ven wrote:

> On Fri, May 28, 2004 at 10:46:18AM -0700, Martin J. Bligh wrote:
> 
>>Personally, I find the argument that it's hardware-specific control code
>>a much better reason for it to belong in the kernel. 
> 
> 
> Is it really hardware specific ??

I have limited knowledge on this, but a great deal of interest.
So, please forgive some stupid questions and wrong assumptions.

The irqbalanced is a user space daemon that attempts to
balance irqs across CPUs. It keeps track of the current
irq counts on the CPUs, and at regular intervals applies
changes to irq binding in order to implement the desired
policy. It achieves a high-level long term balance of irqs
across CPUs.

This is a fairly expensive but generally arch independent
(as long as they support cpu binding of irqs) method to
achieve long term distribution of interrupts.

I think this is best used for fairly balanced (over time)
long running workloads. For short workloads which demonstrate
intense activity in bursts, this won't be as effective.

For fine grained balancing of interrupts, I don't see how
you can avoid implementing something in hw/fw/low level
kernel.

I see networking interrupts requiring fine granularity
balancing, to avoid the potential for dropped packets and
long latencies.  That is, given a busy server that is
seeing a lot of interrupts, fair distribution of the
interrupts is required within a very small amount of time,
and we cannot require a user space daemon that parses the /proc
file system and applies a policy and then rebinds irqs
to different CPUs to run with that frequency.

I would like (with my very limited knowledge) something
that would automatically on every interrupt send it to
the optimum CPU (one that was idle, or least loaded)..
This might involve too much overhead to keep track of,
but some improvement over round robining of the interrupts
coming in should be possible..

Does this make sense?

thanks,
Nivedita

Re: CONFIG_IRQBALANCE for AMD64?

[Jeff Garzik]

Nivedita Singhvi wrote:
> I see networking interrupts requiring fine granularity
> balancing, to avoid the potential for dropped packets and
> long latencies.  That is, given a busy server that is
> seeing a lot of interrupts, fair distribution of the
> interrupts is required within a very small amount of time,
> and we cannot require a user space daemon that parses the /proc
> file system and applies a policy and then rebinds irqs
> to different CPUs to run with that frequency.


Network is one of the areas where you _don't_ want to be constantly 
pointing your NIC's irq to different CPUs.

Cache affinity, packet re-ordering problems, and other fun ensue.

	Jeff

Re: CONFIG_IRQBALANCE for AMD64?

[Nivedita Singhvi]

Jeff Garzik wrote:
> Nivedita Singhvi wrote:

> Network is one of the areas where you _don't_ want to be constantly 
> pointing your NIC's irq to different CPUs.
> 
> Cache affinity, packet re-ordering problems, and other fun ensue.
> 
>     Jeff

I agree that there is a tradeoff point - we need the cache
affinity, but you also want to employ the other CPUs which
might be idle in the system - so there is a timeframe that
is ideal to have one CPU pick up interrupts, and then move
on to another - but that timeframe is surely shorter than
then every 5 mins or so that the irq daemon might run,
correct?

Also, yep, packet re-ordering is an issue for TCP connections,
and we do have mechanisms in the kernel to address it to some
extent - so we don't trigger fast retransmit unnecessarily.
And if you have a lot of UDP and other traffic, or short lived
connections, then you are unnecessarily subjecting them to
a less ideal environment.

I agree that this is very tricky stuff to optimize, and I
certainly don't know how to go about picking the best
heuristic here, but I am wondering how the frequency with
which a user space daemon might run (and you don't want it
to run too often given the cost) would be able to handle
something that does require finer granularity. Or is it
everybody's understanding that fine grained balancing is
not needed?

thanks,
Nivedita

Re: CONFIG_IRQBALANCE for AMD64?

[Jeff Garzik]

Long term, when NIC hardware is more advanced, we'll want something that 
divides sockets/connections/etc. into per-CPU clusters...  something 
that's friendly to both SMP and NUMA+SMP configurations.

Or if enterprising companies get a clue, and open their NIC firmware, I 
bet we could do this now.

	Jeff

Re: CONFIG_IRQBALANCE for AMD64?

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 972 bytes --]

On Fri, May 28, 2004 at 12:51:42PM -0700, Nivedita Singhvi wrote:
> The irqbalanced is a user space daemon that attempts to
> balance irqs across CPUs. It keeps track of the current
> irq counts on the CPUs, and at regular intervals applies
> changes to irq binding in order to implement the desired
> policy. It achieves a high-level long term balance of irqs
> across CPUs.
> 
> This is a fairly expensive but generally arch independent
> (as long as they support cpu binding of irqs) method to
> achieve long term distribution of interrupts.

it's not THAT expensive. Really.


> I think this is best used for fairly balanced (over time)
> long running workloads. For short workloads which demonstrate
> intense activity in bursts, this won't be as effective.

intense bursts average out... and to be honest, in bursts the last thing you
want to do is move it to different cpus all the time since then you get so
much cross cpu cache misses and you are far slower....


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

CONFIG_IRQBALANCE for AMD64?

[Thomas Zehetbauer]

[-- Attachment #1: Type: text/plain, Size: 423 bytes --]

I wonder why there is no in-kernel irq balancing for the AMD64
architecture yet. I guess this shouldn't be much different to the i386
code. Someone willing to explain/provide a patch?

Tom
-- 
  T h o m a s   Z e h e t b a u e r   ( TZ251 )
  PGP encrypted mail preferred - KeyID 96FFCB89
      finger thomasz@hostmaster.org for key

Windows 98 supports real multitasking - it can boot and crash simultaneously.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 481 bytes --]

Re: CONFIG_IRQBALANCE for AMD64?

[Jeff Garzik]

Thomas Zehetbauer wrote:
> I wonder why there is no in-kernel irq balancing for the AMD64
> architecture yet. I guess this shouldn't be much different to the i386
> code. Someone willing to explain/provide a patch?


Why do you think you need in-kernel irq balancing?

Does userspace irqbalanced not work for you?

	Jeff

Re: CONFIG_IRQBALANCE for AMD64?

[Thomas Zehetbauer]

[-- Attachment #1: Type: text/plain, Size: 729 bytes --]

On Don, 2004-05-27 at 07:13, Jeff Garzik wrote:
> Why do you think you need in-kernel irq balancing?

Very good question indeed, why is it there for the i386 architecture at
all?

I just don't like the idea of unnecessary branching for the AMD64
architecture increasing maintenance and testing affords and thereby
generating less maintained and tested code.

> Does userspace irqbalanced not work for you?

Seems to work, just like the i386 irqbalanced before it has been
obsoleted by CONFIG_IRQBALANCE

Tom

-- 
  T h o m a s   Z e h e t b a u e r   ( TZ251 )
  PGP encrypted mail preferred - KeyID 96FFCB89
      finger thomasz@hostmaster.org for key

Press any key to continue or any other key to quit.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 481 bytes --]

Re: CONFIG_IRQBALANCE for AMD64?

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 166 bytes --]


> Seems to work, just like the i386 irqbalanced before it has been
> obsoleted by CONFIG_IRQBALANCE

irqbalanced has NOT been obsoleted by CONFIG_IRQBALANCE.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: CONFIG_IRQBALANCE for AMD64?

[Chris Wedgwood]

On Thu, May 27, 2004 at 06:50:25PM +0200, Arjan van de Ven wrote:

> irqbalanced has NOT been obsoleted by CONFIG_IRQBALANCE.

On Fri, May 28, 2004 at 03:03:34AM +1000, Anton Blanchard wrote:

> > Seems to work, just like the i386 irqbalanced before it has been
> > obsoleted by CONFIG_IRQBALANCE
>
> No, CONFIG_IRQBALANCE is an x86 specific hack.



Why do we have CONFIG_IRQBALANCE at all then?



  --cw

Re: CONFIG_IRQBALANCE for AMD64?

[Anton Blanchard]

 
> Seems to work, just like the i386 irqbalanced before it has been
> obsoleted by CONFIG_IRQBALANCE

No, CONFIG_IRQBALANCE is an x86 specific hack.

Anton

Re: CONFIG_IRQBALANCE for AMD64?

[Thomas Zehetbauer]

Unfortunately I couldn't find much info on the topic, could you please
provide some more information? Why is there a kirqd at all?

What are the differences/advantages/disadvantages between the kirqd and
the user space implementation?

TIA
Tom

Re: CONFIG_IRQBALANCE for AMD64?

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 843 bytes --]

On Fri, 2004-05-28 at 00:36, Thomas Zehetbauer wrote:
> Unfortunately I couldn't find much info on the topic, could you please
> provide some more information? Why is there a kirqd at all?

For people with old distributions that didn't include irqbalanced yet ;)

> 
> What are the differences/advantages/disadvantages between the kirqd and
> the user space implementation?

the userspace implementation implements a more sophisticated algorithm
for balancing irq's (well to be fair, since its in userspace it's easier
to do this so it's not kirqd's failt that it doesn't have that). 
While I didn't test kirqd to compare, we did see quite a difference
(order of percents) in a specweb load between a naive algorithm in
kirqbalanced and the current algorithm. Otoh few real life workloads get
as many interrupts as specweb ;)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]