Re: irqbalance mandatory on SMP kernels?

["Martin J. Bligh" <mbligh@mbligh.org>]

Erik Mouw wrote:
> On Tue, Apr 18, 2006 at 08:19:17PM +0200, Arjan van de Ven wrote:
> 
>>>but spreading IRQ's across all of the CPU's doesn't seem like it's
>>>ever the right answer.
>>
>>well it is in some cases, imagine having 2 cpus and 2 gige nics that are
>>very busy doing webserving. That's an obvious case where 1-nic-per-cpu
>>ends up doing the right thing... the way it ends up is that each nic has
>>a full cpu for itself and it's own apaches... almost fully independent
>>of the other one. Now if you moved both irqs to the same cpu, the
>>apaches would follow, because if they didn't then you'd be bouncing
>>their data *all the time*. And at that point the other cpu will become
>>bored ;)
> 
> 
> So what happens with a dual amd64 system where each CPU has its "own"
> NIC? Something like this:
> 
> 
> MEM0 <--> CPU0 <--- HT ---> CPU1 <--> MEM1
>            ^                 ^
>            |                 |
>            HT                HT
>            |                 |
>            v                 v
>       PCI bridge0      PCI1 bridge
>            ^                 ^
>            |                 |
>           PCI               PCI
>            |                 |
>            v                 v
>        GigE NIC0         GigE NIC1
> 
> The "best" approach would be to run an Apache on each CPU using local
> memory and NIC and having the IRQs handled by the local CPU. Does the
> irq balancer allow such a configuraion, or would it be hamperd by the
> process scheduler deciding to run both Apaches on a single CPU?

The trouble is that we're not smart enough to redirect receiving traffic
back to the correct CPU, I think. You'd need to configure different IP
addrs on the same subnet to each NIC, and have intelligent bonding for
outbound traffic.

Then you'd need NUMA locality of IRQs, by bonding them to their closest
CPUs, which is something that's easily done inside the kernel, but is
harder in userspace. but I'm not getting into that silly pissing contest
again. Either mechanism *could* do it. It's just about creating sensbile
APIs and defaults, both of which we're not good at doing as a community.

M.