Question: How to distribute the interrupts over multiple cores?

Question: How to distribute the interrupts over multiple cores?

[Lenky Gao]

Hi everyone,

I faced the same problem as this URL:
http://stackoverflow.com/questions/13641440/unable-to-distrubute-the-interrupts-over-multiple-cores

The question have been closed with no answer. :(

My machine have 2x Xeon E5504 processors and running vanilla kernel
3.6.11 in CentOS 6.0.

Could anyone give me some tips?

PS:
[root@localhost ~]# cat /proc/cmdline
ro root=/dev/mapper/vg_localhost-lv_root rd_NO_LUKS LANG=en_US.UTF-8
rd_NO_MD quiet rd_LVM_LV=vg_localhost/lv_swap
SYSFONT=latarcyrheb-sun16 rhgb rd_LVM_LV=vg_localhost/lv_root
KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM apic=debug pci=nomsi
[root@localhost ~]# uname -a
Linux localhost.lenkydomain 3.6.11 #2 SMP Fri Mar 22 21:13:41 CST 2013
x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# echo 6 > /proc/irq/25/smp_affinity
[root@localhost ~]# cat /proc/irq/25/smp_affinity
000006
[root@localhost ~]# cat /proc/interrupts | grep eth9
  25:       5824      13914        159        128        210
0          0          0  IR-IO-APIC-fasteoi   eth9
[root@localhost ~]# cat /proc/interrupts | grep eth9
  25:       5824      13927        159        128        210
0          0          0  IR-IO-APIC-fasteoi   eth9
[root@localhost ~]# cat /proc/interrupts | grep eth9
  25:       5824      13939        159        128        210
0          0          0  IR-IO-APIC-fasteoi   eth9
[root@localhost ~]# cat /proc/interrupts | grep eth9
  25:       5824      13973        159        128        210
0          0          0  IR-IO-APIC-fasteoi   eth9


-- 
Regards,

Lenky

Re: Question: How to distribute the interrupts over multiple cores?

[Lin Feng]

Hi Gao,

On 03/25/2013 10:33 AM, Lenky Gao wrote:
> [root@localhost ~]# echo 6 > /proc/irq/25/smp_affinity
> [root@localhost ~]# cat /proc/irq/25/smp_affinity
> 000006
Seems you bind the nic irq to second and third cpu for the bit mask
you set is 110, so now eth9's irq is working on the 3rd cpu.

Have you ever tried irqbalance service? It may help to balance irq if the 
irq workload of 3rd cpu is too heavy.

thanks,
linfeng
> [root@localhost ~]# cat /proc/interrupts | grep eth9
>   25:       5824      13914        159        128        210
> 0          0          0  IR-IO-APIC-fasteoi   eth9
> [root@localhost ~]# cat /proc/interrupts | grep eth9
>   25:       5824      13927        159        128        210
> 0          0          0  IR-IO-APIC-fasteoi   eth9
> [root@localhost ~]# cat /proc/interrupts | grep eth9
>   25:       5824      13939        159        128        210
> 0          0          0  IR-IO-APIC-fasteoi   eth9
> [root@localhost ~]# cat /proc/interrupts | grep eth9
>   25:       5824      13973        159        128        210
> 0          0          0  IR-IO-APIC-fasteoi   eth9

Re: Question: How to distribute the interrupts over multiple cores?

[Lenky Gao]

> Seems you bind the nic irq to second and third cpu for the bit mask
> you set is 110, so now eth9's irq is working on the 3rd cpu.

> Have you ever tried irqbalance service? It may help to balance irq if the
> irq workload of 3rd cpu is too heavy.


I need the interrupts distribute over multiple cores, such as the
second and third, so i set the mask parameter to 0x3. But only one
core is handling the interrupts. As you can see, just the second cpu's
irq number has increased.

The irqbalance service has been stopped.

-- 
Regards,

Lenky

Re: Question: How to distribute the interrupts over multiple cores?

[Lin Feng]

Hi,

On 03/25/2013 11:18 AM, Lenky Gao wrote:
> The irqbalance service has been stopped.
So try start irqbalance to see what happen?
It should help to give what you want ;-)

thanks,
linfeng

Re: Question: How to distribute the interrupts over multiple cores?

[Lenky Gao]

> On 03/25/2013 11:18 AM, Lenky Gao wrote:
>> The irqbalance service has been stopped.
> So try start irqbalance to see what happen?
> It should help to give what you want ;-)

Using the irqbalance service to dynamically change the IRQ-bound? It's
seems a software solution. In my old machine, there is no this
problem. I need some information to explain why. Maybe is the
apic_physflat mode only support distribute the interrupts to one
processor, but i am not exactly sure if this is right? Or my E5504
Machine's hardware issue?

-- 
Regards,

Lenky

Re: Question: How to distribute the interrupts over multiple cores?

[Lin Feng]

Hi,

On 03/25/2013 11:44 AM, Lenky Gao wrote:
>> On 03/25/2013 11:18 AM, Lenky Gao wrote:
>>> The irqbalance service has been stopped.
>> So try start irqbalance to see what happen?
>> It should help to give what you want ;-)
> 
> Using the irqbalance service to dynamically change the IRQ-bound? It's
> seems a software solution. In my old machine, there is no this
> problem. I need some information to explain why. Maybe is the

Do you mean on your old machine the irq will be distributed automatically 
among the cpus set by smp_affinity?

Sorry, I forgot one thing that the irqbalance service will break the mask
set by smp_affinity. So I don't know if now we support the feature
that balance among the cpus set by smp_affinity.
 
In my old conception when we set smp_affinity the irqbalance service
should be stopped, so you are right before, and the kernel will choose
one cpu in your mask for the IRQ and bounded from now on if there is no
irqbalance service.

thanks,
linfeng 

Re: Question: How to distribute the interrupts over multiple cores?

[Lenky Gao]

> Do you mean on your old machine the irq will be distributed automatically
> among the cpus set by smp_affinity?
>

Yes. My another machine's interrupts are as follows:

[root@localhost ~]# cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 23
model name	: Pentium(R) Dual-Core  CPU      E6700  @ 3.20GHz
...
[root@localhost ~]# echo 3 > /proc/irq/18/smp_affinity
[root@localhost ~]# cat /proc/irq/18/smp_affinity
3
[root@localhost ~]# cat /proc/interrupts | grep eth2
  18:        455       458   IO-APIC-fasteoi   eth2
[root@localhost ~]# cat /proc/interrupts | grep eth2
  18:        463       467   IO-APIC-fasteoi   eth2
[root@localhost ~]# cat /proc/interrupts | grep eth2
  18:        471       476   IO-APIC-fasteoi   eth2


-- 
Regards,

Lenky

Re: Question: How to distribute the interrupts over multiple cores?

[Lin Feng]

On 03/25/2013 02:46 PM, Lenky Gao wrote:
>> Do you mean on your old machine the irq will be distributed automatically
>> among the cpus set by smp_affinity?
>>
> 
> Yes. My another machine's interrupts are as follows:
And without irqbalance service? It sounds weird to me..

thanks,
linfeng

> 
> [root@localhost ~]# cat /proc/cpuinfo
> processor	: 0
> vendor_id	: GenuineIntel
> cpu family	: 6
> model		: 23
> model name	: Pentium(R) Dual-Core  CPU      E6700  @ 3.20GHz
> ...
> [root@localhost ~]# echo 3 > /proc/irq/18/smp_affinity
> [root@localhost ~]# cat /proc/irq/18/smp_affinity
> 3
> [root@localhost ~]# cat /proc/interrupts | grep eth2
>   18:        455       458   IO-APIC-fasteoi   eth2
> [root@localhost ~]# cat /proc/interrupts | grep eth2
>   18:        463       467   IO-APIC-fasteoi   eth2
> [root@localhost ~]# cat /proc/interrupts | grep eth2
>   18:        471       476   IO-APIC-fasteoi   eth2
> 
> 

Re: Question: How to distribute the interrupts over multiple cores?

[Lenky Gao]

> And without irqbalance service? It sounds weird to me..

The irqbalance service has been stopped.


-- 
Regards,

Lenky

Re: Question: How to distribute the interrupts over multiple cores?

[Lenky Gao]

I have found a comment in function physflat_cpu_mask_to_apicid to explain why.

static unsigned int physflat_cpu_mask_to_apicid(const struct cpumask *cpumask)
{
	int cpu;

	/*
	 * We're using fixed IRQ delivery, can only return one phys APIC ID.
	 * May as well be the first.
	 */
...

This is mean i can not distribute the interrupts over multiple cores
when the machine have more than 8 cores?


-- 
Regards,

Lenky

Re: Question: How to distribute the interrupts over multiple cores?

[Lin Feng]

Hi,

On 03/25/2013 05:00 PM, Lenky Gao wrote:
> I have found a comment in function physflat_cpu_mask_to_apicid to explain why.
> 
> static unsigned int physflat_cpu_mask_to_apicid(const struct cpumask *cpumask)
> {
> 	int cpu;
> 
> 	/*
> 	 * We're using fixed IRQ delivery, can only return one phys APIC ID.
> 	 * May as well be the first.
> 	 */
> ...
> 
> This is mean i can not distribute the interrupts over multiple cores
> when the machine have more than 8 cores?
> 
> 
Sorry I can't find physflat_cpu_mask_to_apicid() in latest Linux tree, seems it has
been removed out.. But just consider what you said that it's limited by "8 cores"
IMHO it's impossible or it is a bug.

thanks,
linfeng

Re: Question: How to distribute the interrupts over multiple cores?

[Lenky Gao]

> Sorry I can't find physflat_cpu_mask_to_apicid() in latest Linux tree, seems it has
> been removed out.. But just consider what you said that it's limited by "8 cores"
> IMHO it's impossible or it is a bug.

Here:
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/arch/x86/kernel/apic/apic_flat_64.c?id=v2.6.30.8#n281



-- 
Regards,

Lenky