[PATCH] CPEI gets warning at kernel/irq/migration.c:27/move_masked_irq()

[PATCH] CPEI gets warning at kernel/irq/migration.c:27/move_masked_irq()

[Hidetoshi Seto]

Hi,

While running my MCA test (hardware error injection) on 2.6.19,
I got some warning like following:

> BUG: warning at kernel/irq/migration.c:27/move_masked_irq()
>
> Call Trace:
>  [<a000000100013d20>] show_stack+0x40/0xa0
>                                 sp=e00000006b2578d0 bsp=e00000006b2510b0
>  [<a000000100013db0>] dump_stack+0x30/0x60
>                                 sp=e00000006b257aa0 bsp=e00000006b251098
>  [<a0000001000de430>] move_masked_irq+0xb0/0x240
>                                 sp=e00000006b257aa0 bsp=e00000006b251070
>  [<a0000001000de6a0>] move_native_irq+0xe0/0x180
>                                 sp=e00000006b257aa0 bsp=e00000006b251040
>  [<a00000010004ff50>] iosapic_end_level_irq+0x30/0xe0
>                                 sp=e00000006b257aa0 bsp=e00000006b251020
>  [<a0000001000d94d0>] __do_IRQ+0x170/0x400
>                                 sp=e00000006b257aa0 bsp=e00000006b250fd8
>  [<a0000001000116f0>] ia64_handle_irq+0x1b0/0x260
>                                 sp=e00000006b257aa0 bsp=e00000006b250fa8
>  [<a00000010000c3a0>] ia64_leave_kernel+0x0/0x280
>                                 sp=e00000006b257aa0 bsp=e00000006b250fa8
>  [<a000000100690cf0>] _spin_unlock_irqrestore+0x30/0x60
>                                 sp=e00000006b257c70 bsp=e00000006b250f90

It comes from:

[kernel/irq/migration.c]
  26         if (CHECK_IRQ_PER_CPU(desc->status)) {
  27                 WARN_ON(1);
  28                 return;
  29         }

By putting some printk in kernel, I found that irqbalance is trying to
move CPEI which is handled as PER_CPU irq. That's why.

CPEI(Corrected Platform Error Interrupt) is ia64 specific irq, is
allowed to pin to particular processor which selected by the platform, and
even it is PER_CPU but it has set_affinity handler (=iosapic_set_affinity)
as same as other IO-SAPIC-level interrupts. (I don't know why, but
I guess that there would be typical situation where the handler for
migration is needed, such as hotplug - the processor going to be
offline/hot-removed.)

To shut up this warning, there are 2 way at least:
 a) fix CPEI stuff
 b) prohibit setting affinity to PER_CPU irq

I'm not sure what stuff of CPEI need to be fixed, but I think that
returning error to attempting move PER_CPU irq is useful for all
applications since it will never work.

Following small patch takes b) style.
It works, the warning disappeared and irqbalance still runs well.

Thanks,
H.Seto

Signed-off-by: Hidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>

---
 kernel/irq/proc.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Index: linux-2.6.19/kernel/irq/proc.c
===================================================================
--- linux-2.6.19.orig/kernel/irq/proc.c
+++ linux-2.6.19/kernel/irq/proc.c
@@ -54,7 +54,8 @@ static int irq_affinity_write_proc(struc
 	unsigned int irq = (int)(long)data, full_count = count, err;
 	cpumask_t new_value, tmp;

-	if (!irq_desc[irq].chip->set_affinity || no_irq_affinity)
+	if (!irq_desc[irq].chip->set_affinity || no_irq_affinity ||
+				CHECK_IRQ_PER_CPU(irq_desc[irq].status))
 		return -EIO;

 	err = cpumask_parse_user(buffer, count, new_value);

Re: [PATCH] CPEI gets warning at kernel/irq/migration.c:27/move_masked_irq()

[Andrew Morton]

On Tue, 05 Dec 2006 16:35:06 +0900
Hidetoshi Seto <seto.hidetoshi@jp.fujitsu.com> wrote:

> Hi,
> 
> While running my MCA test (hardware error injection) on 2.6.19,
> I got some warning like following:
> 
> > BUG: warning at kernel/irq/migration.c:27/move_masked_irq()
> >
> > Call Trace:
> >  [<a000000100013d20>] show_stack+0x40/0xa0
> >                                 sp=e00000006b2578d0 bsp=e00000006b2510b0
> >  [<a000000100013db0>] dump_stack+0x30/0x60
> >                                 sp=e00000006b257aa0 bsp=e00000006b251098
> >  [<a0000001000de430>] move_masked_irq+0xb0/0x240
> >                                 sp=e00000006b257aa0 bsp=e00000006b251070
> >  [<a0000001000de6a0>] move_native_irq+0xe0/0x180
> >                                 sp=e00000006b257aa0 bsp=e00000006b251040
> >  [<a00000010004ff50>] iosapic_end_level_irq+0x30/0xe0
> >                                 sp=e00000006b257aa0 bsp=e00000006b251020
> >  [<a0000001000d94d0>] __do_IRQ+0x170/0x400
> >                                 sp=e00000006b257aa0 bsp=e00000006b250fd8
> >  [<a0000001000116f0>] ia64_handle_irq+0x1b0/0x260
> >                                 sp=e00000006b257aa0 bsp=e00000006b250fa8
> >  [<a00000010000c3a0>] ia64_leave_kernel+0x0/0x280
> >                                 sp=e00000006b257aa0 bsp=e00000006b250fa8
> >  [<a000000100690cf0>] _spin_unlock_irqrestore+0x30/0x60
> >                                 sp=e00000006b257c70 bsp=e00000006b250f90
> 
> It comes from:
> 
> [kernel/irq/migration.c]
>   26         if (CHECK_IRQ_PER_CPU(desc->status)) {
>   27                 WARN_ON(1);
>   28                 return;
>   29         }
> 
> By putting some printk in kernel, I found that irqbalance is trying to
> move CPEI which is handled as PER_CPU irq. That's why.
> 
> CPEI(Corrected Platform Error Interrupt) is ia64 specific irq, is
> allowed to pin to particular processor which selected by the platform, and
> even it is PER_CPU but it has set_affinity handler (=iosapic_set_affinity)
> as same as other IO-SAPIC-level interrupts. (I don't know why, but
> I guess that there would be typical situation where the handler for
> migration is needed, such as hotplug - the processor going to be
> offline/hot-removed.)
> 
> To shut up this warning, there are 2 way at least:
>  a) fix CPEI stuff
>  b) prohibit setting affinity to PER_CPU irq
> 
> I'm not sure what stuff of CPEI need to be fixed, but I think that
> returning error to attempting move PER_CPU irq is useful for all
> applications since it will never work.
> 
> Following small patch takes b) style.
> It works, the warning disappeared and irqbalance still runs well.
> 
> Thanks,
> H.Seto
> 
> Signed-off-by: Hidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
> 
> ---
>  kernel/irq/proc.c |    3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> Index: linux-2.6.19/kernel/irq/proc.c
> ===================================================================
> --- linux-2.6.19.orig/kernel/irq/proc.c
> +++ linux-2.6.19/kernel/irq/proc.c
> @@ -54,7 +54,8 @@ static int irq_affinity_write_proc(struc
>  	unsigned int irq = (int)(long)data, full_count = count, err;
>  	cpumask_t new_value, tmp;
> 
> -	if (!irq_desc[irq].chip->set_affinity || no_irq_affinity)
> +	if (!irq_desc[irq].chip->set_affinity || no_irq_affinity ||
> +				CHECK_IRQ_PER_CPU(irq_desc[irq].status))
>  		return -EIO;

It'd be nice if we could just teach the userspace balancer to not try to
move perpcu IRQs?

otoh, the patch is super-cheap.   Arjan?

Re: [PATCH] CPEI gets warning at kernel/irq/migration.c:27/move_masked_irq()

[Arjan van de Ven]

> It'd be nice if we could just teach the userspace balancer to not try to
> move perpcu IRQs?
> 
> otoh, the patch is super-cheap.   Arjan?

I can fix irqbalance no problem, however I like the kernel approach as
well, since it's not just irqbalance that moves irqs, sysadmins tend to
do it as well....  so how about both?

One thing we probably should do, and that would help irqbalance
immensely, is to export a bitmask for which cpus an interrupt CAN go to,
next to where the current binding interface is. I'll check into that

Hidetoshi: would it be possible to send me a /proc/interrupts file of
that machine?


-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org

Re: [PATCH] CPEI gets warning at kernel/irq/migration.c:27/move_masked_irq()

[Hidetoshi Seto]

Arjan van de Ven wrote:
>> It'd be nice if we could just teach the userspace balancer to not try to
>> move perpcu IRQs?
>>
>> otoh, the patch is super-cheap.   Arjan?
> 
> I can fix irqbalance no problem, however I like the kernel approach as
> well, since it's not just irqbalance that moves irqs, sysadmins tend to
> do it as well....  so how about both?

Thanks for positive comments.

> One thing we probably should do, and that would help irqbalance
> immensely, is to export a bitmask for which cpus an interrupt CAN go to,
> next to where the current binding interface is. I'll check into that
> 
> Hidetoshi: would it be possible to send me a /proc/interrupts file of
> that machine?

Here you are.
This machine is not special. I guess all ia64 machine will show
few or more but almost same lines. CPEI is #30.

$ cat /proc/interrupts
           CPU0       CPU1
 28:          0          0          LSAPIC  cpe_poll
 29:          0          0          LSAPIC  cmc_poll
 30:          0          0  IO-SAPIC-level  cpe_hndlr
 31:          0          0          LSAPIC  cmc_hndlr
 34:       4643    1881067   IO-SAPIC-edge  ide0
 39:          0          0  IO-SAPIC-level  acpi
 45:        148        560   IO-SAPIC-edge  serial
 48:          0          0  IO-SAPIC-level  uhci_hcd
 49:          0          0  IO-SAPIC-level  uhci_hcd
 50:          0          0  IO-SAPIC-level  ehci_hcd
 51:      21131    1257790  IO-SAPIC-level  eth0
 53:      21778     458314  IO-SAPIC-level  ioc0
 54:          0         44  IO-SAPIC-level  ioc1
232:          0          0          LSAPIC  mca_rdzv
238:          0          0          LSAPIC  perfmon
239:  214832233  214832003          LSAPIC  timer
240:          0          0          LSAPIC  mca_wkup
254:     209193        611          LSAPIC  IPI
ERR:          0

Thanks,
H.Seto