[-rt BUG] scheduling with irqs disabled: swapper

[-rt BUG] scheduling with irqs disabled: swapper

[john stultz]

Hey Ingo, All,
	We had the following bug reported on bootup on one of our boxes (it
was a 4way I believe) running -rt22. So far it seems to be a one-off
but I figured I'd post it to see if anyone had a clue.

thanks
-john

...
usbmon: debugfs is not available
usbcore: registered new driver hiddev
usbcore: registered new driver usbhid
drivers/usb/input/hid-core.c: v2.6:USB HID core driver
mice: PS/2 mouse device common for all mice
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
NET: Registered protocol family 2
BUG: scheduling with irqs disabled: swapper/0x00010002/0
caller is rt_lock_slowlock+0xd2/0x139
 [<c030fb85>] schedule+0x60/0xd0 (8)
 [<c0310787>] rt_lock_slowlock+0xd2/0x139 (12)
 [<c0310e2d>] __lock_text_start+0x1d/0x1e (72)
 [<c022a57f>] i8042_interrupt+0x2c/0x1f0 (4)
 [<c013edb1>] misrouted_irq+0x85/0x118 (36)
 [<c013ef08>] note_interrupt+0x43/0x9f (36)
 [<c013e395>] __do_IRQ+0xc2/0xf9 (16)

Re: [-rt BUG] scheduling with irqs disabled: swapper

[Steven Rostedt]

On Fri, 2006-05-26 at 15:53 -0700, john stultz wrote:
> Hey Ingo, All,
> 	We had the following bug reported on bootup on one of our boxes (it
> was a 4way I believe) running -rt22. So far it seems to be a one-off
> but I figured I'd post it to see if anyone had a clue.

I'm assuming this is a i386.  Also I'm assuming that frame pointers was
not compiled in since the stack is a little suspicious.

Anyway, could you show the /proc/interrupts of this machine.  I'm
curious if the i8042 isn't sharing an interrupt with something with
NODELAY in it.

-- Steve


> ...
> usbmon: debugfs is not available
> usbcore: registered new driver hiddev
> usbcore: registered new driver usbhid
> drivers/usb/input/hid-core.c: v2.6:USB HID core driver
> mice: PS/2 mouse device common for all mice
> md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
> md: bitmap version 4.39
> NET: Registered protocol family 2
> BUG: scheduling with irqs disabled: swapper/0x00010002/0
> caller is rt_lock_slowlock+0xd2/0x139
>  [<c030fb85>] schedule+0x60/0xd0 (8)
>  [<c0310787>] rt_lock_slowlock+0xd2/0x139 (12)
>  [<c0310e2d>] __lock_text_start+0x1d/0x1e (72)
>  [<c022a57f>] i8042_interrupt+0x2c/0x1f0 (4)
>  [<c013edb1>] misrouted_irq+0x85/0x118 (36)
>  [<c013ef08>] note_interrupt+0x43/0x9f (36)
>  [<c013e395>] __do_IRQ+0xc2/0xf9 (16)

Re: [-rt BUG] scheduling with irqs disabled: swapper

[john stultz]

On Fri, 2006-05-26 at 21:14 -0400, Steven Rostedt wrote:
> On Fri, 2006-05-26 at 15:53 -0700, john stultz wrote:
> > Hey Ingo, All,
> > 	We had the following bug reported on bootup on one of our boxes (it
> > was a 4way I believe) running -rt22. So far it seems to be a one-off
> > but I figured I'd post it to see if anyone had a clue.
> 
> I'm assuming this is a i386.  Also I'm assuming that frame pointers was
> not compiled in since the stack is a little suspicious.
> 
> Anyway, could you show the /proc/interrupts of this machine.  I'm
> curious if the i8042 isn't sharing an interrupt with something with
> NODELAY in it.

Here ya go:
            CPU0       CPU1       CPU2       CPU3
   0:       8796    3868607        275     531673  IO-APIC-edge   [........N/  0]  pit
   2:          0          0          0          0  XT-PIC         [........N/  0]  cascade
   3:          5        620          2        229  IO-APIC-edge   [........./ 63]  serial
   8:          0          1          0          0  IO-APIC-edge   [........./  0]  rtc
  11:          0          0          0          0  IO-APIC-edge   [........./  0]  acpi
  19:        120          0          0          1  IO-APIC-level  [........./  0]  ohci_hcd:usb1, ohci_hcd:usb2
  24:         57          9          5      46795  IO-APIC-level  [........./  0]  eth0
  26:       1396      14537          0        702  IO-APIC-level  [........./  0]  ioc0
 NMI:          0          0          0          0
 LOC:    6907796    4419008    4415669    4413513
 ERR:          0
 MIS:          0

thanks
-john

Re: [-rt BUG] scheduling with irqs disabled: swapper

[Steven Rostedt]

On Sat, 2006-05-27 at 17:13 -0700, john stultz wrote:
> On Fri, 2006-05-26 at 21:14 -0400, Steven Rostedt wrote:
> > On Fri, 2006-05-26 at 15:53 -0700, john stultz wrote:
> > > Hey Ingo, All,
> > > 	We had the following bug reported on bootup on one of our boxes (it
> > > was a 4way I believe) running -rt22. So far it seems to be a one-off
> > > but I figured I'd post it to see if anyone had a clue.
> > 
> > I'm assuming this is a i386.  Also I'm assuming that frame pointers was
> > not compiled in since the stack is a little suspicious.
> > 
> > Anyway, could you show the /proc/interrupts of this machine.  I'm
> > curious if the i8042 isn't sharing an interrupt with something with
> > NODELAY in it.
> 
> Here ya go:
>             CPU0       CPU1       CPU2       CPU3
>    0:       8796    3868607        275     531673  IO-APIC-edge   [........N/  0]  pit
>    2:          0          0          0          0  XT-PIC         [........N/  0]  cascade
>    3:          5        620          2        229  IO-APIC-edge   [........./ 63]  serial
>    8:          0          1          0          0  IO-APIC-edge   [........./  0]  rtc
>   11:          0          0          0          0  IO-APIC-edge   [........./  0]  acpi
>   19:        120          0          0          1  IO-APIC-level  [........./  0]  ohci_hcd:usb1, ohci_hcd:usb2
>   24:         57          9          5      46795  IO-APIC-level  [........./  0]  eth0
>   26:       1396      14537          0        702  IO-APIC-level  [........./  0]  ioc0
>  NMI:          0          0          0          0
>  LOC:    6907796    4419008    4415669    4413513
>  ERR:          0
>  MIS:          0

Thanks, but I was looking more into the code, and I'm wondering...
Does this machine have "irqfixup" or "irqpoll" set in the kernel command
line?

I think that -rt doesn't support it yet.  That is, it can call a handler
from interrupt context, which should have been a thread.

Let me know if that was the case.

-- Steve

Re: [-rt BUG] scheduling with irqs disabled: swapper

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> Thanks, but I was looking more into the code, and I'm wondering... 
> Does this machine have "irqfixup" or "irqpoll" set in the kernel 
> command line?
> 
> I think that -rt doesn't support it yet.  That is, it can call a 
> handler from interrupt context, which should have been a thread.
> 
> Let me know if that was the case.

the backtrace shows misrouted_irq(), which is only called if "irqfixup" 
is enabled. That indeed isnt supported in -rt yet.

	Ingo

Re: [-rt BUG] scheduling with irqs disabled: swapper

[john stultz]

On Sun, 2006-05-28 at 08:40 +0200, Ingo Molnar wrote:
> * Steven Rostedt <rostedt@goodmis.org> wrote:
> 
> > Thanks, but I was looking more into the code, and I'm wondering... 
> > Does this machine have "irqfixup" or "irqpoll" set in the kernel 
> > command line?
> > 
> > I think that -rt doesn't support it yet.  That is, it can call a 
> > handler from interrupt context, which should have been a thread.
> > 
> > Let me know if that was the case.
> 
> the backtrace shows misrouted_irq(), which is only called if "irqfixup" 
> is enabled. That indeed isnt supported in -rt yet.

Ugh. You and Steven are right. We've been bitten by this a few times,
but we thought we got rid of that option on all of our boxes. I guess
one slipped by.

Anyway, thanks for pointing that out. Would you consider a patch like
the following so that folks don't continue to slip over this?

thanks
-john

Index: rtdev/kernel/irq/spurious.c
===================================================================
--- rtdev.orig/kernel/irq/spurious.c
+++ rtdev/kernel/irq/spurious.c
@@ -194,6 +194,11 @@ __setup("noirqdebug", noirqdebug_setup);
 
 static int __init irqfixup_setup(char *str)
 {
+#ifdef CONFIG_PREEMPT_RT
+	printk(KERN_WARNING "irqfixup boot option not supported "
+		"w/ CONFIG_PREEMT_RT\n");
+	return 1;
+#endif
 	irqfixup = 1;
 	printk(KERN_WARNING "Misrouted IRQ fixup support enabled.\n");
 	printk(KERN_WARNING "This may impact system performance.\n");
@@ -204,6 +209,11 @@ __setup("irqfixup", irqfixup_setup);
 
 static int __init irqpoll_setup(char *str)
 {
+#ifdef CONFIG_PREEMPT_RT
+	printk(KERN_WARNING "irqpoll boot option not supported "
+		"w/ CONFIG_PREEMT_RT\n");
+	return 1;
+#endif
 	irqfixup = 2;
 	printk(KERN_WARNING "Misrouted IRQ fixup and polling support "
 				"enabled\n");

Re: [-rt BUG] scheduling with irqs disabled: swapper

[Ingo Molnar]

* john stultz <johnstul@us.ibm.com> wrote:

> > the backtrace shows misrouted_irq(), which is only called if "irqfixup" 
> > is enabled. That indeed isnt supported in -rt yet.
> 
> Ugh. You and Steven are right. We've been bitten by this a few times, 
> but we thought we got rid of that option on all of our boxes. I guess 
> one slipped by.
> 
> Anyway, thanks for pointing that out. Would you consider a patch like 
> the following so that folks don't continue to slip over this?

sure, i've applied it. (fixed a small typo in the printout)

	Ingo