[Xenomai-help] Oops with ifconfig on Sequioa AMCC 440EPX board

[Xenomai-help] Oops with ifconfig on Sequioa AMCC 440EPX board

[Wolfgang Grandegger]

Hello,

I get the following Oops with a Xenomai patch kernel when configuring
the EEPRO100 on the PCI slot:

 -bash-3.2# ifconfig eth2 172.16.0.20
------------[ cut here ]------------
Badness at c0025db0 [verbose debug info unavailable]
NIP: c0025db0 LR: c01b5658 CTR: c0190738
REGS: cfaa9d20 TRAP: 0700   Not tainted  (2.6.26.3-dirty)
MSR: 00029000 <EE,ME>  CR: 44002044  XER: 00000004
TASK = cf82b5a0[1052] 'ifconfig' THREAD: cfaa8000
NIP [c0025db0] local_bh_enable+0x84/0x9c
LR [c01b5658] dev_set_rx_mode+0x34/0x48
Call Trace:
[cfaa9dd0] [c01b54bc] __dev_set_rx_mode+0x40/0xb8 (unreliable)
[cfaa9de0] [c01b5658] dev_set_rx_mode+0x34/0x48
[cfaa9e00] [c01b83f8] dev_open+0xd8/0x114
[cfaa9e10] [c01b7244] dev_change_flags+0x8c/0x1b8
[cfaa9e30] [c01f7f58] devinet_ioctl+0x63c/0x77c
[cfaa9ea0] [c01f86f8] inet_ioctl+0xcc/0xf8
[cfaa9eb0] [c01aa878] sock_ioctl+0x60/0x2ec
[cfaa9ed0] [c00ba598] vfs_ioctl+0x34/0x98
[cfaa9ee0] [c00ba8fc] do_vfs_ioctl+0x300/0x430
[cfaa9f10] [c00baa6c] sys_ioctl+0x40/0x74
[cfaa9f40] [c000e1ac] ret_from_syscall+0x0/0x3c
Instruction dump:
80010014 83e1000c 7c0803a6 38210010 4e800020 4bfdde81 4bffffdc 3d20c02f
39297ba0 80090154 7c000034 5400d97e <0f000000> 2f800000 419eff94 38000001
-bash-3.2#
Message from syslogd@ at Thu Jan  1 01:01:21 1970 ...
sequoia666 kernel: ------------[ cut here ]------------

-bash-3.2# uname -a
Linux sequoia666 2.6.26.3-dirty
-bash-3.2# cat /proc/ipipe/version
2.2-04
-bash-3.2# cat /proc/xenomai/version
2.4.5

The NIP is at:

(gdb) l *0xc0025db0
0xc0025db0 is in local_bh_enable (kernel/softirq.c:141).
136	#ifdef CONFIG_TRACE_IRQFLAGS
137		unsigned long flags;
138	
139		WARN_ON_ONCE(in_irq());
140	#endif
141		WARN_ON_ONCE(irqs_disabled());
142	
143	#ifdef CONFIG_TRACE_IRQFLAGS
144		local_irq_save(flags);
145	#endif

Any idea where the problem could be?

TIA.

Wolfgang.

Re: [Xenomai-help] Oops with ifconfig on Sequioa AMCC 440EPX board

[Wolfgang Grandegger]

Wolfgang Grandegger wrote:
> Hello,
> 
> I get the following Oops with a Xenomai patch kernel when configuring
> the EEPRO100 on the PCI slot:
> 
>  -bash-3.2# ifconfig eth2 172.16.0.20
> ------------[ cut here ]------------
> Badness at c0025db0 [verbose debug info unavailable]
> NIP: c0025db0 LR: c01b5658 CTR: c0190738
> REGS: cfaa9d20 TRAP: 0700   Not tainted  (2.6.26.3-dirty)
> MSR: 00029000 <EE,ME>  CR: 44002044  XER: 00000004
> TASK = cf82b5a0[1052] 'ifconfig' THREAD: cfaa8000
> NIP [c0025db0] local_bh_enable+0x84/0x9c
> LR [c01b5658] dev_set_rx_mode+0x34/0x48
> Call Trace:
> [cfaa9dd0] [c01b54bc] __dev_set_rx_mode+0x40/0xb8 (unreliable)
> [cfaa9de0] [c01b5658] dev_set_rx_mode+0x34/0x48
> [cfaa9e00] [c01b83f8] dev_open+0xd8/0x114
> [cfaa9e10] [c01b7244] dev_change_flags+0x8c/0x1b8
> [cfaa9e30] [c01f7f58] devinet_ioctl+0x63c/0x77c
> [cfaa9ea0] [c01f86f8] inet_ioctl+0xcc/0xf8
> [cfaa9eb0] [c01aa878] sock_ioctl+0x60/0x2ec
> [cfaa9ed0] [c00ba598] vfs_ioctl+0x34/0x98
> [cfaa9ee0] [c00ba8fc] do_vfs_ioctl+0x300/0x430
> [cfaa9f10] [c00baa6c] sys_ioctl+0x40/0x74
> [cfaa9f40] [c000e1ac] ret_from_syscall+0x0/0x3c
> Instruction dump:
> 80010014 83e1000c 7c0803a6 38210010 4e800020 4bfdde81 4bffffdc 3d20c02f
> 39297ba0 80090154 7c000034 5400d97e <0f000000> 2f800000 419eff94 38000001
> -bash-3.2#
> Message from syslogd@ at Thu Jan  1 01:01:21 1970 ...
> sequoia666 kernel: ------------[ cut here ]------------
> 
> -bash-3.2# uname -a
> Linux sequoia666 2.6.26.3-dirty
> -bash-3.2# cat /proc/ipipe/version
> 2.2-04
> -bash-3.2# cat /proc/xenomai/version
> 2.4.5
> 
> The NIP is at:
> 
> (gdb) l *0xc0025db0
> 0xc0025db0 is in local_bh_enable (kernel/softirq.c:141).
> 136	#ifdef CONFIG_TRACE_IRQFLAGS
> 137		unsigned long flags;
> 138	
> 139		WARN_ON_ONCE(in_irq());
> 140	#endif
> 141		WARN_ON_ONCE(irqs_disabled());
> 142	
> 143	#ifdef CONFIG_TRACE_IRQFLAGS
> 144		local_irq_save(flags);
> 145	#endif
> 
> Any idea where the problem could be?

I got a bit further with this problem. It is related to cascaded UIC
interrupts. The following patch inspired by qe_ic.c fixes the oops but
it still hangs the system if using the EEPRO100 with RTnet.

diff --git a/arch/powerpc/sysdev/uic.c b/arch/powerpc/sysdev/uic.c
index 625b275..65c50d0 100644
--- a/arch/powerpc/sysdev/uic.c
+++ b/arch/powerpc/sysdev/uic.c
@@ -125,6 +125,18 @@ static void uic_mask_ack_irq(unsigned int virq)
 	spin_unlock_irqrestore(&uic->lock, flags);
 }

+#ifdef CONFIG_IPIPE
+void __ipipe_uic_cascade_irq(unsigned int virq)
+{
+	
+	struct pt_regs regs;    /* Contents not used. */
+
+	ipipe_trace_irq_entry(virq);
+        __ipipe_handle_irq(virq, &regs);
+        ipipe_trace_irq_exit(virq);
+}
+#endif
+
 static int uic_set_irq_type(unsigned int virq, unsigned int flow_type)
 {
 	struct uic *uic = get_irq_chip_data(virq);
@@ -239,7 +251,11 @@ void uic_irq_cascade(unsigned int virq, struct
irq_desc *desc)
 	src = 32 - ffs(msr);

 	subvirq = irq_linear_revmap(uic->irqhost, src);
+#ifdef CONFIG_IPIPE
+	__ipipe_uic_cascade_irq(subvirq);
+#else
 	generic_handle_irq(subvirq);
+#endif

 uic_irq_ret:
 	spin_lock(&desc->lock);


Wolfgang.

Re: [Xenomai-help] Oops with ifconfig on Sequioa AMCC 440EPX board

[Gilles Chanteperdrix]

Wolfgang Grandegger wrote:
> Wolfgang Grandegger wrote:
>> Hello,
>>
>> I get the following Oops with a Xenomai patch kernel when configuring
>> the EEPRO100 on the PCI slot:
>>
>>  -bash-3.2# ifconfig eth2 172.16.0.20
>> ------------[ cut here ]------------
>> Badness at c0025db0 [verbose debug info unavailable]
>> NIP: c0025db0 LR: c01b5658 CTR: c0190738
>> REGS: cfaa9d20 TRAP: 0700   Not tainted  (2.6.26.3-dirty)
>> MSR: 00029000 <EE,ME>  CR: 44002044  XER: 00000004
>> TASK = cf82b5a0[1052] 'ifconfig' THREAD: cfaa8000
>> NIP [c0025db0] local_bh_enable+0x84/0x9c
>> LR [c01b5658] dev_set_rx_mode+0x34/0x48
>> Call Trace:
>> [cfaa9dd0] [c01b54bc] __dev_set_rx_mode+0x40/0xb8 (unreliable)
>> [cfaa9de0] [c01b5658] dev_set_rx_mode+0x34/0x48
>> [cfaa9e00] [c01b83f8] dev_open+0xd8/0x114
>> [cfaa9e10] [c01b7244] dev_change_flags+0x8c/0x1b8
>> [cfaa9e30] [c01f7f58] devinet_ioctl+0x63c/0x77c
>> [cfaa9ea0] [c01f86f8] inet_ioctl+0xcc/0xf8
>> [cfaa9eb0] [c01aa878] sock_ioctl+0x60/0x2ec
>> [cfaa9ed0] [c00ba598] vfs_ioctl+0x34/0x98
>> [cfaa9ee0] [c00ba8fc] do_vfs_ioctl+0x300/0x430
>> [cfaa9f10] [c00baa6c] sys_ioctl+0x40/0x74
>> [cfaa9f40] [c000e1ac] ret_from_syscall+0x0/0x3c
>> Instruction dump:
>> 80010014 83e1000c 7c0803a6 38210010 4e800020 4bfdde81 4bffffdc 3d20c02f
>> 39297ba0 80090154 7c000034 5400d97e <0f000000> 2f800000 419eff94 38000001
>> -bash-3.2#
>> Message from syslogd@ at Thu Jan  1 01:01:21 1970 ...
>> sequoia666 kernel: ------------[ cut here ]------------
>>
>> -bash-3.2# uname -a
>> Linux sequoia666 2.6.26.3-dirty
>> -bash-3.2# cat /proc/ipipe/version
>> 2.2-04
>> -bash-3.2# cat /proc/xenomai/version
>> 2.4.5
>>
>> The NIP is at:
>>
>> (gdb) l *0xc0025db0
>> 0xc0025db0 is in local_bh_enable (kernel/softirq.c:141).
>> 136	#ifdef CONFIG_TRACE_IRQFLAGS
>> 137		unsigned long flags;
>> 138	
>> 139		WARN_ON_ONCE(in_irq());
>> 140	#endif
>> 141		WARN_ON_ONCE(irqs_disabled());
>> 142	
>> 143	#ifdef CONFIG_TRACE_IRQFLAGS
>> 144		local_irq_save(flags);
>> 145	#endif
>>
>> Any idea where the problem could be?
> 
> I got a bit further with this problem. It is related to cascaded UIC
> interrupts. The following patch inspired by qe_ic.c fixes the oops but
> it still hangs the system if using the EEPRO100 with RTnet.

If there are cascaded irqs, maybe you should do like the I-pipe for arm
does: decode the parent irq and trigger the cascaded irq ahead of the
pipeline ?


-- 
					    Gilles.

Re: [Xenomai-help] Oops with ifconfig on Sequioa AMCC 440EPX board

[Philippe Gerum]

Wolfgang Grandegger wrote:
> Hello,
> 
> I get the following Oops with a Xenomai patch kernel when configuring
> the EEPRO100 on the PCI slot:
> 
>  -bash-3.2# ifconfig eth2 172.16.0.20
> ------------[ cut here ]------------
> Badness at c0025db0 [verbose debug info unavailable]
> NIP: c0025db0 LR: c01b5658 CTR: c0190738
> REGS: cfaa9d20 TRAP: 0700   Not tainted  (2.6.26.3-dirty)
> MSR: 00029000 <EE,ME>  CR: 44002044  XER: 00000004
> TASK = cf82b5a0[1052] 'ifconfig' THREAD: cfaa8000
> NIP [c0025db0] local_bh_enable+0x84/0x9c
> LR [c01b5658] dev_set_rx_mode+0x34/0x48
> Call Trace:
> [cfaa9dd0] [c01b54bc] __dev_set_rx_mode+0x40/0xb8 (unreliable)
> [cfaa9de0] [c01b5658] dev_set_rx_mode+0x34/0x48
> [cfaa9e00] [c01b83f8] dev_open+0xd8/0x114
> [cfaa9e10] [c01b7244] dev_change_flags+0x8c/0x1b8
> [cfaa9e30] [c01f7f58] devinet_ioctl+0x63c/0x77c
> [cfaa9ea0] [c01f86f8] inet_ioctl+0xcc/0xf8
> [cfaa9eb0] [c01aa878] sock_ioctl+0x60/0x2ec
> [cfaa9ed0] [c00ba598] vfs_ioctl+0x34/0x98
> [cfaa9ee0] [c00ba8fc] do_vfs_ioctl+0x300/0x430
> [cfaa9f10] [c00baa6c] sys_ioctl+0x40/0x74
> [cfaa9f40] [c000e1ac] ret_from_syscall+0x0/0x3c
> Instruction dump:
> 80010014 83e1000c 7c0803a6 38210010 4e800020 4bfdde81 4bffffdc 3d20c02f
> 39297ba0 80090154 7c000034 5400d97e <0f000000> 2f800000 419eff94 38000001
> -bash-3.2#
> Message from syslogd@ at Thu Jan  1 01:01:21 1970 ...
> sequoia666 kernel: ------------[ cut here ]------------
> 
> -bash-3.2# uname -a
> Linux sequoia666 2.6.26.3-dirty
> -bash-3.2# cat /proc/ipipe/version
> 2.2-04
> -bash-3.2# cat /proc/xenomai/version
> 2.4.5
> 
> The NIP is at:
> 
> (gdb) l *0xc0025db0
> 0xc0025db0 is in local_bh_enable (kernel/softirq.c:141).
> 136	#ifdef CONFIG_TRACE_IRQFLAGS
> 137		unsigned long flags;
> 138	
> 139		WARN_ON_ONCE(in_irq());
> 140	#endif
> 141		WARN_ON_ONCE(irqs_disabled());
> 142	
> 143	#ifdef CONFIG_TRACE_IRQFLAGS
> 144		local_irq_save(flags);
> 145	#endif
> 
> Any idea where the problem could be?
> 

As you pointed out, there was an issue with cascaded UICs. The following patch
should fix it (will be merged in the upcoming 2.2-05):

diff --git a/arch/powerpc/sysdev/uic.c b/arch/powerpc/sysdev/uic.c
index 4eff1ab..bd91669 100644
--- a/arch/powerpc/sysdev/uic.c
+++ b/arch/powerpc/sysdev/uic.c
@@ -241,7 +241,16 @@ void uic_irq_cascade(unsigned int virq, struct irq_desc *desc)
 	src = 32 - ffs(msr);

 	subvirq = irq_linear_revmap(uic->irqhost, src);
+#ifdef CONFIG_IPIPE
+	{
+		struct pt_regs regs;    /* Contents not used. */
+		ipipe_trace_irq_entry(subvirq);
+		__ipipe_handle_irq(subvirq, &regs);
+		ipipe_trace_irq_exit(subvirq);
+	}
+#else
 	generic_handle_irq(subvirq);
+#endif

 uic_irq_ret:
 	spin_lock(&desc->lock);

-- 
Philippe.

Re: [Xenomai-help] Oops with ifconfig on Sequioa AMCC 440EPX board

[Steven A. Falco]

Philippe Gerum wrote:
> Wolfgang Grandegger wrote:
>> Hello,
>>
>> I get the following Oops with a Xenomai patch kernel when configuring
>> the EEPRO100 on the PCI slot:
...
>> Any idea where the problem could be?
>>
> 
> As you pointed out, there was an issue with cascaded UICs. The following patch
> should fix it (will be merged in the upcoming 2.2-05):

Thanks!  The patch does indeed fix it.  One comment, though.  I had
turned on spinlock debugging as well as adding your patch.  That
combination produces the bugs below.  Once I turn off spinlock
debugging (with the patch still included), then things work properly.

So, I don't know if this bug is totally unrelated to Ipipe.  I'll
investigate and post again with the results.

	Steve


PPC 4xx OCP EMAC driver, version 3.54
MAL v2 /plb/mcmal, 2 TX channels, 2 RX channels
ZMII /plb/opb/emac-zmii@domain.hid initialized
RGMII /plb/opb/emac-rgmii@domain.hid initialized with MDIO support
/plb/opb/emac-rgmii@domain.hid: input 0 in RGMII mode
BUG: spinlock bad magic on CPU#0, swapper/1
 lock: cf8161ac, .magic: 00000000, .owner: <none>/-1, .owner_cpu: 0
Call Trace:
[cf82dc20] [c000651c] show_stack+0x44/0x1ac (unreliable)
[cf82dc60] [c01823fc] spin_bug+0x7c/0xc4
[cf82dc80] [c0182608] _raw_spin_lock+0x94/0x180
[cf82dcb0] [c02ab880] _spin_lock_bh+0x20/0x34
[cf82dcd0] [c0240048] dev_mc_add+0x2c/0x84
[cf82dcf0] [c01ac7a8] emac_configure+0x32c/0x588
[cf82dd20] [c02b17c0] 0xc02b17c0
[cf82de10] [c022a0d4] of_platform_device_probe+0x5c/0x84
[cf82de30] [c01a5bd4] driver_probe_device+0xb8/0x1e8
[cf82de50] [c01a5d88] __driver_attach+0x84/0x88
[cf82de70] [c01a50b0] bus_for_each_dev+0x5c/0x98
[cf82dea0] [c01a59dc] driver_attach+0x24/0x34
[cf82deb0] [c01a5708] bus_add_driver+0x1d8/0x258
[cf82dee0] [c01a5f98] driver_register+0x5c/0x158
[cf82df00] [c0229fa8] of_register_driver+0x54/0x70
[cf82df10] [c036d0d0] emac_init+0x1a8/0x1d4
[cf82df50] [c03561b4] kernel_init+0xac/0x294
[cf82dff0] [c000eb60] kernel_thread+0x44/0x60
BUG: spinlock lockup on CPU#0, swapper/1, cf8161ac
Call Trace:
[cf82dc40] [c000651c] show_stack+0x44/0x1ac (unreliable)
[cf82dc80] [c01826c4] _raw_spin_lock+0x150/0x180
[cf82dcb0] [c02ab880] _spin_lock_bh+0x20/0x34
[cf82dcd0] [c0240048] dev_mc_add+0x2c/0x84
[cf82dcf0] [c01ac7a8] emac_configure+0x32c/0x588
[cf82dd20] [c02b17c0] 0xc02b17c0
[cf82de10] [c022a0d4] of_platform_device_probe+0x5c/0x84
[cf82de30] [c01a5bd4] driver_probe_device+0xb8/0x1e8
[cf82de50] [c01a5d88] __driver_attach+0x84/0x88
[cf82de70] [c01a50b0] bus_for_each_dev+0x5c/0x98
[cf82dea0] [c01a59dc] driver_attach+0x24/0x34
[cf82deb0] [c01a5708] bus_add_driver+0x1d8/0x258
[cf82dee0] [c01a5f98] driver_register+0x5c/0x158
[cf82df00] [c0229fa8] of_register_driver+0x54/0x70
[cf82df10] [c036d0d0] emac_init+0x1a8/0x1d4
[cf82df50] [c03561b4] kernel_init+0xac/0x294
[cf82dff0] [c000eb60] kernel_thread+0x44/0x60

Re: [Xenomai-help] Oops with ifconfig on Sequioa AMCC 440EPX board

[Steven A. Falco]

Steven A. Falco wrote:
> Philippe Gerum wrote:
>> Wolfgang Grandegger wrote:
>>> Hello,
>>>
>>> I get the following Oops with a Xenomai patch kernel when configuring
>>> the EEPRO100 on the PCI slot:
> ...
>>> Any idea where the problem could be?
>>>
>> As you pointed out, there was an issue with cascaded UICs. The following patch
>> should fix it (will be merged in the upcoming 2.2-05):
> 
> Thanks!  The patch does indeed fix it.  One comment, though.  I had
> turned on spinlock debugging as well as adding your patch.  That
> combination produces the bugs below.  Once I turn off spinlock
> debugging (with the patch still included), then things work properly.
> 
> So, I don't know if this bug is totally unrelated to Ipipe.  I'll
> investigate and post again with the results.
> 
> 	Steve
> 
> 

Turns out the spinlock bug is totally unrelated to Ipipe, as per the following:

http://kerneltrap.org/mailarchive/linux-netdev/2008/8/13/2926664/thread

So, I believe your patch completely fixes the issue I initially reported.

	Thanks,
	Steve