Oops in fs_enet driver with preempt-rt

Oops in fs_enet driver with preempt-rt

[Rune Torgersen]

Hi
I am trying to get preempt-rt (v2.6.25.4-rt1) to work on a Freescale
8280 CPU (arch/powerpc)

I get the follwing oops when trying to enable ethernet:
(looks like a null pointer dereference in
drivers/net/fs_enet/fs_enet-main.c:106)

Unable to handle kernel paging request for data at address 0x00000000
Faulting instruction address: 0xc0197efc
Oops: Kernel access of bad area, sig: 11 [#1]
PREEMPT Innovative Systems ApMax
Modules linked in:
NIP: c0197efc LR: c0197c48 CTR: c01986c4
REGS: df84de90 TRAP: 0300   Not tainted  (2.6.25.4-rt1)
MSR: 00009032 <EE,ME,IR,DR>  CR: 42004022  XER: 00000000
DAR: 00000000, DSISR: 20000000
TASK = df82ea70[7] 'sirq-net-rx/0' THREAD: df84c000
GPR00: e105a350 df84df40 df82ea70 df37f000 00000011 c034ed90 c034ed70
00000000
GPR08: 00000000 e105a340 00000000 00000009 0000df80 ffffffff 0fffd000
00000028
GPR16: 00400558 00800000 ffffffff df3e71e0 00000011 00000000 00000000
00000000
GPR24: df37f000 00000000 00000000 00000000 fffb9e82 0000012c 00000011
df37f5a0
NIP [c0197efc] fs_enet_rx_napi+0x2f0/0x3ac
LR [c0197c48] fs_enet_rx_napi+0x3c/0x3ac
Call Trace:
[df84df40] [c024a1b4] __spin_unlock+0x18/0x3c (unreliable)
[df84df80] [c01d9d28] net_rx_action+0x84/0x1ac
[df84dfa0] [c0022f18] ksoftirqd+0x158/0x25c
[df84dfd0] [c0033fa0] kthread+0x4c/0x88
[df84dff0] [c000fb34] kernel_thread+0x44/0x60
Instruction dump:
73202000 3b7b0008 41820008 837f00bc 813f00a0 7f03c378 81290034 7d2903a6
4e800421 7f96a000 409c00a8 7c0004ac <a01b0000> 0c000000 4c00012c
5419043e
---[ end trace c548f7dabfe5b3de ]---


0xc0197efc is in fs_enet_rx_napi (include/asm/io.h:124).
119             DEF_MMIO_OUT(name, u##size, __stringify(insn)"%U0%X0
%1,%0")
120     #define DEF_MMIO_OUT_LE(name, size, insn) \
121             DEF_MMIO_OUT(name, u##size, __stringify(insn)" %1,0,%2")
122
123     DEF_MMIO_IN_BE(in_8,     8, lbz);
124     DEF_MMIO_IN_BE(in_be16, 16, lhz);
125     DEF_MMIO_IN_BE(in_be32, 32, lwz);
126     DEF_MMIO_IN_LE(in_le16, 16, lhbrx);
127     DEF_MMIO_IN_LE(in_le32, 32, lwbrx);
128

0xc0197c48 is in fs_enet_rx_napi
(drivers/net/fs_enet/fs_enet-main.c:106).
101              * These get messed up if we get called due to a busy
condition.
102              */
103             bdp = fep->cur_rx;
104
105             /* clear RX status bits for napi*/
106             (*fep->ops->napi_clear_rx_event)(dev);
107
108             while (((sc = CBDR_SC(bdp)) & BD_ENET_RX_EMPTY) == 0) {
109                     curidx = bdp - fep->rx_bd_base;
110

RE: Oops in fs_enet driver with preempt-rt

[Rune Torgersen]

Rune Torgersen wrote:
> Hi
> I am trying to get preempt-rt (v2.6.25.4-rt1) to work on a Freescale
> 8280 CPU (arch/powerpc) 
> 
> I get the follwing oops when trying to enable ethernet:
> (looks like a null pointer dereference in
> drivers/net/fs_enet/fs_enet-main.c:106)

And if I disable NAPI I get this one instead:

Bringing up the eth1 interface...Unable to handle kernel paging request
for data at address 0x00000000
Faulting instruction address: 0xc019706c
Oops: Kernel access of bad area, sig: 11 [#1]
PREEMPT Innovative Systems ApMax
Modules linked in:
NIP: c019706c LR: c0196d84 CTR: c0198790
REGS: ef2c3cb0 TRAP: 0300   Not tainted  (2.6.25.4-rt1)
MSR: 00009032 <EE,ME,IR,DR>  CR: 42004222  XER: 00000000
DAR: 00000000, DSISR: 20000000
TASK = ef2ac590[114] 'ifconfig' THREAD: ef2c2000
GPR00: 00000000 ef2c3d60 ef2ac590 ef37f000 00000008 00000000 ef2c3d5c
0000008c
GPR08: ef2c3d38 00000008 ef2c3d40 00000001 22000222 10021454 00000000
100a37b0
GPR16: 10090000 ef3e71e0 ef3e71e0 00000000 10070000 00000008 100192c0
00008914
GPR24: ef37f000 ef37f5a0 00000000 c0196d34 00000000 ef37f5a0 fffffff4
ef2973a0
NIP [c019706c] fs_enet_interrupt+0x338/0x5f4
LR [c0196d84] fs_enet_interrupt+0x50/0x5f4
Call Trace:
[ef2c3d60] [c024a270] __spin_unlock_irqrestore+0x20/0x4c (unreliable)
[ef2c3db0] [c0049a04] request_irq+0xb8/0xfc
[ef2c3de0] [c0198044] fs_enet_open+0x8c/0x210
[ef2c3e00] [c01d9b4c] dev_open+0x80/0xe8
[ef2c3e10] [c01d89ac] dev_change_flags+0xc8/0x19c
[ef2c3e30] [c021bfb0] devinet_ioctl+0x284/0x6f0
[ef2c3e90] [c021c8fc] inet_ioctl+0xc0/0xf4
[ef2c3ea0] [c01cbec4] sock_ioctl+0x200/0x23c
[ef2c3ec0] [c00810c0] vfs_ioctl+0x44/0xa8
[ef2c3ee0] [c0081520] do_vfs_ioctl+0x3fc/0x440
[ef2c3f10] [c00815a4] sys_ioctl+0x40/0x70
[ef2c3f40] [c000fcf0] ret_from_syscall+0x0/0x38
--- Exception: c01 at 0xff6a37c
    LR = 0xffecd58
Instruction dump:
7c0004ac b01c0000 72e02000 3b9c0008 41820008 839d00bc 813d00a0 7f03c378
81290034 7d2903a6 4e800421 7c0004ac <a01c0000> 0c000000 4c00012c
5417043e
---[ end trace 61ff5e5c97ddf707 ]---

Seems to be caused by line 471 in fs_enet_main:

0xc0196d84 is in fs_enet_interrupt
(drivers/net/fs_enet/fs_enet-main.c:473).
468                     if (fpi->use_napi)
469                             int_clr_events &= ~fep->ev_napi_rx;
470
471                     (*fep->ops->clear_int_events)(dev,
int_clr_events);
472
473                     if (int_events & fep->ev_err)
474                             (*fep->ops->ev_error)(dev, int_events);
475
476                     if (int_events & fep->ev_rx) {
477                             if (!fpi->use_napi)
(gdb)     

Re: Oops in fs_enet driver with preempt-rt

[Scott Wood]

Rune Torgersen wrote:
> 0xc0196d84 is in fs_enet_interrupt
> (drivers/net/fs_enet/fs_enet-main.c:473).
> 468                     if (fpi->use_napi)
> 469                             int_clr_events &= ~fep->ev_napi_rx;
> 470
> 471                     (*fep->ops->clear_int_events)(dev,
> int_clr_events);
> 472
> 473                     if (int_events & fep->ev_err)
> 474                             (*fep->ops->ev_error)(dev, int_events);
> 475
> 476                     if (int_events & fep->ev_rx) {
> 477                             if (!fpi->use_napi)
> (gdb)     

Do you have shared interrupt debugging turned on?  That breaks this 
driver, and a patch to remove the shared flag was nacked in favor of 
actually fixing the driver, which I haven't gotten around to.

-Scott

RE: Oops in fs_enet driver with preempt-rt

[Rune Torgersen]

Scott Wood wrote:
> Rune Torgersen wrote:
>> 0xc0196d84 is in fs_enet_interrupt
>> (drivers/net/fs_enet/fs_enet-main.c:473).
>> 468                     if (fpi->use_napi)
>> 469                             int_clr_events &= ~fep->ev_napi_rx;
>> 470 471                     (*fep->ops->clear_int_events)(dev,
>> int_clr_events); 472
>> 473                     if (int_events & fep->ev_err)
>> 474                             (*fep->ops->ev_error)(dev,
>> int_events); 475 476                     if (int_events &
>> fep->ev_rx) { 477                             if (!fpi->use_napi)
>> (gdb)
> 
> Do you have shared interrupt debugging turned on?  That breaks this
> driver, and a patch to remove the shared flag was nacked in favor of
> actually fixing the driver, which I haven't gotten around to.
> 
> -Scott

Thanks!!
That worked. Now I just have to get highmem support...

RE: Oops in fs_enet driver with preempt-rt

[Rune Torgersen]

Scott Wood wrote:
> Rune Torgersen wrote:
>> 0xc0196d84 is in fs_enet_interrupt
>> (drivers/net/fs_enet/fs_enet-main.c:473).
>> 468                     if (fpi->use_napi)
>> 469                             int_clr_events &= ~fep->ev_napi_rx;
>> 470 471                     (*fep->ops->clear_int_events)(dev,
>> int_clr_events); 472
>> 473                     if (int_events & fep->ev_err)
>> 474                             (*fep->ops->ev_error)(dev,
>> int_events); 475 476                     if (int_events &
>> fep->ev_rx) { 477                             if (!fpi->use_napi)
>> (gdb)
> 
> Do you have shared interrupt debugging turned on?  That breaks this
> driver, and a patch to remove the shared flag was nacked in favor of
> actually fixing the driver, which I haven't gotten around to.
> 
> -Scott

Hmm.... ttyCPM1 output seems to be kinda garbled... Any ideas?
Incomplete locking between printk and userland use of ttyCPM1 as
console?

kjour[  OK  ]   nal
 stdrtiag. nCom it mnteivalr5 s cones
dXT3EFS  n soa7,dint rnae jolrnau
ElT3-Xs: founmed tilefystsm weth irdeoed ratadmod .
ek/*    [  Removing /var/run/* and /var/locp... [  OK  ]
Creating new /var/run/utmlogin /fastboot and /forcefsck...      OK  ]
Removing possible /etc/noBringing up the loopback interface...  [  OK  ]
OK  ]
OK  ] Entering runlevel: 3 hostname to 9919_unit1_c
Starting the i2c_8266 driver: i2
_82c6: 6odume llcenie 'snspucifeed'itai ts nernkl.
                                                  e[ OK ]

Re: Oops in fs_enet driver with preempt-rt

[Scott Wood]

Rune Torgersen wrote:
> Hmm.... ttyCPM1 output seems to be kinda garbled... Any ideas?
> Incomplete locking between printk and userland use of ttyCPM1 as
> console?

s/incomplete/nonexistent/ :-P

Try acquiring port->lock in cpm_uart_console_write.

-Scott

RE: Oops in fs_enet driver with preempt-rt

[Rune Torgersen]

Scott Wood wrote:
> Rune Torgersen wrote:
>> Hmm.... ttyCPM1 output seems to be kinda garbled... Any ideas?
>> Incomplete locking between printk and userland use of ttyCPM1 as
>> console?
> 
> s/incomplete/nonexistent/ :-P
> 
> Try acquiring port->lock in cpm_uart_console_write.
> 
> -Scott

That worked. Console output is sane again.

Re: Oops in fs_enet driver with preempt-rt

[Kumar Gala]

On May 20, 2008, at 11:31 AM, Rune Torgersen wrote:

> Scott Wood wrote:
>> Rune Torgersen wrote:
>>> Hmm.... ttyCPM1 output seems to be kinda garbled... Any ideas?
>>> Incomplete locking between printk and userland use of ttyCPM1 as
>>> console?
>>
>> s/incomplete/nonexistent/ :-P
>>
>> Try acquiring port->lock in cpm_uart_console_write.
>>
>> -Scott
>
> That worked. Console output is sane again.

I'm hoping to see some patches related to these fixes :)

- k

RE: Oops in fs_enet driver with preempt-rt

[Rune Torgersen]

Kumar Gala wrote:
>> Rune Torgersen wrote:
>> That worked. Console output is sane again.
> 
> I'm hoping to see some patches related to these fixes :)

Working on it.