Re: Kernel Panics in the network stack

[Kevin Constantine <kevin.constantine@gmail.com>]

Kevin Constantine wrote:
> On 12/11/2009 01:58 PM, Eric Dumazet wrote:
>> Le 11/12/2009 22:50, Kevin Constantine a écrit :
>>> On 12/11/2009 01:39 PM, Eric Dumazet wrote:
>>>> Le 11/12/2009 22:09, Kevin Constantine a écrit :
>>>>> Hey Everyone-
>>>>>
>>>>> I've been playing with an ARM based linuxstamp
>>>>> http://opencircuits.com/Linuxstamp, and I've been seeing kernel panics
>>>>> with both 2.6.28.3, and 2.6.30 within an hour or so of turning the
>>>>> linuxstamp on.  The stack traces always seem to point at functions
>>>>> related to networking.  I've pasted a couple of the crash outputs 
>>>>> below.
>>>>>    The linuxstamp isn't typically doing anything when the crashes 
>>>>> occur,
>>>>> in fact it'll crash even if I haven't logged in.
>>>>>
>>>>> If I ifconfig the interface down, the linuxstamp stays up 
>>>>> indefinitely.
>>>>>    Any pointers in one direction or another would be much appreciated.
>>>>>
>>>>> I'm not sure if this is the right audience to help out or if the arm
>>>>> lists might be better.  But in any event, any help would be really
>>>>> appreciated.
>>>>>
>>>>>
>>>>> linuxstamp login: Unable to handle kernel paging request at virtual
>>>>> address 183cb7b0
>>>>> pgd = c0004000
>>>>> [183cb7b0] *pgd=00000000
>>>>> Internal error: Oops: 0 [#1] PREEMPT
>>>>> Modules linked in:
>>>>> CPU: 0    Not tainted  (2.6.30-00002-g0148992 #13)
>>>>> PC is at 0x183cb7b0
>>>>> LR is at __udp4_lib_rcv+0x43c/0x72c
>>>>
>>>> Could you disassemble your vmlinux file, __udp4_lib_rcv function
>>>> around LR
>>>> <c024ff4c>, to see which function was called ? This function then 
>>>> called
>>>> a wrong pointer (0x183cb7b0 not a kernel pointer)
>>>>
>>>> Maybe a kernel stack corruption, or bad ram, ...
>>>
>>> The vmlinux file I'm using has probably changed a number of times since
>>> then.  I'll get a fresh stack trace and disassemble that one.

Here's a new panic.  What would you like from the disassembled binary?

debian:~# Internal error: Oops - undefined instruction: 0 [#1] PREEMPT
Modules linked in: spidev atmel_spi
CPU: 0    Not tainted  (2.6.30-00002-g0148992 #16)
PC is at netif_receive_skb+0x284/0x2e8
LR is at kmem_cache_free+0x20/0x64
pc : [<c0214ec4>]    lr : [<c0089608>]    psr: a0000013
sp : c037feb0  ip : c037fe70  fp : c0384e60
r10: 00000008  r9 : c03bad00  r8 : 00000000
r7 : c1d2a800  r6 : c03a077c  r5 : c1e14980  r4 : c03bace0
r3 : c1d2a800  r2 : 00000062  r1 : c1d2a800  r0 : c1e14980
Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
Control: c000717f  Table: 21d58000  DAC: 00000017
Process swapper (pid: 0, stack limit = 0xc037e268)
Stack: (0xc037feb0 to 0xc0380000)
fea0:                                     00000000 c037fec0 00000001 
c039e54c
fec0: 00083674 00000040 00000000 c039e534 c039e530 c0214fb4 fefff000 
c039e54c
fee0: 00000040 c037e000 0000012c c039e530 c03bacf0 00083676 c039e540 
c0213764
ff00: 00000001 00000103 0000000c c037e000 00000001 c03a8678 00000000 
0000000a
ff20: 00000000 c0040358 c037e000 2001ccb8 00000000 00000018 00000000 
00000018
ff40: 00000002 00000001 c037e000 2001ccb8 00000000 c0040428 00000018 
c0022060
ff60: 00000000 ffffffff fefff000 c0022a3c 00000000 00000001 00000080 
60000013
ff80: c00243a4 c037e000 c0381e7c c00243a4 c03a3ac8 41129200 2001ccb8 
00000000
ffa0: fefff800 c037ffb8 c00243e0 c00243ec 60000013 ffffffff c00243a4 
c0024368
ffc0: c03ab174 c03a3a90 c001ed30 c0381cc8 2001ccec c00088d4 c0008434 
00000000
ffe0: 00000000 c001ed30 c0007175 c03a3af8 c001f134 20008034 00000000 
00000000
[<c0214ec4>] (netif_receive_skb+0x284/0x2e8) from [<c0214fb4>] 
(process_backlog+0x8c/0xd8)
[<c0214fb4>] (process_backlog+0x8c/0xd8) from [<c0213764>] 
(net_rx_action+0x68/0x170)
[<c0213764>] (net_rx_action+0x68/0x170) from [<c0040358>] 
(__do_softirq+0x74/0x104)
[<c0040358>] (__do_softirq+0x74/0x104) from [<c0040428>] 
(irq_exit+0x40/0x58)
[<c0040428>] (irq_exit+0x40/0x58) from [<c0022060>] (_text+0x60/0x78)
[<c0022060>] (_text+0x60/0x78) from [<c0022a3c>] (__irq_svc+0x3c/0x80)
Exception stack(0xc037ff70 to 0xc037ffb8)
ff60:                                     00000000 00000001 00000080 
60000013
ff80: c00243a4 c037e000 c0381e7c c00243a4 c03a3ac8 41129200 2001ccb8 
00000000
ffa0: fefff800 c037ffb8 c00243e0 c00243ec 60000013 ffffffff 

[<c0022a3c>] (__irq_svc+0x3c/0x80) from [<c00243e0>] 
(default_idle+0x3c/0x54)
[<c00243e0>] (default_idle+0x3c/0x54) from [<c0024368>] 
(cpu_idle+0x48/0x84)
[<c0024368>] (cpu_idle+0x48/0x84) from [<c00088d4>] 
(start_kernel+0x208/0x254)
[<c00088d4>] (start_kernel+0x208/0x254) from [<20008034>] (0x20008034)
Code: 0a000007 e1a00005 e1a03007 e5951018 (e1a02006)
Kernel panic - not syncing: Fatal exception in interrupt
[<c002895c>] (unwind_backtrace+0x0/0xdc) from [<c02b1dfc>] 
(panic+0x3c/0x120)
[<c02b1dfc>] (panic+0x3c/0x120) from [<c0026e60>] (die+0x154/0x180)
[<c0026e60>] (die+0x154/0x180) from [<c0026f30>] (baddataabort+0x0/0xac)
[<c0026f30>] (baddataabort+0x0/0xac) from [<c037fe9c>] (0xc037fe9c)