iptables causing kernel panic - part II

iptables causing kernel panic - part II

[Matt Ayres]

Hi,

I am seeing the following panic's from the latest 3.0-testing.  SCTP is 
disabled entirely and the only bugfix patches beyond 2.6.16.13 affect 
SCTP for the most part.

Here is my trace, I was not able to get the Code line unfortunately so I 
don't have specific line numbers.  I'd like re-confirmation this is not 
specific to Xen. If it is not I will be sending this over to LKML.

Panic:

Unable to handle kernel paging request at virtual addr
c03d96a9
06099000 -> *pde = 00000001:a75cb001
005a0000 -> *pme = 00000000:eb2f0067
0faf0000 -> *pte = 00000000:00000000
Oops: 0000 [#1]
CPU:    0
EIP:    0061:[<c03d96a9>]    Not tainted VLI
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010286   (2.6.16.13-xen #1)
eax: 00000002   ebx: c8635020   ecx: c8635020   edx: 00000000
esi: d143a000   edi: 00000000   ebp: d143e560   esp: c0517d8c
ds: 007b   es: 007b   ss: 0069
Stack: <0>c04d056c 00000000 c8635020 00000000 cf4a8000 c96e2800 d143a000 
0000000
00000001 00000000 c0517e2c 80000000 c03acc04 c96e2800 c03da724 c0517e70
00000002 cf4a8000 c96e2800 c04d0540 00000000 c03a2e50 00000002 c0517e70
Call Trace:
[<c03acc04>] ip_forward_finish+0x0/0x36
[<c03da724>] ipt_hook+0x1c/0x20
[<c03a2e50>] nf_iterate+0x2c/0x5e
[<c03acc04>] ip_forward_finish+0x0/0x36
[<c03acc04>] ip_forward_finish+0x0/0x36
[<c03a2f4b>] nf_hook_slow+0x3c/0xc3
[<c03acc04>] ip_forward_finish+0x0/0x36
[<c03acdd8>] ip_forward+0x19e/0x22e
[<c03acc04>] ip_forward_finish+0x0/0x36
[<c03abcf7>] ip_rcv+0x40e/0x48f
[<c037dcb5>] netif_receive_skb+0x255/0x294
[<c02e82e6>] tg3_poll+0x532/0x76c
[<c037bd82>] net_rx_action+0xaa/0x17c

ksymoops output:

 >>EIP; c03d96a9 <ipt_do_table+ad/2d0>   <=====

 >>esp; c0517d8c <init_thread_union+1d8c/2000>

Trace; c03acc04 <ip_forward_finish+0/36>
Trace; c03da724 <ipt_hook+1c/20>
Trace; c03a2e50 <nf_iterate+2c/5e>
Trace; c03acc04 <ip_forward_finish+0/36>
Trace; c03acc04 <ip_forward_finish+0/36>
Trace; c03a2f4b <nf_hook_slow+3c/c3>
Trace; c03acc04 <ip_forward_finish+0/36>
Trace; c03acdd8 <ip_forward+19e/22e>
Trace; c03acc04 <ip_forward_finish+0/36>
Trace; c03abcf7 <ip_rcv+40e/48f>
Trace; c037dcb5 <netif_receive_skb+255/294>
Trace; c02e82e6 <tg3_poll+532/76c>
Trace; c037bd82 <net_rx_action+aa/17c>


Question on this, swapper seems to be the "process" the majority of the 
time.  Could this be from netfilter conntrack being overallocated for 
the RAM available to dom0 and since netfilter can't be swapped out it 
causes a panic... or should the kernel just "know" this and swap out the 
apps.

Thank you,
Matt Ayres

Re: iptables causing kernel panic - part II

[Keir Fraser]

On 15 May 2006, at 16:35, Matt Ayres wrote:

> Here is my trace, I was not able to get the Code line unfortunately so 
> I don't have specific line numbers.  I'd like re-confirmation this is 
> not specific to Xen. If it is not I will be sending this over to LKML.

There are no Xen-specific functions in the backtrace. This doesn't mean 
it isn't a Xen bug (since we make various pervasive functions that are 
used all over the place) but it does make it less likely. Certainly 
there's nothing in the backtrace that we can use to track down the bug. 
We would need a line number, or a kernel image or something like that.

> Question on this, swapper seems to be the "process" the majority of 
> the time.  Could this be from netfilter conntrack being overallocated 
> for the RAM available to dom0 and since netfilter can't be swapped out 
> it causes a panic... or should the kernel just "know" this and swap 
> out the apps.

"swapper" is a misnomer. Actually that is the idle process, so it 
simply indicates that the processor was idle when that network 
interrupt occurred. The swapper is usually called kswapd.

  -- Keir

Re: iptables causing kernel panic - part II

[Matt Ayres]

Keir Fraser wrote:
> 
> On 15 May 2006, at 16:35, Matt Ayres wrote:
> 
>> Here is my trace, I was not able to get the Code line unfortunately so 
>> I don't have specific line numbers.  I'd like re-confirmation this is 
>> not specific to Xen. If it is not I will be sending this over to LKML.
> 
> There are no Xen-specific functions in the backtrace. This doesn't mean 
> it isn't a Xen bug (since we make various pervasive functions that are 
> used all over the place) but it does make it less likely. Certainly 
> there's nothing in the backtrace that we can use to track down the bug. 
> We would need a line number, or a kernel image or something like that.
> 

I'll try to get the Code line next time.  Is there a more preferred 
method than using ksymoops? It seems ksymoops is depreciated as a 
/proc/ksyms doesn't even exist on my system and Fedora has stopped 
shipping RPM's for it.  I do have the vmlinux file and System.map if 
that would help.

Sometimes the panic includes xen_idle, ex:

EIP is at ipt_do_table+0xad/0x2d0
eax: 00000002   ebx: cadc221e   ecx: cadc221e   edx: 00000000
esi: d13f6000   edi: 00004000   ebp: d13f9c20   esp: c0525dd0
ds: 007b   es: 007b   ss: 0069
Process swapper (pid: 0, threadinfo=c0524000 task=c04a8c00)
Stack: <0>c04dc0ac 00000000 cadc221e 00000000 c83a1000 cfd60000 d13f6000 
00000000
       00000001 00000000 c0525e70 80000000 c03aeca8 cfd60000 c03e1e18 
c0525eb4
       00000002 c83a1000 cfd60000 c04dc080 00000000 c03a4420 00000002 
c0525eb4
Call Trace:
[<c03aeca8>] ip_forward_finish+0x0/0x36
[<c03e1e18>] ipt_hook+0x1c/0x20
[<c03a4420>] nf_iterate+0x2c/0x5e
[<c03aeca8>] ip_forward_finish+0x0/0x36
[<c03aeca8>] ip_forward_finish+0x0/0x36
PORT HISTORY> next
[<c03a451b>] nf_hook_slow+0x3c/0xc3
[<c03aeca8>] ip_forward_finish+0x0/0x36
[<c03aee7c>] ip_forward+0x19e/0x22e
[<c03aeca8>] ip_forward_finish+0x0/0x36
[<c03add9b>] ip_rcv+0x40e/0x48f
[<c037f289>] netif_receive_skb+0x2b9/0x2f7
[<c037f382>] process_backlog+0xbb/0x14b
[<c037d2ee>] net_rx_action+0xaa/0x17c
[<c011ea03>] __do_softirq+0x73/0xf0
[<c011eac0>] do_softirq+0x40/0x64
[<c010606b>] do_IRQ+0x1f/0x25
[<c02fdfa4>] evtchn_do_upcall+0x60/0x96
[<c0104a2c>] hypervisor_callback+0x2c/0x34
[<c010342e>] xen_idle+0x5e/0x7d
[<c0103509>] cpu_idle+0xbc/0xd5
[<c05264e0>] start_kernel+0x344/0x34b
Code: 89 44 24 18 89 c6 8b 44 24 40 8b 6c 24 18 03 74 83 0c 03 6c 83 20 
c7 44 24 0c 00 00 00 00 0f b7 54 24 06 8b 5c 24 08 89 54 24 1c <8a> 4e 
53 88 4c 24 23 8b 46 08 0f b6 c9 23 43 0c 3b 06 89 c8 0f
<0>Kernel panic - not syncing: Fatal exception in interrupt
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.

I have the Code line for this, but unfortunately I do not have the 
System.map that corresponds to this particular kernel.