Kernel Oops with kernel 2.4.25 and iptables 1.2.9 in __unexpect_related

Kernel Oops with kernel 2.4.25 and iptables 1.2.9 in __unexpect_related

[Peter Olsson]

[-- Attachment #1: Type: text/plain, Size: 4593 bytes --]

After running my firewall without any problems for about 90 days, I suddenly got a Kernel Oops.
When running it through ksymoops I received the result below.

I've read a few comments about a bug in the newnat code in unexpect_related, but what I can see
this should be fixed a long time ago, and if I compare the unexpect_related calls in the ip_conntrack_core.c
from version 2.4.25 til 2.4.26 they work in exactly the same way. Could there be something else causing
this problem? I don't think it's hardware related, I've been using the same hardware, but with kernel 2.4.27
for about a year before this happened, and it's been working really good.

Best regards,

Peter Olsson

----

ksymoops on i686 2.4.25.  Options used
     -V (specified)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.25/ (default)
     -m /boot/System.map-2.4.25 (default)

Unable to handle kernel NULL pointer dereference at virtual address 00000004 printing eip:
c026f410
*pde = 00000000
Oops: 0002
CPU:    0
EIP:    0010:[<c026f410>]  Tainted: P
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010286
eax: 00000000   ebx: e1499a80     ecx: e1499a8c       edx: 00000000
esi: f7c34b8c   edi: f7c34b80     ebp: c2477380       esp: c036bcfc
ds: 0018   es: 0018   ss: 0018
Process swapper (pid: 0, stackpage=c036b000)
Stack: 00000000 f7dd1000 e1499a80 c036a000 c036a000 c0271bc0 e1499a80 e1499a80
       c0272ee0 e1499a80 e1499a80 e6c9c500 f7c34b80 f7c34bc0 00000000 c02704fd
       e6c9c500 c9648c80 cd6dd2d9 00000001 f7c34b80 c036bddc e6c9c500 020749d4
Call Trace:    [<c0271bc0>] [<c0272ee0>] [<c02704fd>] [<c027067c>] [<c022fac6>]
  [<c023d148>] [<c023be60>] [<c022f75e>] [<c023be60>] [<c023be60>] [<c022fa8f>]
  [<c023be60>] [<c023bce6>] [<c023be60>] [<c0227889>] [<c0222fbf>] [<c0227889>]
  [<c0227985>] [<f99a5f68>] [<c0227adf>] [<c011dafb>] [<c0108bad>] [<c01052c0>]
  [<c01052c0>] [<c01052c0>] [<c01052c0>] [<c01052e9>] [<c0105372>] [<c0105000>]
Code: 89 50 04 89 02 8b 43 0c c7 03 00 00 00 00 c7 43 04 00 00 00

>>EIP; c026f410 <__unexpect_related+10/60>   <=====
Trace; c0271bc0 <ip_conntrack_unexpect_related+30/70>
Trace; c0272ee0 <pptp_expectfn+90/b0>
Trace; c02704fd <init_conntrack+40d/420>
Trace; c027067c <ip_conntrack_in+16c/2a0>
Trace; c022fac6 <nf_hook_slow+106/180>
Trace; c023d148 <ip_forward+1f8/260>
Trace; c023be60 <ip_rcv_finish+0/20f>
Trace; c022f75e <nf_iterate+2e/80>
Trace; c023be60 <ip_rcv_finish+0/20f>
Trace; c023be60 <ip_rcv_finish+0/20f>
Trace; c022fa8f <nf_hook_slow+cf/180>
Trace; c023be60 <ip_rcv_finish+0/20f>
Trace; c023bce6 <ip_rcv+3a6/3f0>
Trace; c023be60 <ip_rcv_finish+0/20f>
Trace; c0227889 <netif_receive_skb+189/1c0>
Trace; c0222fbf <alloc_skb+ef/1c0>
Trace; c0227889 <netif_receive_skb+189/1c0>
Trace; c0227985 <process_backlog+c5/180>
Trace; f99a5f68 <[tg3]tg3_poll+f8/110>
Trace; c0227adf <net_rx_action+9f/160>
Trace; c011dafb <do_softirq+6b/d0>
Trace; c0108bad <do_IRQ+dd/f0>
Trace; c01052c0 <default_idle+0/40>
Trace; c01052c0 <default_idle+0/40>
Trace; c01052c0 <default_idle+0/40>
Trace; c01052c0 <default_idle+0/40>
Trace; c01052e9 <default_idle+29/40>
Trace; c0105372 <cpu_idle+52/70>
Trace; c0105000 <_stext+0/0>
Code;  c026f410 <__unexpect_related+10/60>
00000000 <_EIP>:
Code;  c026f410 <__unexpect_related+10/60>   <=====
   0:   89 50 04                  mov    %edx,0x4(%eax)   <=====
Code;  c026f413 <__unexpect_related+13/60>
   3:   89 02                     mov    %eax,(%edx)
Code;  c026f415 <__unexpect_related+15/60>
   5:   8b 43 0c                  mov    0xc(%ebx),%eax
Code;  c026f418 <__unexpect_related+18/60>
   8:   c7 03 00 00 00 00         movl   $0x0,(%ebx)
Code;  c026f41e <__unexpect_related+1e/60>
   e:   c7 43 04 00 00 00 00      movl   $0x0,0x4(%ebx)

 <0>Kernel panic: Aiee, killing interrupt handler!

----

This is the actual __unexpect_related code in my ip_contrack_core.c:

static void __unexpect_related(struct ip_conntrack_expect *expect)
{
        DEBUGP("unexpect_related(%p)\n", expect);
        MUST_BE_WRITE_LOCKED(&ip_conntrack_lock);  

        /* we're not allowed to unexpect a confirmed expectation! */
        IP_NF_ASSERT(!expect->sibling);

        /* delete from global and local lists */
        list_del(&expect->list);
        list_del(&expect->expected_list);

        /* decrement expect-count of master conntrack */
        if (expect->expectant)
                expect->expectant->expecting--;

        ip_conntrack_expect_put(expect);
}

[-- Attachment #2: Type: text/html, Size: 8666 bytes --]

Re: Kernel Oops with kernel 2.4.25 and iptables 1.2.9 in __unexpect_related

[Patrick McHardy]

Peter Olsson wrote:
> After running my firewall without any problems for about 90 days, I 
> suddenly got a Kernel Oops.
> When running it through ksymoops I received the result below.
>  
> I've read a few comments about a bug in the newnat code in 
> unexpect_related, but what I can see
> this should be fixed a long time ago, and if I compare the 
> unexpect_related calls in the ip_conntrack_core.c
> from version 2.4.25 til 2.4.26 they work in exactly the same way. Could 
> there be something else causing
> this problem? I don't think it's hardware related, I've been using the 
> same hardware, but with kernel 2.4.27
> for about a year before this happened, and it's been working really good.

We are at 2.4.27-pre5 now ..

It looks like the list is uninitialized in list_del. Have you applied
any other patches besides pptp_conntrack_nat ?

Regards
Patrick

Re: Kernel Oops with kernel 2.4.25 and iptables 1.2.9 in __unexpect_related

[Peter Olsson]

Hi Patrick,

I meant kernel 2.4.17, not 2.4.27 :)

About other patches... I've applied a couple of other patch-o-matic patches,
but pptp_conntrack_nat
is the only one that seem to call anything related to the __unexpect_related
function. I've only applied
patches that were tested ok, not the experimental ones. The
"__unexpect_related" has the following
code in my version of ip_conntrack_core.c.

Thanks for your answer, I'd really appreciate any help I can get on this.

Best regards,
Peter Olsson

----

/* remove one specific expectation from all lists and drop refcount,
 * does _NOT_ delete the timer. */
static void __unexpect_related(struct ip_conntrack_expect *expect)
{
        DEBUGP("unexpect_related(%p)\n", expect);
        MUST_BE_WRITE_LOCKED(&ip_conntrack_lock);

        /* we're not allowed to unexpect a confirmed expectation! */
        IP_NF_ASSERT(!expect->sibling);

        /* delete from global and local lists */
        list_del(&expect->list);
        list_del(&expect->expected_list);

        /* decrement expect-count of master conntrack */
        if (expect->expectant)
                expect->expectant->expecting--;

        ip_conntrack_expect_put(expect);
}



----- Original Message ----- 
From: "Patrick McHardy" <kaber@trash.net>
To: "Peter Olsson" <aa@tbh.se>
Cc: <netfilter-devel@lists.netfilter.org>
Sent: Tuesday, June 15, 2004 10:31 AM
Subject: Re: Kernel Oops with kernel 2.4.25 and iptables 1.2.9 in
__unexpect_related


> Peter Olsson wrote:
> > After running my firewall without any problems for about 90 days, I
> > suddenly got a Kernel Oops.
> > When running it through ksymoops I received the result below.
> >
> > I've read a few comments about a bug in the newnat code in
> > unexpect_related, but what I can see
> > this should be fixed a long time ago, and if I compare the
> > unexpect_related calls in the ip_conntrack_core.c
> > from version 2.4.25 til 2.4.26 they work in exactly the same way. Could
> > there be something else causing
> > this problem? I don't think it's hardware related, I've been using the
> > same hardware, but with kernel 2.4.27
> > for about a year before this happened, and it's been working really
good.
>
> We are at 2.4.27-pre5 now ..
>
> It looks like the list is uninitialized in list_del. Have you applied
> any other patches besides pptp_conntrack_nat ?
>
> Regards
> Patrick
>

Re: Kernel Oops with kernel 2.4.25 and iptables 1.2.9 in __unexpect_related

[Patrick McHardy]

Peter Olsson wrote:
> Hi Patrick,
> 
> I meant kernel 2.4.17, not 2.4.27 :)
> 
> About other patches... I've applied a couple of other patch-o-matic patches,
> but pptp_conntrack_nat
> is the only one that seem to call anything related to the __unexpect_related
> function. I've only applied
> patches that were tested ok, not the experimental ones. The
> "__unexpect_related" has the following
> code in my version of ip_conntrack_core.c.

Well, there is at least one that won't work with pptp_conntrack_nat
(expect-optimize), so it would really help to see a list. When did
you patch your kernel with pom (roughly) ? You also didn't mention
which kernel version you ran when it crashed.

Regards
Patrick

Re: Kernel Oops with kernel 2.4.25 and iptables 1.2.9 in __unexpect_related

[Peter Olsson]

I looked through all applied patches from pom, and it's the list below. The
kernel I'm using now is 2.4.25, and that's the one I was using when it
crashed, I applied these patches sometime in february and it was from
pom-20031219. I guess some of these patches actually were applied in the
2.4.25 kernel source.

                 submitted/61-remove-memsets
                 submitted/64_masquerade-sameip-noflush
                 submitted/69_amanda-helpers
                 submitted/70_expect-evict-order
                 submitted/72_recent_procfs_fix
                 submitted/74_listhelp
                 submitted/75_selective_cleanup
                 submitted/76_conntrack_bucket_sysctl
                 submitted/88_ip_queue-maxlen
                 submitted/90_fw_compat_local-nullbinding
                 pending/40_nf-log
                 pending/40_nf-log-ipv6
                 pending/59_ip_nat_h-unused-var
                 pending/60_ecn_raw_unclone
                 base/HL-ipv6
                 base/iprange
                 base/mport
                 base/NETLINK
                 base/REJECT-ipv6
                 base/SAME
                 base/time
                 base/TTL
                 extra/CLASSIFY
                 extra/CONNMARK
                 extra/cuseeme-nat
                 extra/IPMARK
                 extra/mms-conntrack-nat
                 extra/netfilter-docbook
                 extra/pptp-conntrack-nat
                 extra/rpc
                 extra/rsh
                 extra/string

Regards,

Peter

----- Original Message ----- 
From: "Patrick McHardy" <kaber@trash.net>
To: "Peter Olsson" <aa@tbh.se>
Cc: <netfilter-devel@lists.netfilter.org>
Sent: Tuesday, June 15, 2004 8:06 PM
Subject: Re: Kernel Oops with kernel 2.4.25 and iptables 1.2.9 in
__unexpect_related


> Peter Olsson wrote:
> > Hi Patrick,
> >
> > I meant kernel 2.4.17, not 2.4.27 :)
> >
> > About other patches... I've applied a couple of other patch-o-matic
patches,
> > but pptp_conntrack_nat
> > is the only one that seem to call anything related to the
__unexpect_related
> > function. I've only applied
> > patches that were tested ok, not the experimental ones. The
> > "__unexpect_related" has the following
> > code in my version of ip_conntrack_core.c.
>
> Well, there is at least one that won't work with pptp_conntrack_nat
> (expect-optimize), so it would really help to see a list. When did
> you patch your kernel with pom (roughly) ? You also didn't mention
> which kernel version you ran when it crashed.
>
> Regards
> Patrick
>
>
>
>

Re: Kernel Oops with kernel 2.4.25 and iptables 1.2.9 in __unexpect_related

[Henrik Nordstrom]

On Tue, 15 Jun 2004, Peter Olsson wrote:

> I looked through all applied patches from pom, and it's the list below.

You really ought to be using pom-ng these days..

Maybe updates/05_linux-2.4.26-orphaned_expect.patch is relevant to your 
problem?

Regards
Henrik

Re: Kernel Oops with kernel 2.4.25 and iptables 1.2.9 in __unexpect_related

[Peter Olsson]

Yes, I know. But I just don't have the time for the moment...

I've applied the orphaned_expect patch now, hopefully it will help! :)

Thanks for all help,

Peter Olsson

----- Original Message ----- 
From: "Henrik Nordstrom" <hno@marasystems.com>
To: "Peter Olsson" <aa@tbh.se>
Cc: "Netfilter Developers List" <netfilter-devel@lists.netfilter.org>
Sent: Wednesday, June 16, 2004 12:07 AM
Subject: Re: Kernel Oops with kernel 2.4.25 and iptables 1.2.9 in
__unexpect_related


> On Tue, 15 Jun 2004, Peter Olsson wrote:
>
> > I looked through all applied patches from pom, and it's the list below.
>
> You really ought to be using pom-ng these days..
>
> Maybe updates/05_linux-2.4.26-orphaned_expect.patch is relevant to your
> problem?
>
> Regards
> Henrik
>
>