[Bug 10875] Oops in nf_nat_setup_info

[Bug 10875] Oops in nf_nat_setup_info

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1847 bytes --]

Hello,

Today I have been plaing with the conntrackd utility and noticed it is 
very easy to trigger a kernel oops just by:

conntrackd -d
conntrackd -n
conntrackd -c
conntrackd -c

The oops is here: http://bugzilla.kernel.org/attachment.cgi?id=16414

I was trying to fix it with:

--- nf_nat_core.c	2008-06-06 19:55:25.000000000 +0200
+++ nf_nat_core.c	2008-05-07 01:22:34.000000000 +0200
@@ -153,7 +153,7 @@
         read_lock_bh(&nf_nat_lock);
         hlist_for_each_entry(nat, n, &bysource[h], bysource) {
                 ct = nat->ct;
-               if (ct && same_src(ct, tuple)) {
+               if (same_src(ct, tuple)) {
                         /* Copy source part from reply tuple. */
                         nf_ct_invert_tuplepr(result,
                                        &ct->tuplehash[IP_CT_DIR_REPLY].tuple);


However and I'm not able to find how nat->ct may become NULL in here and 
unfortutunatelly this patch does not help too much as with above fix I get 
a different Oops: http://bugzilla.kernel.org/attachment.cgi?id=16415

(gdb) l *nf_nat_setup_info+0x223
0x783e30de is in nf_nat_setup_info (net/ipv4/netfilter/nf_nat_core.c:154).
149             struct nf_conn_nat *nat;
150             struct nf_conn *ct;
151             struct hlist_node *n;
152
153             read_lock_bh(&nf_nat_lock);
154             hlist_for_each_entry(nat, n, &bysource[h], bysource) {        <- here
155                     ct = nat->ct;
156                     if (ct && same_src(ct, tuple)) {
157                             /* Copy source part from reply tuple. */
158                             nf_ct_invert_tuplepr(result,

All accesses to bysource seem to be protected by the lock_bh so I have no 
concept where to dig next. :( Any idea?

Best regards,


 				Krzysztof Olędzki

Re: [Bug 10875] Oops in nf_nat_setup_info

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 392 bytes --]



On Fri, 6 Jun 2008, Krzysztof Oledzki wrote:

> Hello,
>
> Today I have been plaing with the conntrackd utility and noticed it is very 
> easy to trigger a kernel oops just by:
>
> conntrackd -d
> conntrackd -n
> conntrackd -c
> conntrackd -c

OK, quite often is is enough to run only the first "conntrackd -c" to 
crash the kernel.

Best regards,

 				Krzysztof Olędzki

Re: [Bug 10875] Oops in nf_nat_setup_info

[Patrick McHardy]

Krzysztof Oledzki wrote:
>
>
> On Fri, 6 Jun 2008, Krzysztof Oledzki wrote:
>
>> Hello,
>>
>> Today I have been plaing with the conntrackd utility and noticed it 
>> is very easy to trigger a kernel oops just by:
>>
>> conntrackd -d
>> conntrackd -n
>> conntrackd -c
>> conntrackd -c
>
> OK, quite often is is enough to run only the first "conntrackd -c" to 
> crash the kernel. 

Is that with or without your accounting patch?

Re: [Bug 10875] Oops in nf_nat_setup_info

[Patrick McHardy]

Patrick McHardy wrote:
> Krzysztof Oledzki wrote:
>>
>>
>> On Fri, 6 Jun 2008, Krzysztof Oledzki wrote:
>>
>>> Hello,
>>>
>>> Today I have been plaing with the conntrackd utility and noticed it 
>>> is very easy to trigger a kernel oops just by:
>>>
>>> conntrackd -d
>>> conntrackd -n
>>> conntrackd -c
>>> conntrackd -c
>>
>> OK, quite often is is enough to run only the first "conntrackd -c" to 
>> crash the kernel. 
>
> Is that with or without your accounting patch? 

In case its not, does that kernel include commit 86577c661?

Re: [Bug 10875] Oops in nf_nat_setup_info

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 891 bytes --]



On Sat, 7 Jun 2008, Patrick McHardy wrote:

> Patrick McHardy wrote:
>> Krzysztof Oledzki wrote:
>>> 
>>> 
>>> On Fri, 6 Jun 2008, Krzysztof Oledzki wrote:
>>> 
>>>> Hello,
>>>> 
>>>> Today I have been plaing with the conntrackd utility and noticed it is 
>>>> very easy to trigger a kernel oops just by:
>>>> 
>>>> conntrackd -d
>>>> conntrackd -n
>>>> conntrackd -c
>>>> conntrackd -c
>>> 
>>> OK, quite often is is enough to run only the first "conntrackd -c" to 
>>> crash the kernel. 
>> 
>> Is that with or without your accounting patch? 
>
> In case its not, does that kernel include commit 86577c661?

No, it does not but unfortunately this fix does not solve the crash. 
However, before you spend too much time on this I'll check 2.6.25.6 and 
2.6.26-rc to make sure this problem has not been solved already.

Best regards,

 				Krzysztof Olędzki

Re: [Bug 10875] Oops in nf_nat_setup_info

[Patrick McHardy]

Krzysztof Oledzki wrote:
> On Sat, 7 Jun 2008, Patrick McHardy wrote:
>> In case its not, does that kernel include commit 86577c661?
> 
> No, it does not but unfortunately this fix does not solve the crash. 
> However, before you spend too much time on this I'll check 2.6.25.6 and 
> 2.6.26-rc to make sure this problem has not been solved already.

Thanks, please let me know how it turns out.

Re: [Bug 10875] Oops in nf_nat_setup_info

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 4539 bytes --]



On Sat, 7 Jun 2008, Patrick McHardy wrote:

> Krzysztof Oledzki wrote:
>> On Sat, 7 Jun 2008, Patrick McHardy wrote:
>>> In case its not, does that kernel include commit 86577c661?
>> 
>> No, it does not but unfortunately this fix does not solve the crash. 
>> However, before you spend too much time on this I'll check 2.6.25.6 and 
>> 2.6.26-rc to make sure this problem has not been solved already.
>
> Thanks, please let me know how it turns out.

Clean 2.6.26-rc5 kernel, no additional patches at all.

BUG: unable to handle kernel NULL pointer dereference at 00000032
IP: [<c03d930e>] nf_nat_setup_info+0x219/0x57f
*pde = 00000000 
Oops: 0000 [#1] PREEMPT SMP

Pid: 1414, comm: conntrackd Not tainted (2.6.26-rc5 #1)
EIP: 0060:[<c03d930e>] EFLAGS: 00010282 CPU: 1
EIP is at nf_nat_setup_info+0x219/0x57f
EAX: c05bd47c EBX: f754bcc4 ECX: 0000000c EDX: 00000000
ESI: 0000019e EDI: f1c49bb4 EBP: f1c49bc8 ESP: f1c49b78
  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process conntrackd (pid: 1414, ti=f1c48000 task=f7d31030 task.ti=f1c48000)
Stack: 00000000 f1c49c2c f322b7fc 00000008 0005caaa f1c49bac 0005caaa c0138e70
        0552215c 00000000 117334e0 00005102 c2012108 3aaf4108 0000002c 00000000
        c0139bdc 3aa00780 f50b3474 f1c49c04 00000008 c038e728 0000000a f50b3474 
Call Trace:
  [<c0138e70>] clockevents_program_event+0xca/0xd9
  [<c0139bdc>] tick_program_event+0x30/0x4f
  [<c038e728>] nla_parse+0x5c/0xb0
  [<c039801f>] ctnetlink_change_status+0x190/0x1c6
  [<c03982f0>] ctnetlink_new_conntrack+0x189/0x61f
  [<c0108346>] read_tsc+0x6/0x22
  [<c01367c4>] getnstimeofday+0x32/0xad
  [<c03902d1>] nfnetlink_rcv_msg+0xc1/0xd8
  [<c0390228>] nfnetlink_rcv_msg+0x18/0xd8
  [<c0390210>] nfnetlink_rcv_msg+0x0/0xd8
  [<c038d2ce>] netlink_rcv_skb+0x2d/0x71
  [<c0390205>] nfnetlink_rcv+0x19/0x24
  [<c038d0f5>] netlink_unicast+0x1b3/0x216
  [<c038d892>] netlink_sendmsg+0x237/0x244
  [<c035cf39>] sock_sendmsg+0xb8/0xd1
  [<c013223c>] autoremove_wake_function+0x0/0x2b
  [<c013223c>] autoremove_wake_function+0x0/0x2b
  [<c035d7a0>] sys_sendto+0xfc/0x127
  [<c014f90e>] __pagevec_lru_add_active+0x99/0xa4
  [<c0152909>] __inc_zone_state+0x10/0x61
  [<c045c854>] _spin_unlock+0xc/0x1f
  [<c015421a>] do_wp_page+0x3e7/0x440
  [<c035e02d>] sys_socketcall+0x106/0x196
  [<c0103946>] syscall_call+0x7/0xb
  =======================
Code: e8 25 4e d4 ff 89 e0 25 00 e0 ff ff f6 40 08 04 74 48 e8 2f 1c 08 00 eb 41 8b 1b 85 db 74 1d 8b 03 0f 18 00 90 8b 53 18 8d 4a 0c <8a> 41 26 3a 84 24 8a 00 00 00 75 e2 e9 09 ff ff ff b8 01 00 00 
EIP: [<c03d930e>] nf_nat_setup_info+0x219/0x57f SS:ESP 0068:f1c49b78
---[ end trace 5de3919242e64ed5 ]---
note: conntrackd[1414] exited with preempt_count 1
BUG: scheduling while atomic: conntrackd/1414/0x10000002
Pid: 1414, comm: conntrackd Tainted: G      D   2.6.26-rc5 #1
  [<c045a96f>] schedule+0x9b/0x60b
  [<c015ba07>] free_pages_and_swap_cache+0x6a/0x7e
  [<c011ed44>] __cond_resched+0xf/0x27
  [<c045b00e>] _cond_resched+0x21/0x2a
  [<c0154845>] unmap_vmas+0x47e/0x551
  [<c01573c7>] exit_mmap+0x70/0xf8
  [<c012162a>] mmput+0x1c/0x7e
  [<c0125b4c>] do_exit+0x1dc/0x572
  [<c0104ccb>] die+0x11f/0x124
  [<c01143bf>] do_page_fault+0x4ae/0x567
  [<c01367c4>] getnstimeofday+0x32/0xad
  [<c0113f11>] do_page_fault+0x0/0x567
  [<c045cb3a>] error_code+0x72/0x78
  [<c03d930e>] nf_nat_setup_info+0x219/0x57f
  [<c0138e70>] clockevents_program_event+0xca/0xd9
  [<c0139bdc>] tick_program_event+0x30/0x4f
  [<c038e728>] nla_parse+0x5c/0xb0
  [<c039801f>] ctnetlink_change_status+0x190/0x1c6
  [<c03982f0>] ctnetlink_new_conntrack+0x189/0x61f
  [<c0108346>] read_tsc+0x6/0x22
  [<c01367c4>] getnstimeofday+0x32/0xad
  [<c03902d1>] nfnetlink_rcv_msg+0xc1/0xd8
  [<c0390228>] nfnetlink_rcv_msg+0x18/0xd8
  [<c0390210>] nfnetlink_rcv_msg+0x0/0xd8
  [<c038d2ce>] netlink_rcv_skb+0x2d/0x71
  [<c0390205>] nfnetlink_rcv+0x19/0x24
  [<c038d0f5>] netlink_unicast+0x1b3/0x216
  [<c038d892>] netlink_sendmsg+0x237/0x244
  [<c035cf39>] sock_sendmsg+0xb8/0xd1
  [<c013223c>] autoremove_wake_function+0x0/0x2b
  [<c013223c>] autoremove_wake_function+0x0/0x2b
  [<c035d7a0>] sys_sendto+0xfc/0x127
  [<c014f90e>] __pagevec_lru_add_active+0x99/0xa4
  [<c0152909>] __inc_zone_state+0x10/0x61
  [<c045c854>] _spin_unlock+0xc/0x1f
  [<c015421a>] do_wp_page+0x3e7/0x440
  [<c035e02d>] sys_socketcall+0x106/0x196
  [<c0103946>] syscall_call+0x7/0xb
  =======================

Best regards,

 				Krzysztof Olędzki

Re: [Bug 10875] Oops in nf_nat_setup_info

[Patrick McHardy]

Krzysztof Oledzki wrote:
> 
> 
> On Sat, 7 Jun 2008, Patrick McHardy wrote:
> 
>> Krzysztof Oledzki wrote:
>>> On Sat, 7 Jun 2008, Patrick McHardy wrote:
>>>> In case its not, does that kernel include commit 86577c661?
>>>
>>> No, it does not but unfortunately this fix does not solve the crash. 
>>> However, before you spend too much time on this I'll check 2.6.25.6 
>>> and 2.6.26-rc to make sure this problem has not been solved already.
>>
>> Thanks, please let me know how it turns out.
> 
> Clean 2.6.26-rc5 kernel, no additional patches at all.
> 
> BUG: unable to handle kernel NULL pointer dereference at 00000032
> IP: [<c03d930e>] nf_nat_setup_info+0x219/0x57f
> *pde = 00000000 Oops: 0000 [#1] PREEMPT SMP


How heavily is that system loaded (conntrack-wise)?

Re: [Bug 10875] Oops in nf_nat_setup_info

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1112 bytes --]



On Sat, 7 Jun 2008, Patrick McHardy wrote:

> Krzysztof Oledzki wrote:
>> 
>> 
>> On Sat, 7 Jun 2008, Patrick McHardy wrote:
>> 
>>> Krzysztof Oledzki wrote:
>>>> On Sat, 7 Jun 2008, Patrick McHardy wrote:
>>>>> In case its not, does that kernel include commit 86577c661?
>>>> 
>>>> No, it does not but unfortunately this fix does not solve the crash. 
>>>> However, before you spend too much time on this I'll check 2.6.25.6 and 
>>>> 2.6.26-rc to make sure this problem has not been solved already.
>>> 
>>> Thanks, please let me know how it turns out.
>> 
>> Clean 2.6.26-rc5 kernel, no additional patches at all.
>> 
>> BUG: unable to handle kernel NULL pointer dereference at 00000032
>> IP: [<c03d930e>] nf_nat_setup_info+0x219/0x57f
>> *pde = 00000000 Oops: 0000 [#1] PREEMPT SMP
>
> How heavily is that system loaded (conntrack-wise)?

It is nearly 100% idle (backup). However, I'm able to trigger this bug 
when I try to synchronize connections from an active firewall (it is still 
running a heavily patched 2.6.22 kernel, btw).

Best regards,

 				Krzysztof Olędzki

Re: [Bug 10875] Oops in nf_nat_setup_info

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 577 bytes --]



On Sat, 7 Jun 2008, Patrick McHardy wrote:

> Krzysztof Oledzki wrote:
>> 
>> 
>> On Fri, 6 Jun 2008, Krzysztof Oledzki wrote:
>> 
>>> Hello,
>>> 
>>> Today I have been plaing with the conntrackd utility and noticed it is 
>>> very easy to trigger a kernel oops just by:
>>> 
>>> conntrackd -d
>>> conntrackd -n
>>> conntrackd -c
>>> conntrackd -c
>> 
>> OK, quite often is is enough to run only the first "conntrackd -c" to crash 
>> the kernel. 
>
> Is that with or without your accounting patch?

Without.

Best regards,

 				Krzysztof Olędzki