kernel crash in nf_nat_move_storage

kernel crash in nf_nat_move_storage

[Thomas Woerner]

[-- Attachment #1: Type: text/plain, Size: 995 bytes --]

Hello,

Using port forwarding from port 80 to 21 with nf_conntrack_ftp loaded 
results in a kernel crash, when connecting to port 80 from a remote
host. This seems to be a problem for kernels > 2.6.18 including 2.6.24.

Steps to Reproduce:

host1> iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j DNAT 
--to :21
host1> iptables -t filter -A INPUT -i eth0 -m state --state NEW -m tcp 
-p tcp --dport 21 -j ACCEPT
host1> modprobe ip_conntrack_ftp
host2> telnet host1 80

Attached is the kernel crash log for kernel 2.6.23.9-85.fc8PAE. I was 
told that this kernel crash dump is incomplete, but it took several 
attempts to get a log with more that 5 lines over serial console. The 
kernel seems to die too fast.

Thanks,
Thomas

-- 
Thomas Woerner
Software Engineer            Phone: +49-711-96437-310
Red Hat GmbH                 Fax  : +49-711-96437-111
Hauptstaetterstr. 58         Email: Thomas Woerner <twoerner@redhat.com>
D-70178 Stuttgart            Web  : http://www.redhat.de/

[-- Attachment #2: kernel-oups --]
[-- Type: text/plain, Size: 2924 bytes --]

sh-3.2# BUG: unable to handle kernel NULL pointer dereference at virtual addres4
printing eip: f8fcb087 *pdpt = 0000000037c82001 <1>*pde = 000000013f75d067 
Oops: 0000 [#1] SMP 
Modules linked in: nf_conntrack_ftp ipt_REJECT xt_state iptable_filter xt_tcpudd
CPU:    1
EIP:    0060:[<f8fcb087>]    Not tainted VLI
EFLAGS: 00010202   (2.6.23.9-85.fc8PAE #1)
EIP is at nf_nat_move_storage+0x23/0x69 [nf_nat]
eax: 00000004   ebx: f7e13d04   ecx: f7e13d00   edx: f7e13d00
esi: f7e13d10   edi: 00000000   ebp: f751b000   esp: c078bc84
ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0068
Process swapper (pid: 0, ti=c078b000 task=f7c02c20 task.ti=c38f1000)
Stack: f7885ea0 f8fcb064 00000001 f920c5dc 00000000 0000004c 00000028 00000000 
       00000000 f921d2c0 f751b000 f76418c0 f920a7a5 f9208d73 c078bce8 f8fce1e0 
       00000000 f8fcb9dd f751b000 00000000 f751b000 00000000 00000001 00000000 
Call Trace:
 [<f8fcb064>] nf_nat_move_storage+0x0/0x69 [nf_nat]
 [<f920c5dc>] __nf_ct_ext_add+0x128/0x1bc [nf_conntrack]
 [<f920a7a5>] nf_ct_helper_ext_add+0x9/0x15 [nf_conntrack]
 [<f9208d73>] nf_conntrack_alter_reply+0x73/0x96 [nf_conntrack]
 [<f8fcb9dd>] nf_nat_setup_info+0x3f3/0x54e [nf_nat]
 [<f92000ea>] ipt_dnat_target+0x0/0x14c [iptable_nat]
 [<f920022e>] ipt_dnat_target+0x144/0x14c [iptable_nat]
 [<f920c09d>] tcp_packet+0x9bc/0x9eb [nf_conntrack]
 [<c046760b>] __alloc_pages+0x64/0x2a2
 [<f92000ea>] ipt_dnat_target+0x0/0x14c [iptable_nat]
 [<f8fd759e>] ipt_do_table+0x3f0/0x482 [ip_tables]
 [<f9208ca8>] nf_conntrack_alloc+0x16d/0x1c5 [nf_conntrack]
 [<f920b3d6>] tcp_new+0xd1/0x1a4 [nf_conntrack]
 [<f920c4f8>] __nf_ct_ext_add+0x44/0x1bc [nf_conntrack]
 [<f9200257>] nf_nat_rule_find+0x21/0x5c [iptable_nat]
 [<f920040d>] nf_nat_fn+0x165/0x189 [iptable_nat]
 [<f920048e>] nf_nat_in+0x29/0x9c [iptable_nat]
 [<c05dab54>] ip_rcv_finish+0x0/0x291
 [<c05d5b9c>] nf_iterate+0x38/0x6a
 [<c05dab54>] ip_rcv_finish+0x0/0x291
 [<c05d5d07>] nf_hook_slow+0x4d/0xb5
 [<c05dab54>] ip_rcv_finish+0x0/0x291
 [<c05db261>] ip_rcv+0x20b/0x4ba
 [<c05dab54>] ip_rcv_finish+0x0/0x291
 [<c05be718>] netif_receive_skb+0x2e1/0x346
 [<f8e00e7d>] nv_napi_poll+0x48c/0x61e [forcedeth]
 [<c05c085c>] net_rx_action+0x9a/0x196
 [<c0432d62>] __do_softirq+0x66/0xd3
 [<c04073d5>] do_softirq+0x6c/0xce
 [<c04455e5>] tick_do_update_jiffies64+0x15/0xa8
 [<c04410ff>] ktime_get+0xf/0x2b
 [<c045c9f1>] handle_fasteoi_irq+0x0/0xa6
 [<c0432c25>] irq_exit+0x38/0x6b
 [<c04074d6>] do_IRQ+0x9f/0xb9
 [<c0403ddf>] default_idle+0x0/0x55
 [<c0405b6f>] common_interrupt+0x23/0x28
 [<c0403ddf>] default_idle+0x0/0x55
 [<c0422297>] native_safe_halt+0x2/0x3
 [<c0403e18>] default_idle+0x39/0x55
 [<c040340b>] cpu_idle+0xab/0xcc
 =======================
Code: 64 0f fe ff ff 31 c0 c3 57 56 89 d6 53 8b 90 ec 00 00 00 85 d2 74 0f 8a 4 
EIP: [<f8fcb087>] nf_nat_move_storage+0x23/0x69 [nf_nat] SS:ESP 0068:c078bc84
Kernel panic - not syncing: Fatal exception in interrupt

Re: kernel crash in nf_nat_move_storage

[Patrick McHardy]

Thomas Woerner wrote:
> Hello,
> 
> Using port forwarding from port 80 to 21 with nf_conntrack_ftp loaded 
> results in a kernel crash, when connecting to port 80 from a remote
> host. This seems to be a problem for kernels > 2.6.18 including 2.6.24.
> 
> Steps to Reproduce:
> 
> host1> iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j DNAT 
> --to :21
> host1> iptables -t filter -A INPUT -i eth0 -m state --state NEW -m tcp 
> -p tcp --dport 21 -j ACCEPT
> host1> modprobe ip_conntrack_ftp
> host2> telnet host1 80
> 
> Attached is the kernel crash log for kernel 2.6.23.9-85.fc8PAE. I was 
> told that this kernel crash dump is incomplete, but it took several 
> attempts to get a log with more that 5 lines over serial console. The 
> kernel seems to die too fast.


This is already fixed in 2.6.23.10.

Re: kernel crash in nf_nat_move_storage

[Thomas Woerner]

[-- Attachment #1: Type: text/plain, Size: 1709 bytes --]

Hello Patrick,

after sucessfully testing 2.6.23.14-107.fc8 on my i386 test system, I 
installed 2.6.23.14-107.fc8 on the x86_64 system. At first I was not 
able to reproduce the problem, but after starting the ftp server 
(vsftpd) and using 'echo "quit" | telnet test-x86_64 80' several times, 
I got a backtrace again. Please have a look at the attachment.

Thanks,
Thomas

Patrick McHardy wrote:
> Thomas Woerner wrote:
>> Hello,
>>
>> Using port forwarding from port 80 to 21 with nf_conntrack_ftp loaded 
>> results in a kernel crash, when connecting to port 80 from a remote
>> host. This seems to be a problem for kernels > 2.6.18 including 2.6.24.
>>
>> Steps to Reproduce:
>>
>> host1> iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j DNAT 
>> --to :21
>> host1> iptables -t filter -A INPUT -i eth0 -m state --state NEW -m tcp 
>> -p tcp --dport 21 -j ACCEPT
>> host1> modprobe ip_conntrack_ftp
>> host2> telnet host1 80
>>
>> Attached is the kernel crash log for kernel 2.6.23.9-85.fc8PAE. I was 
>> told that this kernel crash dump is incomplete, but it took several 
>> attempts to get a log with more that 5 lines over serial console. The 
>> kernel seems to die too fast.
> 
> 
> This is already fixed in 2.6.23.10.
> -
> To unsubscribe from this list: send the line "unsubscribe 
> netfilter-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Thomas Woerner
Software Engineer            Phone: +49-711-96437-310
Red Hat GmbH                 Fax  : +49-711-96437-111
Hauptstaetterstr. 58         Email: Thomas Woerner <twoerner@redhat.com>
D-70178 Stuttgart            Web  : http://www.redhat.de/

[-- Attachment #2: oups3 --]
[-- Type: text/plain, Size: 3797 bytes --]

stack segment: 0000 [1] SMP 
CPU 3 
Modules linked in: nf_conntrack_ftp ipt_REJECT ipt_LOG xt_state iptable_filter xt_tcpudp iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nfnetlink ip_tables x_tables cpufreq_ondemand dm_mirror dm_multipath dm_mod snd_intel8x0 snd_ac97_codec ac97_bus snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss cfi_cmdset_0002 cfi_util snd_mixer_oss shpchp jedec_probe firewire_ohci firewire_core cfi_probe gen_probe snd_pcm parport_pc ck804xrom k8temp hwmon parport floppy sr_mod mtd chipreg map_funcs cdrom snd_timer forcedeth snd pcspkr soundcore sg i2c_nforce2 crc_itu_t serio_raw button snd_page_alloc i2c_core pata_amd ata_generic sata_nv libata sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd
Pid: 0, comm: swapper Not tainted 2.6.23.14-107.fc8 #1
RIP: 0010:[<ffffffff8827e069>]  [<ffffffff8827e069>] :nf_nat:nf_nat_move_storage+0x2f/0x8a
RSP: 0018:ffff810001f7f9d0  EFLAGS: 00010206
RAX: 0000000000000008 RBX: ffff81007d306d08 RCX: ffff81007d306d00
RDX: ffff81007d306d00 RSI: ffff81007d306d20 RDI: ffff81007c1f8130
RBP: 73616c636632785c R08: ffff81007c1f8130 R09: 0000000000000000
R10: 000000004646dc9c R11: ffffffff8826a304 R12: ffff81007d306d20
R13: 0000000000000038 R14: 0000000000000001 R15: 0000000000000000
FS:  00002aaaad752260(0000) GS:ffff81007fead380(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00002aaaaace9958 CR3: 000000007c3df000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 0, threadinfo ffff810040f12000, task ffff810001f76820)
Stack:  0000000000000070 ffff81007c1f8130 ffff81007fefb600 ffffffff8826bb32
 0106d7be00000000 0000000000000000 ffffffff882a2640 ffff81007c1f8130
 ffff81007c1f8130 0000000000000000 0000000000000001 ffffffff88269b3f
Call Trace:
 <IRQ>  [<ffffffff8826bb32>] :nf_conntrack:__nf_ct_ext_add+0x136/0x1bc
 [<ffffffff88269b3f>] :nf_conntrack:nf_ct_helper_ext_add+0xd/0x1c
 [<ffffffff88267f1b>] :nf_conntrack:nf_conntrack_alter_reply+0x89/0xb4
 [<ffffffff8827ea95>] :nf_nat:nf_nat_setup_info+0x3f1/0x548
 [<ffffffff88286232>] :iptable_nat:ipt_dnat_target+0x141/0x14c
 [<ffffffff8125d30f>] _write_lock_bh+0x9/0x1c
 [<ffffffff88267a98>] :nf_conntrack:__nf_ct_refresh_acct+0x137/0x178
 [<ffffffff8825ca80>] :ip_tables:ipt_do_table+0x4d6/0x592
 [<ffffffff8827e102>] :nf_nat:nf_nat_protocol_register+0xd/0x4a
 [<ffffffff88286254>] :iptable_nat:nf_nat_rule_find+0x17/0x57
 [<ffffffff8828643c>] :iptable_nat:nf_nat_fn+0x190/0x1bc
 [<ffffffff882864c7>] :iptable_nat:nf_nat_in+0x23/0x9f
 [<ffffffff81209f3d>] nf_iterate+0x41/0x7d
 [<ffffffff8120f610>] ip_rcv_finish+0x0/0x30b
 [<ffffffff8120a0ea>] nf_hook_slow+0x5d/0xc0
 [<ffffffff8120f610>] ip_rcv_finish+0x0/0x30b
 [<ffffffff8120fe6c>] ip_rcv+0x25c/0x58d
 [<ffffffff811efc79>] netif_receive_skb+0x192/0x3ae
 [<ffffffff8102f1ed>] __update_rq_clock+0x1a/0xed
 [<ffffffff880fefe7>] :forcedeth:nv_napi_poll+0x544/0x6cd
 [<ffffffff811f214f>] net_rx_action+0xa8/0x1a3
 [<ffffffff8103c9ed>] __do_softirq+0x55/0xc3
 [<ffffffff8101d667>] ack_apic_level+0x10/0xd9
 [<ffffffff8100cd5c>] call_softirq+0x1c/0x28
 [<ffffffff8100de8d>] do_softirq+0x2c/0x85
 [<ffffffff8103c953>] irq_exit+0x3f/0x84
 [<ffffffff8100e149>] do_IRQ+0x13e/0x161
 [<ffffffff8100adba>] default_idle+0x0/0x3d
 [<ffffffff8100c0e1>] ret_from_intr+0x0/0xa
 <EOI>  [<ffffffff8101bdf7>] lapic_next_event+0x0/0xa
 [<ffffffff8100ade3>] default_idle+0x29/0x3d
 [<ffffffff8100ae8b>] cpu_idle+0x94/0xbc


Code: 48 f7 45 78 80 01 00 00 74 4c 48 c7 c7 e0 18 28 88 e8 87 f2 
RIP  [<ffffffff8827e069>] :nf_nat:nf_nat_move_storage+0x2f/0x8a
 RSP <ffff810001f7f9d0>
Kernel panic - not syncing: Aiee, killing interrupt handler!

Re: kernel crash in nf_nat_move_storage

[Thomas Woerner]

2.6.24 seems to be ok on i386 also, but on x86_64 it is crashing.

Thomas Woerner wrote:
> Hello Patrick,
> 
> after sucessfully testing 2.6.23.14-107.fc8 on my i386 test system, I 
> installed 2.6.23.14-107.fc8 on the x86_64 system. At first I was not 
> able to reproduce the problem, but after starting the ftp server 
> (vsftpd) and using 'echo "quit" | telnet test-x86_64 80' several times, 
> I got a backtrace again. Please have a look at the attachment.
> 
> Thanks,
> Thomas
> 
> Patrick McHardy wrote:
>> Thomas Woerner wrote:
>>> Hello,
>>>
>>> Using port forwarding from port 80 to 21 with nf_conntrack_ftp loaded 
>>> results in a kernel crash, when connecting to port 80 from a remote
>>> host. This seems to be a problem for kernels > 2.6.18 including 2.6.24.
>>>
>>> Steps to Reproduce:
>>>
>>> host1> iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j 
>>> DNAT --to :21
>>> host1> iptables -t filter -A INPUT -i eth0 -m state --state NEW -m 
>>> tcp -p tcp --dport 21 -j ACCEPT
>>> host1> modprobe ip_conntrack_ftp
>>> host2> telnet host1 80
>>>
>>> Attached is the kernel crash log for kernel 2.6.23.9-85.fc8PAE. I was 
>>> told that this kernel crash dump is incomplete, but it took several 
>>> attempts to get a log with more that 5 lines over serial console. The 
>>> kernel seems to die too fast.
>>
>>
>> This is already fixed in 2.6.23.10.
>> -
>> To unsubscribe from this list: send the line "unsubscribe 
>> netfilter-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 


-- 
Thomas Woerner
Software Engineer            Phone: +49-711-96437-310
Red Hat GmbH                 Fax  : +49-711-96437-111
Hauptstaetterstr. 58         Email: Thomas Woerner <twoerner@redhat.com>
D-70178 Stuttgart            Web  : http://www.redhat.de/

Re: kernel crash in nf_nat_move_storage

[Patrick McHardy]

Thomas Woerner wrote:
> 2.6.24 seems to be ok on i386 also, but on x86_64 it is crashing.
> 
> Thomas Woerner wrote:
>> Hello Patrick,
>>
>> after sucessfully testing 2.6.23.14-107.fc8 on my i386 test system, I 
>> installed 2.6.23.14-107.fc8 on the x86_64 system. At first I was not 
>> able to reproduce the problem, but after starting the ftp server 
>> (vsftpd) and using 'echo "quit" | telnet test-x86_64 80' several 
>> times, I got a backtrace again. Please have a look at the attachment.


Thats odd, I can't see a reason why it would crash on x86_64.
Could you add a few printks to nf_nat_move_storage to print
out the pointers?