Re: CLUSTERIP and UML crash, was [Re: --dport in uml crash]

[Oskar Andreasson <oan@frozentux.net>]

[-- Attachment #1.1: Type: text/plain, Size: 5129 bytes --]

Hi Patrick,

You seem to be right yes, regarding module loading/handling. I don't
dare say anything more, but a non-qualified guess is that it feels like
some kind of bug between how the UML kernel handles modules and how
iptables interact?

I've attached 3 more scenarios. 

1. just restarted, issuing 
iptables -t nat -A PREROUTING -j DNAT --to 192.168.10.2
lsmod
result: kernel panic from lsmod

2. just restarted, issuing
iptables -t nat -A POSTROUTING -j MASQUERADE
lsmod
result: kernel panic, exactly the same as first scenario (except for a
couple of addresses). 

3. just restarted, issuing
modprobe ipt_MASQUERADE
modprobe ip_tables
lsmod
iptables -t nat -A POSTROUTING -j MASQUERADE
lsmod
result: the last lsmod results in a kernel panic. 

All modules loaded with modprobe seems to work just fine, until iptables
touches them. 

I just tried starting an nfs kernel server also, which didn't seem to
work either, so it seems mostly like an UML bug. It crashed at lsmod as
well. 

Would you know which way is the best to get in touch with the
uml-people, and how to debug this? My experience with gdb is ...
somewhat limited. I guess now is as bad a time as any to finally sit
down with it :(. Suggestions on what to read are welcome.

On Tue, 2006-10-24 at 16:24 +0200, Patrick McHardy wrote:
> Oskar Andreasson wrote:
> > Hi Patrick,
> > 
> > I'm sorry for the delayed reply, I had to get some sleep. The original
> > problem seems to have been caused by compiling iptables-1.3.6 against
> > libc6 and then running against an old libc5. It works flawless now that
> > they are both the correct version.
> > 
> > However, I got another kernel panic from the CLUSTERIP target
> > (possibly?). If not, please let me know where to get this info to:)
> > 
> > server1:~# iptables -A INPUT -d 192.168.0.5 -p tcp --dport 4444 -j
> > CLUSTERIP --new --hashmode sourceip --clustermac 01:00:00:00:00:20
> > --total-nodes 1 --local-node 1
> > ip_tables: (C) 2000-2006 Netfilter Core Team
> > Kernel panic - not syncing: Kernel mode fault at addr 0xa0be74, ip
> > 0x400d8cbe
> >  
> > EIP: 0073:[<400d8cbe>] CPU: 0 Not tainted ESP: 007b:bfb31118 EFLAGS:
> > 00200246
> >     Not tainted
> > EAX: ffffffda EBX: 00000000 ECX: 40019000 EDX: 00000400
> > ESI: 08050ef0 EDI: 4014e6c0 EBP: bfb3112c DS: 007b ES: 007b
> > a0bc3728:  [<a0045cf8>] notifier_call_chain+0x28/0x50
> > a0bc3744:  [<a0034620>] panic+0x50/0x100
> > a0bc375c:  [<a0013433>] segv+0x203/0x2d0
> > a0bc3804:  [<a00131b2>] segv_handler+0x92/0x110
> > a0bc3828:  [<a0013120>] segv_handler+0x0/0x110
> > a0bc382c:  [<a002a328>] sig_handler_common_skas+0xa8/0xe0
> > a0bc3854:  [<a002619a>] sig_handler+0x4a/0x60
> > a0bc38ac:  [<a012ac26>] vsnprintf+0x396/0x590
> > a0bc38dc:  [<a00172a0>] maybe_map+0x70/0xb0
> > a0bc38e4:  [<a0017259>] maybe_map+0x29/0xb0
> > a0bc3908:  [<a00176c0>] copy_chunk_to_user+0x0/0x30
> > a0bc390c:  [<a00172fe>] do_op_one_page+0x1e/0x60
> > a0bc3924:  [<a0017496>] do_buffer_op+0x156/0x1b0
> > a0bc3934:  [<a00176c0>] copy_chunk_to_user+0x0/0x30
> > a0bc3948:  [<a00260e5>] set_signals+0x25/0x30
> > a0bc3954:  [<a007596e>] kmem_cache_alloc+0x2e/0x50
> > a0bc3968:  [<a00abe11>] proc_alloc_inode+0x41/0x80
> > a0bc3984:  [<a0093615>] get_new_inode_fast+0x25/0xe0
> > a0bc39c0:  [<a00abf9d>] proc_get_inode+0xfd/0x180
> > a0bc39c8:  [<a00923eb>] d_rehash+0x3b/0x40
> > a0bc39d8:  [<a00aed07>] proc_lookup+0x97/0xa0
> > a0bc3a00:  [<a0017259>] maybe_map+0x29/0xb0
> > a0bc3a24:  [<a00176c0>] copy_chunk_to_user+0x0/0x30
> > a0bc3a28:  [<a00172fe>] do_op_one_page+0x1e/0x60
> > a0bc3a40:  [<a00173d2>] do_buffer_op+0x92/0x1b0
> > a0bc3a50:  [<a00176c0>] copy_chunk_to_user+0x0/0x30
> > a0bc3a5c:  [<a00176c0>] copy_chunk_to_user+0x0/0x30
> > a0bc3a7c:  [<a0027a64>] setjmp_wrapper+0x34/0x40
> > a0bc3a9c:  [<a0027a48>] setjmp_wrapper+0x18/0x40
> > a0bc3aac:  [<a0087954>] link_path_walk+0x64/0xe0
> > a0bc3ac8:  [<a001791d>] strncpy_from_user_skas+0x9d/0x120
> > a0bc3ad8:  [<a0017820>] strncpy_chunk_from_user+0x0/0x60
> > a0bc3b28:  [<a012abe1>] vsnprintf+0x351/0x590
> > a0bc3b74:  [<a009a5f8>] seq_printf+0x28/0x50
> > a0bc3b90:  [<a0055172>] print_unload_info+0x52/0xd0
> > a0bc3bb0:  [<a0057321>] m_show+0x31/0xa0
> > a0bc3bd8:  [<a0099f23>] seq_read+0xc3/0x340
> > a0bc3c0c:  [<a00789d2>] vfs_read+0xf2/0x1e0
> > a0bc3c38:  [<a0078e18>] sys_read+0x38/0x80
> > a0bc3c60:  [<a0016f07>] handle_syscall+0xf7/0x1d0
> > a0bc3c7c:  [<a0078de0>] sys_read+0x0/0x80
> > a0bc3cb4:  [<a002a168>] userspace+0x1c8/0x2e0
> > a0bc3cec:  [<a0048d10>] ____call_usermodehelper+0x0/0xc0
> > a0bc3cfc:  [<a0048d10>] ____call_usermodehelper+0x0/0xc0
> > a0bc3d04:  [<a0016a5a>] new_thread_handler+0x9a/0xb0
> > a0bc3d48:  [<a00169c0>] new_thread_handler+0x0/0xb0
> > a0bc3d5c:  [<a01e4c41>] kill+0x11/0x20
> 
> This looks UML module-load related. The command works fine here
> (with -i interface, otherwise it complains about a missing device).
> Does it also happen with other auto-loaded modules or when
> manually loading ipt_CLUSTERIP?

[-- Attachment #1.2: uml-iptables-panics.txt --]
[-- Type: text/plain, Size: 8131 bytes --]

server1:~# iptables -t nat -A PREROUTING -j DNAT --to 192.168.10.2
ip_tables: (C) 2000-2006 Netfilter Core Team
Netfilter messages via NETLINK v0.30.
ip_conntrack version 2.4 (159 buckets, 1272 max) - 228 bytes per conntrack
server1:~# lsmod
Module                  Size  Used by
Kernel panic - not syncing: Kernel mode fault at addr 0xa08992, ip 0x400d8cbe
 
EIP: 0073:[<400d8cbe>] CPU: 0 Not tainted ESP: 007b:bf70cd38 EFLAGS: 00200246
     Not tainted
EAX: ffffffda EBX: 00000003 ECX: 4001a000 EDX: 00000400
ESI: 0804a008 EDI: 00000000 EBP: bf70cd4c DS: 007b ES: 007b
a104f728:  [<a0045cf8>] notifier_call_chain+0x28/0x50
a104f744:  [<a0034620>] panic+0x50/0x100
a104f75c:  [<a0013433>] segv+0x203/0x2d0
a104f780:  [<a0029382>] map+0x112/0x160
a104f804:  [<a00131b2>] segv_handler+0x92/0x110
a104f828:  [<a0013120>] segv_handler+0x0/0x110
a104f82c:  [<a002a328>] sig_handler_common_skas+0xa8/0xe0
a104f854:  [<a002619a>] sig_handler+0x4a/0x60
a104f8ac:  [<a012ac26>] vsnprintf+0x396/0x590
a104f8e0:  [<a0017259>] maybe_map+0x29/0xb0
a104f904:  [<a0029462>] wait_stub_done+0x52/0x150
a104f920:  [<a00173d2>] do_buffer_op+0x92/0x1b0
a104f930:  [<a00176c0>] copy_chunk_to_user+0x0/0x30
a104f944:  [<a00260e5>] set_signals+0x25/0x30
a104f960:  [<a0029462>] wait_stub_done+0x52/0x150
a104f980:  [<a0093615>] get_new_inode_fast+0x25/0xe0
a104f9ac:  [<a002c614>] ptrace_setregs+0x14/0x30
a104f9c8:  [<a0028e84>] run_syscall_stub+0xd4/0x2d0
a104f9fc:  [<a0017259>] maybe_map+0x29/0xb0
a104fa20:  [<a00176c0>] copy_chunk_to_user+0x0/0x30
a104fa24:  [<a00172fe>] do_op_one_page+0x1e/0x60
a104fa3c:  [<a00173d2>] do_buffer_op+0x92/0x1b0
a104fa4c:  [<a00176c0>] copy_chunk_to_user+0x0/0x30
a104fa58:  [<a00176c0>] copy_chunk_to_user+0x0/0x30
a104fa78:  [<a0027a64>] setjmp_wrapper+0x34/0x40
a104fa98:  [<a0027a48>] setjmp_wrapper+0x18/0x40
a104fabc:  [<a00170d8>] do_ops+0xf8/0x110
a104fb24:  [<a012abe1>] vsnprintf+0x351/0x590
a104fb70:  [<a009a5f8>] seq_printf+0x28/0x50
a104fb8c:  [<a0055172>] print_unload_info+0x52/0xd0
a104fbac:  [<a0057321>] m_show+0x31/0xa0
a104fbd4:  [<a0099f23>] seq_read+0xc3/0x340
a104fc08:  [<a00789d2>] vfs_read+0xf2/0x1e0
a104fc34:  [<a0078e18>] sys_read+0x38/0x80
a104fc5c:  [<a0016f07>] handle_syscall+0xf7/0x1d0
a104fc78:  [<a0078de0>] sys_read+0x0/0x80
a104fcb0:  [<a002a168>] userspace+0x1c8/0x2e0
a104fd00:  [<a0016b31>] fork_handler+0x91/0xa0
a104fd48:  [<a0016aa0>] fork_handler+0x0/0xa0
a104fd5c:  [<a01e4c41>] kill+0x11/0x20

      
blueflux@laptop4:~/netkit2/kernel$  

---------------

server1:~# iptables -t nat -A POSTROUTING -j MASQUERADE
ip_tables: (C) 2000-2006 Netfilter Core Team
Netfilter messages via NETLINK v0.30.
ip_conntrack version 2.4 (159 buckets, 1272 max) - 228 bytes per conntrack
server1:~# lsmod
Module                  Size  Used by
Kernel panic - not syncing: Kernel mode fault at addr 0xa175f4, ip 0x400d8cbe
 
EIP: 0073:[<400d8cbe>] CPU: 0 Not tainted ESP: 007b:bf7d7608 EFLAGS: 00200246
     Not tainted
EAX: ffffffda EBX: 00000003 ECX: 4001a000 EDX: 00000400
ESI: 0804a008 EDI: 00000000 EBP: bf7d761c DS: 007b ES: 007b
a1707728:  [<a0045cf8>] notifier_call_chain+0x28/0x50
a1707744:  [<a0034620>] panic+0x50/0x100
a170775c:  [<a0013433>] segv+0x203/0x2d0
a1707780:  [<a0029382>] map+0x112/0x160
a1707804:  [<a00131b2>] segv_handler+0x92/0x110
a1707828:  [<a0013120>] segv_handler+0x0/0x110
a170782c:  [<a002a328>] sig_handler_common_skas+0xa8/0xe0
a1707854:  [<a002619a>] sig_handler+0x4a/0x60
a17078ac:  [<a012ac26>] vsnprintf+0x396/0x590
a17078e0:  [<a0017259>] maybe_map+0x29/0xb0
a1707904:  [<a0029462>] wait_stub_done+0x52/0x150
a1707920:  [<a0017438>] do_buffer_op+0xf8/0x1b0
a1707930:  [<a00176c0>] copy_chunk_to_user+0x0/0x30
a1707944:  [<a00260e5>] set_signals+0x25/0x30
a1707960:  [<a0029462>] wait_stub_done+0x52/0x150
a1707980:  [<a0093615>] get_new_inode_fast+0x25/0xe0
a17079ac:  [<a002c614>] ptrace_setregs+0x14/0x30
a17079c8:  [<a0028e84>] run_syscall_stub+0xd4/0x2d0
a17079fc:  [<a0017259>] maybe_map+0x29/0xb0
a1707a20:  [<a00176c0>] copy_chunk_to_user+0x0/0x30
a1707a24:  [<a00172fe>] do_op_one_page+0x1e/0x60
a1707a3c:  [<a00173d2>] do_buffer_op+0x92/0x1b0
a1707a4c:  [<a00176c0>] copy_chunk_to_user+0x0/0x30
a1707a58:  [<a00176c0>] copy_chunk_to_user+0x0/0x30
a1707a78:  [<a0027a64>] setjmp_wrapper+0x34/0x40
a1707a98:  [<a0027a48>] setjmp_wrapper+0x18/0x40
a1707abc:  [<a00170d8>] do_ops+0xf8/0x110
a1707b24:  [<a012abe1>] vsnprintf+0x351/0x590
a1707b70:  [<a009a5f8>] seq_printf+0x28/0x50
a1707b8c:  [<a0055172>] print_unload_info+0x52/0xd0
a1707bac:  [<a0057321>] m_show+0x31/0xa0
a1707bd4:  [<a0099f23>] seq_read+0xc3/0x340
a1707c08:  [<a00789d2>] vfs_read+0xf2/0x1e0
a1707c34:  [<a0078e18>] sys_read+0x38/0x80
a1707c5c:  [<a0016f07>] handle_syscall+0xf7/0x1d0
a1707c78:  [<a0078de0>] sys_read+0x0/0x80
a1707cb0:  [<a002a168>] userspace+0x1c8/0x2e0
a1707d00:  [<a0016b31>] fork_handler+0x91/0xa0
a1707d48:  [<a0016aa0>] fork_handler+0x0/0xa0
a1707d5c:  [<a01e4c41>] kill+0x11/0x20

--------------------

server1:~# modprobe ipt_MASQUERADE
Netfilter messages via NETLINK v0.30.
ip_conntrack version 2.4 (159 buckets, 1272 max) - 228 bytes per conntrack
server1:~# modprobe ip_tables
ip_tables: (C) 2000-2006 Netfilter Core Team
server1:~# lsmod                                       
Module                  Size  Used by
ip_tables              12056  0 [permanent]
ipt_MASQUERADE          2528  0 [permanent]
ip_nat                 15372  1 ipt_MASQUERADE,[permanent]
x_tables               10820  2 ip_tables,ipt_MASQUERADE,[permanent]
ip_conntrack           45916  2 ipt_MASQUERADE,ip_nat,[permanent]
nfnetlink               5368  2 ip_nat,ip_conntrack,[permanent]
server1:~# iptables -t nat -A POSTROUTING -j MASQUERADE
server1:~# lsmod
Module                  Size  Used by
Kernel panic - not syncing: Kernel mode fault at addr 0xa08d35, ip 0x400d8cbe
 
EIP: 0073:[<400d8cbe>] CPU: 0 Not tainted ESP: 007b:bf7ab5d8 EFLAGS: 00200246
    Not tainted
EAX: ffffffda EBX: 00000003 ECX: 4001a000 EDX: 00000400
ESI: 0804a008 EDI: 00000000 EBP: bf7ab5ec DS: 007b ES: 007b
a1597728:  [<a0045cf8>] notifier_call_chain+0x28/0x50
a1597744:  [<a0034620>] panic+0x50/0x100
a159775c:  [<a0013433>] segv+0x203/0x2d0
a1597784:  [<a0028e84>] run_syscall_stub+0xd4/0x2d0
a1597804:  [<a00131b2>] segv_handler+0x92/0x110
a1597828:  [<a0013120>] segv_handler+0x0/0x110
a159782c:  [<a002a328>] sig_handler_common_skas+0xa8/0xe0
a1597854:  [<a002619a>] sig_handler+0x4a/0x60
a15978ac:  [<a012ac26>] vsnprintf+0x396/0x590
a15978e0:  [<a0017259>] maybe_map+0x29/0xb0
a1597904:  [<a0029462>] wait_stub_done+0x52/0x150
a1597920:  [<a0017438>] do_buffer_op+0xf8/0x1b0
a1597930:  [<a00176c0>] copy_chunk_to_user+0x0/0x30
a1597944:  [<a00260e5>] set_signals+0x25/0x30
a1597960:  [<a0029462>] wait_stub_done+0x52/0x150
a1597980:  [<a0093615>] get_new_inode_fast+0x25/0xe0
a15979ac:  [<a002c614>] ptrace_setregs+0x14/0x30
a15979c8:  [<a0028e84>] run_syscall_stub+0xd4/0x2d0
a15979fc:  [<a0017259>] maybe_map+0x29/0xb0
a1597a20:  [<a00176c0>] copy_chunk_to_user+0x0/0x30
a1597a24:  [<a00172fe>] do_op_one_page+0x1e/0x60
a1597a3c:  [<a00173d2>] do_buffer_op+0x92/0x1b0
a1597a4c:  [<a00176c0>] copy_chunk_to_user+0x0/0x30
a1597a58:  [<a00176c0>] copy_chunk_to_user+0x0/0x30
a1597a78:  [<a0027a64>] setjmp_wrapper+0x34/0x40
a1597a98:  [<a0027a48>] setjmp_wrapper+0x18/0x40
a1597abc:  [<a00170d8>] do_ops+0xf8/0x110
a1597b24:  [<a012abe1>] vsnprintf+0x351/0x590
a1597b70:  [<a009a5f8>] seq_printf+0x28/0x50
a1597b8c:  [<a0055172>] print_unload_info+0x52/0xd0
a1597bac:  [<a0057321>] m_show+0x31/0xa0
a1597bd4:  [<a0099f23>] seq_read+0xc3/0x340
a1597c08:  [<a00789d2>] vfs_read+0xf2/0x1e0
a1597c34:  [<a0078e18>] sys_read+0x38/0x80
a1597c5c:  [<a0016f07>] handle_syscall+0xf7/0x1d0
a1597c78:  [<a0078de0>] sys_read+0x0/0x80
a1597cb0:  [<a002a168>] userspace+0x1c8/0x2e0
a1597d00:  [<a0016b31>] fork_handler+0x91/0xa0
a1597d48:  [<a0016aa0>] fork_handler+0x0/0xa0
a1597d5c:  [<a01e4c41>] kill+0x11/0x20


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 191 bytes --]