problem with conntrack utility and kernel 2.6.14

problem with conntrack utility and kernel 2.6.14

[Deti Fliegl]

Hi there,

Reading /proc/net/ip_conntrack seems to lock the table as long as being 
read which causes delays and loss in network traffic. Now I'm trying to 
use the conntrack utility from the subversion repository to list the 
conntrack table. This in turn prints out some "Unknown Attribute 5" 
lines and what's even worse it runs very often into a segmentation fault at

recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000008}, 
msg_iov(1)=[{"\260\0\0\0\0\1\0\6\0\0\0\0\0\0\0\0\2\0\0\0004\0\2\200\24"..., 
8192}], msg_controllen=0, msg_flags=0}, 0) = 176
write(2, "nfnl_parse_attr: deficit (4) len"..., 39nfnl_parse_attr: 
deficit (4) len (0).

) = 39

Maybe I'm wrong but it seems to happen due to a race condition when 
conntracking entries are being updated by the kernel... (if you like to 
reproduce this: just keep a high bandwidth connection open where byte 
counters in the conntrack tavle have to be updated very often...)


Deti

Re: problem with conntrack utility and kernel 2.6.14

[Pablo Neira]

Deti Fliegl wrote:
> Reading /proc/net/ip_conntrack seems to lock the table as long as being 
> read which causes delays and loss in network traffic. Now I'm trying to 
> use the conntrack utility from the subversion repository to list the 
> conntrack table. This in turn prints out some "Unknown Attribute 5" 
> lines and what's even worse it runs very often into a segmentation fault at

kernel version? userspace tools and libraries versions?

-- 
Pablo

Re: problem with conntrack utility and kernel 2.6.14

[Deti Fliegl]

Pablo Neira wrote:
> Deti Fliegl wrote:
> 
>> Reading /proc/net/ip_conntrack seems to lock the table as long as 
>> being read which causes delays and loss in network traffic. Now I'm 
>> trying to use the conntrack utility from the subversion repository to 
>> list the conntrack table. This in turn prints out some "Unknown 
>> Attribute 5" lines and what's even worse it runs very often into a 
>> segmentation fault at
> 
> 
> kernel version? userspace tools and libraries versions?
see above: kernel 2.6.14 - tools from svn @ now-30min

Re: problem with conntrack utility and kernel 2.6.14

[Pablo Neira]

Deti Fliegl wrote:
> Reading /proc/net/ip_conntrack seems to lock the table as long as being 
> read which causes delays and loss in network traffic. Now I'm trying to 
> use the conntrack utility from the subversion repository to list the 
> conntrack table. This in turn prints out some "Unknown Attribute 5" 
> lines and what's even worse it runs very often into a segmentation fault at
> 
> recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000008}, 
> msg_iov(1)=[{"\260\0\0\0\0\1\0\6\0\0\0\0\0\0\0\0\2\0\0\0004\0\2\200\24"..., 
> 8192}], msg_controllen=0, msg_flags=0}, 0) = 176
> write(2, "nfnl_parse_attr: deficit (4) len"..., 39nfnl_parse_attr: 
> deficit (4) len (0).
> 
> ) = 39

This problem was already fixed days ago in libnetfilter_conntrack on Oct 
17, see SVN. I'm not able to reproduce what you're reporting. Please 
send me a gdb backtrace, together with other extra info. Are you running 
conntrack on a x86?

> Maybe I'm wrong but it seems to happen due to a race condition when 
> conntracking entries are being updated by the kernel... (if you like to 
> reproduce this: just keep a high bandwidth connection open where byte 
> counters in the conntrack tavle have to be updated very often...)

No, that doesn't make too much sense to me.

-- 
Pablo

Re: problem with conntrack utility and kernel 2.6.14

[Deti Fliegl]

Pablo Neira wrote:
> This problem was already fixed days ago in libnetfilter_conntrack on Oct 
> 17, see SVN. I'm not able to reproduce what you're reporting. Please 
What I'm doing is to request a file from a local webserver in an endless 
loop.

> send me a gdb backtrace, together with other extra info. Are you running 
> conntrack on a x86?
It's a x64_64 system.

Backtrace from gdb:

tcp      6 57 TIME_WAIT src=127.0.0.1 dst=127.0.0.1 sport=47582 dport=80 
packets=6 bytes=427 src=127.0.0.1 dst=127.0.0.1 sport=80 dport=47582 
packets=4 bytes=677 [ASSURED] mark=0 use=1
tcp      6 42 TIME_WAIT src=127.0.0.1 dst=127.0.0.1 sport=44717 dport=80 
packets=6 bytes=427 src=127.0.0.1 dst=127.0.0.1 sport=80 dport=44717 
packets=4 bytes=677 [ASSURED] mark=0 use=1
tcp      6 52 TIME_WAIT src=127.0.0.1 dst=127.0.0.1 sport=46652 dport=80 
packets=6 bytes=427 src=127.0.0.1 dst=127.0.0.1 sport=80 dport=46652 
packets=4 bytes=677 [ASSURED] mark=0 use=1
tcp      6 60 TIME_WAIT src=127.0.0.1 dst=127.0.0.1 sport=48037 dport=80 
packets=6 bytes=427 src=127.0.0.1 dst=127.0.0.1 sport=80 dport=48037 
packets=4 bytes=677 [ASSURED] mark=0 use=1
tcp      6 36 TIME_WAIT src=127.0.0.1 dst=127.0.0.1 sport=43471 dport=80 
packets=6 bytes=427 src=127.0.0.1 dst=127.0.0.1 sport=80 dport=43471 
packets=4 bytes=677 [ASSURED] mark=0 use=1
nfnl_parse_attr: deficit (4) len (0).


Program received signal SIGSEGV, Segmentation fault.
0x00002aaaab0f6aa5 in parse_protoinfo () from 
/usr/local/lib//libnetfilter_conntrack_tcp.so
(gdb) bt
#0  0x00002aaaab0f6aa5 in parse_protoinfo () from 
/usr/local/lib//libnetfilter_conntrack_tcp.so
#1  0x00002aaaaabc41dc in nfct_conntrack_netlink_handler (cth=0x586010, 
nlh=0x7fffff803760, arg=<value optimized out>) at 
libnetfilter_conntrack.c:376
#2  0x00002aaaaabc37f0 in callback_handler (nladdr=<value optimized 
out>, n=0x7fffff803608, arg=<value optimized out>) at 
libnetfilter_conntrack.c:57
#3  0x00002aaaaacc8702 in nfnl_listen (nfnlh=0x586010, 
handler=0x2aaaaabc37c0 <callback_handler>, jarg=0x586010) at 
libnfnetlink.c:349
#4  0x00002aaaaabc4f8c in __nfct_dump_conntrack_table (cth=0x586010, 
zero=<value optimized out>) at libnetfilter_conntrack.c:851
#5  0x0000000000403242 in main (argc=2, argv=0x7fffff805a48) at 
conntrack.c:913
(gdb)


Subversion Revision is 4394

All libn* directories were built using these commands:

aclocal && libtoolize -f && automake -a && autoconf (as my auto-tools 
are not named auto*-1.6 like autogen.sh wants it)

./configure
make clean all install

Files are from
2005-10-28 10:14 /usr/local/lib/libnetfilter_conntrack_icmp.so.0.0.0
2005-10-28 10:14 /usr/local/lib/libnetfilter_conntrack_sctp.so.0.0.0
2005-10-28 10:14 /usr/local/lib/libnetfilter_conntrack.so.0.0.0
2005-10-28 10:14 /usr/local/lib/libnetfilter_conntrack_tcp.so.0.0.0
2005-10-28 10:14 /usr/local/lib/libnetfilter_conntrack_udp.so.0.0.0
2005-10-28 10:15 /usr/local/lib/libnetfilter_log_libipulog.so.0.0.0
2005-10-28 10:15 /usr/local/lib/libnetfilter_log.so.0.0.0
2005-10-28 10:16 /usr/local/lib/libnetfilter_queue_libipq.so.0.0.0
2005-10-28 10:16 /usr/local/lib/libnetfilter_queue.so.0.0.0
2005-10-28 10:13 /usr/local/lib/libnfnetlink.so.0.0.0

Deti

Re: problem with conntrack utility and kernel 2.6.14

[Pablo Neira]

[-- Attachment #1: Type: text/plain, Size: 638 bytes --]

Hi,

Deti Fliegl wrote:
> Pablo Neira wrote:
> 
>> This problem was already fixed days ago in libnetfilter_conntrack on 
>> Oct 17, see SVN. I'm not able to reproduce what you're reporting. Please 
> 
> What I'm doing is to request a file from a local webserver in an endless 
> loop.
> 
>> send me a gdb backtrace, together with other extra info. Are you 
>> running conntrack on a x86?
> 
> It's a x64_64 system.
> 
> Backtrace from gdb:

Thanks for the very detailed report. The patch fixes some aligment 
issues that I didn't handle properly :(. Please, give a try to the patch 
attached and tell if it fixes your problem.

-- 
Pablo

[-- Attachment #2: x --]
[-- Type: text/plain, Size: 1431 bytes --]

Index: src/libnetfilter_conntrack.c
===================================================================
--- src/libnetfilter_conntrack.c	(revision 4394)
+++ src/libnetfilter_conntrack.c	(working copy)
@@ -424,9 +424,8 @@
 					  struct nlmsghdr *nlh, void *arg)
 {
 	struct nfgenmsg *nfmsg;
-	int min_len = sizeof(struct nfgenmsg) + sizeof(struct nlmsghdr);
 	struct nfattr *attr = NFM_NFA(NLMSG_DATA(nlh));
-	int attrlen = nlh->nlmsg_len - NLMSG_ALIGN(min_len);
+	int attrlen = NLMSG_LENGTH(nlh->nlmsg_len) - NFNL_HEADER_LEN;
 	struct nfct_conntrack ct;
 	unsigned int flags = 0;
 	int type = NFNL_MSG_TYPE(nlh->nlmsg_type), ret = 0;
@@ -435,7 +434,7 @@
 
 	nfmsg = NLMSG_DATA(nlh);
 
-	if (nlh->nlmsg_len < min_len)
+	if (NLMSG_LENGTH(nlh->nlmsg_len) < NFNL_HEADER_LEN)
 		return -EINVAL;
 
 	while (NFA_OK(attr, attrlen)) {
@@ -666,9 +665,8 @@
 				       struct nlmsghdr *nlh, void *arg)
 {
 	struct nfgenmsg *nfmsg;
-	int min_len = sizeof(struct nfgenmsg) + sizeof(struct nlmsghdr);
 	struct nfattr *attr = NFM_NFA(NLMSG_DATA(nlh));
-	int attrlen = nlh->nlmsg_len - NLMSG_ALIGN(min_len);
+	int attrlen = NLMSG_LENGTH(nlh->nlmsg_len) - NFNL_HEADER_LEN;
 	struct nfct_expect exp;
 	int type = NFNL_MSG_TYPE(nlh->nlmsg_type), ret = 0;
 
@@ -676,7 +674,7 @@
 
 	nfmsg = NLMSG_DATA(nlh);
 
-	if (nlh->nlmsg_len < min_len)
+	if (NLMSG_LENGTH(nlh->nlmsg_len) < NFNL_HEADER_LEN)
 		return -EINVAL;
 
 	while (NFA_OK(attr, attrlen)) {

Re: problem with conntrack utility and kernel 2.6.14

[Deti Fliegl]

Pablo Neira wrote:
> Thanks for the very detailed report. The patch fixes some aligment 
> issues that I didn't handle properly :(. Please, give a try to the patch 
> attached and tell if it fixes your problem.
Thanks for the patch (which seemed to be already checked into svn) - 
conntrack now works properly - no seg faults occured :)

The kernel oops still happens - but this time I had access to the 
console (hope that helps):

<4>Badness in __kfree_skb at net/core/skbuff.c:330
<4>
<4>Call Trace:<ffffffff802a2f77>{__kfree_skb+167} 
<ffffffff802bb597>{netlink_recvmsg+279}
<4>       <ffffffff8029cb2b>{sock_recvmsg+315} 
<ffffffff8025a4d8>{n_tty_receive_buf+4392}
<4>       <ffffffff8025a4d8>{n_tty_receive_buf+4392} 
<ffffffff8015d828>{filemap_nopage+424}
<4>       <ffffffff8013003b>{try_to_wake_up+1083} 
<ffffffff8014b2f0>{autoremove_wake_function+0}
<4>       <ffffffff8025bbd9>{pty_write+89} 
<ffffffff8029e38b>{sys_recvmsg+395}
<4>       <ffffffff80130f23>{__wake_up+67} <ffffffff80181208>{vfs_write+344}
<4>       <ffffffff80181343>{sys_write+83} 
<ffffffff8010dc4e>{system_call+126}
<4>
<3>scheduling while atomic: conntrack/0xffffff00/7887
<4>
<4>Call Trace:<ffffffff8031d67d>{schedule+125} 
<ffffffff80181208>{vfs_write+344}
<4>       <ffffffff80181343>{sys_write+83} 
<ffffffff8010dcb8>{sysret_careful+13}
<4>
<4>Badness in __kfree_skb at net/core/skbuff.c:330
<4>
<4>Call Trace:<ffffffff802a2f77>{__kfree_skb+167} 
<ffffffff802bb597>{netlink_recvmsg+279}
<4>       <ffffffff8029cb2b>{sock_recvmsg+315} 
<ffffffff8025a4d8>{n_tty_receive_buf+4392}
<4>       <ffffffff8025a4d8>{n_tty_receive_buf+4392} 
<ffffffff8013578d>{printk+141}
<4>       <ffffffff8014b2f0>{autoremove_wake_function+0} 
<ffffffff8025bbd9>{pty_write+89}
<4>       <ffffffff8029e38b>{sys_recvmsg+395} 
<ffffffff8018e477>{pipe_writev+1319}
<4>       <ffffffff80181208>{vfs_write+344} <ffffffff80181343>{sys_write+83}
<4>       <ffffffff8010dc4e>{system_call+126}
<4>Badness in __kfree_skb at net/core/skbuff.c:330
<4>
<4>Call Trace:<ffffffff802a2f77>{__kfree_skb+167} 
<ffffffff802bb597>{netlink_recvmsg+279}
<4>       <ffffffff8029cb2b>{sock_recvmsg+315} 
<ffffffff8025a4d8>{n_tty_receive_buf+4392}
<4>       <ffffffff8025a4d8>{n_tty_receive_buf+4392} 
<ffffffff8013578d>{printk+141}
<4>       <ffffffff8014b2f0>{autoremove_wake_function+0} 
<ffffffff8025bbd9>{pty_write+89}
<4>       <ffffffff8029e38b>{sys_recvmsg+395} 
<ffffffff8018e477>{pipe_writev+1319}
<4>       <ffffffff80181208>{vfs_write+344} <ffffffff80181343>{sys_write+83}
<4>       <ffffffff8010dc4e>{system_call+126}

... etc...

Re: problem with conntrack utility and kernel 2.6.14

[Pablo Neira]

[-- Attachment #1: Type: text/plain, Size: 413 bytes --]

Hi,

Deti Fliegl wrote:
> Thanks for the patch (which seemed to be already checked into svn) - 
> conntrack now works properly - no seg faults occured :)

Could you give a try to the patch attached and tell me if it fixes the 
problem as well?

> The kernel oops still happens - but this time I had access to the 
> console (hope that helps):

Thanks a lot for the report, I'll have a look at it asap.

-- 
Pablo

[-- Attachment #2: x --]
[-- Type: text/plain, Size: 1900 bytes --]

Index: libnfnetlink/include/libnfnetlink/libnfnetlink.h
===================================================================
--- libnfnetlink/include/libnfnetlink/libnfnetlink.h	(revision 4394)
+++ libnfnetlink/include/libnfnetlink/libnfnetlink.h	(working copy)
@@ -18,8 +21,8 @@
 #define NLMSG_TAIL(nlh) \
 	(((void *) (nlh)) + NLMSG_ALIGN((nlh)->nlmsg_len))
 
-#define NFNL_HEADER_LEN	(NLMSG_LENGTH(sizeof(struct nlmsghdr))	\
-			 +NLMSG_LENGTH(sizeof(struct nfgenmsg)))
+#define NFNL_HEADER_LEN	(NLMSG_ALIGN(sizeof(struct nlmsghdr))	\
+			 +NLMSG_ALIGN(sizeof(struct nfgenmsg)))
 
 #define NFNL_BUFFSIZE		8192
 
Index: libnetfilter_conntrack/src/libnetfilter_conntrack.c
===================================================================
--- libnetfilter_conntrack/src/libnetfilter_conntrack.c	(revision 4398)
+++ libnetfilter_conntrack/src/libnetfilter_conntrack.c	(working copy)
@@ -433,7 +433,7 @@
 {
 	struct nfgenmsg *nfmsg;
 	struct nfattr *attr = NFM_NFA(NLMSG_DATA(nlh));
-	int attrlen = NLMSG_LENGTH(nlh->nlmsg_len) - NFNL_HEADER_LEN;
+	int attrlen = nlh->nlmsg_len - NFNL_HEADER_LEN;
 	struct nfct_conntrack ct;
 	unsigned int flags = 0;
 	int type = NFNL_MSG_TYPE(nlh->nlmsg_type), ret = 0;
@@ -442,7 +442,7 @@
 
 	nfmsg = NLMSG_DATA(nlh);
 
-	if (NLMSG_LENGTH(nlh->nlmsg_len) < NFNL_HEADER_LEN)
+	if (nlh->nlmsg_len < NFNL_HEADER_LEN)
 		return -EINVAL;
 
 	while (NFA_OK(attr, attrlen)) {
@@ -674,7 +674,7 @@
 {
 	struct nfgenmsg *nfmsg;
 	struct nfattr *attr = NFM_NFA(NLMSG_DATA(nlh));
-	int attrlen = NLMSG_LENGTH(nlh->nlmsg_len) - NFNL_HEADER_LEN;
+	int attrlen = nlh->nlmsg_len - NFNL_HEADER_LEN;
 	struct nfct_expect exp;
 	int type = NFNL_MSG_TYPE(nlh->nlmsg_type), ret = 0;
 
@@ -682,7 +682,7 @@
 
 	nfmsg = NLMSG_DATA(nlh);
 
-	if (NLMSG_LENGTH(nlh->nlmsg_len) < NFNL_HEADER_LEN)
+	if (nlh->nlmsg_len < NFNL_HEADER_LEN)
 		return -EINVAL;
 
 	while (NFA_OK(attr, attrlen)) {

Re: problem with conntrack utility and kernel 2.6.14

[Deti Fliegl]

Pablo Neira wrote:
> Could you give a try to the patch attached and tell me if it fixes the 
> problem as well?
Well - Houston, We Have a Problem :(

nfnl_parse_attr: deficit (4) len (0).


Program received signal SIGSEGV, Segmentation fault.
0x00002aaaab0f6aa5 in parse_protoinfo () from 
/usr/local/lib//libnetfilter_conntrack_tcp.so
(gdb) bt
#0  0x00002aaaab0f6aa5 in parse_protoinfo () from 
/usr/local/lib//libnetfilter_conntrack_tcp.so
#1  0x00002aaaaabc41dc in nfct_conntrack_netlink_handler (cth=0x586010, 
nlh=0x7fffffc42d00, arg=<value optimized out>) at 
libnetfilter_conntrack.c:384
#2  0x00002aaaaabc37f0 in callback_handler (nladdr=<value optimized 
out>, n=0x7fffffc42b98, arg=<value optimized out>) at 
libnetfilter_conntrack.c:65
#3  0x00002aaaaacc8702 in nfnl_listen (nfnlh=0x586010, 
handler=0x2aaaaabc37c0 <callback_handler>, jarg=0x586010) at 
libnfnetlink.c:349
#4  0x00002aaaaabc4f8c in __nfct_dump_conntrack_table (cth=0x586010, 
zero=<value optimized out>) at libnetfilter_conntrack.c:857
#5  0x0000000000403242 in main (argc=2, argv=0x7fffffc45028) at 
conntrack.c:913

Re: problem with conntrack utility and kernel 2.6.14

[Pablo Neira]

[-- Attachment #1: Type: text/plain, Size: 633 bytes --]

Deti Fliegl wrote:
> Pablo Neira wrote:
> 
>> Could you give a try to the patch attached and tell me if it fixes the 
>> problem as well?
> 
> Well - Houston, We Have a Problem :(
> 
> nfnl_parse_attr: deficit (4) len (0).

Damn, I'm not able to reproduce this on my x86 box. I tried by stressing 
the conntrack tool with the same method that you've previously described 
with no success, so this must be kind of x86_64 alignment issue.

Please, could you give a try to the patch attached and tell me if it 
fixes the problem? I've reworked the whole netlink message parsing function.

BTW, thanks for the responsiveness.

-- 
Pablo

[-- Attachment #2: x --]
[-- Type: text/plain, Size: 3596 bytes --]

Index: src/libnetfilter_conntrack.c
===================================================================
--- src/libnetfilter_conntrack.c	(revision 4398)
+++ src/libnetfilter_conntrack.c	(working copy)
@@ -431,66 +431,71 @@
 static int nfct_conntrack_netlink_handler(struct nfct_handle *cth, 
 					  struct nlmsghdr *nlh, void *arg)
 {
-	struct nfgenmsg *nfmsg;
-	struct nfattr *attr = NFM_NFA(NLMSG_DATA(nlh));
-	int attrlen = NLMSG_LENGTH(nlh->nlmsg_len) - NFNL_HEADER_LEN;
 	struct nfct_conntrack ct;
 	unsigned int flags = 0;
+	struct nfgenmsg *nfhdr = NLMSG_DATA(nlh);
 	int type = NFNL_MSG_TYPE(nlh->nlmsg_type), ret = 0;
+	int len = nlh->nlmsg_len;
+	struct nfattr *cda[CTA_MAX];
 
+	len -= NLMSG_LENGTH(sizeof(struct nfgenmsg));
+	if (len < 0)
+		return -EINVAL;
+
 	memset(&ct, 0, sizeof(struct nfct_conntrack));
 
-	nfmsg = NLMSG_DATA(nlh);
+	nfnl_parse_attr(cda, CTA_MAX, NFA_DATA(nfhdr), len);
 
-	if (NLMSG_LENGTH(nlh->nlmsg_len) < NFNL_HEADER_LEN)
-		return -EINVAL;
+	if (cda[CTA_TUPLE_ORIG-1])
+		parse_tuple(cda[CTA_TUPLE_ORIG-1], 
+			    &ct.tuple[NFCT_DIR_ORIGINAL]);
+	
+	if (cda[CTA_TUPLE_REPLY-1])
+		parse_tuple(cda[CTA_TUPLE_REPLY-1], 
+			    &ct.tuple[NFCT_DIR_REPLY]);
+	
+	if (cda[CTA_STATUS-1]) {
+		ct.status = ntohl(*(u_int32_t *)NFA_DATA(cda[CTA_STATUS-1]));
+		flags |= NFCT_STATUS;
+	}
 
-	while (NFA_OK(attr, attrlen)) {
-		switch(NFA_TYPE(attr)) {
-		case CTA_TUPLE_ORIG:
-			parse_tuple(attr, &ct.tuple[NFCT_DIR_ORIGINAL]);
-			break;
-		case CTA_TUPLE_REPLY:
-			parse_tuple(attr, &ct.tuple[NFCT_DIR_REPLY]);
-			break;
-		case CTA_STATUS:
-			ct.status = ntohl(*(u_int32_t *)NFA_DATA(attr));
-			flags |= NFCT_STATUS;
-			break;
-		case CTA_PROTOINFO:
-			parse_protoinfo(attr, &ct);
-			flags |= NFCT_PROTOINFO;
-			break;
-		case CTA_TIMEOUT:
-			ct.timeout = ntohl(*(u_int32_t *)NFA_DATA(attr));
-			flags |= NFCT_TIMEOUT;
-			break;
-		case CTA_MARK:
-			ct.mark = ntohl(*(u_int32_t *)NFA_DATA(attr));
-			flags |= NFCT_MARK;
-			break;
-		case CTA_COUNTERS_ORIG:
-			nfct_parse_counters(attr, &ct, NFA_TYPE(attr)-1);
-			flags |= NFCT_COUNTERS_ORIG;
-			break;
-		case CTA_COUNTERS_REPLY:
-			nfct_parse_counters(attr, &ct, NFA_TYPE(attr)-1);
-			flags |= NFCT_COUNTERS_RPLY;
-			break;
-		case CTA_USE:
-			ct.use = ntohl(*(u_int32_t *)NFA_DATA(attr));
-			flags |= NFCT_USE;
-			break;
-		case CTA_ID:
-			ct.id = ntohl(*(u_int32_t *)NFA_DATA(attr));
-			flags |= NFCT_ID;
-			break;
-		default:
-			fprintf(stderr, "Unknown Attribute %d\n", NFA_TYPE(attr));
-			break;
-		}
-		attr = NFA_NEXT(attr, attrlen);
+	if (cda[CTA_PROTOINFO-1]) {
+		parse_protoinfo(cda[CTA_PROTOINFO-1], &ct);
+		flags |= NFCT_PROTOINFO;
 	}
+
+	if (cda[CTA_TIMEOUT-1]) {
+		ct.timeout = ntohl(*(u_int32_t *)NFA_DATA(cda[CTA_TIMEOUT-1]));
+		flags |= NFCT_TIMEOUT;
+	}
+	
+	if (cda[CTA_MARK-1]) {
+		ct.mark = ntohl(*(u_int32_t *)NFA_DATA(cda[CTA_MARK-1]));
+		flags |= NFCT_MARK;
+	}
+	
+	if (cda[CTA_COUNTERS_ORIG-1]) {
+		nfct_parse_counters(cda[CTA_COUNTERS_ORIG-1], &ct, 
+				    NFA_TYPE(cda[CTA_COUNTERS_ORIG-1])-1);
+		flags |= NFCT_COUNTERS_ORIG;
+	}
+
+	if (cda[CTA_COUNTERS_REPLY-1]) {
+		nfct_parse_counters(cda[CTA_COUNTERS_REPLY-1], &ct, 
+				    NFA_TYPE(cda[CTA_COUNTERS_REPLY-1])-1);
+		flags |= NFCT_COUNTERS_RPLY;
+	}
+
+	if (cda[CTA_USE-1]) {
+		ct.use = ntohl(*(u_int32_t *)NFA_DATA(cda[CTA_USE-1]));
+		flags |= NFCT_USE;
+	}
+
+	if (cda[CTA_ID-1]) {
+		ct.id = ntohl(*(u_int32_t *)NFA_DATA(cda[CTA_ID-1]));
+		flags |= NFCT_ID;
+	}
+
 	if (cth->callback)
 		ret = cth->callback((void *) &ct, flags,
 				    typemsg2enum(type, nlh->nlmsg_flags));

Re: problem with conntrack utility and kernel 2.6.14

[Deti Fliegl]

Hi Pablo,

the good thing about your latest patch is that the conntrack tool no 
longer crashes the kernel same test conditions as yesterday: 800Mbit/s 
Bandwidth, 25000 conntrack entries, @1000 connections/s).
Very weird...

Deti

Re: problem with conntrack utility and kernel 2.6.14

[Pablo Neira]

Hi,

JFYI, I just commited the following changes to libconntrack_netfilter:

o Replace misleading flag NFCT_ANY_GROUP by NFCT_ALL_GROUPS
o Update test file to use NFCT_ALL_GROUPS
o Add missing check of CTA_PROTOINFO_TCP that resulted in a segfault in
conjuction with events
o Fix ICMP conntracks output
o Add missing prototype definition of nfct_default_expect_display_id in
libnetfilter_conntrack.h

and the following to conntrack:

o Replace misleading message "Not enough memory" by "Can't open handler"
o New option -i for expectation dumping: conntrack -L expect [-i]
o sed 's/VERSION/CONNTRACK_VERSION/g'
o Fix nfct_open flags, now uses NFCT_ALL_GROUPS when needed
o Bumped version to 0.94

AFAIK these fix the problems concerned with the dumping. The problem 
wasn't 64 bits arch specific at all, actually the matter was that I 
required some fast hardware to reproduce them that I don't have. Thanks 
to Deti Fiegl for providing me such fast hardware via SSH to reproduce 
the bugs ;)

I've been giving some testing to the conntrack tool and seems to work 
fine under big stress situation. If you spot any other bug, please let 
me know. I'm willing to do the first 1.0 official as soon as people 
don't complain about bugs.

-- 
Pablo

Re: problem with conntrack utility and kernel 2.6.14

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3356 bytes --]



On Mon, 31 Oct 2005, Pablo Neira wrote:

<CIACH>
> I've been giving some testing to the conntrack tool and seems to work fine 
> under big stress situation. If you spot any other bug, please let me know. 
> I'm willing to do the first 1.0 official as soon as people don't complain 
> about bugs.

1. "Illegal option `-m'" with "conntrack -E -i"
# conntrack -E -i
conntrack v0.94: Illegal option `-m' with this command

2. Unable to delete conntrack by id:
root@olemx:~# conntrack -L -i|grep id=101
tcp      6 431999 ESTABLISHED src=192.168.0.22 dst=192.168.0.33 sport=1607 dport=22 packets=72520 bytes=4421477 src=192.168.0.33 dst=192.168.0.22 sport=22 dport=1607 packets=101332 bytes=21675629 [ASSURED] mark=0 use=1 id=101
root@olemx:~# conntrack -D -i 101
root@olemx:~# conntrack -L -i|grep id=101
tcp      6 431999 ESTABLISHED src=192.168.0.22 dst=192.168.0.33 sport=1607 dport=22 packets=72549 bytes=4423573 src=192.168.0.33 dst=192.168.0.22 sport=22 dport=1607 packets=101352 bytes=21677725 [ASSURED] mark=0 use=1 id=101

3. "deficit (4) len (0)." in conntrack -E expect:

# conntrack -E expect
0 proto=17 src=192.168.31.255 dst=192.168.1.29 sport=138 dport=138
nfnl_parse_attr: deficit (4) len (0).

0 proto=17 src=192.168.31.255 dst=192.168.1.227 sport=138 dport=138
nfnl_parse_attr: deficit (4) len (0).

4. Wrong formating in conntrack -h (Get... & Update...)

Commands:
-L [table] [-z]         List conntrack or expectation table
-G [table] parameters  Get conntrack or expectation
-D [table] parameters   Delete conntrack or expectation
-I [table] parameters   Create a conntrack or expectation
-U [table] parameters  Update a conntrack
-E [table] [options]    Show events
-F [table]              Flush table

Patch attached & inlined (for easy review):

Index: src/conntrack.c
===================================================================
--- src/conntrack.c	(revision 4397)
+++ src/conntrack.c	(working copy)
@@ -670,13 +670,13 @@
  fprintf(stdout, "Usage: %s [commands] [options]\n", prog);
  fprintf(stdout, "\n");
  fprintf(stdout, "Commands:\n");
-fprintf(stdout, "-L [table] [-z]   	List conntrack or expectation table\n");
-fprintf(stdout, "-G [table] parameters  Get conntrack or expectation\n");
-fprintf(stdout, "-D [table] parameters	Delete conntrack or expectation\n");
-fprintf(stdout, "-I [table] parameters	Create a conntrack or expectation\n");
-fprintf(stdout, "-U [table] parameters  Update a conntrack\n");
-fprintf(stdout, "-E [table] [options]	Show events\n");
-fprintf(stdout, "-F [table]	     	Flush table\n");
+fprintf(stdout, "-L [table] [-z]\t\tList conntrack or expectation table\n");
+fprintf(stdout, "-G [table] parameters\tGet conntrack or expectation\n");
+fprintf(stdout, "-D [table] parameters\tDelete conntrack or expectation\n");
+fprintf(stdout, "-I [table] parameters\tCreate a conntrack or expectation\n");
+fprintf(stdout, "-U [table] parameters\tUpdate a conntrack\n");
+fprintf(stdout, "-E [table] [options]\tShow events\n");
+fprintf(stdout, "-F [table]\t\tFlush table\n");
  fprintf(stdout, "\n");
  fprintf(stdout, "Options:\n");
  fprintf(stdout, "--orig-src ip	     	Source address from original direction\n");

5. Missing information in help/man about possibility of using "-i".

Best regards,

 			Krzysztof Olędzki

[-- Attachment #2: Type: TEXT/PLAIN, Size: 1428 bytes --]

Index: src/conntrack.c
===================================================================
--- src/conntrack.c	(revision 4397)
+++ src/conntrack.c	(working copy)
@@ -670,13 +670,13 @@
 fprintf(stdout, "Usage: %s [commands] [options]\n", prog);
 fprintf(stdout, "\n");
 fprintf(stdout, "Commands:\n");
-fprintf(stdout, "-L [table] [-z]   	List conntrack or expectation table\n");
-fprintf(stdout, "-G [table] parameters  Get conntrack or expectation\n");
-fprintf(stdout, "-D [table] parameters	Delete conntrack or expectation\n");
-fprintf(stdout, "-I [table] parameters	Create a conntrack or expectation\n");
-fprintf(stdout, "-U [table] parameters  Update a conntrack\n");
-fprintf(stdout, "-E [table] [options]	Show events\n");
-fprintf(stdout, "-F [table]	     	Flush table\n");
+fprintf(stdout, "-L [table] [-z]\t\tList conntrack or expectation table\n");
+fprintf(stdout, "-G [table] parameters\tGet conntrack or expectation\n");
+fprintf(stdout, "-D [table] parameters\tDelete conntrack or expectation\n");
+fprintf(stdout, "-I [table] parameters\tCreate a conntrack or expectation\n");
+fprintf(stdout, "-U [table] parameters\tUpdate a conntrack\n");
+fprintf(stdout, "-E [table] [options]\tShow events\n");
+fprintf(stdout, "-F [table]\t\tFlush table\n");
 fprintf(stdout, "\n");
 fprintf(stdout, "Options:\n");
 fprintf(stdout, "--orig-src ip	     	Source address from original direction\n");

Re: problem with conntrack utility and kernel 2.6.14

[Pablo Neira]

Krzysztof Oledzki wrote:
> 1. "Illegal option `-m'" with "conntrack -E -i"
> # conntrack -E -i
> conntrack v0.94: Illegal option `-m' with this command

Wrong error output: this should say `-i'. Fixed.

You can't use -E together with -i. But I think that adding the conntrack 
ID to the event information that is dumped could be worth for accounting 
purposes, so I'll add this to my pending patches for ctnetlink, ok?

> 2. Unable to delete conntrack by id:
> root@olemx:~# conntrack -L -i|grep id=101
> tcp      6 431999 ESTABLISHED src=192.168.0.22 dst=192.168.0.33 
> sport=1607 dport=22 packets=72520 bytes=4421477 src=192.168.0.33 
> dst=192.168.0.22 sport=22 dport=1607 packets=101332 bytes=21675629 
> [ASSURED] mark=0 use=1 id=101
> root@olemx:~# conntrack -D -i 101
> root@olemx:~# conntrack -L -i|grep id=101

You can't kill conntracks *just* by the ID. The connection tracking 
table currently uses the tuple information (source, destination, 
protocol information) to place the conntrack in hashes, same thing to 
perform lookups. Implementing the ability of killing conntracks just by 
its ID would be O(n), so we would need to walk through the buckets until 
we find a matching, not so good. Just a wild thought, how bad would be 
hashing the conntracks by its ID? In that case we could implement this 
feature. So, currently you'll always need the information about the 
source, destination and protocol specific stuff together with the ID.

> tcp      6 431999 ESTABLISHED src=192.168.0.22 dst=192.168.0.33 
> sport=1607 dport=22 packets=72549 bytes=4423573 src=192.168.0.33 
> dst=192.168.0.22 sport=22 dport=1607 packets=101352 bytes=21677725 
> [ASSURED] mark=0 use=1 id=101
> 
> 3. "deficit (4) len (0)." in conntrack -E expect:
> 
> # conntrack -E expect
> 0 proto=17 src=192.168.31.255 dst=192.168.1.29 sport=138 dport=138
> nfnl_parse_attr: deficit (4) len (0).
> 
> 0 proto=17 src=192.168.31.255 dst=192.168.1.227 sport=138 dport=138
> nfnl_parse_attr: deficit (4) len (0).

Fixed in SVN.

> 4. Wrong formating in conntrack -h (Get... & Update...)
> 
> Commands:
> -L [table] [-z]         List conntrack or expectation table
> -G [table] parameters  Get conntrack or expectation
> -D [table] parameters   Delete conntrack or expectation
> -I [table] parameters   Create a conntrack or expectation
> -U [table] parameters  Update a conntrack
> -E [table] [options]    Show events
> -F [table]              Flush table
> 
> Patch attached & inlined (for easy review):

Applied. Thanks.

> 5. Missing information in help/man about possibility of using "-i".

Added -i to the manpage. Thanks for the bug report.

-- 
Pablo

Re: problem with conntrack utility and kernel 2.6.14

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1204 bytes --]



On Tue, 1 Nov 2005, Pablo Neira wrote:
<CUT>
> You can't use -E together with -i. But I think that adding the conntrack ID 
> to the event information that is dumped could be worth for accounting 
> purposes, so I'll add this to my pending patches for ctnetlink, ok?
OK, thank you.

<CUT>
> You can't kill conntracks *just* by the ID. The connection tracking table 
> currently uses the tuple information (source, destination, protocol 
> information) to place the conntrack in hashes, same thing to perform lookups.
So, why do we need this conntrack ID? Only for userspace applications?

> Implementing the ability of killing conntracks just by its ID would be O(n),
True :(

> so we would need to walk through the buckets until we find a matching, not so 
> good. Just a wild thought, how bad would be hashing the conntracks by its ID?

AFAIK quite bad since netfilter code really needs src/dst/proto hashing 
when processing received packet.

> In that case we could implement this feature. So, currently you'll always 
> need the information about the source, destination and protocol specific 
> stuff together with the ID.
OK.

Best regards,

 				Krzysztof Olędzki

Re: problem with conntrack utility and kernel 2.6.14

[Pablo Neira]

Krzysztof Oledzki wrote:
>> You can't kill conntracks *just* by the ID. The connection tracking 
>> table currently uses the tuple information (source, destination, 
>> protocol information) to place the conntrack in hashes, same thing to 
>> perform lookups.
> 
> So, why do we need this conntrack ID? Only for userspace applications?

Because of two reasons:

a) We have to provide a way to uniquely identify a conntrack. On slow 
devices, it could happen that a connection with the same src/dst/proto 
could be re-created in a very short period of time. So the user can be 
sure that he's killing that conntrack, and not a similar that represents 
a re-opened connection.
b) Netlink dumping is not atomic, for that reason we need a way to know 
from which point we stopped dumping the connection tracking table. 
Keeping a pointer to the lastest conntrack processed won't work because 
of the timers. It could happen that a conntrack expires at the time that 
  the process of dumping continues. So the idea is using the ID to 
establish such breakpoint, and continue from the lastest conntrack ID 
processed if it's still present, otherwise we'll process the next 
conntrack ID higher than the lastest processed.

>> so we would need to walk through the buckets until we find a matching, 
>> not so good. Just a wild thought, how bad would be hashing the 
>> conntracks by its ID?
> 
> AFAIK quite bad since netfilter code really needs src/dst/proto hashing 
> when processing received packet.

Agreed, you can drop that idea to the trashcan :(

-- 
Pablo

Re: problem with conntrack utility and kernel 2.6.14

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1690 bytes --]



On Tue, 1 Nov 2005, Pablo Neira wrote:

> Krzysztof Oledzki wrote:
>>> You can't kill conntracks *just* by the ID. The connection tracking table 
>>> currently uses the tuple information (source, destination, protocol 
>>> information) to place the conntrack in hashes, same thing to perform 
>>> lookups.
>> 
>> So, why do we need this conntrack ID? Only for userspace applications?
>
> Because of two reasons:
>
> a) We have to provide a way to uniquely identify a conntrack. On slow 
> devices, it could happen that a connection with the same src/dst/proto could 
> be re-created in a very short period of time. So the user can be sure that 
> he's killing that conntrack, and not a similar that represents a re-opened 
> connection.
So one can pass this id as an additional information? OK. :)

BTW: I think conntrack-tool should inform that conntrack killing fails 
because kernel isn't able to find a matching conntrack.

> b) Netlink dumping is not atomic, for that reason we need a way to know from 
> which point we stopped dumping the connection tracking table. Keeping a 
> pointer to the lastest conntrack processed won't work because of the timers. 
> It could happen that a conntrack expires at the time that  the process of 
> dumping continues. So the idea is using the ID to establish such breakpoint, 
> and continue from the lastest conntrack ID processed if it's still present, 
> otherwise we'll process the next conntrack ID higher than the lastest 
> processed.

But to do this you need to travel the structure from the beginning and so 
dumping may require operations like O(n) anyway?


Best regards,

 				Krzysztof Olędzki

Re: problem with conntrack utility and kernel 2.6.14

[Pablo Neira]

Krzysztof Oledzki wrote:
> On Tue, 1 Nov 2005, Pablo Neira wrote:
> 
>> Krzysztof Oledzki wrote:
>>
>>>> You can't kill conntracks *just* by the ID. The connection tracking 
>>>> table currently uses the tuple information (source, destination, 
>>>> protocol information) to place the conntrack in hashes, same thing 
>>>> to perform lookups.
>>>
>>> So, why do we need this conntrack ID? Only for userspace applications?
>>
>> Because of two reasons:
>>
>> a) We have to provide a way to uniquely identify a conntrack. On slow 
>> devices, it could happen that a connection with the same src/dst/proto 
>> could be re-created in a very short period of time. So the user can be 
>> sure that he's killing that conntrack, and not a similar that 
>> represents a re-opened connection.
> 
> So one can pass this id as an additional information? OK. :)

That is :)

> BTW: I think conntrack-tool should inform that conntrack killing fails 
> because kernel isn't able to find a matching conntrack.

That is what it currently does, are you observing a different behaviour?

# conntrack -D --orig-src 1.1.1.1 --orig-dst 2.2.2.2 -p tcp 
--orig-port-src 2005 --orig-port-dst 21
NFNETLINK answers: No such file or directory
Operation failed: such conntrack doesn't exist

>> b) Netlink dumping is not atomic, for that reason we need a way to 
>> know from which point we stopped dumping the connection tracking 
>> table. Keeping a pointer to the lastest conntrack processed won't work 
>> because of the timers. It could happen that a conntrack expires at the 
>> time that  the process of dumping continues. So the idea is using the 
>> ID to establish such breakpoint, and continue from the lastest 
>> conntrack ID processed if it's still present, otherwise we'll process 
>> the next conntrack ID higher than the lastest processed.
> 
> But to do this you need to travel the structure from the beginning and 
> so dumping may require operations like O(n) anyway?

No, we store the last bucket processed in a temporary control buffer, so 
we can continue from this bucket and then look for the last conntrack 
processed in the list associated to this bucket.

-- 
Pablo

Re: problem with conntrack utility and kernel 2.6.14

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2258 bytes --]



On Tue, 1 Nov 2005, Pablo Neira wrote:

> Krzysztof Oledzki wrote:
>> On Tue, 1 Nov 2005, Pablo Neira wrote:
>> 
>>> Krzysztof Oledzki wrote:
>>> 
>>>>> You can't kill conntracks *just* by the ID. The connection tracking 
>>>>> table currently uses the tuple information (source, destination, 
>>>>> protocol information) to place the conntrack in hashes, same thing to 
>>>>> perform lookups.
>>>> 
>>>> So, why do we need this conntrack ID? Only for userspace applications?
>>> 
>>> Because of two reasons:
>>> 
>>> a) We have to provide a way to uniquely identify a conntrack. On slow 
>>> devices, it could happen that a connection with the same src/dst/proto 
>>> could be re-created in a very short period of time. So the user can be 
>>> sure that he's killing that conntrack, and not a similar that represents a 
>>> re-opened connection.
>> 
>> So one can pass this id as an additional information? OK. :)
>
> That is :)
>
>> BTW: I think conntrack-tool should inform that conntrack killing fails 
>> because kernel isn't able to find a matching conntrack.
>
> That is what it currently does, are you observing a different behaviour?
Unfortunately :(

> # conntrack -D --orig-src 1.1.1.1 --orig-dst 2.2.2.2 -p tcp --orig-port-src 
> 2005 --orig-port-dst 21
> NFNETLINK answers: No such file or directory
> Operation failed: such conntrack doesn't exist

# conntrack -L -i|grep 192.168.130.123
udp      17 20 src=192.168.0.33 dst=192.168.130.123 sport=123 dport=123 packets=1 bytes=76 src=192.168.130.123 dst=192.168.0.33 sport=123 dport=123 packets=1 bytes=76 mark=0 use=1 id=13157

root@olemx:~# conntrack -D --orig-src 192.168.0.33 --orig-dst 
192.168.130.123 -p udp --orig-port-src 123 --orig-port-dst 123 -i 90909

# conntrack -L -i|grep 192.168.130.123
(empty)

# conntrack -D --orig-src 192.168.0.33 --orig-dst 192.168.130.123 -p udp --orig-port-src 123 --orig-port-dst 123 -i 90909
NFNETLINK answers: No such file or directory
Operation failed: sorry, you must be root or get CAP_NET_ADMIN capability to do this

It kills wrong conntrack (id does not match) and display bogus information 
about CAP_NET_ADMIN.

# conntrack -V
conntrack v0.94

Best regards,

 				Krzysztof Olędzki

Re: problem with conntrack utility and kernel 2.6.14

[Pablo Neira]

Krzysztof Oledzki wrote:
> # conntrack -V
> conntrack v0.94

Refresh your working copy, try lastest conntrack (v0.95) and tell if 
everything goes fine.

-- 
Pablo

Re: problem with conntrack utility and kernel 2.6.14

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 309 bytes --]



On Tue, 1 Nov 2005, Pablo Neira wrote:

> Krzysztof Oledzki wrote:
>> # conntrack -V
>> conntrack v0.94
>
> Refresh your working copy, try lastest conntrack (v0.95) and tell if 
> everything goes fine.

# conntrack -V
conntrack v0.95

Still wrong. :(

Best regards,

 				Krzysztof Olędzki

Re: problem with conntrack utility and kernel 2.6.14

[Pablo Neira]

[-- Attachment #1: Type: text/plain, Size: 1181 bytes --]

Krzysztof Oledzki wrote:
>> That is what it currently does, are you observing a different behaviour?
> 
> Unfortunately :(
> 
>> # conntrack -D --orig-src 1.1.1.1 --orig-dst 2.2.2.2 -p tcp 
>> --orig-port-src 2005 --orig-port-dst 21
>> NFNETLINK answers: No such file or directory
>> Operation failed: such conntrack doesn't exist
> 
> # conntrack -L -i|grep 192.168.130.123
> udp      17 20 src=192.168.0.33 dst=192.168.130.123 sport=123 dport=123 
> packets=1 bytes=76 src=192.168.130.123 dst=192.168.0.33 sport=123 
> dport=123 packets=1 bytes=76 mark=0 use=1 id=13157
> 
> root@olemx:~# conntrack -D --orig-src 192.168.0.33 --orig-dst 
> 192.168.130.123 -p udp --orig-port-src 123 --orig-port-dst 123 -i 90909

Fixed in SVN.

> # conntrack -L -i|grep 192.168.130.123
> (empty)
> 
> # conntrack -D --orig-src 192.168.0.33 --orig-dst 192.168.130.123 -p udp 
> --orig-port-src 123 --orig-port-dst 123 -i 90909
> NFNETLINK answers: No such file or directory
> Operation failed: sorry, you must be root or get CAP_NET_ADMIN 
> capability to do this

This error message is related with libnfnetlink, give a try to the patch 
attached. It fixes it. I'll pass it to Harald.

-- 
Pablo

[-- Attachment #2: x --]
[-- Type: text/plain, Size: 343 bytes --]

Index: src/libnfnetlink.c
===================================================================
--- src/libnfnetlink.c	(revision 4416)
+++ src/libnfnetlink.c	(working copy)
@@ -462,7 +462,7 @@
 					}
 					perror("NFNETLINK answers");
 				}
-				return -1;
+				return err->error;
 			}
 			if (answer) {
 				memcpy(answer, h, h->nlmsg_len);

Re: problem with conntrack utility and kernel 2.6.14

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1438 bytes --]



On Tue, 1 Nov 2005, Pablo Neira wrote:

> Krzysztof Oledzki wrote:
>>> That is what it currently does, are you observing a different behaviour?
>> 
>> Unfortunately :(
>> 
>>> # conntrack -D --orig-src 1.1.1.1 --orig-dst 2.2.2.2 -p tcp 
>>> --orig-port-src 2005 --orig-port-dst 21
>>> NFNETLINK answers: No such file or directory
>>> Operation failed: such conntrack doesn't exist
>> 
>> # conntrack -L -i|grep 192.168.130.123
>> udp      17 20 src=192.168.0.33 dst=192.168.130.123 sport=123 dport=123 
>> packets=1 bytes=76 src=192.168.130.123 dst=192.168.0.33 sport=123 dport=123 
>> packets=1 bytes=76 mark=0 use=1 id=13157
>> 
>> root@olemx:~# conntrack -D --orig-src 192.168.0.33 --orig-dst 
>> 192.168.130.123 -p udp --orig-port-src 123 --orig-port-dst 123 -i 90909
>
> Fixed in SVN.
>
>> # conntrack -L -i|grep 192.168.130.123
>> (empty)
>> 
>> # conntrack -D --orig-src 192.168.0.33 --orig-dst 192.168.130.123 -p udp 
>> --orig-port-src 123 --orig-port-dst 123 -i 90909
>> NFNETLINK answers: No such file or directory
>> Operation failed: sorry, you must be root or get CAP_NET_ADMIN capability 
>> to do this
>
> This error message is related with libnfnetlink, give a try to the patch 
> attached. It fixes it. I'll pass it to Harald.

Both problems are now fixed. Thank you.

Best regards,

 				Krzysztof Olędzki

PS: libnfnetlink still has aclocal-1.6 & automake-1.6 in autogen.sh.

Re: problem with conntrack utility and kernel 2.6.14

[Pablo Neira]

Krzysztof Oledzki wrote:
> PS: libnfnetlink still has aclocal-1.6 & automake-1.6 in autogen.s
> h.

Just passed a patch to Harald to fix this. Thanks for the hint.

-- 
Pablo

Re: problem with conntrack utility and kernel 2.6.14

[Deti Fliegl]

Hi Pablo,

thanks for your night shift. Now it is working much more stable as no 
segfaults within conntrack happened.

Now running "while true; do conntrack -L|wc -l; done" together with my 
stresstest script the system crashed after app 20 minutes.

Hm... somehow there must be something wrong with the kernel stuff.

<4>Badness in __kfree_skb at net/core/skbuff.c:330
<4>
<4>Call Trace:<ffffffff802a2f77>{__kfree_skb+167} 
<ffffffff802bb597>{netlink_recvmsg+279}
<4>       <ffffffff8029cb2b>{sock_recvmsg+315} 
<ffffffff8019b400>{update_atime+64}
<4>



Deti

Re: problem with conntrack utility and kernel 2.6.14

[Pablo Neira Ayuso]

[-- Attachment #1: Type: text/plain, Size: 1115 bytes --]

Hi Patrick,

Deti Fliegl wrote:
> The kernel oops still happens - but this time I had access to the
> console (hope that helps):
> 
> <4>Badness in __kfree_skb at net/core/skbuff.c:330
> <4>
> <4>Call Trace:<ffffffff802a2f77>{__kfree_skb+167}
> <ffffffff802bb597>{netlink_recvmsg+279}
> <4>       <ffffffff8029cb2b>{sock_recvmsg+315}
> <ffffffff8025a4d8>{n_tty_receive_buf+4392}
> <4>       <ffffffff8025a4d8>{n_tty_receive_buf+4392}
> <ffffffff8015d828>{filemap_nopage+424}
> <4>       <ffffffff8013003b>{try_to_wake_up+1083}
> <ffffffff8014b2f0>{autoremove_wake_function+0}
> <4>       <ffffffff8025bbd9>{pty_write+89}
> <ffffffff8029e38b>{sys_recvmsg+395}
> <4>       <ffffffff80130f23>{__wake_up+67}
> <ffffffff80181208>{vfs_write+344}
> <4>       <ffffffff80181343>{sys_write+83}
> <ffffffff8010dc4e>{system_call+126}
> <4>
> <3>scheduling while atomic: conntrack/0xffffff00/7887

Romary Sonrier is reporting a similar problem on a i586 box:

https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=409

I can't see a way how the ctnetlink code can trigger that oops at moment
:(. Any clue on this?

-- 
Pablo

[-- Attachment #2: x --]
[-- Type: text/plain, Size: 1492 bytes --]

Dec  1 17:54:55 ipnobox_light_v1-8 Badness in __kfree_skb at net/core/skbuff.c:330
Dec  1 17:54:55 ipnobox_light_v1-8 [<c01ed857>] __kfree_skb+0x87/0xc0
Dec  1 17:54:55 ipnobox_light_v1-8 [<c0212048>] netlink_recvmsg+0xf8/0x220
Dec  1 17:54:55 ipnobox_light_v1-8 [<c01e98f6>] sock_recvmsg+0xe6/0x120
Dec  1 17:54:55 ipnobox_light_v1-8 [<c0126e0f>] file_read_actor+0x6f/0xe0
Dec  1 17:54:55 ipnobox_light_v1-8 [<c0120e90>] autoremove_wake_function+0x0/0x40
Dec  1 17:54:55 ipnobox_light_v1-8 [<c01eadb7>] sys_recvmsg+0xf7/0x1d0
Dec  1 17:54:55 ipnobox_light_v1-8 [<c012a490>] buffered_rmqueue+0xe0/0x1b0
Dec  1 17:54:55 ipnobox_light_v1-8 [<c0113eb2>] current_fs_time+0x42/0x50
Dec  1 17:54:55 ipnobox_light_v1-8 [<c0154343>] inode_update_time+0x33/0xa0
Dec  1 17:54:55 ipnobox_light_v1-8 [<c0147e34>] pipe_writev+0x324/0x4e0
Dec  1 17:54:55 ipnobox_light_v1-8 [<c0148017>] pipe_write+0x27/0x30
Dec  1 17:54:55 ipnobox_light_v1-8 [<c0180892>] copy_from_user+0x32/0x60
Dec  1 17:54:55 ipnobox_light_v1-8 [<c01eaefa>] sys_socketcall+0x6a/0x1e0
Dec  1 17:54:55 ipnobox_light_v1-8 [<c01028a9>] syscall_call+0x7/0xb
Dec  1 17:54:55 ipnobox_light_v1-8 scheduling while atomic:
conntrack/0xffffff00/1576
Dec  1 17:54:55 ipnobox_light_v1-8 [<c026b78d>] schedule+0x52d/0x5f0
Dec  1 17:54:55 ipnobox_light_v1-8 [<c01eaefa>] sys_socketcall+0x6a/0x1e0
Dec  1 17:54:55 ipnobox_light_v1-8 [<c010291a>] work_resched+0x5/0x16

>>uname -a
Linux ipnobox_light_v1-8 2.6.14.3 #1 Mon Nov 28 10:44:25 CET 2005 i586

Re: problem with conntrack utility and kernel 2.6.14

[Patrick McHardy]

Pablo Neira Ayuso wrote:
> Hi Patrick,
> 
> Deti Fliegl wrote:
> 
>>The kernel oops still happens - but this time I had access to the
>>console (hope that helps):
>>
>><4>Badness in __kfree_skb at net/core/skbuff.c:330
>><4>
>><4>Call Trace:<ffffffff802a2f77>{__kfree_skb+167}
>><ffffffff802bb597>{netlink_recvmsg+279}
>><4>       <ffffffff8029cb2b>{sock_recvmsg+315}
>><ffffffff8025a4d8>{n_tty_receive_buf+4392}
>><4>       <ffffffff8025a4d8>{n_tty_receive_buf+4392}
>><ffffffff8015d828>{filemap_nopage+424}
>><4>       <ffffffff8013003b>{try_to_wake_up+1083}
>><ffffffff8014b2f0>{autoremove_wake_function+0}
>><4>       <ffffffff8025bbd9>{pty_write+89}
>><ffffffff8029e38b>{sys_recvmsg+395}
>><4>       <ffffffff80130f23>{__wake_up+67}
>><ffffffff80181208>{vfs_write+344}
>><4>       <ffffffff80181343>{sys_write+83}
>><ffffffff8010dc4e>{system_call+126}
>><4>
>><3>scheduling while atomic: conntrack/0xffffff00/7887
> 
> 
> Romary Sonrier is reporting a similar problem on a i586 box:
> 
> https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=409
> 
> I can't see a way how the ctnetlink code can trigger that oops at moment
> :(. Any clue on this?

Interesting. I guess that rules out bad hardware in Deti's case. I still
can't see how this can happen, but I'm going to have another look.

Re: problem with conntrack utility and kernel 2.6.14

[Deti Fliegl]

Patrick McHardy wrote:
> Interesting. I guess that rules out bad hardware in Deti's case. I still
> can't see how this can happen, but I'm going to have another look.
Well, I was testing on a system bought in june. Tomorrow I will setup a 
brand new box (but same platform) and repeat my stresstests.

Deti

Re: problem with conntrack utility and kernel 2.6.14

[Yasuyuki KOZAKAI]

[-- Attachment #1: Type: Text/Plain, Size: 1667 bytes --]


Hi,

From: Pablo Neira Ayuso <pablo@eurodev.net>
Date: Sun, 04 Dec 2005 03:14:34 +0100

> Hi Patrick,
> 
> Deti Fliegl wrote:
> > The kernel oops still happens - but this time I had access to the
> > console (hope that helps):
> > 
> > <4>Badness in __kfree_skb at net/core/skbuff.c:330
> > <4>
> > <4>Call Trace:<ffffffff802a2f77>{__kfree_skb+167}
> > <ffffffff802bb597>{netlink_recvmsg+279}
> > <4>       <ffffffff8029cb2b>{sock_recvmsg+315}
> > <ffffffff8025a4d8>{n_tty_receive_buf+4392}
> > <4>       <ffffffff8025a4d8>{n_tty_receive_buf+4392}
> > <ffffffff8015d828>{filemap_nopage+424}
> > <4>       <ffffffff8013003b>{try_to_wake_up+1083}
> > <ffffffff8014b2f0>{autoremove_wake_function+0}
> > <4>       <ffffffff8025bbd9>{pty_write+89}
> > <ffffffff8029e38b>{sys_recvmsg+395}
> > <4>       <ffffffff80130f23>{__wake_up+67}
> > <ffffffff80181208>{vfs_write+344}
> > <4>       <ffffffff80181343>{sys_write+83}
> > <ffffffff8010dc4e>{system_call+126}
> > <4>
> > <3>scheduling while atomic: conntrack/0xffffff00/7887

I'm reading some codes around preempt/spinlock and 0xffffff00 seems to mean
local_bh_enable() without local_bh_disable(). But I've not found such bug yet.

> Romary Sonrier is reporting a similar problem on a i586 box:
> 
> https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=409
> 
> I can't see a way how the ctnetlink code can trigger that oops at moment
> :(. Any clue on this?

Deti, Romary, could you try CONFIG_DEBUG_PREEMPT and CONFIG_DEBUG_SPINLOCK ?

And did you apply Pablo's patch (attached in this mail) which fixes refcount
leak ? linux 2.6.14.3 doesn't include this fix. If no, please try it.

Regards,

-- Yasuyuki Kozakai


[-- Attachment #2: proto_find.patch --]
[-- Type: Text/Plain, Size: 2240 bytes --]

Remove proto == NULL checking since ip_conntrack_[nat_]proto_find_get always
returns a valid pointer. Fix missing ip_conntrack_proto_put in some paths.

Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>

Index: netfilter-2.6.14.git/net/ipv4/netfilter/ip_conntrack_netlink.c
===================================================================
--- netfilter-2.6.14.git.orig/net/ipv4/netfilter/ip_conntrack_netlink.c	2005-11-21 15:00:28.000000000 +0100
+++ netfilter-2.6.14.git/net/ipv4/netfilter/ip_conntrack_netlink.c	2005-11-21 15:03:49.000000000 +0100
@@ -59,11 +59,13 @@ ctnetlink_dump_tuples_proto(struct sk_bu
 
 	NFA_PUT(skb, CTA_PROTO_NUM, sizeof(u_int8_t), &tuple->dst.protonum);
 
+	/* If no protocol helper is found, this function will return the
+	 * generic protocol helper, so proto won't *ever* be NULL */
 	proto = ip_conntrack_proto_find_get(tuple->dst.protonum);
-	if (likely(proto && proto->tuple_to_nfattr)) {
+	if (likely(proto->tuple_to_nfattr))
 		ret = proto->tuple_to_nfattr(skb, tuple);
-		ip_conntrack_proto_put(proto);
-	}
+	
+	ip_conntrack_proto_put(proto);
 
 	return ret;
 
@@ -128,9 +130,11 @@ ctnetlink_dump_protoinfo(struct sk_buff 
 
 	struct nfattr *nest_proto;
 	int ret;
-	
-	if (!proto || !proto->to_nfattr)
+
+	if (!proto->to_nfattr) {
+		ip_conntrack_proto_put(proto);
 		return 0;
+	}
 	
 	nest_proto = NFA_NEST(skb, CTA_PROTOINFO);
 
@@ -527,10 +531,10 @@ ctnetlink_parse_tuple_proto(struct nfatt
 
 	proto = ip_conntrack_proto_find_get(tuple->dst.protonum);
 
-	if (likely(proto && proto->nfattr_to_tuple)) {
+	if (likely(proto->nfattr_to_tuple))
 		ret = proto->nfattr_to_tuple(tb, tuple);
-		ip_conntrack_proto_put(proto);
-	}
+	
+	ip_conntrack_proto_put(proto);
 	
 	return ret;
 }
@@ -596,8 +600,6 @@ static int ctnetlink_parse_nat_proto(str
 		return -EINVAL;
 
 	npt = ip_nat_proto_find_get(ct->tuplehash[IP_CT_DIR_ORIGINAL].tuple.dst.protonum);
-	if (!npt)
-		return 0;
 
 	if (!npt->nfattr_to_range) {
 		ip_nat_proto_put(npt);
@@ -957,8 +959,6 @@ ctnetlink_change_protoinfo(struct ip_con
 	nfattr_parse_nested(tb, CTA_PROTOINFO_MAX, attr);
 
 	proto = ip_conntrack_proto_find_get(npt);
-	if (!proto)
-		return -EINVAL;
 
 	if (proto->from_nfattr)
 		err = proto->from_nfattr(tb, ct);

Re: problem with conntrack utility and kernel 2.6.14

[Deti Fliegl]

[-- Attachment #1: Type: text/plain, Size: 468 bytes --]

Yasuyuki KOZAKAI wrote:
> Deti, Romary, could you try CONFIG_DEBUG_PREEMPT and CONFIG_DEBUG_SPINLOCK ?
Done. First I had to enable preemption in general (was switched off).

> 
> And did you apply Pablo's patch (attached in this mail) which fixes refcount
> leak ? linux 2.6.14.3 doesn't include this fix. If no, please try it.
Latest test was done with linux-2.6.15-rc5 which already contains this 
patch.

And here comes your oops...

Sorry it's reproducible.

Deti

[-- Attachment #2: panic.txt --]
[-- Type: text/plain, Size: 10428 bytes --]

----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at kernel/sched.c:2930
invalid operand: 0000 [1] PREEMPT SMP 
CPU 3 
Modules linked in: ip_conntrack_netlink ipt_CLASSIFY sch_htb ipt_mark ipt_CONNMARK af_packet ipt_SAME ipt_state i8xx_tco ipt_pkttype ipt_NOTRACK iptable_raw ipt_LOG ipt_limit ipt_hashlimit ip_nat_ftp evdev ip_conntrack_ftp edd iptable_nat ip_nat ip_conntrack ipt_MARK nfnetlink iptable_mangle iptable_filter ip_tables sg sr_mod i2c_i801 i2c_core hw_random capability commoncap ext3 jbd ide_cd cdrom ide_disk e1000 piix ide_core megaraid_mbox megaraid_mm sd_mod scsi_mod
Pid: 12629, comm: conntrack Not tainted 2.6.15-rc5-smp #1
RIP: 0010:[<ffffffff80129a3c>] <ffffffff80129a3c>{sub_preempt_count+22}
RSP: 0018:ffff81011513bb38  EFLAGS: 00010206
RAX: ffff81011513bfd8 RBX: ffff810117b3be08 RCX: 0000000000000000
RDX: 00000000ffffffff RSI: 0000000000000dc8 RDI: 00000000000000ff
RBP: ffff81011513bb38 R08: 0000000000000000 R09: ffff810115608e80
R10: 0000000000000000 R11: 0000000000000000 R12: ffff810115b867f0
R13: ffff81011a1fed80 R14: ffff81011513bc48 R15: ffff81011654def0
FS:  00002aaaab0f6b00(0000) GS:ffffffff8049a980(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000000054d008 CR3: 000000011569f000 CR4: 00000000000006e0
Process conntrack (pid: 12629, threadinfo ffff81011513a000, task ffff810115076840)
Stack: ffff810115b867c0 ffffffff80135ee0 0000000000000296 ffffffff88184a87 
       ffff81011654dc00 ffff81011513be08 ffff81011a1fed80 ffff81011654dc00 
       ffff810115b867c0 ffffffff80292544 
Call Trace:<ffffffff80135ee0>{local_bh_enable+69} <ffffffff88184a87>{:ip_conntrack_netlink:ctnetlink_dump_table+254}
       <ffffffff80292544>{netlink_dump+148} <ffffffff802927c7>{netlink_recvmsg+305}
       <ffffffff80275302>{sock_recvmsg+273} <ffffffff802f010b>{_spin_unlock_irq+20}
       <ffffffff802eee5f>{thread_return+206} <ffffffff8012975a>{resched_task+78}
       <ffffffff8012b772>{try_to_wake_up+1291} <ffffffff8014540a>{autoremove_wake_function+0}
       <ffffffff80174806>{fget+153} <ffffffff80276938>{sys_recvmsg+357}
       <ffffffff8018041b>{pipe_writev+1293} <ffffffff802eee5f>{thread_return+206}
       <ffffffff8017402e>{vfs_write+326} <ffffffff80174122>{sys_write+69}
       <ffffffff8010da72>{system_call+126} 

Code: 0f 0b 68 51 da 30 80 c2 72 0b 81 ff fe 00 00 00 3e 77 1d 65 
RIP <ffffffff80129a3c>{sub_preempt_count+22} RSP <ffff81011513bb38>
 <3>BUG: soft lockup detected on CPU#1!
CPU 1:
Modules linked in: ip_conntrack_netlink ipt_CLASSIFY sch_htb ipt_mark ipt_CONNMARK af_packet ipt_SAME ipt_state i8xx_tco ipt_pkttype ipt_NOTRACK iptable_raw ipt_LOG ipt_limit ipt_hashlimit ip_nat_ftp evdev ip_conntrack_ftp edd iptable_nat ip_nat ip_conntrack ipt_MARK nfnetlink iptable_mangle iptable_filter ip_tables sg sr_mod i2c_i801 i2c_core hw_random capability commoncap ext3 jbd ide_cd cdrom ide_disk e1000 piix ide_core megaraid_mbox megaraid_mm sd_mod scsi_mod
Pid: 12754, comm: wget Not tainted 2.6.15-rc5-smp #1
RIP: 0010:[<ffffffff802f0208>] <ffffffff802f0208>{_write_lock_irqsave+117}
RSP: 0018:ffff81011525f978  EFLAGS: 00000202
RAX: 0000000001000001 RBX: ffffffff881336d0 RCX: ffff8101170a3a68
RDX: 0000000000000000 RSI: 0000000000000014 RDI: 0000000000000001
RBP: ffff81011525fac8 R08: 0000000000008460 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff81011cc2e078
R13: ffff81011525fbc8 R14: ffff8101173fcd10 R15: ffff81011525fab4
FS:  00002aaaab4726e0(0000) GS:ffffffff8049a880(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000536100 CR3: 0000000115470000 CR4: 00000000000006e0

Call Trace:<ffffffff802f01c7>{_write_lock_irqsave+52} <ffffffff802f0233>{_write_lock_bh+6}
       <ffffffff881278a2>{:ip_conntrack:tcp_packet+152} <ffffffff801598bd>{__alloc_pages+79}
       <ffffffff802bfb17>{inet_fill_ifaddr+558} <ffffffff802f032e>{_read_lock_irqsave+32}
       <ffffffff88125142>{:ip_conntrack:__ip_conntrack_find+17}
       <ffffffff88126cb6>{:ip_conntrack:ip_conntrack_in+1083}
       <ffffffff80295a37>{nf_iterate+87} <ffffffff8029e731>{dst_output+0}
       <ffffffff80295c30>{nf_hook_slow+135} <ffffffff8029e731>{dst_output+0}
       <ffffffff802a0d64>{ip_queue_xmit+1227} <ffffffff80135f14>{local_bh_enable+121}
       <ffffffff802d4da6>{unix_stream_sendmsg+669} <ffffffff80135f14>{local_bh_enable+121}
       <ffffffff80299139>{__ip_route_output_key+1656} <ffffffff80135092>{current_fs_time+82}
       <ffffffff802f032e>{_read_lock_irqsave+32} <ffffffff8018c2ac>{update_atime+69}
       <ffffffff802b389a>{tcp_v4_send_check+153} <ffffffff802afd38>{tcp_transmit_skb+1673}
       <ffffffff802b045e>{tcp_connect+756} <ffffffff802b74c6>{tcp_v4_connect+2328}
       <ffffffff80277db6>{lock_sock+200} <ffffffff802c2550>{inet_stream_connect+186}
       <ffffffff802f0171>{_spin_unlock+19} <ffffffff8015c48e>{cache_alloc_refill+420}
       <ffffffff80174806>{fget+153} <ffffffff80276f9a>{sys_connect+114}
       <ffffffff802f0171>{_spin_unlock+19} <ffffffff80276659>{sock_map_fd+299}
       <ffffffff8010da72>{system_call+126} 
BUG: soft lockup detected on CPU#1!
CPU 1:
Modules linked in: ip_conntrack_netlink ipt_CLASSIFY sch_htb ipt_mark ipt_CONNMARK af_packet ipt_SAME ipt_state i8xx_tco ipt_pkttype ipt_NOTRACK iptable_raw ipt_LOG ipt_limit ipt_hashlimit ip_nat_ftp evdev ip_conntrack_ftp edd iptable_nat ip_nat ip_conntrack ipt_MARK nfnetlink iptable_mangle iptable_filter ip_tables sg sr_mod i2c_i801 i2c_core hw_random capability commoncap ext3 jbd ide_cd cdrom ide_disk e1000 piix ide_core megaraid_mbox megaraid_mm sd_mod scsi_mod
Pid: 12754, comm: wget Not tainted 2.6.15-rc5-smp #1
RIP: 0010:[<ffffffff802f0210>] <ffffffff802f0210>{_write_lock_irqsave+125}
RSP: 0018:ffff81011525f978  EFLAGS: 00000202
RAX: 0000000001000001 RBX: ffffffff881336d0 RCX: ffff8101170a3a68
RDX: 0000000000000000 RSI: 0000000000000014 RDI: 0000000000000001
RBP: ffff81011525fac8 R08: 0000000000008460 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff81011cc2e078
R13: ffff81011525fbc8 R14: ffff8101173fcd10 R15: ffff81011525fab4
FS:  00002aaaab4726e0(0000) GS:ffffffff8049a880(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000536100 CR3: 0000000115470000 CR4: 00000000000006e0

Call Trace:<ffffffff802f01c7>{_write_lock_irqsave+52} <ffffffff802f0233>{_write_lock_bh+6}
       <ffffffff881278a2>{:ip_conntrack:tcp_packet+152} <ffffffff801598bd>{__alloc_pages+79}
       <ffffffff802bfb17>{inet_fill_ifaddr+558} <ffffffff802f032e>{_read_lock_irqsave+32}
       <ffffffff88125142>{:ip_conntrack:__ip_conntrack_find+17}
       <ffffffff88126cb6>{:ip_conntrack:ip_conntrack_in+1083}
       <ffffffff80295a37>{nf_iterate+87} <ffffffff8029e731>{dst_output+0}
       <ffffffff80295c30>{nf_hook_slow+135} <ffffffff8029e731>{dst_output+0}
       <ffffffff802a0d64>{ip_queue_xmit+1227} <ffffffff80135f14>{local_bh_enable+121}
       <ffffffff802d4da6>{unix_stream_sendmsg+669} <ffffffff80135f14>{local_bh_enable+121}
       <ffffffff80299139>{__ip_route_output_key+1656} <ffffffff80135092>{current_fs_time+82}
       <ffffffff802f032e>{_read_lock_irqsave+32} <ffffffff8018c2ac>{update_atime+69}
       <ffffffff802b389a>{tcp_v4_send_check+153} <ffffffff802afd38>{tcp_transmit_skb+1673}
       <ffffffff802b045e>{tcp_connect+756} <ffffffff802b74c6>{tcp_v4_connect+2328}
       <ffffffff80277db6>{lock_sock+200} <ffffffff802c2550>{inet_stream_connect+186}
       <ffffffff802f0171>{_spin_unlock+19} <ffffffff8015c48e>{cache_alloc_refill+420}
       <ffffffff80174806>{fget+153} <ffffffff80276f9a>{sys_connect+114}
       <ffffffff802f0171>{_spin_unlock+19} <ffffffff80276659>{sock_map_fd+299}
       <ffffffff8010da72>{system_call+126} 
BUG: soft lockup detected on CPU#1!
CPU 1:
Modules linked in: ip_conntrack_netlink ipt_CLASSIFY sch_htb ipt_mark ipt_CONNMARK af_packet ipt_SAME ipt_state i8xx_tco ipt_pkttype ipt_NOTRACK iptable_raw ipt_LOG ipt_limit ipt_hashlimit ip_nat_ftp evdev ip_conntrack_ftp edd iptable_nat ip_nat ip_conntrack ipt_MARK nfnetlink iptable_mangle iptable_filter ip_tables sg sr_mod i2c_i801 i2c_core hw_random capability commoncap ext3 jbd ide_cd cdrom ide_disk e1000 piix ide_core megaraid_mbox megaraid_mm sd_mod scsi_mod
Pid: 12754, comm: wget Not tainted 2.6.15-rc5-smp #1
RIP: 0010:[<ffffffff802f0208>] <ffffffff802f0208>{_write_lock_irqsave+117}
RSP: 0018:ffff81011525f978  EFLAGS: 00000202
RAX: 0000000001000001 RBX: ffffffff881336d0 RCX: ffff8101170a3a68
RDX: 0000000000000000 RSI: 0000000000000014 RDI: 0000000000000001
RBP: ffff81011525fac8 R08: 0000000000008460 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff81011cc2e078
R13: ffff81011525fbc8 R14: ffff8101173fcd10 R15: ffff81011525fab4
FS:  00002aaaab4726e0(0000) GS:ffffffff8049a880(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000536100 CR3: 0000000115470000 CR4: 00000000000006e0

Call Trace:<ffffffff802f01c7>{_write_lock_irqsave+52} <ffffffff802f0233>{_write_lock_bh+6}
       <ffffffff881278a2>{:ip_conntrack:tcp_packet+152} <ffffffff801598bd>{__alloc_pages+79}
       <ffffffff802bfb17>{inet_fill_ifaddr+558} <ffffffff802f032e>{_read_lock_irqsave+32}
       <ffffffff88125142>{:ip_conntrack:__ip_conntrack_find+17}
       <ffffffff88126cb6>{:ip_conntrack:ip_conntrack_in+1083}
       <ffffffff80295a37>{nf_iterate+87} <ffffffff8029e731>{dst_output+0}
       <ffffffff80295c30>{nf_hook_slow+135} <ffffffff8029e731>{dst_output+0}
       <ffffffff802a0d64>{ip_queue_xmit+1227} <ffffffff80135f14>{local_bh_enable+121}
       <ffffffff802d4da6>{unix_stream_sendmsg+669} <ffffffff80135f14>{local_bh_enable+121}
       <ffffffff80299139>{__ip_route_output_key+1656} <ffffffff80135092>{current_fs_time+82}
       <ffffffff802f032e>{_read_lock_irqsave+32} <ffffffff8018c2ac>{update_atime+69}
       <ffffffff802b389a>{tcp_v4_send_check+153} <ffffffff802afd38>{tcp_transmit_skb+1673}
       <ffffffff802b045e>{tcp_connect+756} <ffffffff802b74c6>{tcp_v4_connect+2328}
       <ffffffff80277db6>{lock_sock+200} <ffffffff802c2550>{inet_stream_connect+186}
       <ffffffff802f0171>{_spin_unlock+19} <ffffffff8015c48e>{cache_alloc_refill+420}
       <ffffffff80174806>{fget+153} <ffffffff80276f9a>{sys_connect+114}
       <ffffffff802f0171>{_spin_unlock+19} <ffffffff80276659>{sock_map_fd+299}
       <ffffffff8010da72>{system_call+126} 

Re: problem with conntrack utility and kernel 2.6.14

[Patrick McHardy]

[-- Attachment #1: Type: text/plain, Size: 189 bytes --]

Deti Fliegl wrote:
> Latest test was done with linux-2.6.15-rc5 which already contains this 
> patch.
> 
> And here comes your oops...

Found it (I hope) :)

Can you try this patch please?

[-- Attachment #2: x --]
[-- Type: text/plain, Size: 1411 bytes --]

[NETFILTER]: Fix unbalanced read_unlock_bh in ctnetlink

NFA_NEST calls NFA_PUT which jumps to nfattr_failure if the skb has no
room left. We call read_unlock_bh at nfattr_failure for the NFA_PUT inside
the locked section, so move NFA_NEST inside the locked section too.

Signed-off-by: Patrick McHardy <kaber@trash.net>

---
commit cd85228eea7c7ab9d701090e3dc9643397cf271d
tree e3fa7a6a24c5b199d311a9f74c312fee3b18eae7
parent 96c75906027f008ed3a4058a606938901e9c6d99
author Patrick McHardy <kaber@trash.net> Sun, 04 Dec 2005 20:56:05 +0100
committer Patrick McHardy <kaber@trash.net> Sun, 04 Dec 2005 20:56:05 +0100

 net/ipv4/netfilter/ip_conntrack_proto_tcp.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/net/ipv4/netfilter/ip_conntrack_proto_tcp.c b/net/ipv4/netfilter/ip_conntrack_proto_tcp.c
index aeb7353..e7fa29e 100644
--- a/net/ipv4/netfilter/ip_conntrack_proto_tcp.c
+++ b/net/ipv4/netfilter/ip_conntrack_proto_tcp.c
@@ -341,9 +341,10 @@ static int tcp_print_conntrack(struct se
 static int tcp_to_nfattr(struct sk_buff *skb, struct nfattr *nfa,
 			 const struct ip_conntrack *ct)
 {
-	struct nfattr *nest_parms = NFA_NEST(skb, CTA_PROTOINFO_TCP);
+	struct nfattr *nest_parms;
 	
 	read_lock_bh(&tcp_lock);
+	nest_parms = NFA_NEST(skb, CTA_PROTOINFO_TCP);
 	NFA_PUT(skb, CTA_PROTOINFO_TCP_STATE, sizeof(u_int8_t),
 		&ct->proto.tcp.state);
 	read_unlock_bh(&tcp_lock);

Re: problem with conntrack utility and kernel 2.6.14

[Yasuyuki KOZAKAI]

From: Patrick McHardy <kaber@trash.net>
Date: Sun, 04 Dec 2005 20:56:14 +0100

> Deti Fliegl wrote:
> > Latest test was done with linux-2.6.15-rc5 which already contains this 
> > patch.
> > 
> > And here comes your oops...

Thanks a lot, Deti! This BUG_ON() is what I expected.

> Found it (I hope) :)

Great! I'll be able to sleep well tonight.

-- Yasuyuki Kozakai

Re: problem with conntrack utility and kernel 2.6.14 - still with 2.6.14.4

[Deti Fliegl]

... it seems that somehow the same old conntrack bug still occures with 
   kernel 2.6.14.4 released today. I thought the patches would go into 
the release trunk? Did I miss something?

Deti

Re: problem with conntrack utility and kernel 2.6.14 - still with 2.6.14.4

[Pablo Neira Ayuso]

Deti Fliegl wrote:
> ... it seems that somehow the same old conntrack bug still occures with
>   kernel 2.6.14.4 released today. I thought the patches would go into
> the release trunk? Did I miss something?

No. Unfortunately the patches will go into 2.6.14.5. The current
revision have enough patches in it :(

-- 
Pablo

Re: problem with conntrack utility and kernel 2.6.14 - still with 2.6.14.4

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 875 bytes --]



On Thu, 15 Dec 2005, Deti Fliegl wrote:

> ... it seems that somehow the same old conntrack bug still occures with 
> kernel 2.6.14.4 released today. I thought the patches would go into the 
> release trunk? Did I miss something?

You may use my private patch-set for 2.6.14.4 
(ftp://ftp.ans.pl/pub/patches/patch-ole-2.6.14.4-o1.gz):

+ 0020: iptables-PreroutingFilter-2611
+ 0030: IPv6-OptionalSIT-26
+ 0050: VGACanDo64KB-Option-26
+ 0060: 3c59x-ShowEthID-26
+ 0061: tulip-8021q-2.6.13.2-szpajder
+ 0070: 2.6.14-libata-passthru
+ 0101: proto_find
+ 0102: ctnetlink-fix-unbalancedread_unlock_bh
+ 0103: CTA_PROTO_NUM-u_int8_t
+ 0120: workaround-for-pnp-device-interrupt
+ 0130: vlan-CHECKSUM_HW
+ 0140: ip_gre-csum

Patches 0101, 0102 and 0103 have the most critical fixes for the 
nf_netlink/conntrack.

Best regards,


 			Krzysztof Olędzki

Re: Major problem with conntrack utility and kernel 2.6.14.3

[Patrick McHardy]

[-- Attachment #1: Type: text/plain, Size: 570 bytes --]

Romary Sonrier wrote:
> Hi evrybody,
> 
> Next week, my main goal at work will to help you to solve this issue.
> I can reproduce easly the kernel oops with my boxs... I still need few hours 
> to be able to print the kernel oops on my serial port (because it's an 
> embedded system). 
> As soon as possible i will send you the whole messages of the kernel oops.
> Is there a mean to help you more quicly ? using some tools like strace or gdb?
> I'm far from being on geek okernel deguging.... ;-)

Please try this patch, I hope it will fix the problems you're seeing.

[-- Attachment #2: x --]
[-- Type: text/plain, Size: 1411 bytes --]

[NETFILTER]: Fix unbalanced read_unlock_bh in ctnetlink

NFA_NEST calls NFA_PUT which jumps to nfattr_failure if the skb has no
room left. We call read_unlock_bh at nfattr_failure for the NFA_PUT inside
the locked section, so move NFA_NEST inside the locked section too.

Signed-off-by: Patrick McHardy <kaber@trash.net>

---
commit cd85228eea7c7ab9d701090e3dc9643397cf271d
tree e3fa7a6a24c5b199d311a9f74c312fee3b18eae7
parent 96c75906027f008ed3a4058a606938901e9c6d99
author Patrick McHardy <kaber@trash.net> Sun, 04 Dec 2005 20:56:05 +0100
committer Patrick McHardy <kaber@trash.net> Sun, 04 Dec 2005 20:56:05 +0100

 net/ipv4/netfilter/ip_conntrack_proto_tcp.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/net/ipv4/netfilter/ip_conntrack_proto_tcp.c b/net/ipv4/netfilter/ip_conntrack_proto_tcp.c
index aeb7353..e7fa29e 100644
--- a/net/ipv4/netfilter/ip_conntrack_proto_tcp.c
+++ b/net/ipv4/netfilter/ip_conntrack_proto_tcp.c
@@ -341,9 +341,10 @@ static int tcp_print_conntrack(struct se
 static int tcp_to_nfattr(struct sk_buff *skb, struct nfattr *nfa,
 			 const struct ip_conntrack *ct)
 {
-	struct nfattr *nest_parms = NFA_NEST(skb, CTA_PROTOINFO_TCP);
+	struct nfattr *nest_parms;
 	
 	read_lock_bh(&tcp_lock);
+	nest_parms = NFA_NEST(skb, CTA_PROTOINFO_TCP);
 	NFA_PUT(skb, CTA_PROTOINFO_TCP_STATE, sizeof(u_int8_t),
 		&ct->proto.tcp.state);
 	read_unlock_bh(&tcp_lock);

Re: Major problem with conntrack utility and kernel 2.6.14.3

[Deti Fliegl]

Patrick McHardy wrote:
> Please try this patch, I hope it will fix the problems you're seeing.
Good news: 3h uptime with stresstest active. Looks very promising. Will 
continue test until 12h uptime.

Very good work, thanks a lot!

Deti

Re: Major problem with conntrack utility and kernel 2.6.14.3

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1027 bytes --]



On Sun, 4 Dec 2005, Patrick McHardy wrote:

> Romary Sonrier wrote:
>> Hi evrybody,
>> 
>> Next week, my main goal at work will to help you to solve this issue.
>> I can reproduce easly the kernel oops with my boxs... I still need few 
>> hours to be able to print the kernel oops on my serial port (because it's 
>> an embedded system). As soon as possible i will send you the whole messages 
>> of the kernel oops.
>> Is there a mean to help you more quicly ? using some tools like strace or 
>> gdb?
>> I'm far from being on geek okernel deguging.... ;-)
>
> Please try this patch, I hope it will fix the problems you're seeing.
>

This patch also fixed my problem (kernel BUG at kernel/sched.c:2833). 
Thank you.

Is it possible that this patch and "Fix refcount leak ip_conntrack/nat_proto" 
(http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=00cb277a4a1fb76aafb2fb28aa99f30546e619c5) 
will go to next 2.6.14-stable kernel?

Best regards,

 				Krzysztof Olędzki

Re: Major problem with conntrack utility and kernel 2.6.14.3

[Patrick McHardy]

Krzysztof Oledzki wrote:
> On Sun, 4 Dec 2005, Patrick McHardy wrote:
> 
>> Please try this patch, I hope it will fix the problems you're seeing.
>>
> 
> This patch also fixed my problem (kernel BUG at kernel/sched.c:2833).
> Thank you.
> 
> Is it possible that this patch and "Fix refcount leak
> ip_conntrack/nat_proto"
> (http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=00cb277a4a1fb76aafb2fb28aa99f30546e619c5)
> will go to next 2.6.14-stable kernel?

I'll submit them to -stable tonight.

Re: problem with conntrack utility and kernel 2.6.14

[Deti Fliegl]

Hi Pablo,

the problem seems to occur in conjunction with high bandwidths 
(>500Mbit/s) and not with the number of connections. I am testing on a 
dual processor x86_64 system with hyperthreading enabled (4 CPUs 
detected by the kernel). It seems that information from the connection 
tracking table gets garbled when conntrack is being called.

Even kernel oopses  with complete lockups are happening:

<4>Badness in __kfree_skb at net/core/skbuff.c:330
<4>
<4>Call Trace:<ffffffff802a2f77>{__kfree_skb+167} 
<ffffffff802bb597>{netlink_recvmsg+279}
<4>       <ffffffff8029cb2b>{sock_recvmsg+315} 
<ffffffff8025a4d8>{n_tty_receive_buf+4392}
<4>       <ffffffff8025a4d8>{n_tty_receive_buf+4392} 
<ffffffff8014b2f0>{autoremove_wake_function+0}
<4>       <ffffffff8025bbd9>{pty_write+89} 
<ffffffff8029e38b>{sys_recvmsg+395}
<4>       <ffffffff80130f23>{__wake_up+67} <ffffffff80181208>{vfs_write+344}
<4>       <ffffffff80181343>{sys_write+83} 
<ffffffff8010dc4e>{system_call+126}
<4>
<3>scheduling while atomic: conntrack/0xffffff00/7454
<4>
<4>Call Trace:<ffffffff8031d67d>{schedule+125}

.... sorry trace ends here as I've got no console access.


Hope you find the problem,

Deti