conntrackd segfault on EPSV IPv6 ftp command when using ftp ExpectationSync

conntrackd segfault on EPSV IPv6 ftp command when using ftp ExpectationSync

[Bill Fink]

[not sure whether to send to netfilter or netfilter-devel,
so sending to both, but trim replies as appropriate]

I am trying to use the ftp ExpectationSync capability of conntrackd
for both IPv4 and IPv6 for connections through a pair of bridged
firewalls (primary / hot backup).  I have the following config
snippet in conntrackd.conf:

	Options {
		ExpectationSync {
			ftp
			sip
			ras	# for H.323
			q.931	# for H.323
			h.245	# for H.323
		}
	}

For IPv4, things work as expected.  But when I try the basic
analogous IPv6 test to the suggested IPv4 test from the
documentation:

x100ssd2% nc 2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx 21
220 FTP Server ready.
USER anonymous
331 Anonymous login ok, send your complete email address as your password
PASS bill@
230-
                *** Welcome to this anonymous ftp server! ***
 
     You are user 1 out of a maximum of 10 authorized anonymous logins.
     The current time here is Thu Jul 04 23:40:51 2013.
     If you experience any problems here, contact : root@localhost
 
 
230 Anonymous login ok, restrictions apply.
EPSV
229 Entering Extended Passive Mode (|||1584|)

As soon as I enter the EPSV command, I get the following
conntrackd segfault:

Jul  5 00:41:06 sen-fw1 kernel: [274422.060695] conntrackd[4821]: segfault at 0 ip 000000000040c660 sp 00007fffebb098a8 error 4 in conntrackd[400000+3d000]

I am using a Fedora 17 3.7.3-101.fc17.x86_64 kernel with
conntrack-tools-1.4.0-1.fc17.x86_64.

I had to use the attached patch to get "conntrackd -R" to resync
both IPv4 and IPv6 (enabled with a "Family IPv4-IPv6" entry in
conntrackd.conf).  It works well for me for the basic ct table,
but I'm not sure about the expect table part since I can't really
exercise it due to the segfault.  Note the segfault also occurs
with the original unpatched conntrackd, so it's not related to
my patch.

Any help would be greatly appreciated.

					-Thanks

					-Bill

P.S.  I am not subscribed to either netfilter or netfilter-devel.



Patch to add IPv6 to "conntrackd -R":
------------------------------------------------------------------------

diff -Nurp conntrack-tools-1.4.0.orig/src/netlink.c conntrack-tools-1.4.0/src/netlink.c
--- conntrack-tools-1.4.0.orig/src/netlink.c	2012-09-21 10:06:07.000000000 -0400
+++ conntrack-tools-1.4.0/src/netlink.c	2013-07-04 23:32:36.302310719 -0400
@@ -148,7 +148,16 @@ void nl_resize_socket_buffer(struct nfct
 
 int nl_dump_conntrack_table(struct nfct_handle *h)
 {
-	return nfct_query(h, NFCT_Q_DUMP, &CONFIG(family));
+	int fam, ret;
+
+	if (!CONFIG(both_ipv4_ipv6))
+		return nfct_query(h, NFCT_Q_DUMP, &CONFIG(family));
+	fam = AF_INET;
+	ret = nfct_query(h, NFCT_Q_DUMP, &fam);
+	if (ret < 0)
+		return ret;
+	fam = AF_INET6;
+	return nfct_query(h, NFCT_Q_DUMP, &fam);
 }
 
 static int
@@ -380,7 +389,16 @@ int nl_get_expect(struct nfct_handle *h,
 
 int nl_dump_expect_table(struct nfct_handle *h)
 {
-	return nfexp_query(h, NFCT_Q_DUMP, &CONFIG(family));
+	int fam, ret;
+
+	if (!CONFIG(both_ipv4_ipv6))
+		return nfexp_query(h, NFCT_Q_DUMP, &CONFIG(family));
+	fam = AF_INET;
+	ret = nfexp_query(h, NFCT_Q_DUMP, &fam);
+	if (ret < 0)
+		return ret;
+	fam = AF_INET6;
+	return nfexp_query(h, NFCT_Q_DUMP, &fam);
 }
 
 int nl_flush_expect_table(struct nfct_handle *h)
diff -Nurp conntrack-tools-1.4.0.orig/src/read_config_yy.y conntrack-tools-1.4.0/src/read_config_yy.y
--- conntrack-tools-1.4.0.orig/src/read_config_yy.y	2012-09-21 10:06:07.000000000 -0400
+++ conntrack-tools-1.4.0/src/read_config_yy.y	2013-03-20 18:47:36.391160857 -0400
@@ -1193,10 +1193,27 @@ scheduler_line : T_PRIO T_NUMBER
 
 family : T_FAMILY T_STRING
 {
-	if (strncmp($2, "IPv6", strlen("IPv6")) == 0)
+	if (strncmp($2, "IPv6-IPv4", strlen("IPv6-IPv4")) == 0) {
 		conf.family = AF_INET6;
-	else
+		conf.both_ipv4_ipv6 = 1;
+	}
+	else if (strncmp($2, "IPv6", strlen("IPv6")) == 0) {
+		conf.family = AF_INET6;
+		conf.both_ipv4_ipv6 = 0;
+	}
+	else if (strncmp($2, "IPv4-IPv6", strlen("IPv4-IPv6")) == 0) {
+		conf.family = AF_INET;
+		conf.both_ipv4_ipv6 = 1;
+	}
+	else if (strncmp($2, "IPv4", strlen("IPv4")) == 0) {
 		conf.family = AF_INET;
+		conf.both_ipv4_ipv6 = 0;
+	}
+	else {
+		print_err(CTD_CFG_WARN, "%s is not a valid Family, "
+					"ignoring", $2);
+		break;
+	}
 };
 
 event_iterations_limit : T_EVENT_ITER_LIMIT T_NUMBER
@@ -1864,8 +1881,10 @@ init_config(char *filename)
 	fclose(fp);
 
 	/* default to IPv4 */
-	if (CONFIG(family) == 0)
+	if (CONFIG(family) == 0) {
 		CONFIG(family) = AF_INET;
+		CONFIG(both_ipv4_ipv6) = 0;
+	}
 
 	/* set to default is not specified */
 	if (strcmp(CONFIG(lockfile), "") == 0)

Re: conntrackd segfault on EPSV IPv6 ftp command when using ftp ExpectationSync

[Florian Westphal]

Bill Fink <billfink@mindspring.com> wrote:
> 230 Anonymous login ok, restrictions apply.
> EPSV
> 229 Entering Extended Passive Mode (|||1584|)
> 
> As soon as I enter the EPSV command, I get the following
> conntrackd segfault:
> 
> Jul  5 00:41:06 sen-fw1 kernel: [274422.060695] conntrackd[4821]: segfault at 0 ip 000000000040c660 sp 00007fffebb098a8 error 4 in conntrackd[400000+3d000]

#0  0x000000000040f217 in jhash2 (k=0x0, length=4, initval=0) at
../include/jhash.h:99
99                      a += k[0];
(gdb) bt f
#0  0x000000000040f217 in jhash2 (k=0x0, length=4, initval=0) at ../include/jhash.h:99
        a = 2654435769 b = 2654435769 c = 0 len = 4
#1  0x000000000040f564 in ct_filter_hash6 (data=0x0, table=0x16ef630) at filter.c:57
#2  0x000000000040ad34 in hashtable_hash (table=0x16ef630, data=0x0) at hash.c:63
#3  0x000000000040fd19 in __ct_filter_test_ipv6 (f=0x16eeba0, ct=0x1703760) at filter.c:265
id_src = 51 id_dst = 24051376 src = 0x1703760 dst = 0x0

NULL deref in __ct_filter_test_ipv6.  Doesn't happen for ipv4 because
nfct_get_attr_u32() return 0, but nfct_get_attr() returns NULL instead.

@@ -261,8 +264,8 @@ __ct_filter_test_ipv6(struct ct_filter *f, const
		struct nf_conntrack *ct)
        src = nfct_get_attr(ct, ATTR_ORIG_IPV6_SRC);
        dst = nfct_get_attr(ct, ATTR_REPL_IPV6_SRC);

-       id_src = hashtable_hash(f->h6, src);
-       id_dst = hashtable_hash(f->h6, dst);
+       id_src = src ? hashtable_hash(f->h6, src) : 0;
+       id_dst = dst ? hashtable_hash(f->h6, dst) : 0;


Not sure if this is enough, there are other callers
of nfct_get_attr() that don't check for NULL.

Re: conntrackd segfault on EPSV IPv6 ftp command when using ftp ExpectationSync

[Bill Fink]

[Please Cc: me on replies as I am not subscribed]

Florian,

First, many thanks for the quick fix!

On Fri, 5 Jul 2013, Florian Westphal wrote:

> Bill Fink <billfink@xxxxxxxxxxxxxx> wrote:
> > 230 Anonymous login ok, restrictions apply.
> > EPSV
> > 229 Entering Extended Passive Mode (|||1584|)
> > 
> > As soon as I enter the EPSV command, I get the following
> > conntrackd segfault:
> > 
> > Jul  5 00:41:06 sen-fw1 kernel: [274422.060695] conntrackd[4821]: segfault at 0 ip 000000000040c660 sp 00007fffebb098a8 error 4 in conntrackd[400000+3d000]
> 
> #0  0x000000000040f217 in jhash2 (k=0x0, length=4, initval=0) at
> ../include/jhash.h:99
> 99                      a += k[0];
> (gdb) bt f
> #0  0x000000000040f217 in jhash2 (k=0x0, length=4, initval=0) at ../include/jhash.h:99
>         a = 2654435769 b = 2654435769 c = 0 len = 4
> #1  0x000000000040f564 in ct_filter_hash6 (data=0x0, table=0x16ef630) at filter.c:57
> #2  0x000000000040ad34 in hashtable_hash (table=0x16ef630, data=0x0) at hash.c:63
> #3  0x000000000040fd19 in __ct_filter_test_ipv6 (f=0x16eeba0, ct=0x1703760) at filter.c:265
> id_src = 51 id_dst = 24051376 src = 0x1703760 dst = 0x0
> 
> NULL deref in __ct_filter_test_ipv6.  Doesn't happen for ipv4 because
> nfct_get_attr_u32() return 0, but nfct_get_attr() returns NULL instead.
> 
> @@ -261,8 +264,8 @@ __ct_filter_test_ipv6(struct ct_filter *f, const
> 		struct nf_conntrack *ct)
>         src = nfct_get_attr(ct, ATTR_ORIG_IPV6_SRC);
>         dst = nfct_get_attr(ct, ATTR_REPL_IPV6_SRC);
> 
> -       id_src = hashtable_hash(f->h6, src);
> -       id_dst = hashtable_hash(f->h6, dst);
> +       id_src = src ? hashtable_hash(f->h6, src) : 0;
> +       id_dst = dst ? hashtable_hash(f->h6, dst) : 0;
> 
> 
> Not sure if this is enough, there are other callers
> of nfct_get_attr() that don't check for NULL.

This cured my immediate problem.  conntrackd no longer segfaults
and I now get IPv6 expectations.

[root@sen-fw1 ~]# conntrackd -i expect
proto=6 src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy sport=0 dport=23046 mask-src=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff mask-dst=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff sport=0 dport=65535 master-src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx master-dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy sport=38142 dport=21 class=0 helper=ftp [active since 44s]

I will now continue further testing.

I did need my patch to successfully resync the IPv6 expectations
from the kernel via "conntrackd -R" after flushing the conntrackd
cache via "conntrackd -f".

I guess I should submit my patch as an RFC patch to get
comments on it.

					-Thanks

					-Bill

Re: conntrackd segfault on EPSV IPv6 ftp command when using ftp ExpectationSync

[Bill Fink]

On Fri, 5 Jul 2013, Bill Fink wrote:

> [Please Cc: me on replies as I am not subscribed]
> 
> On Fri, 5 Jul 2013, Florian Westphal wrote:
> 
> > Bill Fink <billfink@xxxxxxxxxxxxxx> wrote:
> > > 230 Anonymous login ok, restrictions apply.
> > > EPSV
> > > 229 Entering Extended Passive Mode (|||1584|)
> > > 
> > > As soon as I enter the EPSV command, I get the following
> > > conntrackd segfault:
> > > 
> > > Jul  5 00:41:06 sen-fw1 kernel: [274422.060695] conntrackd[4821]: segfault at 0 ip 000000000040c660 sp 00007fffebb098a8 error 4 in conntrackd[400000+3d000]
> > 
> > #0  0x000000000040f217 in jhash2 (k=0x0, length=4, initval=0) at
> > ../include/jhash.h:99
> > 99                      a += k[0];
> > (gdb) bt f
> > #0  0x000000000040f217 in jhash2 (k=0x0, length=4, initval=0) at ../include/jhash.h:99
> >         a = 2654435769 b = 2654435769 c = 0 len = 4
> > #1  0x000000000040f564 in ct_filter_hash6 (data=0x0, table=0x16ef630) at filter.c:57
> > #2  0x000000000040ad34 in hashtable_hash (table=0x16ef630, data=0x0) at hash.c:63
> > #3  0x000000000040fd19 in __ct_filter_test_ipv6 (f=0x16eeba0, ct=0x1703760) at filter.c:265
> > id_src = 51 id_dst = 24051376 src = 0x1703760 dst = 0x0
> > 
> > NULL deref in __ct_filter_test_ipv6.  Doesn't happen for ipv4 because
> > nfct_get_attr_u32() return 0, but nfct_get_attr() returns NULL instead.
> > 
> > @@ -261,8 +264,8 @@ __ct_filter_test_ipv6(struct ct_filter *f, const
> > 		struct nf_conntrack *ct)
> >         src = nfct_get_attr(ct, ATTR_ORIG_IPV6_SRC);
> >         dst = nfct_get_attr(ct, ATTR_REPL_IPV6_SRC);
> > 
> > -       id_src = hashtable_hash(f->h6, src);
> > -       id_dst = hashtable_hash(f->h6, dst);
> > +       id_src = src ? hashtable_hash(f->h6, src) : 0;
> > +       id_dst = dst ? hashtable_hash(f->h6, dst) : 0;
> > 
> > 
> > Not sure if this is enough, there are other callers
> > of nfct_get_attr() that don't check for NULL.
> 
> This cured my immediate problem.  conntrackd no longer segfaults
> and I now get IPv6 expectations.
> 
> [root@sen-fw1 ~]# conntrackd -i expect
> proto=6 src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy sport=0 dport=23046 mask-src=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff mask-dst=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff sport=0 dport=65535 master-src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx master-dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy sport=38142 dport=21 class=0 helper=ftp [active since 44s]
> 
> I will now continue further testing.

While definitely making progress, the next problem is that while
the active firewall sees the IPv6 ftp expectation, it is not
successfully synced to the backup firewall, and the following
error appears in the conntrackd.log on the backup firewall:

[Fri Jul  5 16:28:50 2013] (pid=5128) [ERROR] inject-add2: Invalid argument
Fri Jul  5 16:28:50 2013	300 proto=6 src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy sport=0 dport=11645 mask-src=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff mask-dst=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff sport=0 dport=65535 master-src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx master-dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy sport=46231 dport=21 class=0 helper=ftp

I don't see anything wrong with the above, which matches exactly
the IPv6 ftp expectation seen on the primary firewall:

[root@sen-fw1 ~]# conntrackd -i expect
proto=6 src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy sport=0 dport=11645 mask-src=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff mask-dst=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff sport=0 dport=65535 master-src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx master-dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy sport=46231 dport=21 class=0 helper=ftp [active since 185s]

I started looking at external_inject_exp_new() in external_inject.c,
where the inject-add2 error presumably comes from, but I haven't
gotten too far yet since I'm not that familiar with the code.
Anyone have any ideas about what might be wrong?

						-Thanks

						-Bill

Re: conntrackd segfault on EPSV IPv6 ftp command when using ftp ExpectationSync

[Pablo Neira Ayuso]

Hi,

On Fri, Jul 05, 2013 at 02:03:12AM -0400, Bill Fink wrote:
> [not sure whether to send to netfilter or netfilter-devel,
> so sending to both, but trim replies as appropriate]
> 
> I am trying to use the ftp ExpectationSync capability of conntrackd
> for both IPv4 and IPv6 for connections through a pair of bridged
> firewalls (primary / hot backup).  I have the following config
> snippet in conntrackd.conf:
> 
> 	Options {
> 		ExpectationSync {
> 			ftp
> 			sip
> 			ras	# for H.323
> 			q.931	# for H.323
> 			h.245	# for H.323
> 		}
> 	}
> 
> For IPv4, things work as expected.  But when I try the basic
> analogous IPv6 test to the suggested IPv4 test from the
> documentation:
> 
> x100ssd2% nc 2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx 21
> 220 FTP Server ready.
> USER anonymous
> 331 Anonymous login ok, send your complete email address as your password
> PASS bill@
> 230-
>                 *** Welcome to this anonymous ftp server! ***
>  
>      You are user 1 out of a maximum of 10 authorized anonymous logins.
>      The current time here is Thu Jul 04 23:40:51 2013.
>      If you experience any problems here, contact : root@localhost
>  
>  
> 230 Anonymous login ok, restrictions apply.
> EPSV
> 229 Entering Extended Passive Mode (|||1584|)
> 
> As soon as I enter the EPSV command, I get the following
> conntrackd segfault:
> 
> Jul  5 00:41:06 sen-fw1 kernel: [274422.060695] conntrackd[4821]: segfault at 0 ip 000000000040c660 sp 00007fffebb098a8 error 4 in conntrackd[400000+3d000]

I have pushed this patch to fix this issue.

http://git.netfilter.org/conntrack-tools/commit/?id=479a37a549abf197ce59a4ae1666d8cba80fe977

Thanks Florian for diagnosing this, and you for reporting.

> I am using a Fedora 17 3.7.3-101.fc17.x86_64 kernel with
> conntrack-tools-1.4.0-1.fc17.x86_64.
> 
> I had to use the attached patch to get "conntrackd -R" to resync
> both IPv4 and IPv6 (enabled with a "Family IPv4-IPv6" entry in
> conntrackd.conf).  It works well for me for the basic ct table,
> but I'm not sure about the expect table part since I can't really
> exercise it due to the segfault.  Note the segfault also occurs
> with the original unpatched conntrackd, so it's not related to
> my patch.

For this, I have applied the following patch:

http://git.netfilter.org/conntrack-tools/commit/?id=e2c6576e775652c35d336afa0551676339c6a793

Let me know.

Re: conntrackd segfault on EPSV IPv6 ftp command when using ftp ExpectationSync

[Bill Fink]

On Sat, 6 Jul 2013, Pablo Neira Ayuso wrote:

> On Fri, Jul 05, 2013 at 02:03:12AM -0400, Bill Fink wrote:
> > [not sure whether to send to netfilter or netfilter-devel,
> > so sending to both, but trim replies as appropriate]
> > 
> > I am trying to use the ftp ExpectationSync capability of conntrackd
> > for both IPv4 and IPv6 for connections through a pair of bridged
> > firewalls (primary / hot backup).  I have the following config
> > snippet in conntrackd.conf:
> > 
> > 	Options {
> > 		ExpectationSync {
> > 			ftp
> > 			sip
> > 			ras	# for H.323
> > 			q.931	# for H.323
> > 			h.245	# for H.323
> > 		}
> > 	}
> > 
> > For IPv4, things work as expected.  But when I try the basic
> > analogous IPv6 test to the suggested IPv4 test from the
> > documentation:
> > 
> > x100ssd2% nc 2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx 21
> > 220 FTP Server ready.
> > USER anonymous
> > 331 Anonymous login ok, send your complete email address as your password
> > PASS bill@
> > 230-
> >                 *** Welcome to this anonymous ftp server! ***
> >  
> >      You are user 1 out of a maximum of 10 authorized anonymous logins.
> >      The current time here is Thu Jul 04 23:40:51 2013.
> >      If you experience any problems here, contact : root@localhost
> >  
> >  
> > 230 Anonymous login ok, restrictions apply.
> > EPSV
> > 229 Entering Extended Passive Mode (|||1584|)
> > 
> > As soon as I enter the EPSV command, I get the following
> > conntrackd segfault:
> > 
> > Jul  5 00:41:06 sen-fw1 kernel: [274422.060695] conntrackd[4821]: segfault at 0 ip 000000000040c660 sp 00007fffebb098a8 error 4 in conntrackd[400000+3d000]
> 
> I have pushed this patch to fix this issue.
> 
> http://git.netfilter.org/conntrack-tools/commit/?id=479a37a549abf197ce59a4ae1666d8cba80fe977
> 
> Thanks Florian for diagnosing this, and you for reporting.

Thanks!  I have tested this and it does fix the segfault.

> > I am using a Fedora 17 3.7.3-101.fc17.x86_64 kernel with
> > conntrack-tools-1.4.0-1.fc17.x86_64.
> > 
> > I had to use the attached patch to get "conntrackd -R" to resync
> > both IPv4 and IPv6 (enabled with a "Family IPv4-IPv6" entry in
> > conntrackd.conf).  It works well for me for the basic ct table,
> > but I'm not sure about the expect table part since I can't really
> > exercise it due to the segfault.  Note the segfault also occurs
> > with the original unpatched conntrackd, so it's not related to
> > my patch.
> 
> For this, I have applied the following patch:
> 
> http://git.netfilter.org/conntrack-tools/commit/?id=e2c6576e775652c35d336afa0551676339c6a793

I also tested this and it fixes the IPv6 kernel resync issue.

> Let me know.

I still have the remaining problem that the IPv6 expectation
is not successfully synced from the primary firewall to the
backup firewall.  I see the following error in conntrackd.log
on the backup firewall:

[Sun Jul  7 01:56:38 2013] (pid=24763) [ERROR] inject-add2: Invalid argument
Sun Jul  7 01:56:38 2013	300 proto=6 src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy sport=0 dport=39767 mask-src=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff mask-dst=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff sport=0 dport=65535 master-src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx master-dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy sport=37484 dport=21 class=0 helper=ftp

This exactly matches the IPv6 expectation on the primary firewall:

[root@sen-fw1 ~]# conntrackd -i expect
proto=6 src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy sport=0 dport=39767 mask-src=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff mask-dst=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff sport=0 dport=65535 master-src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx master-dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy sport=37484 dport=21 class=0 helper=ftp [active since 9s]

IPv4 expectations are working fine.

I tried to track down the error, and followed the error path:

	external_inject_exp_new() ->
	  nl_create_expect()->
	    nfexp_query() ->
	      nfnl_query() ->
		nfnl_catch() ->
		  nfnl_process() ->
		    nfnl_step() ->
		      nfnl_is_error() because
			nlh->nlmsg_type == NLMSG_ERROR

but I wasn't sure how to proceed further.

					-Thanks

					-Bill

Re: conntrackd segfault on EPSV IPv6 ftp command when using ftp ExpectationSync

[Bill Fink]

On Sun, 7 Jul 2013, Bill Fink wrote:

> I still have the remaining problem that the IPv6 expectation
> is not successfully synced from the primary firewall to the
> backup firewall.  I see the following error in conntrackd.log
> on the backup firewall:
> 
> [Sun Jul  7 01:56:38 2013] (pid=24763) [ERROR] inject-add2: Invalid argument
> Sun Jul  7 01:56:38 2013	300 proto=6 src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy sport=0 dport=39767 mask-src=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff mask-dst=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff sport=0 dport=65535 master-src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx master-dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy sport=37484 dport=21 class=0 helper=ftp
> 
> This exactly matches the IPv6 expectation on the primary firewall:
> 
> [root@sen-fw1 ~]# conntrackd -i expect
> proto=6 src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy sport=0 dport=39767 mask-src=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff mask-dst=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff sport=0 dport=65535 master-src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx master-dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy sport=37484 dport=21 class=0 helper=ftp [active since 9s]
> 
> IPv4 expectations are working fine.
> 
> I tried to track down the error, and followed the error path:
> 
> 	external_inject_exp_new() ->
> 	  nl_create_expect()->
> 	    nfexp_query() ->
> 	      nfnl_query() ->
> 		nfnl_catch() ->
> 		  nfnl_process() ->
> 		    nfnl_step() ->
> 		      nfnl_is_error() because
> 			nlh->nlmsg_type == NLMSG_ERROR
> 
> but I wasn't sure how to proceed further.

I made some more progress, but still haven't found the root
cause.  I put some printks in the nf_conntrack_netlink and
nf_conntrack_ipv6 modules and tracked the error path there:

	ctnetlink_new_expect() ->
	  ctnetlink_create_expect() ->
	    ctnetlink_parse_expect_nat() ->
	      ctnetlink_parse_tuple() ->
		ctnetlink_parse_tuple_ip() ->
		  l3proto->nlattr_to_tuple() ->
		    ipv6_nlattr_to_tuple() which fails

because tb[CTA_IP_V6_SRC] and tb[CTA_IP_V6_DST] are null.

But I'm currently stuck there.  Where are tb[CTA_IP_V6_SRC]
and tb[CTA_IP_V6_DST] supposed to be set?  Is it in conntrackd
during the build of the netlink message, or somewhere in one
of the conntrack kernel modules during processing of the
netlink message?  Is there a map somewhere of the control
flow betwwen conntrackd and the conntrack kernel modules?

I was also a little confused by the call to ctnetlink_parse_expect_nat()
since I'm not using NAT.

					-Thanks

					-Bill

Re: conntrackd segfault on EPSV IPv6 ftp command when using ftp ExpectationSync

[Pablo Neira Ayuso]

[-- Attachment #1: Type: text/plain, Size: 2845 bytes --]

On Tue, Jul 09, 2013 at 01:30:23AM -0400, Bill Fink wrote:
> On Sun, 7 Jul 2013, Bill Fink wrote:
> 
> > I still have the remaining problem that the IPv6 expectation
> > is not successfully synced from the primary firewall to the
> > backup firewall.  I see the following error in conntrackd.log
> > on the backup firewall:
> > 
> > [Sun Jul  7 01:56:38 2013] (pid=24763) [ERROR] inject-add2: Invalid argument
> > Sun Jul  7 01:56:38 2013	300 proto=6 src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy sport=0 dport=39767 mask-src=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff mask-dst=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff sport=0 dport=65535 master-src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx master-dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy sport=37484 dport=21 class=0 helper=ftp
> > 
> > This exactly matches the IPv6 expectation on the primary firewall:
> > 
> > [root@sen-fw1 ~]# conntrackd -i expect
> > proto=6 src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy sport=0 dport=39767 mask-src=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff mask-dst=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff sport=0 dport=65535 master-src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx master-dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy sport=37484 dport=21 class=0 helper=ftp [active since 9s]
> > 
> > IPv4 expectations are working fine.
> > 
> > I tried to track down the error, and followed the error path:
> > 
> > 	external_inject_exp_new() ->
> > 	  nl_create_expect()->
> > 	    nfexp_query() ->
> > 	      nfnl_query() ->
> > 		nfnl_catch() ->
> > 		  nfnl_process() ->
> > 		    nfnl_step() ->
> > 		      nfnl_is_error() because
> > 			nlh->nlmsg_type == NLMSG_ERROR
> > 
> > but I wasn't sure how to proceed further.
> 
> I made some more progress, but still haven't found the root
> cause.  I put some printks in the nf_conntrack_netlink and
> nf_conntrack_ipv6 modules and tracked the error path there:
> 
> 	ctnetlink_new_expect() ->
> 	  ctnetlink_create_expect() ->
> 	    ctnetlink_parse_expect_nat() ->
> 	      ctnetlink_parse_tuple() ->
> 		ctnetlink_parse_tuple_ip() ->
> 		  l3proto->nlattr_to_tuple() ->
> 		    ipv6_nlattr_to_tuple() which fails
> 
> because tb[CTA_IP_V6_SRC] and tb[CTA_IP_V6_DST] are null.
> 
> But I'm currently stuck there.  Where are tb[CTA_IP_V6_SRC]
> and tb[CTA_IP_V6_DST] supposed to be set?  Is it in conntrackd
> during the build of the netlink message, or somewhere in one
> of the conntrack kernel modules during processing of the
> netlink message?  Is there a map somewhere of the control
> flow betwwen conntrackd and the conntrack kernel modules?
> 
> I was also a little confused by the call to ctnetlink_parse_expect_nat()
> since I'm not using NAT.

That's a good clue, thanks. Please, give a try to the attached kernel
patch.

Regards.

[-- Attachment #2: 0001-netfilter-ctnetlink-fix-incorrect-NAT-expectation-du.patch --]
[-- Type: text/x-diff, Size: 1269 bytes --]

>From 08fd3987afa49e092d3165a1b965a7a466cd3dc2 Mon Sep 17 00:00:00 2001
From: Pablo Neira Ayuso <pablo@netfilter.org>
Date: Tue, 9 Jul 2013 20:16:39 +0200
Subject: [PATCH] netfilter: ctnetlink: fix incorrect NAT expectation dumping

nf_ct_expect_alloc leaves unset the expectation NAT fields. However,
ctnetlink_exp_dump_expect expects them to be zeroed in case they are
not used, which may not be the case. This results in dumping the NAT
tuple of the expectation when it should not.

Fix it by zeroing the NAT fields of the expectation.

Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
---
 net/netfilter/nf_conntrack_expect.c |    5 +++++
 1 file changed, 5 insertions(+)

diff --git a/net/netfilter/nf_conntrack_expect.c b/net/netfilter/nf_conntrack_expect.c
index c63b618..4fd1ca9 100644
--- a/net/netfilter/nf_conntrack_expect.c
+++ b/net/netfilter/nf_conntrack_expect.c
@@ -293,6 +293,11 @@ void nf_ct_expect_init(struct nf_conntrack_expect *exp, unsigned int class,
 		       sizeof(exp->tuple.dst.u3) - len);
 
 	exp->tuple.dst.u.all = *dst;
+
+#ifdef CONFIG_NF_NAT_NEEDED
+	memset(&exp->saved_addr, 0, sizeof(exp->saved_addr));
+	memset(&exp->saved_proto, 0, sizeof(exp->saved_proto));
+#endif
 }
 EXPORT_SYMBOL_GPL(nf_ct_expect_init);
 
-- 
1.7.10.4

Re: conntrackd segfault on EPSV IPv6 ftp command when using ftp ExpectationSync

[Bill Fink]

On Tue, 9 Jul 2013, Pablo Neira Ayuso wrote:

> On Tue, Jul 09, 2013 at 01:30:23AM -0400, Bill Fink wrote:
> > On Sun, 7 Jul 2013, Bill Fink wrote:
> > 
> > > I still have the remaining problem that the IPv6 expectation
> > > is not successfully synced from the primary firewall to the
> > > backup firewall.  I see the following error in conntrackd.log
> > > on the backup firewall:
> > > 
> > > [Sun Jul  7 01:56:38 2013] (pid=24763) [ERROR] inject-add2: Invalid argument
> > > Sun Jul  7 01:56:38 2013	300 proto=6 src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy sport=0 dport=39767 mask-src=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff mask-dst=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff sport=0 dport=65535 master-src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx master-dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy sport=37484 dport=21 class=0 helper=ftp
> > > 
> > > This exactly matches the IPv6 expectation on the primary firewall:
> > > 
> > > [root@sen-fw1 ~]# conntrackd -i expect
> > > proto=6 src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy sport=0 dport=39767 mask-src=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff mask-dst=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff sport=0 dport=65535 master-src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx master-dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy sport=37484 dport=21 class=0 helper=ftp [active since 9s]
> > > 
> > > IPv4 expectations are working fine.
> > > 
> > > I tried to track down the error, and followed the error path:
> > > 
> > > 	external_inject_exp_new() ->
> > > 	  nl_create_expect()->
> > > 	    nfexp_query() ->
> > > 	      nfnl_query() ->
> > > 		nfnl_catch() ->
> > > 		  nfnl_process() ->
> > > 		    nfnl_step() ->
> > > 		      nfnl_is_error() because
> > > 			nlh->nlmsg_type == NLMSG_ERROR
> > > 
> > > but I wasn't sure how to proceed further.
> > 
> > I made some more progress, but still haven't found the root
> > cause.  I put some printks in the nf_conntrack_netlink and
> > nf_conntrack_ipv6 modules and tracked the error path there:
> > 
> > 	ctnetlink_new_expect() ->
> > 	  ctnetlink_create_expect() ->
> > 	    ctnetlink_parse_expect_nat() ->
> > 	      ctnetlink_parse_tuple() ->
> > 		ctnetlink_parse_tuple_ip() ->
> > 		  l3proto->nlattr_to_tuple() ->
> > 		    ipv6_nlattr_to_tuple() which fails
> > 
> > because tb[CTA_IP_V6_SRC] and tb[CTA_IP_V6_DST] are null.
> > 
> > But I'm currently stuck there.  Where are tb[CTA_IP_V6_SRC]
> > and tb[CTA_IP_V6_DST] supposed to be set?  Is it in conntrackd
> > during the build of the netlink message, or somewhere in one
> > of the conntrack kernel modules during processing of the
> > netlink message?  Is there a map somewhere of the control
> > flow betwwen conntrackd and the conntrack kernel modules?
> > 
> > I was also a little confused by the call to ctnetlink_parse_expect_nat()
> > since I'm not using NAT.
> 
> That's a good clue, thanks. Please, give a try to the attached kernel
> patch.

>From 08fd3987afa49e092d3165a1b965a7a466cd3dc2 Mon Sep 17 00:00:00 2001
From: Pablo Neira Ayuso <pablo@netfilter.org>
Date: Tue, 9 Jul 2013 20:16:39 +0200
Subject: [PATCH] netfilter: ctnetlink: fix incorrect NAT expectation dumping

nf_ct_expect_alloc leaves unset the expectation NAT fields. However,
ctnetlink_exp_dump_expect expects them to be zeroed in case they are
not used, which may not be the case. This results in dumping the NAT
tuple of the expectation when it should not.

Fix it by zeroing the NAT fields of the expectation.

Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
---
 net/netfilter/nf_conntrack_expect.c |    5 +++++
 1 file changed, 5 insertions(+)

diff --git a/net/netfilter/nf_conntrack_expect.c b/net/netfilter/nf_conntrack_expect.c
index c63b618..4fd1ca9 100644
--- a/net/netfilter/nf_conntrack_expect.c
+++ b/net/netfilter/nf_conntrack_expect.c
@@ -293,6 +293,11 @@ void nf_ct_expect_init(struct nf_conntrack_expect *exp, unsigned int class,
 		       sizeof(exp->tuple.dst.u3) - len);
 
 	exp->tuple.dst.u.all = *dst;
+
+#ifdef CONFIG_NF_NAT_NEEDED
+	memset(&exp->saved_addr, 0, sizeof(exp->saved_addr));
+	memset(&exp->saved_proto, 0, sizeof(exp->saved_proto));
+#endif
 }
 EXPORT_SYMBOL_GPL(nf_ct_expect_init);
 
Almost there.  With the above patch, I now successfully get
IPv6 expectations on the backup firewall.  Unfortunately they're
not quite right.  On the backup firewall, the expectation src-IP
is the same as the dst-IP (either IPv4 or IPv6).

Primary firewall:

[root@sen-fw1 linux-3.7.3-101.fc17.x86_64]# conntrack -L expect
251 proto=6 src=192.168.218.199 dst=192.168.28.198 sport=0 dport=54705 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=192.168.218.199 master-dst=192.168.28.198 sport=56877 dport=21 class=0 helper=ftp
conntrack v1.4.0 (conntrack-tools): 1 expectations have been shown.

Backup firewall:

[root@sen-fw2 linux-3.7.3-101.fc17.x86_64]# conntrack -L expect
245 proto=6 src=192.168.28.198 dst=192.168.28.198 sport=0 dport=54705 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=192.168.218.199 master-dst=192.168.28.198 sport=56877 dport=21 class=0 helper=ftp
conntrack v1.4.0 (conntrack-tools): 1 expectations have been shown.

This was an unfortunate side affect of the patch to fix the
conntrackd segfault problem.  If I use Florian's version
of the fix segfault patch rather than Pablo's then all is
good.

Primary firewall:

[root@sen-fw1 linux-3.7.3-101.fc17.x86_64]# conntrack -L expect
270 proto=6 src=192.168.218.199 dst=192.168.28.198 sport=0 dport=58224 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=192.168.218.199 master-dst=192.168.28.198 sport=52814 dport=21 class=0 helper=ftp
conntrack v1.4.0 (conntrack-tools): 1 expectations have been shown.

Backup firewall:

[root@sen-fw2 linux-3.7.3-101.fc17.x86_64]# conntrack -L expect
262 proto=6 src=192.168.218.199 dst=192.168.28.198 sport=0 dport=58224 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=192.168.218.199 master-dst=192.168.28.198 sport=52814 dport=21 class=0 helper=ftp
conntrack v1.4.0 (conntrack-tools): 1 expectations have been shown.

Primary firewall:

[root@sen-fw1 linux-3.7.3-101.fc17.x86_64]# conntrack -f ipv6 -L expect
285 proto=6 src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:b8 dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:bf sport=0 dport=13031 mask-src=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff mask-dst=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff sport=0 dport=65535 master-src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:b8 master-dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:bf sport=56784 dport=21 class=0 helper=ftp
conntrack v1.4.0 (conntrack-tools): 1 expectations have been shown.

Backup firewall:

[root@sen-fw2 linux-3.7.3-101.fc17.x86_64]# conntrack -f ipv6 -L expect
270 proto=6 src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:b8 dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:bf sport=0 dport=13031 mask-src=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff mask-dst=ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff sport=0 dport=65535 master-src=2001:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:b8 master-dst=2001:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:bf sport=56784 dport=21 class=0 helper=ftp
conntrack v1.4.0 (conntrack-tools): 1 expectations have been shown.

Primary firewall and backup firewall expectations exactly match
(both IPv4 and IPv6).

						-Thanks again

						-Bill

Re: conntrackd segfault on EPSV IPv6 ftp command when using ftp ExpectationSync

[Pablo Neira Ayuso]

[-- Attachment #1: Type: text/plain, Size: 1502 bytes --]

On Wed, Jul 10, 2013 at 05:58:15AM -0400, Bill Fink wrote:
> Almost there.  With the above patch, I now successfully get
> IPv6 expectations on the backup firewall.  Unfortunately they're
> not quite right.  On the backup firewall, the expectation src-IP
> is the same as the dst-IP (either IPv4 or IPv6).
> 
> Primary firewall:
> 
> [root@sen-fw1 linux-3.7.3-101.fc17.x86_64]# conntrack -L expect
> 251 proto=6 src=192.168.218.199 dst=192.168.28.198 sport=0 dport=54705 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=192.168.218.199 master-dst=192.168.28.198 sport=56877 dport=21 class=0 helper=ftp
> conntrack v1.4.0 (conntrack-tools): 1 expectations have been shown.
> 
> Backup firewall:
> 
> [root@sen-fw2 linux-3.7.3-101.fc17.x86_64]# conntrack -L expect
> 245 proto=6 src=192.168.28.198 dst=192.168.28.198 sport=0 dport=54705 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=192.168.218.199 master-dst=192.168.28.198 sport=56877 dport=21 class=0 helper=ftp
> conntrack v1.4.0 (conntrack-tools): 1 expectations have been shown.
> 
> This was an unfortunate side affect of the patch to fix the
> conntrackd segfault problem.  If I use Florian's version
> of the fix segfault patch rather than Pablo's then all is
> good.

Thanks for the information, however, we still need to get working back
the filtering support.

Could you give a try to the following patch, please?

It applies on top of conntrack-tools master branch, thanks.

[-- Attachment #2: 0001-conntrackd-add-specific-filtering-for-expectations.patch --]
[-- Type: text/x-diff, Size: 8971 bytes --]

>From 964b9a11a15f27dc4393d07dd9971f65c9fe998c Mon Sep 17 00:00:00 2001
From: Pablo Neira Ayuso <pablo@netfilter.org>
Date: Wed, 10 Jul 2013 23:35:22 +0200
Subject: [PATCH] conntrackd: add specific filtering for expectations

This patch adds exp_filter_conntrack, that allows us to filter expectations
based on its master conntrack. Previously, the daemon was relying on the
existing ct_filter_conntrack, however, we cannot use it since the master
tuple of the expectation has neither reply tuple nor state.

This partially reverts (479a37a conntrackd: fix crash with IPv6 expectation
in the filtering code) as it was incorrectly setting the reply tuple of the
master conntrack.

Reported-by: Bill Fink <billfink@mindspring.com>
Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
---
 include/filter.h |    1 +
 src/ctnl.c       |   37 ++++---------
 src/filter.c     |  154 +++++++++++++++++++++++++++++++++++++++++++++++++++---
 3 files changed, 158 insertions(+), 34 deletions(-)

diff --git a/include/filter.h b/include/filter.h
index 3c7c8cc..e1cf671 100644
--- a/include/filter.h
+++ b/include/filter.h
@@ -51,6 +51,7 @@ void ct_filter_set_logic(struct ct_filter *f,
 			 enum ct_filter_type type,
 			 enum ct_filter_logic logic);
 int ct_filter_conntrack(const struct nf_conntrack *ct, int userspace);
+int exp_filter_conntrack(const struct nf_conntrack *master);
 
 struct exp_filter;
 struct nf_expect;
diff --git a/src/ctnl.c b/src/ctnl.c
index 9e1cfa1..c37c38a 100644
--- a/src/ctnl.c
+++ b/src/ctnl.c
@@ -211,35 +211,14 @@ out:
 		return NFCT_CB_CONTINUE;
 }
 
-static const struct nf_conntrack *exp_get_master_ct(struct nf_expect *exp)
-{
-	struct nf_conntrack *master =
-		(struct nf_conntrack *)nfexp_get_attr(exp, ATTR_EXP_MASTER);
-
-	/* The function ct_filter_conntrack needs the source address of the
-	 * reply tuple, emulate it.
-	 */
-	switch (nfct_get_attr_u8(master, ATTR_L3PROTO)) {
-	case AF_INET:
-		nfct_set_attr_u32(master, ATTR_REPL_IPV4_SRC,
-				  nfct_get_attr_u32(master, ATTR_IPV4_DST));
-		break;
-	case AF_INET6:
-		nfct_set_attr(master, ATTR_REPL_IPV6_SRC,
-			      nfct_get_attr(master, ATTR_IPV6_DST));
-		break;
-	}
-
-	return master;
-}
-
 static int exp_event_handler(const struct nlmsghdr *nlh,
 			     enum nf_conntrack_msg_type type,
 			     struct nf_expect *exp,
 			     void *data)
 {
 	int origin_type;
-	const struct nf_conntrack *master = exp_get_master_ct(exp);
+	const struct nf_conntrack *master =
+		nfexp_get_attr(exp, ATTR_EXP_MASTER);
 
 	STATE(stats).nl_events_received++;
 
@@ -247,7 +226,7 @@ static int exp_event_handler(const struct nlmsghdr *nlh,
 		STATE(stats).nl_events_filtered++;
 		goto out;
 	}
-	if (ct_filter_conntrack(master, 1))
+	if (exp_filter_conntrack(master))
 		return NFCT_CB_CONTINUE;
 
 	origin_type = origin_find(nlh);
@@ -296,12 +275,13 @@ static int dump_handler(enum nf_conntrack_msg_type type,
 static int exp_dump_handler(enum nf_conntrack_msg_type type,
 			    struct nf_expect *exp, void *data)
 {
-	const struct nf_conntrack *master = exp_get_master_ct(exp);
+	const struct nf_conntrack *master =
+		nfexp_get_attr(exp, ATTR_EXP_MASTER);
 
 	if (!exp_filter_find(STATE(exp_filter), exp))
 		return NFCT_CB_CONTINUE;
 
-	if (ct_filter_conntrack(master, 1))
+	if (exp_filter_conntrack(master))
 		return NFCT_CB_CONTINUE;
 
 	switch(type) {
@@ -329,12 +309,13 @@ static int get_handler(enum nf_conntrack_msg_type type,
 static int exp_get_handler(enum nf_conntrack_msg_type type,
 			   struct nf_expect *exp, void *data)
 {
-	const struct nf_conntrack *master = exp_get_master_ct(exp);
+	const struct nf_conntrack *master =
+		nfexp_get_attr(exp, ATTR_EXP_MASTER);
 
 	if (!exp_filter_find(STATE(exp_filter), exp))
 		return NFCT_CB_CONTINUE;
 
-	if (ct_filter_conntrack(master, 1))
+	if (exp_filter_conntrack(master))
 		return NFCT_CB_CONTINUE;
 
 	STATE(get_retval) = 1;
diff --git a/src/filter.c b/src/filter.c
index e21cfde..c160c1a 100644
--- a/src/filter.c
+++ b/src/filter.c
@@ -279,13 +279,10 @@ __ct_filter_test_mask4(const void *ptr, const void *ct)
 		(elem->ip & elem->mask) == (dst & elem->mask));
 }
 
-static int
-__ct_filter_test_mask6(const void *ptr, const void *ct)
+static inline int
+vector_elem_cmp_ipv6(const struct ct_filter_netmask_ipv6 *elem,
+		     const uint32_t *src, const uint32_t *dst)
 {
-	const struct ct_filter_netmask_ipv6 *elem = ptr;
-	const uint32_t *src = nfct_get_attr(ct, ATTR_ORIG_IPV6_SRC);
-	const uint32_t *dst = nfct_get_attr(ct, ATTR_REPL_IPV6_SRC);
-
 	return (((elem->ip[0] & elem->mask[0]) == (src[0] & elem->mask[0]) &&
 		 (elem->ip[1] & elem->mask[1]) == (src[1] & elem->mask[1]) &&
 		 (elem->ip[2] & elem->mask[2]) == (src[2] & elem->mask[2]) &&
@@ -297,6 +294,16 @@ __ct_filter_test_mask6(const void *ptr, const void *ct)
 }
 
 static int
+__ct_filter_test_mask6(const void *ptr, const void *ct)
+{
+	const struct ct_filter_netmask_ipv6 *elem = ptr;
+	const uint32_t *src = nfct_get_attr(ct, ATTR_ORIG_IPV6_SRC);
+	const uint32_t *dst = nfct_get_attr(ct, ATTR_REPL_IPV6_SRC);
+
+	return vector_elem_cmp_ipv6(elem, src, dst);
+}
+
+static int
 __ct_filter_test_state(struct ct_filter *f, const struct nf_conntrack *ct)
 {
 	uint16_t val = 0;
@@ -407,6 +414,141 @@ int ct_filter_conntrack(const struct nf_conntrack *ct, int userspace)
 	return 0;
 }
 
+static int exp_filter_ipv4(const void *ptr, const void *master)
+{
+	const struct ct_filter_netmask_ipv4 *elem = ptr;
+	const uint32_t src = nfct_get_attr_u32(master, ATTR_IPV4_SRC);
+	const uint32_t dst = nfct_get_attr_u32(master, ATTR_IPV4_DST);
+
+	return ((elem->ip & elem->mask) == (src & elem->mask) ||
+		(elem->ip & elem->mask) == (dst & elem->mask));
+}
+
+static int exp_filter_ipv6(const void *ptr, const void *master)
+{
+	const struct ct_filter_netmask_ipv6 *elem = ptr;
+	const uint32_t *src = nfct_get_attr(master, ATTR_IPV6_SRC);
+	const uint32_t *dst = nfct_get_attr(master, ATTR_IPV6_DST);
+
+	return vector_elem_cmp_ipv6(elem, src, dst);
+}
+
+static inline int
+exp_filter_test_ipv4(struct ct_filter *f, const struct nf_conntrack *ct)
+{
+	int id_src, id_dst;
+	uint32_t src, dst;
+
+	src = nfct_get_attr_u32(ct, ATTR_IPV4_SRC);
+	dst = nfct_get_attr_u32(ct, ATTR_IPV4_DST);
+
+	id_src = hashtable_hash(f->h, &src);
+	id_dst = hashtable_hash(f->h, &dst);
+
+	return hashtable_find(f->h, &src, id_src) ||
+	       hashtable_find(f->h, &dst, id_dst);
+}
+
+static inline int
+exp_filter_test_ipv6(struct ct_filter *f, const struct nf_conntrack *ct)
+{
+	int id_src, id_dst;
+	const uint32_t *src, *dst;
+
+	src = nfct_get_attr(ct, ATTR_IPV6_SRC);
+	dst = nfct_get_attr(ct, ATTR_IPV6_DST);
+
+	id_src = hashtable_hash(f->h6, src);
+	id_dst = hashtable_hash(f->h6, dst);
+
+	return hashtable_find(f->h6, src, id_src) ||
+	       hashtable_find(f->h6, dst, id_dst);
+}
+
+static int exp_filter_check(struct ct_filter *f, const struct nf_conntrack *ct)
+{
+	int ret, protonum = nfct_get_attr_u8(ct, ATTR_L4PROTO);
+
+	/* no event filtering at all */
+	if (f == NULL)
+		return 1;
+
+	if (f->logic[CT_FILTER_L4PROTO] != -1) {
+		ret = test_bit_u32(protonum, f->l4protomap);
+		if (ret ^ f->logic[CT_FILTER_L4PROTO])
+			return 0;
+	}
+
+	if (f->logic[CT_FILTER_ADDRESS] != -1) {
+		switch (nfct_get_attr_u8(ct, ATTR_L3PROTO)) {
+		case AF_INET:
+			ret = vector_iterate(f->v, ct, exp_filter_ipv4);
+			if (ret ^ f->logic[CT_FILTER_ADDRESS])
+				return 0;
+			ret = exp_filter_test_ipv4(f, ct);
+			if (ret ^ f->logic[CT_FILTER_ADDRESS])
+				return 0;
+			break;
+		case AF_INET6:
+			ret = vector_iterate(f->v6, ct, exp_filter_ipv6);
+			if (ret ^ f->logic[CT_FILTER_ADDRESS])
+				return 0;
+			ret = exp_filter_test_ipv6(f, ct);
+			if (ret ^ f->logic[CT_FILTER_ADDRESS])
+				return 0;
+			break;
+		default:
+			break;
+		}
+	}
+
+	return 1;
+}
+
+static inline int exp_filter_sanity_check(const struct nf_conntrack *master)
+{
+	if (!nfct_attr_is_set(master, ATTR_L3PROTO)) {
+		dlog(LOG_ERR, "missing layer 3 protocol");
+		return 0;
+	}
+
+	switch (nfct_get_attr_u8(master, ATTR_L3PROTO)) {
+	case AF_INET:
+		if (!nfct_attr_is_set(master, ATTR_IPV4_SRC) ||
+		    !nfct_attr_is_set(master, ATTR_IPV4_DST)) {
+		    	dlog(LOG_ERR, "missing IPv4 address. "
+				      "You forgot to load "
+				      "nf_conntrack_ipv4?");
+			return 0;
+		}
+		break;
+	case AF_INET6:
+		if (!nfct_attr_is_set(master, ATTR_IPV6_SRC) ||
+		    !nfct_attr_is_set(master, ATTR_IPV6_DST)) {
+		    	dlog(LOG_ERR, "missing IPv6 address. "
+				      "You forgot to load "
+				      "nf_conntrack_ipv6?");
+			return 0;
+		}
+		break;
+	}
+	return 1;
+}
+
+int exp_filter_conntrack(const struct nf_conntrack *master)
+{
+	if (!exp_filter_sanity_check(master))
+		return 1;
+
+	/* The master tuple of the expectation has neither reply tuple nor
+	 * state, use specific filtering function for this case.
+	 */
+	if (!exp_filter_check(STATE(us_filter), master))
+		return 1;
+
+	return 0;
+}
+
 struct exp_filter {
 	struct list_head 	list;
 };
-- 
1.7.10.4

Re: conntrackd segfault on EPSV IPv6 ftp command when using ftp ExpectationSync

[Pablo Neira Ayuso]

[-- Attachment #1: Type: text/plain, Size: 1736 bytes --]

On Thu, Jul 11, 2013 at 12:08:20AM +0200, Pablo Neira Ayuso wrote:
> On Wed, Jul 10, 2013 at 05:58:15AM -0400, Bill Fink wrote:
> > Almost there.  With the above patch, I now successfully get
> > IPv6 expectations on the backup firewall.  Unfortunately they're
> > not quite right.  On the backup firewall, the expectation src-IP
> > is the same as the dst-IP (either IPv4 or IPv6).
> > 
> > Primary firewall:
> > 
> > [root@sen-fw1 linux-3.7.3-101.fc17.x86_64]# conntrack -L expect
> > 251 proto=6 src=192.168.218.199 dst=192.168.28.198 sport=0 dport=54705 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=192.168.218.199 master-dst=192.168.28.198 sport=56877 dport=21 class=0 helper=ftp
> > conntrack v1.4.0 (conntrack-tools): 1 expectations have been shown.
> > 
> > Backup firewall:
> > 
> > [root@sen-fw2 linux-3.7.3-101.fc17.x86_64]# conntrack -L expect
> > 245 proto=6 src=192.168.28.198 dst=192.168.28.198 sport=0 dport=54705 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=192.168.218.199 master-dst=192.168.28.198 sport=56877 dport=21 class=0 helper=ftp
> > conntrack v1.4.0 (conntrack-tools): 1 expectations have been shown.
> > 
> > This was an unfortunate side affect of the patch to fix the
> > conntrackd segfault problem.  If I use Florian's version
> > of the fix segfault patch rather than Pablo's then all is
> > good.
> 
> Thanks for the information, however, we still need to get working back
> the filtering support.
> 
> Could you give a try to the following patch, please?
> 
> It applies on top of conntrack-tools master branch, thanks.

There are still some downsides in the previous solution, please, give
a try to this patch instead.

Thanks.

[-- Attachment #2: 0001-conntrackd-simplify-expectation-filtering.patch --]
[-- Type: text/x-diff, Size: 8598 bytes --]

>From 586382d9a8389ee553db019fd9be14a8a7c0b8ec Mon Sep 17 00:00:00 2001
From: Pablo Neira Ayuso <pablo@netfilter.org>
Date: Thu, 11 Jul 2013 00:43:20 +0200
Subject: [PATCH] conntrackd: simplify expectation filtering

This patch simplifies the expectation filtering by looking up for the
master conntrack. If it does not exists, then we assume that we don't
want this expectation either.

This simplification also fixes the current broken expectation filtering,
since the master conntrack from expectations has neither reply tuple
nor state, however, the filtering code assumes the opposite.

This partially reverts (479a37a conntrackd: fix crash with IPv6 expectation
in the filtering code) since it was incorrectly setting the reply tuple
of the master conntrack.

Thanks to Bill Fink for providing feedback to resolve this issue.

Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
---
 include/filter.h      |    1 +
 include/internal.h    |    1 +
 src/cache-ct.c        |   11 +++++++++--
 src/ctnl.c            |   37 +++++++++----------------------------
 src/filter.c          |   45 +++++++++++++++++++++++++++++++++++++++++++++
 src/internal_bypass.c |    6 ++++++
 src/internal_cache.c  |   11 +++++++++++
 7 files changed, 82 insertions(+), 30 deletions(-)

diff --git a/include/filter.h b/include/filter.h
index 3c7c8cc..d0acd96 100644
--- a/include/filter.h
+++ b/include/filter.h
@@ -51,6 +51,7 @@ void ct_filter_set_logic(struct ct_filter *f,
 			 enum ct_filter_type type,
 			 enum ct_filter_logic logic);
 int ct_filter_conntrack(const struct nf_conntrack *ct, int userspace);
+int ct_filter_master(const struct nf_conntrack *master);
 
 struct exp_filter;
 struct nf_expect;
diff --git a/include/internal.h b/include/internal.h
index 2ba9714..1a796a7 100644
--- a/include/internal.h
+++ b/include/internal.h
@@ -40,6 +40,7 @@ struct internal_handler {
 		void	(*new)(struct nf_expect *exp, int origin_type);
 		void	(*upd)(struct nf_expect *exp, int origin_type);
 		int	(*del)(struct nf_expect *exp, int origin_type);
+		int	(*find)(const struct nf_conntrack *master);
 
 		void	(*dump)(int fd, int type);
 		void	(*populate)(struct nf_expect *exp);
diff --git a/src/cache-ct.c b/src/cache-ct.c
index a538215..f86d143 100644
--- a/src/cache-ct.c
+++ b/src/cache-ct.c
@@ -88,14 +88,21 @@ cache_ct_hash(const void *data, const struct hashtable *table)
 	return ret;
 }
 
+/* master conntrack of expectations have no ID */
+static inline int
+cache_ct_cmp_id(const struct nf_conntrack *ct1, const struct nf_conntrack *ct2)
+{
+	return nfct_attr_is_set(ct2, ATTR_ID) ?
+	       nfct_get_attr_u32(ct1, ATTR_ID) == nfct_get_attr_u32(ct2, ATTR_ID) : 1;
+}
+
 static int cache_ct_cmp(const void *data1, const void *data2)
 {
 	const struct cache_object *obj = data1;
 	const struct nf_conntrack *ct = data2;
 
 	return nfct_cmp(obj->ptr, ct, NFCT_CMP_ORIG) &&
-	       nfct_get_attr_u32(obj->ptr, ATTR_ID) ==
-	       nfct_get_attr_u32(ct, ATTR_ID);
+	       cache_ct_cmp_id(obj->ptr, ct);
 }
 
 static void *cache_ct_alloc(void)
diff --git a/src/ctnl.c b/src/ctnl.c
index 9e1cfa1..10b5f4c 100644
--- a/src/ctnl.c
+++ b/src/ctnl.c
@@ -211,35 +211,14 @@ out:
 		return NFCT_CB_CONTINUE;
 }
 
-static const struct nf_conntrack *exp_get_master_ct(struct nf_expect *exp)
-{
-	struct nf_conntrack *master =
-		(struct nf_conntrack *)nfexp_get_attr(exp, ATTR_EXP_MASTER);
-
-	/* The function ct_filter_conntrack needs the source address of the
-	 * reply tuple, emulate it.
-	 */
-	switch (nfct_get_attr_u8(master, ATTR_L3PROTO)) {
-	case AF_INET:
-		nfct_set_attr_u32(master, ATTR_REPL_IPV4_SRC,
-				  nfct_get_attr_u32(master, ATTR_IPV4_DST));
-		break;
-	case AF_INET6:
-		nfct_set_attr(master, ATTR_REPL_IPV6_SRC,
-			      nfct_get_attr(master, ATTR_IPV6_DST));
-		break;
-	}
-
-	return master;
-}
-
 static int exp_event_handler(const struct nlmsghdr *nlh,
 			     enum nf_conntrack_msg_type type,
 			     struct nf_expect *exp,
 			     void *data)
 {
 	int origin_type;
-	const struct nf_conntrack *master = exp_get_master_ct(exp);
+	const struct nf_conntrack *master =
+		nfexp_get_attr(exp, ATTR_EXP_MASTER);
 
 	STATE(stats).nl_events_received++;
 
@@ -247,7 +226,7 @@ static int exp_event_handler(const struct nlmsghdr *nlh,
 		STATE(stats).nl_events_filtered++;
 		goto out;
 	}
-	if (ct_filter_conntrack(master, 1))
+	if (ct_filter_master(master))
 		return NFCT_CB_CONTINUE;
 
 	origin_type = origin_find(nlh);
@@ -296,12 +275,13 @@ static int dump_handler(enum nf_conntrack_msg_type type,
 static int exp_dump_handler(enum nf_conntrack_msg_type type,
 			    struct nf_expect *exp, void *data)
 {
-	const struct nf_conntrack *master = exp_get_master_ct(exp);
+	const struct nf_conntrack *master =
+		nfexp_get_attr(exp, ATTR_EXP_MASTER);
 
 	if (!exp_filter_find(STATE(exp_filter), exp))
 		return NFCT_CB_CONTINUE;
 
-	if (ct_filter_conntrack(master, 1))
+	if (ct_filter_master(master))
 		return NFCT_CB_CONTINUE;
 
 	switch(type) {
@@ -329,12 +309,13 @@ static int get_handler(enum nf_conntrack_msg_type type,
 static int exp_get_handler(enum nf_conntrack_msg_type type,
 			   struct nf_expect *exp, void *data)
 {
-	const struct nf_conntrack *master = exp_get_master_ct(exp);
+	const struct nf_conntrack *master =
+		nfexp_get_attr(exp, ATTR_EXP_MASTER);
 
 	if (!exp_filter_find(STATE(exp_filter), exp))
 		return NFCT_CB_CONTINUE;
 
-	if (ct_filter_conntrack(master, 1))
+	if (ct_filter_master(master))
 		return NFCT_CB_CONTINUE;
 
 	STATE(get_retval) = 1;
diff --git a/src/filter.c b/src/filter.c
index e21cfde..8fac71b 100644
--- a/src/filter.c
+++ b/src/filter.c
@@ -407,6 +407,51 @@ int ct_filter_conntrack(const struct nf_conntrack *ct, int userspace)
 	return 0;
 }
 
+static inline int
+ct_filter_master_sanity_check(const struct nf_conntrack *master)
+{
+	if (master == NULL) {
+		dlog(LOG_ERR, "no master tuple in expectation");
+		return 0;
+	}
+
+	if (!nfct_attr_is_set(master, ATTR_L3PROTO)) {
+		dlog(LOG_ERR, "missing layer 3 protocol");
+		return 0;
+	}
+
+	switch (nfct_get_attr_u8(master, ATTR_L3PROTO)) {
+	case AF_INET:
+		if (!nfct_attr_is_set(master, ATTR_IPV4_SRC) ||
+		    !nfct_attr_is_set(master, ATTR_IPV4_DST)) {
+		    	dlog(LOG_ERR, "missing IPv4 address. "
+			     "You forgot to load nf_conntrack_ipv4?");
+			return 0;
+		}
+		break;
+	case AF_INET6:
+		if (!nfct_attr_is_set(master, ATTR_IPV6_SRC) ||
+		    !nfct_attr_is_set(master, ATTR_IPV6_DST)) {
+		    	dlog(LOG_ERR, "missing IPv6 address. "
+			     "You forgot to load nf_conntrack_ipv6?");
+			return 0;
+		}
+		break;
+	}
+	return 1;
+}
+
+int ct_filter_master(const struct nf_conntrack *master)
+{
+	if (!ct_filter_master_sanity_check(master))
+		return 1;
+
+	/* Check if we've got a master conntrack for this expectation in our
+	 * caches. If there is not, we don't want this expectation either.
+	 */
+	return STATE(mode)->internal->exp.find(master) ? 0 : 1;
+}
+
 struct exp_filter {
 	struct list_head 	list;
 };
diff --git a/src/internal_bypass.c b/src/internal_bypass.c
index 1194339..ce2ae46 100644
--- a/src/internal_bypass.c
+++ b/src/internal_bypass.c
@@ -283,6 +283,11 @@ static int internal_bypass_exp_event_del(struct nf_expect *exp, int origin)
 	return 1;
 }
 
+static int internal_bypass_exp_master_find(const struct nf_conntrack *master)
+{
+	return nl_get_conntrack(STATE(get), master) == 0;
+}
+
 struct internal_handler internal_bypass = {
 	.init			= internal_bypass_init,
 	.close			= internal_bypass_close,
@@ -309,5 +314,6 @@ struct internal_handler internal_bypass = {
 		.new			= internal_bypass_exp_event_new,
 		.upd			= internal_bypass_exp_event_upd,
 		.del			= internal_bypass_exp_event_del,
+		.find			= internal_bypass_exp_master_find,
 	},
 };
diff --git a/src/internal_cache.c b/src/internal_cache.c
index ba2d74b..bad31f3 100644
--- a/src/internal_cache.c
+++ b/src/internal_cache.c
@@ -364,6 +364,16 @@ static int internal_cache_exp_event_del(struct nf_expect *exp, int origin)
 	return 1;
 }
 
+static int internal_cache_exp_master_find(const struct nf_conntrack *master)
+{
+	struct cache_object *obj;
+	int id;
+
+	obj = cache_find(STATE(mode)->internal->ct.data,
+			 (struct nf_conntrack *)master, &id);
+	return obj ? 1 : 0;
+}
+
 struct internal_handler internal_cache = {
 	.flags			= INTERNAL_F_POPULATE | INTERNAL_F_RESYNC,
 	.init			= internal_cache_init,
@@ -391,5 +401,6 @@ struct internal_handler internal_cache = {
 		.new			= internal_cache_exp_event_new,
 		.upd			= internal_cache_exp_event_upd,
 		.del			= internal_cache_exp_event_del,
+		.find			= internal_cache_exp_master_find,
 	},
 };
-- 
1.7.10.4

Re: conntrackd segfault on EPSV IPv6 ftp command when using ftp ExpectationSync

[Bill Fink]

On Thu, 11 Jul 2013, Pablo Neira Ayuso wrote:

> On Thu, Jul 11, 2013 at 12:08:20AM +0200, Pablo Neira Ayuso wrote:
> > On Wed, Jul 10, 2013 at 05:58:15AM -0400, Bill Fink wrote:
> > > Almost there.  With the above patch, I now successfully get
> > > IPv6 expectations on the backup firewall.  Unfortunately they're
> > > not quite right.  On the backup firewall, the expectation src-IP
> > > is the same as the dst-IP (either IPv4 or IPv6).
> > > 
> > > Primary firewall:
> > > 
> > > [root@sen-fw1 linux-3.7.3-101.fc17.x86_64]# conntrack -L expect
> > > 251 proto=6 src=192.168.218.199 dst=192.168.28.198 sport=0 dport=54705 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=192.168.218.199 master-dst=192.168.28.198 sport=56877 dport=21 class=0 helper=ftp
> > > conntrack v1.4.0 (conntrack-tools): 1 expectations have been shown.
> > > 
> > > Backup firewall:
> > > 
> > > [root@sen-fw2 linux-3.7.3-101.fc17.x86_64]# conntrack -L expect
> > > 245 proto=6 src=192.168.28.198 dst=192.168.28.198 sport=0 dport=54705 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=192.168.218.199 master-dst=192.168.28.198 sport=56877 dport=21 class=0 helper=ftp
> > > conntrack v1.4.0 (conntrack-tools): 1 expectations have been shown.
> > > 
> > > This was an unfortunate side affect of the patch to fix the
> > > conntrackd segfault problem.  If I use Florian's version
> > > of the fix segfault patch rather than Pablo's then all is
> > > good.
> > 
> > Thanks for the information, however, we still need to get working back
> > the filtering support.
> > 
> > Could you give a try to the following patch, please?
> > 
> > It applies on top of conntrack-tools master branch, thanks.
> 
> There are still some downsides in the previous solution, please, give
> a try to this patch instead.

The firewalls are now in production, so I don't have the same freedom
I did previously.  I'll check the patch out sometime after hours.
Normally, this weekend would be a good time, but I'm going to be
away this weekend.  So it might be a few days until I get a chance.

Thanks again for all your (and Florian's) great help!

						-Bill

Re: conntrackd segfault on EPSV IPv6 ftp command when using ftp ExpectationSync

[Bill Fink]

Pablo,

On Thu, 11 Jul 2013, Pablo Neira Ayuso wrote:

> On Thu, Jul 11, 2013 at 12:08:20AM +0200, Pablo Neira Ayuso wrote:
> > On Wed, Jul 10, 2013 at 05:58:15AM -0400, Bill Fink wrote:
> > > Almost there.  With the above patch, I now successfully get
> > > IPv6 expectations on the backup firewall.  Unfortunately they're
> > > not quite right.  On the backup firewall, the expectation src-IP
> > > is the same as the dst-IP (either IPv4 or IPv6).
> > > 
> > > Primary firewall:
> > > 
> > > [root@sen-fw1 linux-3.7.3-101.fc17.x86_64]# conntrack -L expect
> > > 251 proto=6 src=192.168.218.199 dst=192.168.28.198 sport=0 dport=54705 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=192.168.218.199 master-dst=192.168.28.198 sport=56877 dport=21 class=0 helper=ftp
> > > conntrack v1.4.0 (conntrack-tools): 1 expectations have been shown.
> > > 
> > > Backup firewall:
> > > 
> > > [root@sen-fw2 linux-3.7.3-101.fc17.x86_64]# conntrack -L expect
> > > 245 proto=6 src=192.168.28.198 dst=192.168.28.198 sport=0 dport=54705 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=192.168.218.199 master-dst=192.168.28.198 sport=56877 dport=21 class=0 helper=ftp
> > > conntrack v1.4.0 (conntrack-tools): 1 expectations have been shown.
> > > 
> > > This was an unfortunate side affect of the patch to fix the
> > > conntrackd segfault problem.  If I use Florian's version
> > > of the fix segfault patch rather than Pablo's then all is
> > > good.
> > 
> > Thanks for the information, however, we still need to get working back
> > the filtering support.
> > 
> > Could you give a try to the following patch, please?
> > 
> > It applies on top of conntrack-tools master branch, thanks.
> 
> There are still some downsides in the previous solution, please, give
> a try to this patch instead.

It appears to work pretty well and does fix the src-IP issue!

I did notice a couple of other things but they're not necessarily
related to these patches.  I noticed that my long lived BGP tcp
connection was getting some duplicate conntrackd ct entries (both
IPv4 and IPv6).  The duplicate ct entry occurred 60 seconds after
the original, and once I saw a second duplicate ct IPv4 entry,
again with about a 60 second separation.

And on the expectation entries, they seem to have a lifetime
of 300 seconds.  The IPv6 expectation entries are deleted after
the 300 seconds as expected, but the IPv4 expectation entries
actually go away in a minute or less.  I don't know if that
is expected behavior or not.  Note I was leaving the ftp
control connection open the entire time.  Also it may have
just been my imagination, but it seemed if I queried it more
often with "conntrack -L expect" it would stick around somewhat
longer (but would go away within the minute).

As I mentioned in my previous e-mail, I will be away for the
weekend.

					-Thanks

					-Bill

Re: conntrackd segfault on EPSV IPv6 ftp command when using ftp ExpectationSync

[Pablo Neira Ayuso]

On Fri, Jul 12, 2013 at 03:01:14AM -0400, Bill Fink wrote:
> Pablo,
> 
> On Thu, 11 Jul 2013, Pablo Neira Ayuso wrote:
> 
> > On Thu, Jul 11, 2013 at 12:08:20AM +0200, Pablo Neira Ayuso wrote:
> > > On Wed, Jul 10, 2013 at 05:58:15AM -0400, Bill Fink wrote:
> > > > Almost there.  With the above patch, I now successfully get
> > > > IPv6 expectations on the backup firewall.  Unfortunately they're
> > > > not quite right.  On the backup firewall, the expectation src-IP
> > > > is the same as the dst-IP (either IPv4 or IPv6).
> > > > 
> > > > Primary firewall:
> > > > 
> > > > [root@sen-fw1 linux-3.7.3-101.fc17.x86_64]# conntrack -L expect
> > > > 251 proto=6 src=192.168.218.199 dst=192.168.28.198 sport=0 dport=54705 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=192.168.218.199 master-dst=192.168.28.198 sport=56877 dport=21 class=0 helper=ftp
> > > > conntrack v1.4.0 (conntrack-tools): 1 expectations have been shown.
> > > > 
> > > > Backup firewall:
> > > > 
> > > > [root@sen-fw2 linux-3.7.3-101.fc17.x86_64]# conntrack -L expect
> > > > 245 proto=6 src=192.168.28.198 dst=192.168.28.198 sport=0 dport=54705 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=192.168.218.199 master-dst=192.168.28.198 sport=56877 dport=21 class=0 helper=ftp
> > > > conntrack v1.4.0 (conntrack-tools): 1 expectations have been shown.
> > > > 
> > > > This was an unfortunate side affect of the patch to fix the
> > > > conntrackd segfault problem.  If I use Florian's version
> > > > of the fix segfault patch rather than Pablo's then all is
> > > > good.
> > > 
> > > Thanks for the information, however, we still need to get working back
> > > the filtering support.
> > > 
> > > Could you give a try to the following patch, please?
> > > 
> > > It applies on top of conntrack-tools master branch, thanks.
> > 
> > There are still some downsides in the previous solution, please, give
> > a try to this patch instead.
> 
> It appears to work pretty well and does fix the src-IP issue!

Thanks for testing and reporting back, I have pushed the patch to
master.

> I did notice a couple of other things but they're not necessarily
> related to these patches.  I noticed that my long lived BGP tcp
> connection was getting some duplicate conntrackd ct entries (both
> IPv4 and IPv6).  The duplicate ct entry occurred 60 seconds after
> the original, and once I saw a second duplicate ct IPv4 entry,
> again with about a 60 second separation.

That's strange. Please, tell me if this happening in the primary
and/or the backup. If you are using the cache mode, also dump the
caches via -i and -e to see the content.

> And on the expectation entries, they seem to have a lifetime
> of 300 seconds.  The IPv6 expectation entries are deleted after
> the 300 seconds as expected, but the IPv4 expectation entries
> actually go away in a minute or less.  I don't know if that
> is expected behavior or not.  Note I was leaving the ftp
> control connection open the entire time.  Also it may have
> just been my imagination, but it seemed if I queried it more
> often with "conntrack -L expect" it would stick around somewhat
> longer (but would go away within the minute).

Expectations depends on the master conntrack, if the master is
released, the expectations will be released too. You may be noticing
that effect.

You can use `conntrack -E` and `conntrack -E exp` to verify this.

Regards.

Re: conntrackd segfault on EPSV IPv6 ftp command when using ftp ExpectationSync

[Bill Fink]

On Mon, 15 Jul 2013, Pablo Neira Ayuso wrote:

> On Fri, Jul 12, 2013 at 03:01:14AM -0400, Bill Fink wrote:
> > Pablo,
> > 
> > On Thu, 11 Jul 2013, Pablo Neira Ayuso wrote:
> > 
> > > On Thu, Jul 11, 2013 at 12:08:20AM +0200, Pablo Neira Ayuso wrote:
> > > > On Wed, Jul 10, 2013 at 05:58:15AM -0400, Bill Fink wrote:
> > > > > Almost there.  With the above patch, I now successfully get
> > > > > IPv6 expectations on the backup firewall.  Unfortunately they're
> > > > > not quite right.  On the backup firewall, the expectation src-IP
> > > > > is the same as the dst-IP (either IPv4 or IPv6).
> > > > > 
> > > > > Primary firewall:
> > > > > 
> > > > > [root@sen-fw1 linux-3.7.3-101.fc17.x86_64]# conntrack -L expect
> > > > > 251 proto=6 src=192.168.218.199 dst=192.168.28.198 sport=0 dport=54705 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=192.168.218.199 master-dst=192.168.28.198 sport=56877 dport=21 class=0 helper=ftp
> > > > > conntrack v1.4.0 (conntrack-tools): 1 expectations have been shown.
> > > > > 
> > > > > Backup firewall:
> > > > > 
> > > > > [root@sen-fw2 linux-3.7.3-101.fc17.x86_64]# conntrack -L expect
> > > > > 245 proto=6 src=192.168.28.198 dst=192.168.28.198 sport=0 dport=54705 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=192.168.218.199 master-dst=192.168.28.198 sport=56877 dport=21 class=0 helper=ftp
> > > > > conntrack v1.4.0 (conntrack-tools): 1 expectations have been shown.
> > > > > 
> > > > > This was an unfortunate side affect of the patch to fix the
> > > > > conntrackd segfault problem.  If I use Florian's version
> > > > > of the fix segfault patch rather than Pablo's then all is
> > > > > good.
> > > > 
> > > > Thanks for the information, however, we still need to get working back
> > > > the filtering support.
> > > > 
> > > > Could you give a try to the following patch, please?
> > > > 
> > > > It applies on top of conntrack-tools master branch, thanks.
> > > 
> > > There are still some downsides in the previous solution, please, give
> > > a try to this patch instead.
> > 
> > It appears to work pretty well and does fix the src-IP issue!
> 
> Thanks for testing and reporting back, I have pushed the patch to
> master.
> 
> > I did notice a couple of other things but they're not necessarily
> > related to these patches.  I noticed that my long lived BGP tcp
> > connection was getting some duplicate conntrackd ct entries (both
> > IPv4 and IPv6).  The duplicate ct entry occurred 60 seconds after
> > the original, and once I saw a second duplicate ct IPv4 entry,
> > again with about a 60 second separation.
> 
> That's strange. Please, tell me if this happening in the primary
> and/or the backup. If you are using the cache mode, also dump the
> caches via -i and -e to see the content.

I'm not using the external cache.  The duplicate BGP conntrackd
entries seem to have multiplied (this is on the primary with the
"conntrackd -i" command):

tcp      6 ESTABLISHED src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=61888 dport=179 src=aaa.aaa.aaa.ccc dst=aaa.aaa.aaa.bbb sport=179 dport=61888 [ASSURED] mark=0 [active since 18464s]
tcp      6 ESTABLISHED src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=61888 dport=179 src=aaa.aaa.aaa.ccc dst=aaa.aaa.aaa.bbb sport=179 dport=61888 [ASSURED] mark=0 [active since 245497s]
tcp      6 ESTABLISHED src=2001:xxxx:xxxx:xxxx::yyyy dst=2001:xxxx:xxxx:xxxx::zzzz sport=49166 dport=179 src=2001:xxxx:xxxx:xxxx::zzzz dst=2001:xxxx:xxxx:xxxx::yyyy sport=179 dport=49166 [ASSURED] mark=0 [active since 271387s]
tcp      6 ESTABLISHED src=2001:xxxx:xxxx:xxxx::yyyy dst=2001:xxxx:xxxx:xxxx::zzzz sport=49166 dport=179 src=2001:xxxx:xxxx:xxxx::zzzz dst=2001:xxxx:xxxx:xxxx::yyyy sport=179 dport=49166 [ASSURED] mark=0 [active since 271447s]
tcp      6 ESTABLISHED src=2001:xxxx:xxxx:xxxx::yyyy dst=2001:xxxx:xxxx:xxxx::zzzz sport=49166 dport=179 src=2001:xxxx:xxxx:xxxx::zzzz dst=2001:xxxx:xxxx:xxxx::yyyy sport=179 dport=49166 [ASSURED] mark=0 [active since 271508s]
tcp      6 ESTABLISHED src=2001:xxxx:xxxx:xxxx::yyyy dst=2001:xxxx:xxxx:xxxx::zzzz sport=49166 dport=179 src=2001:xxxx:xxxx:xxxx::zzzz dst=2001:xxxx:xxxx:xxxx::yyyy sport=179 dport=49166 [ASSURED] mark=0 [active since 271568s]
tcp      6 ESTABLISHED src=2001:xxxx:xxxx:xxxx::yyyy dst=2001:xxxx:xxxx:xxxx::zzzz sport=49166 dport=179 src=2001:xxxx:xxxx:xxxx::zzzz dst=2001:xxxx:xxxx:xxxx::yyyy sport=179 dport=49166 [ASSURED] mark=0 [active since 271628s]
tcp      6 ESTABLISHED src=2001:xxxx:xxxx:xxxx::yyyy dst=2001:xxxx:xxxx:xxxx::zzzz sport=49166 dport=179 src=2001:xxxx:xxxx:xxxx::zzzz dst=2001:xxxx:xxxx:xxxx::yyyy sport=179 dport=49166 [ASSURED] mark=0 [active since 271689s]
tcp      6 ESTABLISHED src=2001:xxxx:xxxx:xxxx::yyyy dst=2001:xxxx:xxxx:xxxx::zzzz sport=49166 dport=179 src=2001:xxxx:xxxx:xxxx::zzzz dst=2001:xxxx:xxxx:xxxx::yyyy sport=179 dport=49166 [ASSURED] mark=0 [active since 271749s]
tcp      6 ESTABLISHED src=2001:xxxx:xxxx:xxxx::yyyy dst=2001:xxxx:xxxx:xxxx::zzzz sport=49166 dport=179 src=2001:xxxx:xxxx:xxxx::zzzz dst=2001:xxxx:xxxx:xxxx::yyyy sport=179 dport=49166 [ASSURED] [active since 271803s]

> > And on the expectation entries, they seem to have a lifetime
> > of 300 seconds.  The IPv6 expectation entries are deleted after
> > the 300 seconds as expected, but the IPv4 expectation entries
> > actually go away in a minute or less.  I don't know if that
> > is expected behavior or not.  Note I was leaving the ftp
> > control connection open the entire time.  Also it may have
> > just been my imagination, but it seemed if I queried it more
> > often with "conntrack -L expect" it would stick around somewhat
> > longer (but would go away within the minute).
> 
> Expectations depends on the master conntrack, if the master is
> released, the expectations will be released too. You may be noticing
> that effect.

I wasn't expecting that since I was leaving the ftp control
connection open.

> You can use `conntrack -E` and `conntrack -E exp` to verify this.

I couldn't get filtering to work with expect:

[root@sen-fw1 ~]# conntrack -E expect -d aaa.aaa.aaa.ccc
conntrack v1.4.0 (conntrack-tools): Illegal option `--dst' with this command
Try `conntrack -h' or 'conntrack --help' for more information.

[root@sen-fw1 ~]# conntrack -E expect --tuple-dst aaa.aaa.aaa.ccc
conntrack v1.4.0 (conntrack-tools): Illegal option `--tuple-dst' with this command
Try `conntrack -h' or 'conntrack --help' for more information.

But with the help of grep:

x100ssd2% nc aaa.aaa.aaa.ccc 21
220 FTP Server ready.

gives:

    [NEW] tcp      6 120 SYN_SENT src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=49840 dport=21 [UNREPLIED] src=aaa.aaa.aaa.ccc dst=aaa.aaa.aaa.bbb sport=21 dport=49840 helper=ftp
 [UPDATE] tcp      6 60 SYN_RECV src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=49840 dport=21 src=aaa.aaa.aaa.ccc dst=aaa.aaa.aaa.bbb sport=21 dport=49840 helper=ftp
 [UPDATE] tcp      6 432000 ESTABLISHED src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=49840 dport=21 src=aaa.aaa.aaa.ccc dst=aaa.aaa.aaa.bbb sport=21 dport=49840 [ASSURED] helper=ftp

Then doing:

USER anonymous
331 Anonymous login ok, send your complete email address as your password
PASS bill@
230-
...
230 Anonymous login ok, restrictions apply.
PASV
227 Entering Passive Mode (aaa,aaa,aaa,ccc,175,61).

yields:

    [NEW] 300 proto=6 src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=0 dport=44861 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=aaa.aaa.aaa.bbb master-dst=aaa.aaa.aaa.ccc sport=49840 dport=21 class=0 helper=ftp

Not doing anything but waiting, a little while later:

[DESTROY] tcp      6 src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=49840 dport=21 src=aaa.aaa.aaa.ccc dst=aaa.aaa.aaa.bbb sport=21 dport=49840 [ASSURED]
 [UPDATE] tcp      6 433405 ESTABLISHED src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=49840 dport=21 src=aaa.aaa.aaa.ccc dst=aaa.aaa.aaa.bbb sport=21 dport=49840 [ASSURED] mark=0 helper=ftp

together with:

[DESTROY] 273 proto=6 src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=0 dport=44861 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=aaa.aaa.aaa.bbb master-dst=aaa.aaa.aaa.ccc sport=49840 dport=21 class=0 helper=ftp

So that's why the expect goes away.  Remember this didn't happen
with IPv6 (it didn't go away until the 300 seconds expired).

Is the DESTROY/UPDATE on the master ftp connection normal when
the ftp control connection hasn't been closed?

At least for ftp it's not a big problem in practice since the
expectation seems to be used almost immediately after creation
for normal ftp transfers.

					-Thanks

					-Bill

Re: conntrackd segfault on EPSV IPv6 ftp command when using ftp ExpectationSync

[Pablo Neira Ayuso]

Hi Bill,

On Tue, Jul 16, 2013 at 01:55:03AM -0400, Bill Fink wrote:
> On Mon, 15 Jul 2013, Pablo Neira Ayuso wrote:
> 
> > On Fri, Jul 12, 2013 at 03:01:14AM -0400, Bill Fink wrote:
> > > Pablo,
> > > 
> > > On Thu, 11 Jul 2013, Pablo Neira Ayuso wrote:
> > > 
> > > > On Thu, Jul 11, 2013 at 12:08:20AM +0200, Pablo Neira Ayuso wrote:
> > > > > On Wed, Jul 10, 2013 at 05:58:15AM -0400, Bill Fink wrote:
> > > > > > Almost there.  With the above patch, I now successfully get
> > > > > > IPv6 expectations on the backup firewall.  Unfortunately they're
> > > > > > not quite right.  On the backup firewall, the expectation src-IP
> > > > > > is the same as the dst-IP (either IPv4 or IPv6).
> > > > > > 
> > > > > > Primary firewall:
> > > > > > 
> > > > > > [root@sen-fw1 linux-3.7.3-101.fc17.x86_64]# conntrack -L expect
> > > > > > 251 proto=6 src=192.168.218.199 dst=192.168.28.198 sport=0 dport=54705 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=192.168.218.199 master-dst=192.168.28.198 sport=56877 dport=21 class=0 helper=ftp
> > > > > > conntrack v1.4.0 (conntrack-tools): 1 expectations have been shown.
> > > > > > 
> > > > > > Backup firewall:
> > > > > > 
> > > > > > [root@sen-fw2 linux-3.7.3-101.fc17.x86_64]# conntrack -L expect
> > > > > > 245 proto=6 src=192.168.28.198 dst=192.168.28.198 sport=0 dport=54705 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=192.168.218.199 master-dst=192.168.28.198 sport=56877 dport=21 class=0 helper=ftp
> > > > > > conntrack v1.4.0 (conntrack-tools): 1 expectations have been shown.
> > > > > > 
> > > > > > This was an unfortunate side affect of the patch to fix the
> > > > > > conntrackd segfault problem.  If I use Florian's version
> > > > > > of the fix segfault patch rather than Pablo's then all is
> > > > > > good.
> > > > > 
> > > > > Thanks for the information, however, we still need to get working back
> > > > > the filtering support.
> > > > > 
> > > > > Could you give a try to the following patch, please?
> > > > > 
> > > > > It applies on top of conntrack-tools master branch, thanks.
> > > > 
> > > > There are still some downsides in the previous solution, please, give
> > > > a try to this patch instead.
> > > 
> > > It appears to work pretty well and does fix the src-IP issue!
> > 
> > Thanks for testing and reporting back, I have pushed the patch to
> > master.
> > 
> > > I did notice a couple of other things but they're not necessarily
> > > related to these patches.  I noticed that my long lived BGP tcp
> > > connection was getting some duplicate conntrackd ct entries (both
> > > IPv4 and IPv6).  The duplicate ct entry occurred 60 seconds after
> > > the original, and once I saw a second duplicate ct IPv4 entry,
> > > again with about a 60 second separation.
> > 
> > That's strange. Please, tell me if this happening in the primary
> > and/or the backup. If you are using the cache mode, also dump the
> > caches via -i and -e to see the content.
> 
> I'm not using the external cache.  The duplicate BGP conntrackd
> entries seem to have multiplied (this is on the primary with the
> "conntrackd -i" command):
> 
> tcp      6 ESTABLISHED src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=61888 dport=179 src=aaa.aaa.aaa.ccc dst=aaa.aaa.aaa.bbb sport=179 dport=61888 [ASSURED] mark=0 [active since 18464s]
> tcp      6 ESTABLISHED src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=61888 dport=179 src=aaa.aaa.aaa.ccc dst=aaa.aaa.aaa.bbb sport=179 dport=61888 [ASSURED] mark=0 [active since 245497s]
> tcp      6 ESTABLISHED src=2001:xxxx:xxxx:xxxx::yyyy dst=2001:xxxx:xxxx:xxxx::zzzz sport=49166 dport=179 src=2001:xxxx:xxxx:xxxx::zzzz dst=2001:xxxx:xxxx:xxxx::yyyy sport=179 dport=49166 [ASSURED] mark=0 [active since 271387s]
> tcp      6 ESTABLISHED src=2001:xxxx:xxxx:xxxx::yyyy dst=2001:xxxx:xxxx:xxxx::zzzz sport=49166 dport=179 src=2001:xxxx:xxxx:xxxx::zzzz dst=2001:xxxx:xxxx:xxxx::yyyy sport=179 dport=49166 [ASSURED] mark=0 [active since 271447s]
> tcp      6 ESTABLISHED src=2001:xxxx:xxxx:xxxx::yyyy dst=2001:xxxx:xxxx:xxxx::zzzz sport=49166 dport=179 src=2001:xxxx:xxxx:xxxx::zzzz dst=2001:xxxx:xxxx:xxxx::yyyy sport=179 dport=49166 [ASSURED] mark=0 [active since 271508s]
> tcp      6 ESTABLISHED src=2001:xxxx:xxxx:xxxx::yyyy dst=2001:xxxx:xxxx:xxxx::zzzz sport=49166 dport=179 src=2001:xxxx:xxxx:xxxx::zzzz dst=2001:xxxx:xxxx:xxxx::yyyy sport=179 dport=49166 [ASSURED] mark=0 [active since 271568s]
> tcp      6 ESTABLISHED src=2001:xxxx:xxxx:xxxx::yyyy dst=2001:xxxx:xxxx:xxxx::zzzz sport=49166 dport=179 src=2001:xxxx:xxxx:xxxx::zzzz dst=2001:xxxx:xxxx:xxxx::yyyy sport=179 dport=49166 [ASSURED] mark=0 [active since 271628s]
> tcp      6 ESTABLISHED src=2001:xxxx:xxxx:xxxx::yyyy dst=2001:xxxx:xxxx:xxxx::zzzz sport=49166 dport=179 src=2001:xxxx:xxxx:xxxx::zzzz dst=2001:xxxx:xxxx:xxxx::yyyy sport=179 dport=49166 [ASSURED] mark=0 [active since 271689s]
> tcp      6 ESTABLISHED src=2001:xxxx:xxxx:xxxx::yyyy dst=2001:xxxx:xxxx:xxxx::zzzz sport=49166 dport=179 src=2001:xxxx:xxxx:xxxx::zzzz dst=2001:xxxx:xxxx:xxxx::yyyy sport=179 dport=49166 [ASSURED] mark=0 [active since 271749s]
> tcp      6 ESTABLISHED src=2001:xxxx:xxxx:xxxx::yyyy dst=2001:xxxx:xxxx:xxxx::zzzz sport=49166 dport=179 src=2001:xxxx:xxxx:xxxx::zzzz dst=2001:xxxx:xxxx:xxxx::yyyy sport=179 dport=49166 [ASSURED] [active since 271803s]

Interesting. Can you apply the following patch? It should help to
debug the issue. Basically, it displays the conntrack unique ID. They
should have different IDs.

> > > And on the expectation entries, they seem to have a lifetime
> > > of 300 seconds.  The IPv6 expectation entries are deleted after
> > > the 300 seconds as expected, but the IPv4 expectation entries
> > > actually go away in a minute or less.  I don't know if that
> > > is expected behavior or not.  Note I was leaving the ftp
> > > control connection open the entire time.  Also it may have
> > > just been my imagination, but it seemed if I queried it more
> > > often with "conntrack -L expect" it would stick around somewhat
> > > longer (but would go away within the minute).
> > 
> > Expectations depends on the master conntrack, if the master is
> > released, the expectations will be released too. You may be noticing
> > that effect.
> 
> I wasn't expecting that since I was leaving the ftp control
> connection open.
> 
> > You can use `conntrack -E` and `conntrack -E exp` to verify this.
> 
> I couldn't get filtering to work with expect:
> 
> [root@sen-fw1 ~]# conntrack -E expect -d aaa.aaa.aaa.ccc
> conntrack v1.4.0 (conntrack-tools): Illegal option `--dst' with this command
> Try `conntrack -h' or 'conntrack --help' for more information.
> 
> [root@sen-fw1 ~]# conntrack -E expect --tuple-dst aaa.aaa.aaa.ccc
> conntrack v1.4.0 (conntrack-tools): Illegal option `--tuple-dst' with this command
> Try `conntrack -h' or 'conntrack --help' for more information.
> 
> But with the help of grep:
> 
> x100ssd2% nc aaa.aaa.aaa.ccc 21
> 220 FTP Server ready.
> 
> gives:
> 
>     [NEW] tcp      6 120 SYN_SENT src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=49840 dport=21 [UNREPLIED] src=aaa.aaa.aaa.ccc dst=aaa.aaa.aaa.bbb sport=21 dport=49840 helper=ftp
>  [UPDATE] tcp      6 60 SYN_RECV src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=49840 dport=21 src=aaa.aaa.aaa.ccc dst=aaa.aaa.aaa.bbb sport=21 dport=49840 helper=ftp
>  [UPDATE] tcp      6 432000 ESTABLISHED src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=49840 dport=21 src=aaa.aaa.aaa.ccc dst=aaa.aaa.aaa.bbb sport=21 dport=49840 [ASSURED] helper=ftp
> 
> Then doing:
> 
> USER anonymous
> 331 Anonymous login ok, send your complete email address as your password
> PASS bill@
> 230-
> ...
> 230 Anonymous login ok, restrictions apply.
> PASV
> 227 Entering Passive Mode (aaa,aaa,aaa,ccc,175,61).
> 
> yields:
> 
>     [NEW] 300 proto=6 src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=0 dport=44861 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=aaa.aaa.aaa.bbb master-dst=aaa.aaa.aaa.ccc sport=49840 dport=21 class=0 helper=ftp
> 
> Not doing anything but waiting, a little while later:

Can you catch the last event before this one below? I'd like to know
in what TCP state the flow was before the destroy event.

> [DESTROY] tcp      6 src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=49840 dport=21 src=aaa.aaa.aaa.ccc dst=aaa.aaa.aaa.bbb sport=21 dport=49840 [ASSURED]
>  [UPDATE] tcp      6 433405 ESTABLISHED src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=49840 dport=21 src=aaa.aaa.aaa.ccc dst=aaa.aaa.aaa.bbb sport=21 dport=49840 [ASSURED] mark=0 helper=ftp
> 
> together with:
> 
> [DESTROY] 273 proto=6 src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=0 dport=44861 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=aaa.aaa.aaa.bbb master-dst=aaa.aaa.aaa.ccc sport=49840 dport=21 class=0 helper=ftp
> 
> So that's why the expect goes away.  Remember this didn't happen
> with IPv6 (it didn't go away until the 300 seconds expired).
> 
> Is the DESTROY/UPDATE on the master ftp connection normal when
> the ftp control connection hasn't been closed?

That's not normal. I guess you have tcp pick enabled:

cat /proc/sys/net/netfilter/nf_conntrack_tcp_loose

shows 1. That allows conntrack to create new entries from the first
packet seen (no mater if it's not a tcp syn packet). It seems that
conntrack is creating a new TCP established flow for some late packet
retransmission. Florian Westphal posted a patch to reduce the default
timeout for pickup flows:

6547a22 netfilter: nf_conntrack: avoid large timeout for mid-stream pickup

That should mitigate this effect. Still, you'll have to trace traffic
from your network to reach the root cause of this. You can gather pcap
traces of the BGP's TCP connection, as it seems reproducible with it,
to simplify things. I think both things are different manifestations
of the same issue.

Thanks for the very detailed report. Let me know how it goes with your
tracing.

Regards.

Re: conntrackd segfault on EPSV IPv6 ftp command when using ftp ExpectationSync

[Pablo Neira Ayuso]

[-- Attachment #1: Type: text/plain, Size: 277 bytes --]

On Tue, Jul 16, 2013 at 11:33:54PM +0200, Pablo Neira Ayuso wrote:
[...]
> Interesting. Can you apply the following patch? It should help to
> debug the issue. Basically, it displays the conntrack unique ID. They
> should have different IDs.

Forgot attachment, here it comes.

[-- Attachment #2: x --]
[-- Type: text/plain, Size: 388 bytes --]

diff --git a/src/cache-ct.c b/src/cache-ct.c
index f86d143..11b0543 100644
--- a/src/cache-ct.c
+++ b/src/cache-ct.c
@@ -152,7 +152,7 @@ static int cache_ct_dump_step(void *data1, void *n)
 			     obj->ptr,
 			     NFCT_T_UNKNOWN, 
 			     container->type,
-			     0);
+			     NFCT_OF_ID);
 
 	for (i = 0; i < obj->cache->num_features; i++) {
 		if (obj->cache->features[i]->dump) {

Re: conntrackd segfault on EPSV IPv6 ftp command when using ftp ExpectationSync

[Bill Fink]

On Tue, 16 Jul 2013, Pablo Neira Ayuso wrote:

> On Tue, Jul 16, 2013 at 01:55:03AM -0400, Bill Fink wrote:
> > On Mon, 15 Jul 2013, Pablo Neira Ayuso wrote:
> > 
> > > On Fri, Jul 12, 2013 at 03:01:14AM -0400, Bill Fink wrote:
> > > 
> > > > I did notice a couple of other things but they're not necessarily
> > > > related to these patches.  I noticed that my long lived BGP tcp
> > > > connection was getting some duplicate conntrackd ct entries (both
> > > > IPv4 and IPv6).  The duplicate ct entry occurred 60 seconds after
> > > > the original, and once I saw a second duplicate ct IPv4 entry,
> > > > again with about a 60 second separation.
> > > 
> > > That's strange. Please, tell me if this happening in the primary
> > > and/or the backup. If you are using the cache mode, also dump the
> > > caches via -i and -e to see the content.
> > 
> > I'm not using the external cache.  The duplicate BGP conntrackd
> > entries seem to have multiplied (this is on the primary with the
> > "conntrackd -i" command):
> > 
> > tcp      6 ESTABLISHED src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=61888 dport=179 src=aaa.aaa.aaa.ccc dst=aaa.aaa.aaa.bbb sport=179 dport=61888 [ASSURED] mark=0 [active since 18464s]
> > tcp      6 ESTABLISHED src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=61888 dport=179 src=aaa.aaa.aaa.ccc dst=aaa.aaa.aaa.bbb sport=179 dport=61888 [ASSURED] mark=0 [active since 245497s]
> > tcp      6 ESTABLISHED src=2001:xxxx:xxxx:xxxx::yyyy dst=2001:xxxx:xxxx:xxxx::zzzz sport=49166 dport=179 src=2001:xxxx:xxxx:xxxx::zzzz dst=2001:xxxx:xxxx:xxxx::yyyy sport=179 dport=49166 [ASSURED] mark=0 [active since 271387s]
> > tcp      6 ESTABLISHED src=2001:xxxx:xxxx:xxxx::yyyy dst=2001:xxxx:xxxx:xxxx::zzzz sport=49166 dport=179 src=2001:xxxx:xxxx:xxxx::zzzz dst=2001:xxxx:xxxx:xxxx::yyyy sport=179 dport=49166 [ASSURED] mark=0 [active since 271447s]
> > tcp      6 ESTABLISHED src=2001:xxxx:xxxx:xxxx::yyyy dst=2001:xxxx:xxxx:xxxx::zzzz sport=49166 dport=179 src=2001:xxxx:xxxx:xxxx::zzzz dst=2001:xxxx:xxxx:xxxx::yyyy sport=179 dport=49166 [ASSURED] mark=0 [active since 271508s]
> > tcp      6 ESTABLISHED src=2001:xxxx:xxxx:xxxx::yyyy dst=2001:xxxx:xxxx:xxxx::zzzz sport=49166 dport=179 src=2001:xxxx:xxxx:xxxx::zzzz dst=2001:xxxx:xxxx:xxxx::yyyy sport=179 dport=49166 [ASSURED] mark=0 [active since 271568s]
> > tcp      6 ESTABLISHED src=2001:xxxx:xxxx:xxxx::yyyy dst=2001:xxxx:xxxx:xxxx::zzzz sport=49166 dport=179 src=2001:xxxx:xxxx:xxxx::zzzz dst=2001:xxxx:xxxx:xxxx::yyyy sport=179 dport=49166 [ASSURED] mark=0 [active since 271628s]
> > tcp      6 ESTABLISHED src=2001:xxxx:xxxx:xxxx::yyyy dst=2001:xxxx:xxxx:xxxx::zzzz sport=49166 dport=179 src=2001:xxxx:xxxx:xxxx::zzzz dst=2001:xxxx:xxxx:xxxx::yyyy sport=179 dport=49166 [ASSURED] mark=0 [active since 271689s]
> > tcp      6 ESTABLISHED src=2001:xxxx:xxxx:xxxx::yyyy dst=2001:xxxx:xxxx:xxxx::zzzz sport=49166 dport=179 src=2001:xxxx:xxxx:xxxx::zzzz dst=2001:xxxx:xxxx:xxxx::yyyy sport=179 dport=49166 [ASSURED] mark=0 [active since 271749s]
> > tcp      6 ESTABLISHED src=2001:xxxx:xxxx:xxxx::yyyy dst=2001:xxxx:xxxx:xxxx::zzzz sport=49166 dport=179 src=2001:xxxx:xxxx:xxxx::zzzz dst=2001:xxxx:xxxx:xxxx::yyyy sport=179 dport=49166 [ASSURED] [active since 271803s]
> 
> Interesting. Can you apply the following patch? It should help to
> debug the issue. Basically, it displays the conntrack unique ID. They
> should have different IDs.

I tried the debugging patch, but the "conntrackd -i" output wasn't
any different (no conntrack unique IDs).

> > > > And on the expectation entries, they seem to have a lifetime
> > > > of 300 seconds.  The IPv6 expectation entries are deleted after
> > > > the 300 seconds as expected, but the IPv4 expectation entries
> > > > actually go away in a minute or less.  I don't know if that
> > > > is expected behavior or not.  Note I was leaving the ftp
> > > > control connection open the entire time.  Also it may have
> > > > just been my imagination, but it seemed if I queried it more
> > > > often with "conntrack -L expect" it would stick around somewhat
> > > > longer (but would go away within the minute).
> > > 
> > > Expectations depends on the master conntrack, if the master is
> > > released, the expectations will be released too. You may be noticing
> > > that effect.
> > 
> > I wasn't expecting that since I was leaving the ftp control
> > connection open.
> > 
> > > You can use `conntrack -E` and `conntrack -E exp` to verify this.
> > 
> > I couldn't get filtering to work with expect:
> > 
> > [root@sen-fw1 ~]# conntrack -E expect -d aaa.aaa.aaa.ccc
> > conntrack v1.4.0 (conntrack-tools): Illegal option `--dst' with this command
> > Try `conntrack -h' or 'conntrack --help' for more information.
> > 
> > [root@sen-fw1 ~]# conntrack -E expect --tuple-dst aaa.aaa.aaa.ccc
> > conntrack v1.4.0 (conntrack-tools): Illegal option `--tuple-dst' with this command
> > Try `conntrack -h' or 'conntrack --help' for more information.
> > 
> > But with the help of grep:
> > 
> > x100ssd2% nc aaa.aaa.aaa.ccc 21
> > 220 FTP Server ready.
> > 
> > gives:
> > 
> >     [NEW] tcp      6 120 SYN_SENT src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=49840 dport=21 [UNREPLIED] src=aaa.aaa.aaa.ccc dst=aaa.aaa.aaa.bbb sport=21 dport=49840 helper=ftp
> >  [UPDATE] tcp      6 60 SYN_RECV src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=49840 dport=21 src=aaa.aaa.aaa.ccc dst=aaa.aaa.aaa.bbb sport=21 dport=49840 helper=ftp
> >  [UPDATE] tcp      6 432000 ESTABLISHED src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=49840 dport=21 src=aaa.aaa.aaa.ccc dst=aaa.aaa.aaa.bbb sport=21 dport=49840 [ASSURED] helper=ftp
> > 
> > Then doing:
> > 
> > USER anonymous
> > 331 Anonymous login ok, send your complete email address as your password
> > PASS bill@
> > 230-
> > ...
> > 230 Anonymous login ok, restrictions apply.
> > PASV
> > 227 Entering Passive Mode (aaa,aaa,aaa,ccc,175,61).
> > 
> > yields:
> > 
> >     [NEW] 300 proto=6 src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=0 dport=44861 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=aaa.aaa.aaa.bbb master-dst=aaa.aaa.aaa.ccc sport=49840 dport=21 class=0 helper=ftp
> > 
> > Not doing anything but waiting, a little while later:
> 
> Can you catch the last event before this one below? I'd like to know
> in what TCP state the flow was before the destroy event.

All the events were listed above in order, so the last event on the
master ftp connection before the following DESTROY/UPDATE was the
UPDATE to the ESTABLISHED state.

> > [DESTROY] tcp      6 src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=49840 dport=21 src=aaa.aaa.aaa.ccc dst=aaa.aaa.aaa.bbb sport=21 dport=49840 [ASSURED]
> >  [UPDATE] tcp      6 433405 ESTABLISHED src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=49840 dport=21 src=aaa.aaa.aaa.ccc dst=aaa.aaa.aaa.bbb sport=21 dport=49840 [ASSURED] mark=0 helper=ftp
> > 
> > together with:
> > 
> > [DESTROY] 273 proto=6 src=aaa.aaa.aaa.bbb dst=aaa.aaa.aaa.ccc sport=0 dport=44861 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=aaa.aaa.aaa.bbb master-dst=aaa.aaa.aaa.ccc sport=49840 dport=21 class=0 helper=ftp
> > 
> > So that's why the expect goes away.  Remember this didn't happen
> > with IPv6 (it didn't go away until the 300 seconds expired).
> > 
> > Is the DESTROY/UPDATE on the master ftp connection normal when
> > the ftp control connection hasn't been closed?
> 
> That's not normal. I guess you have tcp pick enabled:
> 
> cat /proc/sys/net/netfilter/nf_conntrack_tcp_loose

Yes, I have nf_conntrack_tcp_loose set.

> shows 1. That allows conntrack to create new entries from the first
> packet seen (no mater if it's not a tcp syn packet). It seems that
> conntrack is creating a new TCP established flow for some late packet
> retransmission. Florian Westphal posted a patch to reduce the default
> timeout for pickup flows:
> 
> 6547a22 netfilter: nf_conntrack: avoid large timeout for mid-stream pickup
> 
> That should mitigate this effect. Still, you'll have to trace traffic
> from your network to reach the root cause of this. You can gather pcap
> traces of the BGP's TCP connection, as it seems reproducible with it,
> to simplify things. I think both things are different manifestations
> of the same issue.

For whatever reason, I can't reproduce this anymore.  The expectations
now expire after the expected 5 minutes.  I'm not sure what's different
now.  If it happens again, I'll try and get more debugging info.

> Thanks for the very detailed report. Let me know how it goes with your
> tracing.

						-Thanks

						-Bill