2.6.12 netfilter: local packets marked as invalid

2.6.12 netfilter: local packets marked as invalid

[Daniel Drake]

Hi,

Some Gentoo users have reported very long application startup times in 2.6.12.
This seems to be because the applications are attempting to connect to local
ports such as sunrpc/portmap (where these services are not running), but some
packets are being dropped, so the application load just pauses until the
connection times out.

There was a similar problem reported recently ("2.6.12: connection tracking
broken?"), regarding bridge devices. No bridge devices are involved here.

This is easy to reproduce, and the problem exists on both Linux 2.6.12 and
2.6.13-rc2. This was not a problem on Linux 2.6.11.

Taking a simple configuration:

	# iptables -P INPUT ACCEPT
	# iptables -P OUTPUT ACCEPT
	# iptables -P FORWARD ACCEPT

I don't have a webserver running. If I try and telnet to port 80, I
immediately get connection refused, as expected:

	# telnet 127.0.0.1 80
	Trying 127.0.0.1...
	telnet: connect to address 127.0.0.1: Connection refused

I now add another rule, to drop invalid packets, and retry the telnet connection.

	# iptables -A INPUT -m state --state INVALID -j DROP
	# telnet 127.0.0.1 80
	<very long pause>
	telnet: connect to address 127.0.0.1: Connection timed out

During the pause, netstat reports the connection state as SYN_SENT:

Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      1 localhost:39066         localhost:http          SYN_SENT

I now flush the filter table, and make the kernel log invalid packets:

	# iptables -t filter -F
	#iptables -A INPUT -m state --state INVALID -j LOG --log-prefix "inv "

When retrying the telnet test, this appears in the logs:

Jul  8 14:53:04 dsd inv IN=lo OUT=
MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.1
LEN=40 TOS=0x10 PREC=0x00 TTL=64 ID=15 DF PROTO=TCP SPT=80 DPT=58950 WINDOW=0
RES=0x00 ACK RST URGP=0

Does this mean that the kernel thinks its own ACK RST packet is invalid?

There is a Gentoo bug on this here:
http://bugs.gentoo.org/96948
...but I think I got the imporant info into this message.

Let me know if I can provide any more info. For those interested in a
temporary workaround, you can explicitly allow all local traffic, i.e.

	# iptables -A INPUT -i lo -j ACCEPT
	# iptables -A INPUT -m state --state INVALID -j DROP

Thanks.
Daniel

Re: 2.6.12 netfilter: local packets marked as invalid

[Patrick McHardy]

Daniel Drake wrote:
> When retrying the telnet test, this appears in the logs:
> 
> Jul  8 14:53:04 dsd inv IN=lo OUT=
> MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.1
> LEN=40 TOS=0x10 PREC=0x00 TTL=64 ID=15 DF PROTO=TCP SPT=80 DPT=58950 WINDOW=0
> RES=0x00 ACK RST URGP=0
> 
> Does this mean that the kernel thinks its own ACK RST packet is invalid?

I think I know what happens. In 2.6.12 we started dropping the conntrack
reference when a packet leaves IP, so packets on loopback are tracked
twice (LOCAL_OUT/PRE_ROUTING). TCP connection tracking destroys a
conntrack entry when the only reply is an RST. So when the packet is
tracked for the second time in PRE_ROUTING, the conntrack entry can't
be found anymore and the packet is considered invalid.

You could confirm this theory by logging invalid packets in LOCAL_OUT
and in PRE_ROUTING - only PRE_ROUTING should trigger. I'm going to
think about a solution meanwhile.

Regards
Patrick

Re: 2.6.12 netfilter: local packets marked as invalid

[Daniel Drake]

Patrick McHardy wrote:
> You could confirm this theory by logging invalid packets in LOCAL_OUT
> and in PRE_ROUTING - only PRE_ROUTING should trigger. I'm going to
> think about a solution meanwhile.

You'll have to forgive my lack of netfilter knowledge, I set up my firewall
ages ago and haven't really touched it since :)

How can I do this with iptables?

Thanks,
Daniel

Re: 2.6.12 netfilter: local packets marked as invalid

[Patrick McHardy]

[-- Attachment #1: Type: text/plain, Size: 281 bytes --]

Daniel Drake wrote:
> You'll have to forgive my lack of netfilter knowledge, I set up my firewall
> ages ago and haven't really touched it since :)

We decided to revert the responsible change because it caused problems
in other areas as well. This patch should fix your problem.


[-- Attachment #2: x --]
[-- Type: text/plain, Size: 2780 bytes --]

[NETFILTER]: Revert nf_reset change

Revert the nf_reset change that caused so much trouble, drop conntrack
references manually before packets are queued to packet sockets.

Signed-off-by: Phil Oester <kernel@linuxace.com>
Signed-off-by: Patrick McHardy <kaber@trash.net>

---
commit c64e9fafffd203886f6c57149803f1824febd9bb
tree 40a90ab1427c25f7a36109eabd1a53b0a8f144be
parent 9fd73bed06eb5cf309e0cb5cdf3249a54fd3ff22
author Phil Oester <kernel@linuxace.com> Mon, 11 Jul 2005 02:04:08
committer Patrick McHardy <kaber@trash.net> Mon, 11 Jul 2005 02:04:08

 net/ipv4/ip_output.c                         |    9 ---------
 net/ipv4/netfilter/ip_conntrack_standalone.c |    7 +++++++
 net/packet/af_packet.c                       |    6 ++++++
 3 files changed, 13 insertions(+), 9 deletions(-)

diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
--- a/net/ipv4/ip_output.c
+++ b/net/ipv4/ip_output.c
@@ -107,7 +107,6 @@ static int ip_dev_loopback_xmit(struct s
 	newskb->pkt_type = PACKET_LOOPBACK;
 	newskb->ip_summed = CHECKSUM_UNNECESSARY;
 	BUG_TRAP(newskb->dst);
-	nf_reset(newskb);
 	netif_rx(newskb);
 	return 0;
 }
@@ -188,14 +187,6 @@ static inline int ip_finish_output2(stru
 		skb = skb2;
 	}
 
-#ifdef CONFIG_BRIDGE_NETFILTER
-	/* bridge-netfilter defers calling some IP hooks to the bridge layer
-	 * and still needs the conntrack reference.
-	 */
-	if (skb->nf_bridge == NULL)
-#endif
-		nf_reset(skb);
-
 	if (hh) {
 		int hh_alen;
 
diff --git a/net/ipv4/netfilter/ip_conntrack_standalone.c b/net/ipv4/netfilter/ip_conntrack_standalone.c
--- a/net/ipv4/netfilter/ip_conntrack_standalone.c
+++ b/net/ipv4/netfilter/ip_conntrack_standalone.c
@@ -432,6 +432,13 @@ static unsigned int ip_conntrack_defrag(
 				        const struct net_device *out,
 				        int (*okfn)(struct sk_buff *))
 {
+#if !defined(CONFIG_IP_NF_NAT) && !defined(CONFIG_IP_NF_NAT_MODULE)
+	/* Previously seen (loopback)?  Ignore.  Do this before
+           fragment check. */
+	if ((*pskb)->nfct)
+		return NF_ACCEPT;
+#endif
+
 	/* Gather fragments. */
 	if ((*pskb)->nh.iph->frag_off & htons(IP_MF|IP_OFFSET)) {
 		*pskb = ip_ct_gather_frags(*pskb,
diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -274,6 +274,9 @@ static int packet_rcv_spkt(struct sk_buf
 	dst_release(skb->dst);
 	skb->dst = NULL;
 
+	/* drop conntrack reference */
+	nf_reset(skb);
+
 	spkt = (struct sockaddr_pkt*)skb->cb;
 
 	skb_push(skb, skb->data-skb->mac.raw);
@@ -517,6 +520,9 @@ static int packet_rcv(struct sk_buff *sk
 	dst_release(skb->dst);
 	skb->dst = NULL;
 
+	/* drop conntrack reference */
+	nf_reset(skb);
+
 	spin_lock(&sk->sk_receive_queue.lock);
 	po->stats.tp_packets++;
 	__skb_queue_tail(&sk->sk_receive_queue, skb);

Re: 2.6.12 netfilter: local packets marked as invalid

[Daniel Drake]

Patrick McHardy wrote:
> We decided to revert the responsible change because it caused problems
> in other areas as well. This patch should fix your problem.

Thanks, it works. If you decide to revisit this in the future, feel free to
send me a patch and I will help test it.

Daniel

Re: 2.6.12 netfilter: local packets marked as invalid

[David S. Miller]

From: Patrick McHardy <kaber@trash.net>
Date: Tue, 12 Jul 2005 13:58:27 +0200

> Daniel Drake wrote:
> > You'll have to forgive my lack of netfilter knowledge, I set up my firewall
> > ages ago and haven't really touched it since :)
> 
> We decided to revert the responsible change because it caused problems
> in other areas as well. This patch should fix your problem.

Applied.

Now the question is what to do about the 2.6.12.x stable
tree.  I think we put the offending change there, now we
need to revert it there too.  Patrick, could you push this
patch to stable@kernel.org so we can resolve that too?

Thanks a lot.

Re: 2.6.12 netfilter: local packets marked as invalid

[Chris Wright]

* David S. Miller (davem@davemloft.net) wrote:
> From: Patrick McHardy <kaber@trash.net>
> Date: Tue, 12 Jul 2005 13:58:27 +0200
> 
> > Daniel Drake wrote:
> > > You'll have to forgive my lack of netfilter knowledge, I set up my firewall
> > > ages ago and haven't really touched it since :)
> > 
> > We decided to revert the responsible change because it caused problems
> > in other areas as well. This patch should fix your problem.
> 
> Applied.
> 
> Now the question is what to do about the 2.6.12.x stable
> tree.  I think we put the offending change there, now we
> need to revert it there too.  Patrick, could you push this
> patch to stable@kernel.org so we can resolve that too?

There's the first fix in the queue, I can either drop that one, or
patch on top of it.  Dropping what's in the queue[1] is fine for me.
Below's the backport that Daniel sent over this morning (which applies
if I drop what's in the queue).  Patrick, does that look ok?  I didn't
queue this change yet, as I'd prefer it came either from you or with you
Cc'd so you can ack it.

[1] http://www.kernel.org/git/?p=linux/kernel/git/chrisw/stable-queue.git;a=blob;h=77843604cf9af8cf5458d97eb56d5346e6d380b3;hb=9aaf5aa7c4e4b8309997d2b433bf7464280799eb;f=queue/netfilter-connection-tracking.patch

--

[NETFILTER]: Revert nf_reset change

Revert the nf_reset change that caused so much trouble, drop conntrack
references manually before packets are queued to packet sockets.

Adapted for 2.6.12 by Daniel Drake <dsd@gentoo.org>

Signed-off-by: Phil Oester <kernel@linuxace.com>
Signed-off-by: Patrick McHardy <kaber@trash.net>

--- linux-2.6.12/net/ipv4/ip_output.c_orig	2005-07-12 13:42:56.000000000 +0100
+++ linux-2.6.12/net/ipv4/ip_output.c	2005-07-12 13:46:03.000000000 +0100
@@ -111,7 +111,6 @@ static int ip_dev_loopback_xmit(struct s
 #ifdef CONFIG_NETFILTER_DEBUG
 	nf_debug_ip_loopback_xmit(newskb);
 #endif
-	nf_reset(newskb);
 	netif_rx(newskb);
 	return 0;
 }
@@ -196,8 +195,6 @@ static inline int ip_finish_output2(stru
 	nf_debug_ip_finish_output2(skb);
 #endif /*CONFIG_NETFILTER_DEBUG*/
 
-	nf_reset(skb);
-
 	if (hh) {
 		int hh_alen;
 
--- linux-2.6.12/net/ipv4/netfilter/ip_conntrack_standalone.c_orig	2005-07-12 13:43:16.000000000 +0100
+++ linux-2.6.12/net/ipv4/netfilter/ip_conntrack_standalone.c	2005-07-12 13:47:44.000000000 +0100
@@ -432,6 +432,13 @@ static unsigned int ip_conntrack_defrag(
 				        const struct net_device *out,
 				        int (*okfn)(struct sk_buff *))
 {
+#if !defined(CONFIG_IP_NF_NAT) && !defined(CONFIG_IP_NF_NAT_MODULE)
+	/* Previously seen (loopback)?  Ignore.  Do this before
+           fragment check. */
+	if ((*pskb)->nfct)
+		return NF_ACCEPT;
+#endif
+
 	/* Gather fragments. */
 	if ((*pskb)->nh.iph->frag_off & htons(IP_MF|IP_OFFSET)) {
 		*pskb = ip_ct_gather_frags(*pskb,
--- linux-2.6.12/net/packet/af_packet.c_orig	2005-07-12 13:47:38.000000000 +0100
+++ linux-2.6.12/net/packet/af_packet.c	2005-07-12 13:47:44.000000000 +0100
@@ -274,6 +274,9 @@ static int packet_rcv_spkt(struct sk_buf
 	dst_release(skb->dst);
 	skb->dst = NULL;
 
+	/* drop conntrack reference */
+	nf_reset(skb);
+
 	spkt = (struct sockaddr_pkt*)skb->cb;
 
 	skb_push(skb, skb->data-skb->mac.raw);
@@ -517,6 +520,9 @@ static int packet_rcv(struct sk_buff *sk
 	dst_release(skb->dst);
 	skb->dst = NULL;
 
+	/* drop conntrack reference */
+	nf_reset(skb);
+
 	spin_lock(&sk->sk_receive_queue.lock);
 	po->stats.tp_packets++;
 	__skb_queue_tail(&sk->sk_receive_queue, skb);

Re: 2.6.12 netfilter: local packets marked as invalid

[Patrick McHardy]

Chris Wright wrote:
> * David S. Miller (davem@davemloft.net) wrote:
> 
>>Now the question is what to do about the 2.6.12.x stable
>>tree.  I think we put the offending change there, now we
>>need to revert it there too.  Patrick, could you push this
>>patch to stable@kernel.org so we can resolve that too?
> 
> There's the first fix in the queue, I can either drop that one, or
> patch on top of it.  Dropping what's in the queue[1] is fine for me.
> Below's the backport that Daniel sent over this morning (which applies
> if I drop what's in the queue).  Patrick, does that look ok?  I didn't
> queue this change yet, as I'd prefer it came either from you or with you
> Cc'd so you can ack it.
> 
> [1] http://www.kernel.org/git/?p=linux/kernel/git/chrisw/stable-queue.git;a=blob;h=77843604cf9af8cf5458d97eb56d5346e6d380b3;hb=9aaf5aa7c4e4b8309997d2b433bf7464280799eb;f=queue/netfilter-connection-tracking.patch

Daniel's patch is fine, thanks.

ACKed-by: Patrick McHardy <kaber@trash.net>

Regards
Patrick

Re: 2.6.12 netfilter: local packets marked as invalid

[Chris Wright]

* Patrick McHardy (kaber@trash.net) wrote:
> Chris Wright wrote:
> >* David S. Miller (davem@davemloft.net) wrote:
> >
> >>Now the question is what to do about the 2.6.12.x stable
> >>tree.  I think we put the offending change there, now we
> >>need to revert it there too.  Patrick, could you push this
> >>patch to stable@kernel.org so we can resolve that too?
> >
> >There's the first fix in the queue, I can either drop that one, or
> >patch on top of it.  Dropping what's in the queue[1] is fine for me.
> >Below's the backport that Daniel sent over this morning (which applies
> >if I drop what's in the queue).  Patrick, does that look ok?  I didn't
> >queue this change yet, as I'd prefer it came either from you or with you
> >Cc'd so you can ack it.
> >
> >[1] 
> >http://www.kernel.org/git/?p=linux/kernel/git/chrisw/stable-queue.git;a=blob;h=77843604cf9af8cf5458d97eb56d5346e6d380b3;hb=9aaf5aa7c4e4b8309997d2b433bf7464280799eb;f=queue/netfilter-connection-tracking.patch
> 
> Daniel's patch is fine, thanks.
> 
> ACKed-by: Patrick McHardy <kaber@trash.net>

Great, thanks.
-chris