REDIRECT differences in 2.6.12.2

REDIRECT differences in 2.6.12.2

[Dirk Morris]

I've been using REDIRECT to transparently capture TCP traffic (like 
squid), but after upgrading to 2.6.12.2 I'm running into some issues.

My understanding is (in anything from 2.4.x up to 2.6.8.1) the SYN came 
in (from alice) and was redirected to whatever port (on eve), the 
SYN/ACK was sent back from that port and it matchs conntrack which 
'fixes' the packet to look like it was coming from the original 
destination (bob).

This works great 2.4.x and the early 2.6 <2.6.8.1).
I havent tried 2.6.9 - 2.6.11 because we were having some reset problems 
under load, but now we are trying 2.6.12.2.

Here is the 2.6.8.1 output (which is expected):

setup:
[alice] <----> [eve] <----> [bob]
all tcpdump are on the alice <----> eve segment.

[dmorris @ eve] ~ # uname 
-a                                                                        
Linux cartman 2.6.8.1 #1 SMP Fri Apr 29 18:35:22 PDT 2005 i686 GNU/Linux

[dmorris @ alice] ~ # s tcpdump "not port 22 or 
icmp"                                        
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
06:27:28.053491 arp who-has bob tell alice
06:27:28.053635 arp reply bob is-at 00:13:20:3d:a7:74
06:27:28.053640 IP alice.33026 > bob.echo: S 1284341781:1284341781(0) 
win 5840 <mss 1460,sackOK,timestamp 153932448 0,nop,wscale 7>
06:27:28.070878 arp who-has bob tell eve
06:27:28.071684 IP bob.echo > alice.33026: S 1135117817:1135117817(0) 
ack 1284341782 win 5792 <mss 1460,sackOK,timestamp 274943 153932448>
06:27:28.071700 IP alice.33026 > bob.echo: . ack 1 win 5840 
<nop,nop,timestamp 153932467 274943>

[dmorris @ eve] ~ # s cat /proc/net/ip_conntrack | grep "port=7"  
tcp      6 431996 ESTABLISHED src=192.168.2.1 dst=192.168.2.3 
sport=33030 dport=7 src=192.168.2.2 dst=192.168.2.1 sport=9500 
dport=33030 [ASSURED] use=1


Now in 2.6.12.2 the SYN/ACK doesnt seem to get "fixed, " and so all 
connections are immediately reset:

[dmorris @ eve] ~ # uname 
-a                                                                         
Linux cartman 2.6.12.2 #1 SMP Sun Jul 3 17:21:45 PDT 2005 i686 GNU/Linux

[dmorris @ alice] ~ # s tcpdump "not port 22 or 
icmp"                            
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
06:16:00.981041 arp who-has bob tell alice
06:16:00.981117 arp reply bob is-at 00:13:20:3d:a7:74
06:16:00.981122 IP alice.33022 > bob.echo: S 574776601:574776601(0) win 
5840 <mss 1460,sackOK,timestamp 153245271 0,nop,wscale 7>
06:16:00.983896 arp who-has bob tell eve
06:16:00.984228 IP eve.9500 > alice.33022: S 551400716:551400716(0) ack 
574776602 win 5792 <mss 1460,sackOK,timestamp 1983246 153245271>
06:16:00.984249 IP alice.33022 > eve.9500: R 574776602:574776602(0) win 0

~ # s cat /proc/net/ip_conntrack | grep 
"port=7"                                          [dmorris @ cartman]
tcp      6 58 SYN_RECV src=192.168.2.1 dst=192.168.2.3 sport=33028 
dport=7 packets=1 bytes=60 src=192.168.2.2 dst=192.168.2.1 sport=9500 
dport=33028 packets=1 bytes=56 use=1

notice how the syn/ack comes from eve instead of bob, and it only 
matches 1 packet (the SYN I assume)
I suspect the syn/ack isnt matching in the conntrack table for some reason?
Could it be because the syn is marked, but the syn/ack isnt?

Any ideas?

Thanks,
-Dirk






more info:

[dmorris @ eve] ~ # s iptables -t nat 
-nL                                                                 
[dmorris @ cartman]
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination        
REDIRECT   tcp  --  0.0.0.0/0            0.0.0.0/0           MARK match 
!0x1000000/0x1000000 redir ports 9500-9627

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination        

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination 


~ # s cat /proc/net/ip_conntrack | grep 
"port=7"                                          [dmorris @ cartman]
tcp      6 58 SYN_RECV src=192.168.2.1 dst=192.168.2.3 sport=33028 
dport=7 packets=1 bytes=60 src=192.168.2.2 dst=192.168.2.1 sport=9500 
dport=33028 packets=1 bytes=56 mark=0 use=1

Re: REDIRECT differences in 2.6.12.2

[Patrick McHardy]

Dirk Morris wrote:
> notice how the syn/ack comes from eve instead of bob, and it only 
> matches 1 packet (the SYN I assume)
> I suspect the syn/ack isnt matching in the conntrack table for some reason?
> Could it be because the syn is marked, but the syn/ack isnt?

No, that doesn't affect association with a conntrack entry. Please retry
after doing "echo 255 > /proc/sys/net/ipv4/netfilter/ip_ct_log_invalid"
and "modprobe ipt_LOG".

Regards
Patrick

Re: REDIRECT differences in 2.6.12.2

[Dirk Morris]

eve: iptables -t nat -A PREROUTING -p tcp --destination-port 1234 -j 
REDIRECT --to-port 1234
eve: echo 255 > /proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid
eve: nc -p -l 1234

alice: telnet bob 1234

messages on eve (192.168.2.2) shows:

Jul  8 11:47:34 cartman kernel: ip_ct_tcp: invalid packet ignored IN= 
OUT= PHYSIN=eth0 SRC=192.168.2.1 DST=192.168.2.3 LEN=60 TOS=0x10 
PREC=0x00 TTL=64 ID=18927 DF PROTO=TCP SPT=33080 DPT=1234 SEQ=1119072953 
ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT 
(020405B40402080A18B1070F0000000001030307)


192.168.2.1 is alice, 192.168.2.3 is bob

Thanks,
-Dirk

Patrick McHardy wrote:

> Dirk Morris wrote:
>
>> notice how the syn/ack comes from eve instead of bob, and it only 
>> matches 1 packet (the SYN I assume)
>> I suspect the syn/ack isnt matching in the conntrack table for some 
>> reason?
>> Could it be because the syn is marked, but the syn/ack isnt?
>
>
> No, that doesn't affect association with a conntrack entry. Please retry
> after doing "echo 255 > /proc/sys/net/ipv4/netfilter/ip_ct_log_invalid"
> and "modprobe ipt_LOG".
>
> Regards
> Patrick
>

Re: REDIRECT differences in 2.6.12.2

[Patrick McHardy]

Dirk Morris wrote:
> eve: iptables -t nat -A PREROUTING -p tcp --destination-port 1234 -j 
> REDIRECT --to-port 1234
> eve: echo 255 > /proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid
> eve: nc -p -l 1234
> 
> alice: telnet bob 1234
> 
> messages on eve (192.168.2.2) shows:
> 
> Jul  8 11:47:34 cartman kernel: ip_ct_tcp: invalid packet ignored IN= 
> OUT= PHYSIN=eth0 SRC=192.168.2.1 DST=192.168.2.3 LEN=60 TOS=0x10 
> PREC=0x00 TTL=64 ID=18927 DF PROTO=TCP SPT=33080 DPT=1234 SEQ=1119072953 
> ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT 
> (020405B40402080A18B1070F0000000001030307)

You didn't mention anything about a bridge. Please describe your setup
in full detail. And please use IPs instead of names.

Regards
Patrick

Re: REDIRECT differences in 2.6.12.2

[Dirk Morris]

Hey Patrick,

A bit more info:

internet <--> [alice 192.168.1.1]  <-->  [eve 192.168.1.2] <--> [bob 
192.168.1.3]

alice and bob are on 2.6.8.1, alice is running ipmasq so that eve and 
bob can get to the internet.

eve is a bridge:

~ # uname -a                                               
Linux eve 2.6.12.2 #4 SMP Fri Jul 8 11:05:21 PDT 2005 i686 GNU/Linux

~ # s iptables -t nat -nL                                  
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination        
REDIRECT   tcp  --  0.0.0.0/0            0.0.0.0/0           tcp 
dpt:1234 redir
ports 1234

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination        

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination    

~ # cat 
ifconfig                                                                                            

br0       Link encap:Ethernet  HWaddr 00:30:48:82:CC:A0 
          inet addr:192.168.2.2  Bcast:192.168.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3009 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2009 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:245119 (239.3 KiB)  TX bytes:453531 (442.9 KiB)

eth0      Link encap:Ethernet  HWaddr 00:30:48:82:CC:A0 
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2758 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2344 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:262532 (256.3 KiB)  TX bytes:496238 (484.6 KiB)
          Base address:0xc000 Memory:f2000000-f2020000

eth1      Link encap:Ethernet  HWaddr 00:30:48:82:CC:A1 
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:348 errors:0 dropped:0 overruns:0 frame:0
          TX packets:506 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:50201 (49.0 KiB)  TX bytes:58129 (56.7 KiB)
          Base address:0xd100 Memory:f1000000-f10200

~ # brctl show br0   
bridge name     bridge id               STP enabled     interfaces
br0             8000.00304882cca0       no              eth0
                                                        eth1
~ # route -n      
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use 
Iface
192.168.2.0     0.0.0.0         255.255.255.0   U     0      0        0 br0
0.0.0.0         192.168.2.1     0.0.0.0         UG    0      0        0 br0


I can give you lots more info if you like.

Hope this helps!

-Dirk

Patrick McHardy wrote:

> Dirk Morris wrote:
>
>> eve: iptables -t nat -A PREROUTING -p tcp --destination-port 1234 -j 
>> REDIRECT --to-port 1234
>> eve: echo 255 > /proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid
>> eve: nc -p -l 1234
>>
>> alice: telnet bob 1234
>>
>> messages on eve (192.168.2.2) shows:
>>
>> Jul  8 11:47:34 cartman kernel: ip_ct_tcp: invalid packet ignored IN= 
>> OUT= PHYSIN=eth0 SRC=192.168.2.1 DST=192.168.2.3 LEN=60 TOS=0x10 
>> PREC=0x00 TTL=64 ID=18927 DF PROTO=TCP SPT=33080 DPT=1234 
>> SEQ=1119072953 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT 
>> (020405B40402080A18B1070F0000000001030307)
>
>
> You didn't mention anything about a bridge. Please describe your setup
> in full detail. And please use IPs instead of names.
>
> Regards
> Patrick
>

Re: REDIRECT differences in 2.6.12.2

[Patrick McHardy]

[-- Attachment #1: Type: text/plain, Size: 211 bytes --]

Dirk Morris wrote:
> Hey Patrick,
> 
> A bit more info:

Thanks, we already identified the responsible patch, it is
the same problem as in the SO_ORIGINAL_DST thread. This
patch should fix it.

Regards
Patrick


[-- Attachment #2: x --]
[-- Type: text/plain, Size: 2780 bytes --]

[NETFILTER]: Revert nf_reset change

Revert the nf_reset change that caused so much trouble, drop conntrack
references manually before packets are queued to packet sockets.

Signed-off-by: Phil Oester <kernel@linuxace.com>
Signed-off-by: Patrick McHardy <kaber@trash.net>

---
commit c64e9fafffd203886f6c57149803f1824febd9bb
tree 40a90ab1427c25f7a36109eabd1a53b0a8f144be
parent 9fd73bed06eb5cf309e0cb5cdf3249a54fd3ff22
author Phil Oester <kernel@linuxace.com> Mon, 11 Jul 2005 02:04:08
committer Patrick McHardy <kaber@trash.net> Mon, 11 Jul 2005 02:04:08

 net/ipv4/ip_output.c                         |    9 ---------
 net/ipv4/netfilter/ip_conntrack_standalone.c |    7 +++++++
 net/packet/af_packet.c                       |    6 ++++++
 3 files changed, 13 insertions(+), 9 deletions(-)

diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
--- a/net/ipv4/ip_output.c
+++ b/net/ipv4/ip_output.c
@@ -107,7 +107,6 @@ static int ip_dev_loopback_xmit(struct s
 	newskb->pkt_type = PACKET_LOOPBACK;
 	newskb->ip_summed = CHECKSUM_UNNECESSARY;
 	BUG_TRAP(newskb->dst);
-	nf_reset(newskb);
 	netif_rx(newskb);
 	return 0;
 }
@@ -188,14 +187,6 @@ static inline int ip_finish_output2(stru
 		skb = skb2;
 	}
 
-#ifdef CONFIG_BRIDGE_NETFILTER
-	/* bridge-netfilter defers calling some IP hooks to the bridge layer
-	 * and still needs the conntrack reference.
-	 */
-	if (skb->nf_bridge == NULL)
-#endif
-		nf_reset(skb);
-
 	if (hh) {
 		int hh_alen;
 
diff --git a/net/ipv4/netfilter/ip_conntrack_standalone.c b/net/ipv4/netfilter/ip_conntrack_standalone.c
--- a/net/ipv4/netfilter/ip_conntrack_standalone.c
+++ b/net/ipv4/netfilter/ip_conntrack_standalone.c
@@ -432,6 +432,13 @@ static unsigned int ip_conntrack_defrag(
 				        const struct net_device *out,
 				        int (*okfn)(struct sk_buff *))
 {
+#if !defined(CONFIG_IP_NF_NAT) && !defined(CONFIG_IP_NF_NAT_MODULE)
+	/* Previously seen (loopback)?  Ignore.  Do this before
+           fragment check. */
+	if ((*pskb)->nfct)
+		return NF_ACCEPT;
+#endif
+
 	/* Gather fragments. */
 	if ((*pskb)->nh.iph->frag_off & htons(IP_MF|IP_OFFSET)) {
 		*pskb = ip_ct_gather_frags(*pskb,
diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -274,6 +274,9 @@ static int packet_rcv_spkt(struct sk_buf
 	dst_release(skb->dst);
 	skb->dst = NULL;
 
+	/* drop conntrack reference */
+	nf_reset(skb);
+
 	spkt = (struct sockaddr_pkt*)skb->cb;
 
 	skb_push(skb, skb->data-skb->mac.raw);
@@ -517,6 +520,9 @@ static int packet_rcv(struct sk_buff *sk
 	dst_release(skb->dst);
 	skb->dst = NULL;
 
+	/* drop conntrack reference */
+	nf_reset(skb);
+
 	spin_lock(&sk->sk_receive_queue.lock);
 	po->stats.tp_packets++;
 	__skb_queue_tail(&sk->sk_receive_queue, skb);