Re: Connection reset problem

[Michael Gale <michael.gale@pason.com>]

Hey,

	I have seen something like this before and was able to track it down by doing a tcpdump on both sides of the firewall and comparing the two.

The problem was an extra SYN packet was being sent by a Cisco router, which was breaking the connection tracking for that connection. A temp fix was do make the following 
change:

--snip--
if [ -e "/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal" ];
then
     echo "Disabing TCP Window Tracking -- Added by IT"
     echo 1 >
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal
fi
--snip--

Michael Gale


Brian McNally wrote:
> I've got a host that periodically resets connections to clients, and I'm 
> having trouble pinpointing the problem. Here are the details:
> 
> Server:
> 
> 2.6.13 kernel
> CAOS Linux
> iptables 1.3.1
> 
> Relevant iptables rules:
> 
> ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state 
> RELATED,ESTABLISHED
> ACCEPT     tcp  --  0.0.0.0/0            211.7.9.193       tcp dpt:22 
> state NEW,ESTABLISHED
> LOG        all  --  0.0.0.0/0            211.7.9.193       LOG flags 0 
> level 4 prefix `BLOCKED: '
> REJECT     tcp  --  0.0.0.0/0            211.7.9.193       reject-with 
> tcp-reset
> 
> I see connections that should be being tracked as established end up 
> matching the blocked and reject rules. For example, here's what ends up 
> in systems logs for some sample packets:
> 
> Jan 4 09:38:57 serverhost kernel: BLOCKED: IN=eth1 OUT= 
> MAC=00:35:48:71:c6:ea:00:14:f2:dd:f7:46:08:00 SRC=172.28.10.170 
> DST=211.7.9.193 LEN=64 TOS=0x10 PREC=0x00 TTL=63 ID=57445 DF PROTO=TCP
> SPT=51645 DPT=22 WINDOW=16022 RES=0x00 ACK URGP=0
> 
> Jan 4 09:45:21 serverhost kernel: BLOCKED: IN=eth1 OUT= 
> MAC=00:35:48:71:c6:ea:00:14:f2:dd:f7:46:08:00 SRC=172.28.10.170 
> DST=211.7.9.193 LEN=64 TOS=0x10 PREC=0x00 TTL=63 ID=56765 DF PROTO=TCP 
> SPT=36234 DPT=22 WINDOW=7680 RES=0x00 ACK URGP=0
> 
> And, here's a tcpdump snipet from the first of those packets:
> 
> 09:38:57.547888 IP serverhost.example.com.ssh > 172.28.10.170.51645: . 
> 131806:133174(1368) ack 27689 win 540 <nop,nop,timestamp 22101576 
> 816063448>
> 
> 09:38:57.547908 IP serverhost.example.com.ssh > 172.28.10.170.51645: P 
> 133174:133838(664) ack 27689 win 540 <nop,nop,timestamp 22101577 816063448>
> 
> 09:38:57.548366 IP 172.28.10.170.51645 > serverhost.example.com.ssh: . 
> ack 131806 win 16022 <nop,nop,timestamp 816063449 22021269,nop,nop,sack 
> sack 1 {1740668210:1740668874}>
> 
> 09:38:57.548574 IP serverhost.example.com.ssh > 172.28.10.170.51645: R 
> 1675860173:1675860173(0) win 0
> 
> 09:38:57.548579 IP 172.28.10.170.51645 > serverhost.example.com.ssh: . 
> ack 133838 win 15673 <nop,nop,timestamp 816063449 22101576>
> 
> 09:38:57.548740 IP 172.28.10.170.51645 > serverhost.example.com.ssh: . 
> ack 133838 win 0
> 
> * Notice the 4th line, where the connection gets reset by 
> serverhost.example.com.
> 
> So, what's going on here? I've got two theories but I'm certainly open 
> to other ideas.
> 
> 1. Our border firewall (something I don't have direct control over) is 
> somehow mangling the packets so that iptables connection tracking gets 
> thrown off. I'd want more concrete evidence that this is the case before 
> proceeding though.
> 
> 2. Some kernel options (or some collection of kernel options) is causing 
> iptables or TCP/IP to screw up. This is a custom kernel, and not 
> something I set up originally, so it's in an unknown state. If you 
> suspect some kernel variable could reveal the problem, what would you 
> check? I've played around a bit without luck.
> 
> -- 
> Brian McNally
> 
> 

-- 
Michael Gale

Red Hat Certified Engineer
Network Administrator
Pason Systems Corp.