Connection reset problem

Connection reset problem

[Brian McNally]

I've got a host that periodically resets connections to clients, and  
I'm having trouble pinpointing the problem. Here are the details:

Server:

2.6.13 kernel
CAOS Linux
iptables 1.3.1

Relevant iptables rules:

ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state  
RELATED,ESTABLISHED
ACCEPT     tcp  --  0.0.0.0/0            211.7.9.193       tcp dpt:22  
state NEW,ESTABLISHED
LOG        all  --  0.0.0.0/0            211.7.9.193       LOG flags  
0 level 4 prefix `BLOCKED: '
REJECT     tcp  --  0.0.0.0/0            211.7.9.193       reject- 
with tcp-reset

I see connections that should be being tracked as established end up  
matching the blocked and reject rules. For example, here's what ends  
up in systems logs for some sample packets:

Jan 4 09:38:57 serverhost kernel: BLOCKED: IN=eth1 OUT=  
MAC=00:35:48:71:c6:ea:00:14:f2:dd:f7:46:08:00 SRC=172.28.10.170  
DST=211.7.9.193 LEN=64 TOS=0x10 PREC=0x00 TTL=63 ID=57445 DF PROTO=TCP
SPT=51645 DPT=22 WINDOW=16022 RES=0x00 ACK URGP=0

Jan 4 09:45:21 serverhost kernel: BLOCKED: IN=eth1 OUT=  
MAC=00:35:48:71:c6:ea:00:14:f2:dd:f7:46:08:00 SRC=172.28.10.170  
DST=211.7.9.193 LEN=64 TOS=0x10 PREC=0x00 TTL=63 ID=56765 DF  
PROTO=TCP SPT=36234 DPT=22 WINDOW=7680 RES=0x00 ACK URGP=0

And, here's a tcpdump snipet from the first of those packets:

09:38:57.547888 IP serverhost.example.com.ssh >  
172.28.10.170.51645: . 131806:133174(1368) ack 27689 win 540  
<nop,nop,timestamp 22101576 816063448>

09:38:57.547908 IP serverhost.example.com.ssh > 172.28.10.170.51645:  
P 133174:133838(664) ack 27689 win 540 <nop,nop,timestamp 22101577  
816063448>

09:38:57.548366 IP 172.28.10.170.51645 >  
serverhost.example.com.ssh: . ack 131806 win 16022 <nop,nop,timestamp  
816063449 22021269,nop,nop,sack sack 1 {1740668210:1740668874}>

09:38:57.548574 IP serverhost.example.com.ssh > 172.28.10.170.51645:  
R 1675860173:1675860173(0) win 0

09:38:57.548579 IP 172.28.10.170.51645 >  
serverhost.example.com.ssh: . ack 133838 win 15673 <nop,nop,timestamp  
816063449 22101576>

09:38:57.548740 IP 172.28.10.170.51645 >  
serverhost.example.com.ssh: . ack 133838 win 0

* Notice the 4th line, where the connection gets reset by  
serverhost.example.com.

So, what's going on here? I've got two theories but I'm certainly  
open to other ideas.

1. Our border firewall (something I don't have direct control over)  
is somehow mangling the packets so that iptables connection tracking  
gets thrown off. I'd want more concrete evidence that this is the  
case before proceeding though.

2. Some kernel options (or some collection of kernel options) is  
causing iptables or TCP/IP to screw up. This is a custom kernel, and  
not something I set up originally, so it's in an unknown state. If  
you suspect some kernel variable could reveal the problem, what would  
you check? I've played around a bit without luck.

--
Brian McNally

Re: Connection reset problem

[Michael Gale]

Hey,

	I have seen something like this before and was able to track it down by doing a tcpdump on both sides of the firewall and comparing the two.

The problem was an extra SYN packet was being sent by a Cisco router, which was breaking the connection tracking for that connection. A temp fix was do make the following 
change:

--snip--
if [ -e "/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal" ];
then
     echo "Disabing TCP Window Tracking -- Added by IT"
     echo 1 >
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal
fi
--snip--

Michael Gale


Brian McNally wrote:
> I've got a host that periodically resets connections to clients, and I'm 
> having trouble pinpointing the problem. Here are the details:
> 
> Server:
> 
> 2.6.13 kernel
> CAOS Linux
> iptables 1.3.1
> 
> Relevant iptables rules:
> 
> ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state 
> RELATED,ESTABLISHED
> ACCEPT     tcp  --  0.0.0.0/0            211.7.9.193       tcp dpt:22 
> state NEW,ESTABLISHED
> LOG        all  --  0.0.0.0/0            211.7.9.193       LOG flags 0 
> level 4 prefix `BLOCKED: '
> REJECT     tcp  --  0.0.0.0/0            211.7.9.193       reject-with 
> tcp-reset
> 
> I see connections that should be being tracked as established end up 
> matching the blocked and reject rules. For example, here's what ends up 
> in systems logs for some sample packets:
> 
> Jan 4 09:38:57 serverhost kernel: BLOCKED: IN=eth1 OUT= 
> MAC=00:35:48:71:c6:ea:00:14:f2:dd:f7:46:08:00 SRC=172.28.10.170 
> DST=211.7.9.193 LEN=64 TOS=0x10 PREC=0x00 TTL=63 ID=57445 DF PROTO=TCP
> SPT=51645 DPT=22 WINDOW=16022 RES=0x00 ACK URGP=0
> 
> Jan 4 09:45:21 serverhost kernel: BLOCKED: IN=eth1 OUT= 
> MAC=00:35:48:71:c6:ea:00:14:f2:dd:f7:46:08:00 SRC=172.28.10.170 
> DST=211.7.9.193 LEN=64 TOS=0x10 PREC=0x00 TTL=63 ID=56765 DF PROTO=TCP 
> SPT=36234 DPT=22 WINDOW=7680 RES=0x00 ACK URGP=0
> 
> And, here's a tcpdump snipet from the first of those packets:
> 
> 09:38:57.547888 IP serverhost.example.com.ssh > 172.28.10.170.51645: . 
> 131806:133174(1368) ack 27689 win 540 <nop,nop,timestamp 22101576 
> 816063448>
> 
> 09:38:57.547908 IP serverhost.example.com.ssh > 172.28.10.170.51645: P 
> 133174:133838(664) ack 27689 win 540 <nop,nop,timestamp 22101577 816063448>
> 
> 09:38:57.548366 IP 172.28.10.170.51645 > serverhost.example.com.ssh: . 
> ack 131806 win 16022 <nop,nop,timestamp 816063449 22021269,nop,nop,sack 
> sack 1 {1740668210:1740668874}>
> 
> 09:38:57.548574 IP serverhost.example.com.ssh > 172.28.10.170.51645: R 
> 1675860173:1675860173(0) win 0
> 
> 09:38:57.548579 IP 172.28.10.170.51645 > serverhost.example.com.ssh: . 
> ack 133838 win 15673 <nop,nop,timestamp 816063449 22101576>
> 
> 09:38:57.548740 IP 172.28.10.170.51645 > serverhost.example.com.ssh: . 
> ack 133838 win 0
> 
> * Notice the 4th line, where the connection gets reset by 
> serverhost.example.com.
> 
> So, what's going on here? I've got two theories but I'm certainly open 
> to other ideas.
> 
> 1. Our border firewall (something I don't have direct control over) is 
> somehow mangling the packets so that iptables connection tracking gets 
> thrown off. I'd want more concrete evidence that this is the case before 
> proceeding though.
> 
> 2. Some kernel options (or some collection of kernel options) is causing 
> iptables or TCP/IP to screw up. This is a custom kernel, and not 
> something I set up originally, so it's in an unknown state. If you 
> suspect some kernel variable could reveal the problem, what would you 
> check? I've played around a bit without luck.
> 
> -- 
> Brian McNally
> 
> 

-- 
Michael Gale

Red Hat Certified Engineer
Network Administrator
Pason Systems Corp.