Session tracking failure - ssh packets dropped as INVALID

Session tracking failure - ssh packets dropped as INVALID

[John Zornig]

I'm setting up a host based firewall on a CentOS 5.1 system and I'm  
having issues with ssh sessions hanging.
I want to restrict incoming ssh sessions to those originating from a  
particular subnet and interface.

I can connect via ssh, but often when I generate a lot of traffic e.g.  
by cat'ing a large file or running top, the session hangs. By  
selective logging, I have discovered that when a session hangs the  
packets coming to port 22 for that session change from ESTABLISHED to  
INVALID and I have a rule that all INVALID packets are dropped. For  
some reason the connection tracking appears to be faulty. Is this a  
known issue or am I doing something incorrect? I've had this occur on  
a number of systems I'm setting up at the moment all are configured  
similarly.

iptables-save output below:

# Generated by iptables-save v1.3.5 on Sun Feb 10 11:50:18 2008
*filter
:INPUT DROP [36592:4741659]
:FORWARD DROP [0:0]
:OUTPUT DROP [31:11468]
-A INPUT -f -j LOG --log-prefix "FRAGMENT DROPPED "
-A INPUT -f -j DROP
-A INPUT -s 172.18.0.0/255.255.254.0 -i eth0 -p tcp -m state --state  
NEW,ESTABLISHED -m tcp --dport 22 -j LOG --log-prefix "ESTABLISHED -  
TO BE ACCEPTED "
-A INPUT -s 172.18.0.0/255.255.254.0 -i eth0 -p tcp -m state --state  
NEW,ESTABLISHED -m tcp --dport 22 -j ACCEPT
-A INPUT -m state --state INVALID -j LOG --log-prefix "INVALID - TO BE  
DROPPED "
-A INPUT -m state --state INVALID -j DROP
-A INPUT -i lo -j ACCEPT
-A INPUT -m addrtype --dst-type LOCAL -j LOG
-A FORWARD -j LOG
-A OUTPUT -d 172.18.0.0/255.255.254.0 -o eth0 -p tcp -m state --state  
ESTABLISHED -m tcp --sport 22 -j LOG --log-prefix "ESTABLISHED - TO BE  
ACCEPTED "
-A OUTPUT -d 172.18.0.0/255.255.254.0 -o eth0 -p tcp -m state --state  
ESTABLISHED -m tcp --sport 22 -j ACCEPT
-A OUTPUT -m state --state INVALID -j LOG --log-prefix "INVALID - TO  
BE DROPPED "
-A OUTPUT -m state --state INVALID -j DROP
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -j LOG
COMMIT
# Completed on Sun Feb 10 11:50:18 2008


Log excerpt while executing "ls -Rl /" in the ssh session.

Feb 10 13:15:09 logging kernel: ESTABLISHED - TO BE ACCEPTED IN=eth0  
OUT= MAC=00:1d:09:1e:b0:a4:00:d0:00:db:f8:00:08:00 SRC=172.18.0.168  
DST=192.168.154.56 LEN=52 TOS=0x10 PREC=0x00 TTL=61 ID=5833 DF  
PROTO=TCP SPT=50066 DPT=22 WINDOW=25100 RES=0x00 ACK URGP=0
Feb 10 13:15:09 logging kernel: ESTABLISHED - TO BE ACCEPTED IN=eth0  
OUT= MAC=00:1d:09:1e:b0:a4:00:d0:00:db:f8:00:08:00 SRC=172.18.0.168  
DST=192.168.154.56 LEN=52 TOS=0x10 PREC=0x00 TTL=61 ID=30003 DF  
PROTO=TCP SPT=50066 DPT=22 WINDOW=25068 RES=0x00 ACK URGP=0
Feb 10 13:15:09 logging kernel: ESTABLISHED - TO BE ACCEPTED IN=eth0  
OUT= MAC=00:1d:09:1e:b0:a4:00:d0:00:db:f8:00:08:00 SRC=172.18.0.168  
DST=192.168.154.56 LEN=52 TOS=0x10 PREC=0x00 TTL=61 ID=11272 DF  
PROTO=TCP SPT=50066 DPT=22 WINDOW=25044 RES=0x00 ACK URGP=0
Feb 10 13:15:09 logging kernel: ESTABLISHED - TO BE ACCEPTED IN=eth0  
OUT= MAC=00:1d:09:1e:b0:a4:00:d0:00:db:f8:00:08:00 SRC=172.18.0.168  
DST=192.168.154.56 LEN=52 TOS=0x10 PREC=0x00 TTL=61 ID=16325 DF  
PROTO=TCP SPT=50066 DPT=22 WINDOW=24916 RES=0x00 ACK URGP=0
Feb 10 13:15:09 logging kernel: ESTABLISHED - TO BE ACCEPTED IN=eth0  
OUT= MAC=00:1d:09:1e:b0:a4:00:d0:00:db:f8:00:08:00 SRC=172.18.0.168  
DST=192.168.154.56 LEN=52 TOS=0x10 PREC=0x00 TTL=61 ID=23358 DF  
PROTO=TCP SPT=50066 DPT=22 WINDOW=24748 RES=0x00 ACK URGP=0
Feb 10 13:15:09 logging kernel: INVALID - TO BE DROPPED IN=eth0 OUT=  
MAC=00:1d:09:1e:b0:a4:00:d0:00:db:f8:00:08:00 SRC=172.18.0.168  
DST=192.168.154.56 LEN=64 TOS=0x10 PREC=0x00 TTL=61 ID=22690 DF  
PROTO=TCP SPT=50066 DPT=22 WINDOW=23572 RES=0x00 ACK URGP=0
Feb 10 13:15:09 logging kernel: INVALID - TO BE DROPPED IN=eth0 OUT=  
MAC=00:1d:09:1e:b0:a4:00:d0:00:db:f8:00:08:00 SRC=172.18.0.168  
DST=192.168.154.56 LEN=64 TOS=0x10 PREC=0x00 TTL=61 ID=9482 DF  
PROTO=TCP SPT=50066 DPT=22 WINDOW=23572 RES=0x00 ACK URGP=0
Feb 10 13:15:09 logging kernel: INVALID - TO BE DROPPED IN=eth0 OUT=  
MAC=00:1d:09:1e:b0:a4:00:d0:00:db:f8:00:08:00 SRC=172.18.0.168  
DST=192.168.154.56 LEN=64 TOS=0x10 PREC=0x00 TTL=61 ID=22670 DF  
PROTO=TCP SPT=50066 DPT=22 WINDOW=23572 RES=0x00 ACK URGP=0
Feb 10 13:15:09 logging kernel: INVALID - TO BE DROPPED IN=eth0 OUT=  
MAC=00:1d:09:1e:b0:a4:00:d0:00:db:f8:00:08:00 SRC=172.18.0.168  
DST=192.168.154.56 LEN=64 TOS=0x10 PREC=0x00 TTL=61 ID=26191 DF  
PROTO=TCP SPT=50066 DPT=22 WINDOW=27668 RES=0x00 ACK URGP=0
Feb 10 13:15:09 logging kernel: INVALID - TO BE DROPPED IN=eth0 OUT=  
MAC=00:1d:09:1e:b0:a4:00:d0:00:db:f8:00:08:00 SRC=172.18.0.168  
DST=192.168.154.56 LEN=112 TOS=0x10 PREC=0x00 TTL=61 ID=31011 DF  
PROTO=TCP SPT=50066 DPT=22 WINDOW=27668 RES=0x00 ACK PSH URGP=0
Feb 10 13:15:09 logging kernel: INVALID - TO BE DROPPED IN=eth0 OUT=  
MAC=00:1d:09:1e:b0:a4:00:d0:00:db:f8:00:08:00 SRC=172.18.0.168  
DST=192.168.154.56 LEN=64 TOS=0x10 PREC=0x00 TTL=61 ID=19328 DF  
PROTO=TCP SPT=50066 DPT=22 WINDOW=31764 RES=0x00 ACK URGP=0
Feb 10 13:15:09 logging kernel: INVALID - TO BE DROPPED IN=eth0 OUT=  
MAC=00:1d:09:1e:b0:a4:00:d0:00:db:f8:00:08:00 SRC=172.18.0.168  
DST=192.168.154.56 LEN=64 TOS=0x10 PREC=0x00 TTL=61 ID=11844 DF  
PROTO=TCP SPT=50066 DPT=22 WINDOW=33252 RES=0x00 ACK URGP=0



JZ

Re: Session tracking failure - ssh packets dropped as INVALID

[Eray Aslan]

On 10.02.2008 05:29, John Zornig wrote:
> I can connect via ssh, but often when I generate a lot of traffic e.g. 
> by cat'ing a large file or running top, the session hangs. By selective 
> logging, I have discovered that when a session hangs the packets coming 
> to port 22 for that session change from ESTABLISHED to INVALID and I 
> have a rule that all INVALID packets are dropped. For some reason the 
> connection tracking appears to be faulty. Is this a known issue or am I 
> doing something incorrect? I've had this occur on a number of systems 
> I'm setting up at the moment all are configured similarly.

No word of advice unfortunately but I have bitten by packets getting 
dropped by the INVALID rule as well.  In the end, I have disabled the 
rule that drops INVALID packets.

On 15.11.2007 Jozsef Kadlezsic wrote to a similar question:

> Please enable full internal logging in netfilter and make sure at least 
> one loggin target module is loaded in and record by tcpdump one full 
> TCP session where such packets occurs. Then send me the generated kernel 
> log and the dump file so that I could analyze it. 

I did not have the time to debug it.  Maybe you can.  In addition, I 
have seen reports that increasing timeouts may help 
(ip_conntrack_tcp_timeout_close_wait, ip_conntrack_tcp_timeout_close, 
ip_conntrack_tcp_timeout_fin_wait, ip_conntrack_tcp_timeout_last_ack). 
It did not help for me and of course this is just a work around.  The 
real problem lies elsewhere.

-- 
Eray

Re: Session tracking failure - ssh packets dropped as INVALID

[Martijn Lievaart]

Eray Aslan wrote:
> On 10.02.2008 05:29, John Zornig wrote:
>> I can connect via ssh, but often when I generate a lot of traffic 
>> e.g. by cat'ing a large file or running top, the session hangs. By 
>> selective logging, I have discovered that when a session hangs the 
>> packets coming to port 22 for that session change from ESTABLISHED to 
>> INVALID and I have a rule that all INVALID packets are dropped. For 
>> some reason the connection tracking appears to be faulty. Is this a 
>> known issue or am I doing something incorrect? I've had this occur on 
>> a number of systems I'm setting up at the moment all are configured 
>> similarly.
>
> No word of advice unfortunately but I have bitten by packets getting 
> dropped by the INVALID rule as well. In the end, I have disabled the 
> rule that drops INVALID packets.
>
> On 15.11.2007 Jozsef Kadlezsic wrote to a similar question:
>
>> Please enable full internal logging in netfilter and make sure at 
>> least one loggin target module is loaded in and record by tcpdump one 
>> full TCP session where such packets occurs. Then send me the 
>> generated kernel log and the dump file so that I could analyze it. 
>
> I did not have the time to debug it. Maybe you can. In addition, I 
> have seen reports that increasing timeouts may help 
> (ip_conntrack_tcp_timeout_close_wait, ip_conntrack_tcp_timeout_close, 
> ip_conntrack_tcp_timeout_fin_wait, ip_conntrack_tcp_timeout_last_ack). 
> It did not help for me and of course this is just a work around. The 
> real problem lies elsewhere.
>

Enabling conntrack_be_liberal seems to have solved the issue for me.

HTH,
M4

Re: Session tracking failure - ssh packets dropped as INVALID

[John Zornig]

Thanks Martijn,


On 11/02/2008, at 6:16 AM, Martijn Lievaart wrote:

>
> Enabling conntrack_be_liberal seems to have solved the issue for me.
>


echo "1" > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal

seems to work for my systems. No more INVALID packets.

Some of the netfilter firewall how-to guides I've found say that  
ip_conntrack_tcp_be_liberal should never be set on a firewall. So I'm  
still left wondering if I'm working around a bug and my firewall is  
not as strict as it should be, or are these how-to's all wrong.

Regards,
JZ

Re: Session tracking failure - ssh packets dropped as INVALID

[Philip Craig]

John Zornig wrote:
> echo "1" > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal
> 
> seems to work for my systems. No more INVALID packets.
> 
> Some of the netfilter firewall how-to guides I've found say that  
> ip_conntrack_tcp_be_liberal should never be set on a firewall. So I'm  
> still left wondering if I'm working around a bug and my firewall is  
> not as strict as it should be, or are these how-to's all wrong.

This is just a workaround for a bug.

Are you using a recent kernel?

It would be good to work with Jozsef to get this fixed if you are able
to help debug it.  Also report it on the netfilter-devel list.

To debug, I think the first steps are to enable logging with:

	modprobe ipt_LOG
	echo 255 >/proc/sys/net/netfilter/nf_conntrack_log_invalid

and check your syslog for errors.

Also get a tcpdump of the ssh traffic with something like:

	tcpdump -i eth0 -s 0 -w dump.pcap tcp port 22

Re: Session tracking failure - ssh packets dropped as INVALID

[Jozsef Kadlecsik]

On Mon, 11 Feb 2008, Philip Craig wrote:

> John Zornig wrote:
> > echo "1" > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal
> 
> This is just a workaround for a bug.
> 
> Are you using a recent kernel?
> 
> It would be good to work with Jozsef to get this fixed if you are able
> to help debug it.  Also report it on the netfilter-devel list.
> 
> To debug, I think the first steps are to enable logging with:
> 
> 	modprobe ipt_LOG
> 	echo 255 >/proc/sys/net/netfilter/nf_conntrack_log_invalid
> 
> and check your syslog for errors.
> 
> Also get a tcpdump of the ssh traffic with something like:
> 
> 	tcpdump -i eth0 -s 0 -w dump.pcap tcp port 22

Exactly. By enabling ip_conntrack_tcp_be_liberal you just mask the 
problem. And without dumped traffic to analyze I cannot do anything to 
improve the kernel.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Session tracking failure - ssh packets dropped as INVALID

[Martijn Lievaart]

Jozsef Kadlecsik wrote:
> On Mon, 11 Feb 2008, Philip Craig wrote:
>
>   
>> John Zornig wrote:
>>     
>>> echo "1" > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal
>>>       
>> This is just a workaround for a bug.
>>
>> Are you using a recent kernel?
>>     

I'm not the OP, so FWIW. I see this on two machines where I also have 
root. Both have older kernels. One runs Xen, so upgrading is not easy. 
I'll try to upgrade the other to see if the problem goes away.

HTH,
M4