Connection tracking looses packets?

Connection tracking looses packets?

[Justin Schoeman]

Hi all,

I am having problems with a firewall here.  The basic configuration 
works perfectly, but we have occasional problems with DNAT.  Long 
running sessions (we have found the problem on ftp, http, pop and smtp 
sessions) occasionally stall, and transfer no more data.  Looking at the 
firewall logs, I see:

INVALID NAT: IN=eth3 OUT= MAC=00:02:55:d6:eb:ae:00:11:bc:25:96:80:08:00 
SRC=196.x.x.x DST=10.50.18.22 LEN=52 TOS=0x00 PREC=0x00 TTL=117 ID=37995 
PROTO=TCP SPT=57952 DPT=80 WINDOW=65535 RES=0x00 ACK URGP=0

(This is logged by a -m state --state INVALID rule in the mangle table.) 
The packet is then not natted, but drops into the INPUT chain for the 
firewall itself, where it is dropped.

This seems to be the problem, as there are always one or more of these 
messages on a stalled connection.

Looking at /proc/net/ip_conntrack immediately after the above message is 
logged, I see:

tcp      6 295 ESTABLISHED src=196.x.x.x dst=10.50.18.22 sport=57952 
dport=80 packets=5 bytes=633 src=192.168.1.22 dst=196.x.x.x sport=80 
dport=57952 packets=12 bytes=13793 [ASSURED] mark=0 use=1

So the connection seems to be valid.

Any idea why this packet may be dropped, or are there other possible 
reasons why the connection may be stalling?

Googling reveals a significant number of people having problems with 
random stalls within a connection, but so far I have not seen a single 
answer to the problem.

Any help/advice will be greatly appreciated.

Thanks,

Justin

RE: Connection tracking looses packets?

[Sietse van Zanen]

The problem is, that there is no easy answer.
 
The packets are probably dropped due to incorrect sequence numbers or TCP flags. Or they might arrive untimely. Netfilter will then not see them as part of an established connection and mark them invalid.
 
The point is, that many application servers (especially those from that horrible company in Redmond) do not fully adhere RFC's, or they are configured incorrectly. This often causes issues with firewalls dropping packes. There also might be incorrectly configured networking devices (routers, concetrators etc) in between the connection end-points.
 
I would say the change is about 0-1% that it is netfilter causing the issue. It just inspects all the packet conforming RFC's, the way it should be done. There is therefor a 99-100% change the problem lies elsewhere and most likely it's incorrectly working or configured clients and/or servers.
 
Though you probably will not find this answer useful in solving your problem, I hope it still clarifies some things.
 
-Sietse
 
 
________________________________

From: netfilter-bounces@lists.netfilter.org on behalf of Justin Schoeman
Sent: Wed 24-May-06 15:25
To: netfilter@lists.netfilter.org
Subject: Connection tracking looses packets?



Hi all,

I am having problems with a firewall here.  The basic configuration
works perfectly, but we have occasional problems with DNAT.  Long
running sessions (we have found the problem on ftp, http, pop and smtp
sessions) occasionally stall, and transfer no more data.  Looking at the
firewall logs, I see:

INVALID NAT: IN=eth3 OUT= MAC=00:02:55:d6:eb:ae:00:11:bc:25:96:80:08:00
SRC=196.x.x.x DST=10.50.18.22 LEN=52 TOS=0x00 PREC=0x00 TTL=117 ID=37995
PROTO=TCP SPT=57952 DPT=80 WINDOW=65535 RES=0x00 ACK URGP=0

(This is logged by a -m state --state INVALID rule in the mangle table.)
The packet is then not natted, but drops into the INPUT chain for the
firewall itself, where it is dropped.

This seems to be the problem, as there are always one or more of these
messages on a stalled connection.

Looking at /proc/net/ip_conntrack immediately after the above message is
logged, I see:

tcp      6 295 ESTABLISHED src=196.x.x.x dst=10.50.18.22 sport=57952
dport=80 packets=5 bytes=633 src=192.168.1.22 dst=196.x.x.x sport=80
dport=57952 packets=12 bytes=13793 [ASSURED] mark=0 use=1

So the connection seems to be valid.

Any idea why this packet may be dropped, or are there other possible
reasons why the connection may be stalling?

Googling reveals a significant number of people having problems with
random stalls within a connection, but so far I have not seen a single
answer to the problem.

Any help/advice will be greatly appreciated.

Thanks,

Justin

Re: Connection tracking looses packets?

[Pascal Hambourg]

Hello,

Justin Schoeman a écrit :
> 
> (This is logged by a -m state --state INVALID rule in the mangle table.) 
> The packet is then not natted, but drops into the INPUT chain for the 
> firewall itself, where it is dropped.

Yes, packets marked in the INVALID state by the connection tracking are 
not handled by NAT.


> Any idea why this packet may be dropped, or are there other possible 
> reasons why the connection may be stalling?

If the reason is incorrect sequence numbers with a kernel >= 2.6.9 or 
including patch tcp-window-tracking, you can try to enable (value > 0) 
the parameter /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal 
(net.ipv4.netfilter.ip_conntrack_tcp_be_liberal in sysctl) :

   ip_conntrack_tcp_be_liberal
                 when enabled, only out of window reset (RST) segments
                 are marked as INVALID; when disabled (default), all
                 out of window packets are marked as INVALID.

Connection tracking looses packets?

[Mike Polek]

Hi,

For anyone interested, I was attempting to set up a load balancing
FTP server, which I've done before. I had a brand new kernel,
installed the nfct patches, etc. Not knowing that there had
been a tcp-window-tracking patch to the kernel, I couldn't figure
out why the first passive connection would always work, but
later passive connections would fail. After poking around, I could
see the (*pskb)->nfct pointer was NULL, and I was completely
stumped.

Thanks to Pascal Hambourg's post, I tried

echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal

and magically, everything works perfectly, just like the old server.

I hope this helps others who are load balancing ftp servers.

Thanks to everyone who contributes to Open Source and these
mailing lists!!!

Mike Polek
Pictage, Inc.