conntrack INVALID packets

conntrack INVALID packets

[Roman Ledovskiy]

Hi,

I have linux firewall in between internet and my network, diagram is
following:

Upstream
   |
Firewall server
   |
my network

"My network" hosts many different email/web servers.

Firewall server is not a bridge, it has 2 interfaces (one to upstream, one
to internal network)
Not much kernel configuration, kernel is default kernel from Centos5 x64
repo: 2.6.18-8.1.10.el5

On this server I get quite a lot of INVALID packets, about 0.07% of all and
I can't figure out why..

Iptables output is quite big so can't post it here, but the logic is simple
- inspect NEW packets, accept RELATED/ESTABLISHED, drop INVALID.


Conntrack is not full:
net.ipv4.netfilter.ip_conntrack_count = 5888
net.ipv4.netfilter.ip_conntrack_max = 65536

server has a lot of free memory also.

It is not ip_conntrack_tcp_be_liberal or ip_conntrack_checksum (changing
these values doesn't help)
 
There is definitely something wrong because I see INVALID packets not only
from outside -> my network, but from my network -> reverse also (although a
lot less)

The only thing I can think of is that one of "ip_conntrack_*_timeout*"
values is reached and conntrack removes connection (and after that there is
one more packet in regards to that deleted connection)

Could anybody tell possible reasons for packet being marked as INVALID
besides those mentioned above.


Any help is appreciated.



Thanks
Roman Ledovskiy

Re: conntrack INVALID packets

[Olivier Sessink]

Roman Ledovskiy wrote:
> Hi,
> 
> I have linux firewall in between internet and my network, diagram is
> following:
> 
> Upstream
>    |
> Firewall server
>    |
> my network
> 
> "My network" hosts many different email/web servers.
> 
> Firewall server is not a bridge, it has 2 interfaces (one to upstream, one
> to internal network)
> Not much kernel configuration, kernel is default kernel from Centos5 x64
> repo: 2.6.18-8.1.10.el5
> 
> On this server I get quite a lot of INVALID packets, about 0.07% of all and
> I can't figure out why..

same thing here. most of the INVALID packets seem to have RST or FIN ACK
set. So what I did, I increased the timeouts to setup a connection, and
I increased the timeouts for connection closing. That helped a lot.

echo "240" >
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_close_wait
echo "60" > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_close
echo "240" > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_fin_wait
echo "60" > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_last_ack

regards,
	Olivier

Re: conntrack INVALID packets

[Xesc Arbona]

> Olivier Sessink wrote:
>> Roman Ledovskiy wrote:
>> Hi,
>> 
>> I have linux firewall in between internet and my network, diagram is
>> following:
>> 
>> Upstream
>>    |
>> Firewall server
>>    |
>> my network
>>  
>> "My network" hosts many different email/web servers.
>> 
>> Firewall server is not a bridge, it has 2 interfaces (one to upstream, one
>> to internal network)
>> Not much kernel configuration, kernel is default kernel from Centos5 x64
>> repo: 2.6.18-8.1.10.el5
>> 
>> On this server I get quite a lot of INVALID packets, about 0.07% of all and
>> I can't figure out why..
>
> same thing here. most of the INVALID packets seem to have RST or FIN ACK
> set. So what I did, I increased the timeouts to setup a connection, and
> I increased the timeouts for connection closing. That helped a lot.
>

Hi,

I have a similar problem, but increasing the timeouts didn't help me. Our situation is:

We have several Windows backend webservers on an internal network, connected to a Debian machine (kernel 2.6.18.dfsg.1-24) which has a direct connection to Internet and acts as Reverse-Proxy. No traffic is forwarded, incoming requests are proxied to the backend servers. We use iptables 1.3.6.0debian1-5 and ferm as firewall on the same machine.

We have been experiencing timeouts between the proxy and the backend servers. I've looked at the firewall configuration, and several packets were dropped because itpables marks their state as INVALID. On the INPUT chain I've changed the rules:

  mod state state INVALID DROP;
  mod state state (ESTABLISHED RELATED) ACCEPT;

by

  mod state state INVALID goto DEBUG; 
  mod state state (ESTABLISHED RELATED) ACCEPT;

and created a DEBUG function that logs information about the packets. Most packets being marked as INVALID are RST, but also SYN-ACK, FIN-ACK and ACK. 

/proc/sys/net/ipv4/netfilter/ip_conntrack_count is 6177
/proc/sys/net/ipv4/netfilter/ip_conntrack_max is 32768

Jun  2 17:58:27 proxy003 kernel: TCP DEBUG: IN=eth0 OUT= MAC=00:0c:29:4a:b9:e8:00:0c:29:0e:d8:f8:08:00 SRC=10.231.0.55 DST=10.231.0.203 LEN=64 TOS=0x00 PREC=0x00 TTL=128 ID=27512 PROTO=TCP SPT=82 DPT=40508 WINDOW=64240 RES=0x00 ACK SYN URGP=0
Jun  2 17:58:31 proxy003 kernel: TCP DEBUG: IN=eth0 OUT= MAC=00:0c:29:4a:b9:e8:00:0c:29:0e:d8:f8:08:00 SRC=10.231.0.55 DST=10.231.0.203 LEN=64 TOS=0x00 PREC=0x00 TTL=128 ID=27893 PROTO=TCP SPT=82 DPT=40574 WINDOW=64240 RES=0x00 ACK SYN URGP=0
Jun  2 17:58:31 proxy003 kernel: TCP DEBUG: IN=eth0 OUT= MAC=00:0c:29:4a:b9:e8:00:0c:29:0e:d8:f8:08:00 SRC=10.231.0.55 DST=10.231.0.203 LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=27903 DF PROTO=TCP SPT=82 DPT=40574 WINDOW=63875 RES=0x00 ACK URGP=0
Jun  2 17:58:31 proxy003 kernel: TCP DEBUG: IN=eth0 OUT= MAC=00:0c:29:4a:b9:e8:00:0c:29:0e:d8:f8:08:00 SRC=10.231.0.55 DST=10.231.0.203 LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=27904 DF PROTO=TCP SPT=82 DPT=40574 WINDOW=63875 RES=0x00 ACK FIN URGP=0

* Does someone know what can be causing this? and how to solve it?

* When sending a SYN packet (via OUTPUT chain) to a backend server, shouldn't a connection appear as state NEW on /proc/net/ip_conntrack? And when a SYN-ACK comes from the backend server, shoudln't be marked as ESTABLISHED?

Thanks a lot!

--
Xesc

Re: conntrack INVALID packets

[Jozsef Kadlecsik]

On Tue, 2 Jun 2009, Xesc Arbona wrote:

> > Olivier Sessink wrote:
> >> Roman Ledovskiy wrote:
> >> Hi,
> >> 
> >> I have linux firewall in between internet and my network, diagram is
> >> following:
> >> 
> >> Upstream
> >>    |
> >> Firewall server
> >>    |
> >> my network
> >>  
> >> "My network" hosts many different email/web servers.
> >> 
> >> Firewall server is not a bridge, it has 2 interfaces (one to upstream, one
> >> to internal network)
> >> Not much kernel configuration, kernel is default kernel from Centos5 x64
> >> repo: 2.6.18-8.1.10.el5
> >> 
> >> On this server I get quite a lot of INVALID packets, about 0.07% of all and
> >> I can't figure out why..
> >
> > same thing here. most of the INVALID packets seem to have RST or FIN ACK
> > set. So what I did, I increased the timeouts to setup a connection, and
> > I increased the timeouts for connection closing. That helped a lot.
> 
> I have a similar problem, but increasing the timeouts didn't help me. 
> Our situation is:
> 
> We have several Windows backend webservers on an internal network, 
> connected to a Debian machine (kernel 2.6.18.dfsg.1-24) which has a 
> direct connection to Internet and acts as Reverse-Proxy. 

The kernel release 2.6.18 is pretty old and a lot of TCP connection 
tracking improvements has been added since then. I don't think that 
tweaking the timeout parameters would really help, the best would be to 
upgrade to a recent kernel.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary