Iptables find invalid packets

Iptables find invalid packets

[Dimitri GOURDON]

Hi all,

I've setup LVS on a box using Keepalived (and Iptables) to load balance 
traffic between 2 web servers. I have a problem :

A lot of TCP packets with FIN or RST flags (all I think) from clients 
are dropped by Iptables as state INVALID. The consequence is that I have 
a lot of connection in FIN_WAIT state (shown by netstat) on the 2 web 
servers...

I have reproduced this on my lab...

I have sniff packets with tcpdump to see flags, ACK number,... I've 
found nothing bad.

I have tried to accept these packets with Iptables and then, all my 
connections are terminated in a normal way (only 1-2 connection(s) stay 
in FIN_WAIT on web servers).

I know I can play with TCP timeout to reduce this behavior, but I'll 
prefer correct the problem. I don't understand why these packets are 
invalid if when I accept them, the connections end normaly...

Google is NOT my friend on this issue.

Is someone can help me ???

Dimitri Gourdon
Linux Web Administrator
Itool systems

PS : Sorry for my poor english !

Re: Iptables find invalid packets

[Bernhard Bock]

Hi Dimitri,

Dimitri GOURDON wrote:
> A lot of TCP packets with FIN or RST flags (all I think) from clients 
> are dropped by Iptables as state INVALID. The consequence is that I have 
> a lot of connection in FIN_WAIT state (shown by netstat) on the 2 web 
> servers...
[...]
> Is someone can help me ???

Sounds a bit like the problem I've asked about a few days ago. Try to 
increase the hashsize of nf_conntrack. My INVALID packets vanished after 
this (at least as long as I wasn't using conntrackd for stateful failover).

best regards
Bernhard

Re: Iptables find invalid packets

[Dimitri GOURDON]

I will try but don't believe it :

I've reproduce the same behavior in my lab with only 1 client... so... I 
try  ;)

Dimitri

Bernhard Bock a écrit :
> Hi Dimitri,
>
> Dimitri GOURDON wrote:
>> A lot of TCP packets with FIN or RST flags (all I think) from clients 
>> are dropped by Iptables as state INVALID. The consequence is that I 
>> have a lot of connection in FIN_WAIT state (shown by netstat) on the 
>> 2 web servers...
> [...]
>> Is someone can help me ???
>
> Sounds a bit like the problem I've asked about a few days ago. Try to 
> increase the hashsize of nf_conntrack. My INVALID packets vanished 
> after this (at least as long as I wasn't using conntrackd for stateful 
> failover).
>
> best regards
> Bernhard

Re: Iptables find invalid packets

[Bernhard Bock]

Dimitri GOURDON wrote:
> There are 2 parameters that can perhaps help me :
> 
> cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max
> 65520
> cat /proc/sys/net/ipv4/netfilter/ip_conntrack_buckets
> 8190
> 
> I can modify the first on the fly (the problem is the same : no change)
> But not the second...

Depending on your kernel version, you may have to set the value when 
loading the module. Try to insert the following line in 
/etc/modprobe.conf (or similar, depending on your linux distribution):

options nf_conntrack hashsize=16384

best regards
Bernhard

Re: Iptables find invalid packets

[Dimitri GOURDON]

Bernhard Bock a écrit :
> Dimitri GOURDON wrote:
>> There are 2 parameters that can perhaps help me :
>>
>> cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max
>> 65520
>> cat /proc/sys/net/ipv4/netfilter/ip_conntrack_buckets
>> 8190
>>
>> I can modify the first on the fly (the problem is the same : no change)
>> But not the second...
>
> Depending on your kernel version, you may have to set the value when 
> loading the module. Try to insert the following line in 
> /etc/modprobe.conf (or similar, depending on your linux distribution):
>
> options nf_conntrack hashsize=16384
>
> best regards
> Bernhard
I work on Debian and I don't find similar file or this kind of parameters...

Re: Iptables find invalid packets

[Dimitri GOURDON]

Bernhard Bock a écrit :
> Dimitri GOURDON wrote:
>> There are 2 parameters that can perhaps help me :
>>
>> cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max
>> 65520
>> cat /proc/sys/net/ipv4/netfilter/ip_conntrack_buckets
>> 8190
>>
>> I can modify the first on the fly (the problem is the same : no change)
>> But not the second...
>
> Depending on your kernel version, you may have to set the value when 
> loading the module. Try to insert the following line in 
> /etc/modprobe.conf (or similar, depending on your linux distribution):
>
> options nf_conntrack hashsize=16384
>
> best regards
> Bernhard
Finally, I've successed to modify the 2 parameters. I have respectively 
modify them with 131072 and 16384. And the problem stay entire...

Thanks.

Have you other ideas ???

Re: Iptables find invalid packets

[Bernhard Bock]

Hi,

Dimitri GOURDON wrote:
> Finally, I've successed to modify the 2 parameters. I have respectively 
> modify them with 131072 and 16384. And the problem stay entire...

Well, it certainly worked for me. You may try to increase these values 
further. Apart from this, I do not have any idea. I do, however, use a 
newer kernel (2.6.25). So upgrading your kernel - as proposed by others 
- may help. YMMV.

best regards
Bernhard

Re: Iptables find invalid packets

[Vladislav Kurz]

On Monday 21 of July 2008, Dimitri GOURDON wrote:
> Hi all,
>
> I've setup LVS on a box using Keepalived (and Iptables) to load balance
> traffic between 2 web servers. I have a problem :
>
> A lot of TCP packets with FIN or RST flags (all I think) from clients
> are dropped by Iptables as state INVALID. The consequence is that I have
> a lot of connection in FIN_WAIT state (shown by netstat) on the 2 web
> servers...

I have similar problem, and asked about it here. I was told to try newer 
kernel (I run debian stable - 2.6.18). However I didn't upgrade yet, but If 
you run the same kernel as I do and upgrade would help you I'd like to here 
about that.

-- 
Regards
        Vladislav Kurz

Re: Iptables find invalid packets

[Vladislav Kurz]

On Monday 21 of July 2008, you wrote:
> Vladislav Kurz a écrit :
> > On Monday 21 of July 2008, Dimitri GOURDON wrote:
> >> Hi all,
> >>
> >> I've setup LVS on a box using Keepalived (and Iptables) to load balance
> >> traffic between 2 web servers. I have a problem :
> >>
> >> A lot of TCP packets with FIN or RST flags (all I think) from clients
> >> are dropped by Iptables as state INVALID. The consequence is that I have
> >> a lot of connection in FIN_WAIT state (shown by netstat) on the 2 web
> >> servers...
> >
> > I have similar problem, and asked about it here. I was told to try newer
> > kernel (I run debian stable - 2.6.18). However I didn't upgrade yet, but
> > If you run the same kernel as I do and upgrade would help you I'd like to
> > here about that.
>
> I run 2.6.18-4-bigmem kernel. I've passed just a little to test a more
> recent but I stop because I've encountered problem with some iptables
> rules...

Temporary workaround is only to LOG invalid packets instead of DROP. The 
system then becomes quite usable. Anyway try newer kernel if you can.
Or describe more in detail what problems with what rules did you have.


-- 
regards
        Vladislav Kurz

Re: Iptables find invalid packets

[Vladislav Kurz]

On Monday 21 of July 2008, Dimitri GOURDON wrote:
> Vladislav Kurz a écrit :
> > On Monday 21 of July 2008, you wrote:
> >> Vladislav Kurz a écrit :
> >>> On Monday 21 of July 2008, Dimitri GOURDON wrote:
> >>>> Hi all,
> >>>>
> >>>> I've setup LVS on a box using Keepalived (and Iptables) to load
> >>>> balance traffic between 2 web servers. I have a problem :
> >>>>
> >>>> A lot of TCP packets with FIN or RST flags (all I think) from clients
> >>>> are dropped by Iptables as state INVALID. The consequence is that I
> >>>> have a lot of connection in FIN_WAIT state (shown by netstat) on the 2
> >>>> web servers...
> >>>
> >>> I have similar problem, and asked about it here. I was told to try
> >>> newer kernel (I run debian stable - 2.6.18). However I didn't upgrade
> >>> yet, but If you run the same kernel as I do and upgrade would help you
> >>> I'd like to here about that.
> >>
> >> I run 2.6.18-4-bigmem kernel. I've passed just a little to test a more
> >> recent but I stop because I've encountered problem with some iptables
> >> rules...
> >
> > Temporary workaround is only to LOG invalid packets instead of DROP. The
> > system then becomes quite usable. Anyway try newer kernel if you can.
> > Or describe more in detail what problems with what rules did you have.
>
> Here are my rules :
>
> $IPTABLES is iptables bin file
> $EXT is external interface
> $IP_V1 is the virtual IP clients reach
>
>
> $IPTABLES -N LOG_INVALID
> $IPTABLES -A LOG_INVALID -j LOG --log-prefix '[iptables_invalid] : '
> $IPTABLES -A LOG_INVALID -j DROP
>
> $IPTABLES -A INPUT -i $EXT -p TCP --dport 443 -d $IP_V1 -m state --state
> NEW,ESTABLISHED,RELATED -j ACCEPT $IPTABLES -A INPUT -i $EXT -p TCP
> --dport 443 -d $IP_V1 -m state --state INVALID -j LOG_INVALID
>
> I don't want to accept invalid packet because they are suppose to be
> invalid... All (I think) FIN and RST flagged packets from clients are
> dropped.

I was told that 2.6.18 has some bug in conntrack which causes valid packets to 
be marked as invalid. So you probably want to accept some of those invalid 
packets. Anyway try this setting it may lower the number of invalid packets:

echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal

However in my case even after this I see a lot of invalid packets.


-- 
Regards
        Vladislav Kurz