Re: conntrackd failover works partially, was Re: conntrack performance test results in INVALID packets

[Pablo Neira Ayuso <pablo@netfilter.org>]

Bernhard Bock wrote:
> Pablo Neira Ayuso wrote:
>> I though that your problem was that you cannot even recover the flows in
>> the first failover, but it seems to me that you have triggered several
>> fail-overs between the nodes. There's no way to hit this in a clean
>> session - ie. empty connection tracking table. 
> 
> Well, there are several thousand connections established and teared down
> on the primary node before the secondary nodes takes over, but as far as
> I can tell there is no "bouncing" between the nodes. So, there's no
> empty connection tracking table at failover time:
> 
> 1. Stop conntrackd
> 2. Clear conntrack table
> 3. Restart Fedora iptables service (see below)
> 4. Start conntrackd
> -> 0 connections
> 5. Start traffic
> -> lots of connections
> 6. fail-over

OK

>> If you are triggering several fail-overs with unclean session, the new
>> script should help. So please, give it a try. It will take you a couple
>> of minutes to get it working.
> 
> Your script makes things worse for me, as it drops a lot of traffic on
> switchover.

Hm, the new script does exactly the same when the node becomes primary
as it used to do script_master.sh, so I cannot find a reason why the new
script does it worst.

> In my setup, it helps a lot to let INVALID packets pass for a couple of
> seconds after switchover and return to the “normal” policy only after
> this time. I coded this into my keepalived scripts. During this time,
> some state recovers and most of the sessions actually work afterwards.

This is a horrible workaround :(

> With a “hard” failover, nearly all sessions get lost.

During the fail-over, keepalived recovers the virtual IPs and conntrackd
commits the states into the kernel. The commit takes very short but you
can still lose some packets if the state is not yet present in the
kernel - thus, these packets are logged as invalid and dropped as we
don't find any matching state (with a sane stateful rule-set, of
course). *However*, the TCP sessions should recover as the peer or the
server retransmits the packet in short, so I don't understand why you
lose nearly all the sessions.

Is the firewall sending RST packets to the peer/server to close
connections? If so, I remember a similar report with a RHEL kernel:

http://www.mail-archive.com/netfilter-failover@lists.netfilter.org/msg00065.html

> One more thing I just noticed: It is not sufficient to clear the
> conntrack table with 'conntrack -F'. I have to unload and reload the
> iptables kernel modules to make it work again. This is done by the
> Fedora init scripts for iptables. Without this, after a "broken"
> fail-over, the machine keeps dropping some (few) packets even without
> conntrackd and a second node involved. After reloading the modules,
> everything's fine again. I guess this hints towards searching in the
> kernel space and not in the conntrack-tools?!

conntrack -F should be enough, there's something wrong in the kernel.
There were other issues related with nat.

There are three patches that should hit -stable for 2.6.26 soon that are
not directly related but that are worth to have:

http://marc.info/?l=netfilter-devel&m=121907870404717&w=2
http://marc.info/?l=netfilter-devel&m=121907870504722&w=2
http://marc.info/?l=netfilter-devel&m=121907870604726&w=2

There were other issues related with NAT but they are fixed in 2.6.26,
however, I'm not sure if fedora is a real 2.6.26 kernel.

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/239215

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers