Re: conntrackd failover works partially, was Re: conntrack performance test results in INVALID packets

[Bernhard Bock <mailinglists@bock.nu>]

Pablo,

Pablo Neira Ayuso wrote:
> I though that your problem was that you cannot even recover the flows in
> the first failover, but it seems to me that you have triggered several
> fail-overs between the nodes. There's no way to hit this in a clean
> session - ie. empty connection tracking table. 

Well, there are several thousand connections established and teared down
on the primary node before the secondary nodes takes over, but as far as
I can tell there is no "bouncing" between the nodes. So, there's no
empty connection tracking table at failover time:

1. Stop conntrackd
2. Clear conntrack table
3. Restart Fedora iptables service (see below)
4. Start conntrackd
-> 0 connections
5. Start traffic
-> lots of connections
6. fail-over

> If you are triggering several fail-overs with unclean session, the new
> script should help. So please, give it a try. It will take you a couple
> of minutes to get it working.

Your script makes things worse for me, as it drops a lot of traffic on
switchover.

In my setup, it helps a lot to let INVALID packets pass for a couple of
seconds after switchover and return to the “normal” policy only after
this time. I coded this into my keepalived scripts. During this time,
some state recovers and most of the sessions actually work afterwards.
With a “hard” failover, nearly all sessions get lost.


One more thing I just noticed: It is not sufficient to clear the
conntrack table with 'conntrack -F'. I have to unload and reload the
iptables kernel modules to make it work again. This is done by the
Fedora init scripts for iptables. Without this, after a "broken"
fail-over, the machine keeps dropping some (few) packets even without
conntrackd and a second node involved. After reloading the modules,
everything's fine again. I guess this hints towards searching in the
kernel space and not in the conntrack-tools?!

Best regards
Bernhard