Re: failing fail-over - commit still in progress

[Pierre-Philipp Braun <pbraun@nethence.com>]

On 8/21/23 12:26, Pablo Neira Ayuso wrote:
> As I said before, you have to have a stateful ruleset which does not
> pick up states from the middle.

I am now filtering both interfaces, the front-facing and the internal one, on the FORWARD chain.

table inet filter {
         chain input {
                 type filter hook input priority filter; policy accept;
         }

         chain forward {
                 type filter hook forward priority filter; policy drop;
                 ip protocol icmp accept
                 ct state invalid log prefix "INVALID: " drop
                 ct state established,related,new accept
                 log prefix "DROP POLICY: "
         }

         chain output {
                 type filter hook output priority filter; policy accept;
         }
}
table ip nat {
         chain postrouting {
                 type nat hook postrouting priority srcnat; policy accept;
                 ip saddr 10.1.0.0/16 oif "eth0" snat to 217.19.208.157
         }

         chain prerouting {
                 type nat hook prerouting priority dstnat; policy accept;
                 iif "eth0" tcp dport 50 dnat to 10.1.0.50:22
         }
}

However it looks like I am still tracking in the middle.

> nftables is completely irrelevant in this picture. State
> synchronization relies on ctnetlink and userspace conntrackd for state
> synchronization. nftables is only the packet classification framework.

I was just wondering if I absolutely had to use the iptables example from the testcase sample.
I notice that example has additional SYN flag.
Basically I am doing things the other way around, DNAT instead of SNAT.

> Rule of thumb is: You have disable nf_conntrack_tcp_loose from
> conntrack and a stateful ruleset which drops packets that are in
> invalid state.

tcp_loose is zero and as for the stateful ruleset, I am still not sure.

> Otherwise, state synchronization does not make sense because conntrack
> can pick connections from the middle, ie. you can implement "poor man"
> failover and let conntrack recover the history from the middle.

This seems to be what is happening (correct me if I am wrong).  I sometimes manage to successfully fail-over after checking carefully that things are in order, namely that the state is there on internal and external cache respectively.  Anyhow the commit error remains.

> Can you see events on the active node with `conntrack -E`?

No, and I have to restart conntrackd to actually be able to see the states (I guess thanks to `StartupResync on`).

> Did you debug with tcpdump on both ends to check to see if conntrackd
> delivers the synchronization messages?

Yes I noticed an UDP datagram that is larger than the other ones.

> What do conntrackd stats tell you? There is a good number of options
> that allow you debug your setup.

right after a successful fail-over

root@vrrp1:~# conntrackd -s
cache internal:
current active connections:                2
connections created:                       4    failed:            0
connections updated:                       0    failed:            0
connections destroyed:                     2    failed:            0

cache external:
current active connections:                2
connections created:                       4    failed:            0
connections updated:                       0    failed:            0
connections destroyed:                     2    failed:            0

traffic processed:
                    0 Bytes                         0 Pckts

UDP traffic (active device=eth2):
                 8384 Bytes sent                 8216 Bytes recv
                  499 Pckts sent                  498 Pckts recv
                    0 Error send                    0 Error recv

message tracking:
                    0 Malformed msgs                    0 Lost msgs

root@vrrp2:~# conntrackd -s
cache internal:
current active connections:                2
connections created:                       2    failed:            0
connections updated:                       0    failed:            0
connections destroyed:                     0    failed:            0

cache external:
current active connections:                2
connections created:                       6    failed:            0
connections updated:                       0    failed:            0
connections destroyed:                     4    failed:            0

traffic processed:
                    0 Bytes                         0 Pckts

UDP traffic (active device=eth2):
                 8552 Bytes sent                 8704 Bytes recv
                  519 Pckts sent                  519 Pckts recv
                    0 Error send                    0 Error recv

message tracking:
                    0 Malformed msgs                    0 Lost msgs

but the commit error is always there in the logs.