conntrackd questions

conntrackd questions

[Marco]

I'm setting up a virtual environment to test conntrackd in an
active/backup configuration using FTFW. While it mostly works fine,
I'm seeing some issues and I don't know whether they are expected or
not.
I'm using keepalived to handle the VIPs, and the provided
primary-backup.sh script to manage conntrackd cache synchronization.

This is my setup:


  external VIP: 10.0.0.200
 eth0 (.201)      eth0 (.202)
+-----+          +-----+
| FW1 |          | FW2 |
+-----+          +-----+
 eth1 (.2)        eth1 (.3)
  internal VIP: 192.168.1.1


      +--------+
      | client |
      +--------+


Both firewalls are configured to perform SNAT --to-source 10.0.0.200
for the traffic coming from the client.

The client generates traffic continuously with this simple script:

while true; do
  for i in www.site1.com www.site2.com www.site3.com www.site4.com
www.site5.com www.site6.com; do
    wget -O/dev/null http://$i
  done
done

To do failover, I stop keepalived on FW1 so FW2 becomes active. To do
failback, I restart it so FW1 is again the active.

What happens is that sometimes (apparently randomly) the client's wget
gets stuck after a failover or a failback, and does not even time out;
I have to stop it with ctrl-c.

Here are my configuration files:

keepalived.conf on FW1:

vrrp_sync_group G1 {   # must be before vrrp_instance declaration
  group {
    E1
    I1
  }
  notify_master "/etc/conntrackd/primary-backup.sh primary"
  notify_backup "/etc/conntrackd/primary-backup.sh backup"
  notify_fault "/etc/conntrackd/primary-backup.sh fault"
}

vrrp_instance E1 {
    interface eth0
    state MASTER
    virtual_router_id 61
    priority 100
    advert_int 1
    authentication {
      auth_type PASS
      auth_pass papas_con_tomate
    }
    virtual_ipaddress {
        10.0.0.200/24 dev eth0
    }
}

vrrp_instance I1 {
    interface eth1
    state MASTER
    virtual_router_id 62
    priority 100
    advert_int 1
    authentication {
      auth_type PASS
      auth_pass papas_con_tomate
    }
    virtual_ipaddress {
        192.168.1.1/24 dev eth1
    }
}

On fw2, it's the same but with state BACKUP and priority 50.

conntrackd.conf on FW1  (for simplicity, I'm not using a dedicated
link for now):

Sync {
	Mode FTFW {
		DisableExternalCache Off
	}
	Multicast {
		IPv4_address 225.0.0.50
		Group 3780
		IPv4_interface 192.168.1.2
		Interface eth1
		SndSocketBuffer 1249280
		RcvSocketBuffer 1249280
		Checksum on
	}
}
General {
	Nice -20
	HashSize 32768
	HashLimit 131072
	LogFile on
	Syslog on
	LockFile /var/lock/conntrack.lock
	UNIX {
		Path /var/run/conntrackd.ctl
		Backlog 20
	}
	NetlinkBufferSize 2097152
	NetlinkBufferSizeMaxGrowth 8388608
	Filter From Userspace {
		Protocol Accept {
			TCP
			UDP
			ICMP # This requires a Linux kernel >= 2.6.31
		}
		Address Ignore {
        IPv4_address 127.0.0.1 # loopback
        IPv4_address 10.0.0.202
        IPv4_address 10.0.0.200
        IPv4_address 192.168.1.2
        IPv4_address 192.168.1.1
		}
	}
}

on FW2:

Sync {
	Mode FTFW {
		DisableExternalCache Off
	}
	Multicast {
		IPv4_address 225.0.0.50
		Group 3780
		IPv4_interface 192.168.1.3
		Interface eth1
		SndSocketBuffer 1249280
		RcvSocketBuffer 1249280
		Checksum on
	}
}
General {
	Nice -20
	HashSize 32768
	HashLimit 131072
	LogFile on
	Syslog on
	LockFile /var/lock/conntrack.lock
	UNIX {
		Path /var/run/conntrackd.ctl
		Backlog 20
	}
	NetlinkBufferSize 2097152
	NetlinkBufferSizeMaxGrowth 8388608
	Filter From Userspace {
		Protocol Accept {
			TCP
			UDP
			ICMP # This requires a Linux kernel >= 2.6.31
		}
		Address Ignore {
        IPv4_address 127.0.0.1 # loopback
        IPv4_address 10.0.0.203
        IPv4_address 10.0.0.200
        IPv4_address 192.168.1.3
        IPv4_address 192.168.1.1
		}
	}
}


Thanks in advance for any help.

Re: conntrackd questions

[Pablo Neira Ayuso]

Hi Marco,

On Tue, Feb 19, 2013 at 03:18:09PM +0100, Marco wrote:
> I'm setting up a virtual environment to test conntrackd in an
> active/backup configuration using FTFW. While it mostly works fine,
> I'm seeing some issues and I don't know whether they are expected or
> not.
> I'm using keepalived to handle the VIPs, and the provided
> primary-backup.sh script to manage conntrackd cache synchronization.
> 
> This is my setup:
> 
> 
>   external VIP: 10.0.0.200
>  eth0 (.201)      eth0 (.202)
> +-----+          +-----+
> | FW1 |          | FW2 |
> +-----+          +-----+
>  eth1 (.2)        eth1 (.3)
>   internal VIP: 192.168.1.1
> 
> 
>       +--------+
>       | client |
>       +--------+
> 
> 
> Both firewalls are configured to perform SNAT --to-source 10.0.0.200
> for the traffic coming from the client.
> 
> The client generates traffic continuously with this simple script:
> 
> while true; do
>   for i in www.site1.com www.site2.com www.site3.com www.site4.com
> www.site5.com www.site6.com; do
>     wget -O/dev/null http://$i
>   done
> done
> 
> To do failover, I stop keepalived on FW1 so FW2 becomes active. To do
> failback, I restart it so FW1 is again the active.
> 
> What happens is that sometimes (apparently randomly) the client's wget
> gets stuck after a failover or a failback, and does not even time out;
> I have to stop it with ctrl-c.
> 
> Here are my configuration files:
> 
> keepalived.conf on FW1:
> 
> vrrp_sync_group G1 {   # must be before vrrp_instance declaration
>   group {
>     E1
>     I1
>   }
>   notify_master "/etc/conntrackd/primary-backup.sh primary"
>   notify_backup "/etc/conntrackd/primary-backup.sh backup"
>   notify_fault "/etc/conntrackd/primary-backup.sh fault"
> }
> 
> vrrp_instance E1 {
>     interface eth0
>     state MASTER
>     virtual_router_id 61
>     priority 100
>     advert_int 1
>     authentication {
>       auth_type PASS
>       auth_pass papas_con_tomate
>     }
>     virtual_ipaddress {
>         10.0.0.200/24 dev eth0
>     }
> }
> 
> vrrp_instance I1 {
>     interface eth1
>     state MASTER
>     virtual_router_id 62
>     priority 100
>     advert_int 1
>     authentication {
>       auth_type PASS
>       auth_pass papas_con_tomate
>     }
>     virtual_ipaddress {
>         192.168.1.1/24 dev eth1
>     }
> }
> 
> On fw2, it's the same but with state BACKUP and priority 50.
> 
> conntrackd.conf on FW1  (for simplicity, I'm not using a dedicated
> link for now):
> 
> Sync {
> 	Mode FTFW {
> 		DisableExternalCache Off
> 	}
> 	Multicast {
> 		IPv4_address 225.0.0.50
> 		Group 3780
> 		IPv4_interface 192.168.1.2
> 		Interface eth1
> 		SndSocketBuffer 1249280
> 		RcvSocketBuffer 1249280
> 		Checksum on
> 	}
> }
> General {
> 	Nice -20
> 	HashSize 32768
> 	HashLimit 131072
> 	LogFile on
> 	Syslog on
> 	LockFile /var/lock/conntrack.lock
> 	UNIX {
> 		Path /var/run/conntrackd.ctl
> 		Backlog 20
> 	}
> 	NetlinkBufferSize 2097152
> 	NetlinkBufferSizeMaxGrowth 8388608
> 	Filter From Userspace {
> 		Protocol Accept {
> 			TCP
> 			UDP
> 			ICMP # This requires a Linux kernel >= 2.6.31
> 		}
> 		Address Ignore {
>         IPv4_address 127.0.0.1 # loopback
>         IPv4_address 10.0.0.202
>         IPv4_address 10.0.0.200
>         IPv4_address 192.168.1.2
>         IPv4_address 192.168.1.1
> 		}
> 	}
> }
> on FW2:
> 
> Sync {
> 	Mode FTFW {
> 		DisableExternalCache Off
> 	}
> 	Multicast {
> 		IPv4_address 225.0.0.50
> 		Group 3780
> 		IPv4_interface 192.168.1.3
> 		Interface eth1
> 		SndSocketBuffer 1249280
> 		RcvSocketBuffer 1249280
> 		Checksum on
> 	}
> }
> General {
> 	Nice -20
> 	HashSize 32768
> 	HashLimit 131072
> 	LogFile on
> 	Syslog on
> 	LockFile /var/lock/conntrack.lock
> 	UNIX {
> 		Path /var/run/conntrackd.ctl
> 		Backlog 20
> 	}
> 	NetlinkBufferSize 2097152
> 	NetlinkBufferSizeMaxGrowth 8388608
> 	Filter From Userspace {
> 		Protocol Accept {
> 			TCP
> 			UDP
> 			ICMP # This requires a Linux kernel >= 2.6.31
> 		}
> 		Address Ignore {
>         IPv4_address 127.0.0.1 # loopback
>         IPv4_address 10.0.0.203
>         IPv4_address 10.0.0.200
>         IPv4_address 192.168.1.3
>         IPv4_address 192.168.1.1
> 		}
> 	}
> }
> 
> 
> Thanks in advance for any help.

There are several things that you can check to troubleshoot
conntrackd:

1) Log files: /var/log/conntrackd.log and /var/log/keepalived.log
2) See if entries are actually synchronized via `conntrackd -i' and
`conntrackd -e' and other statistics.
3) echo 255 > /proc/sys/net/netfilter/nf_conntrack_log_invalid, to
see if the connection tracking system is marking packets as invalid.

I also suggest you to read the official documentation:

http://conntrack-tools.netfilter.org/manual.html

Regards.

Re: conntrackd questions

[Marco]

2013/2/19 Pablo Neira Ayuso <pablo@netfilter.org>:

> There are several things that you can check to troubleshoot
> conntrackd:
>
> 1) Log files: /var/log/conntrackd.log and /var/log/keepalived.log
> 2) See if entries are actually synchronized via `conntrackd -i' and
> `conntrackd -e' and other statistics.
> 3) echo 255 > /proc/sys/net/netfilter/nf_conntrack_log_invalid, to
> see if the connection tracking system is marking packets as invalid.
>
> I also suggest you to read the official documentation:
>
> http://conntrack-tools.netfilter.org/manual.html

I had read the official documentation. I will keep testing to see if I
can come up with more information.

On a related note, am I understanding correctly that if external cache
is disabled, then there's no need to use the notify_* scripts in
keepalived.conf? or it's harmless to use them anyway?

Thanks.

Re: conntrackd questions

[Marco]

2013/2/19 Pablo Neira Ayuso <pablo@netfilter.org>:

> There are several things that you can check to troubleshoot
> conntrackd:

Ok, I think I see what's happening. To understand, I captured traffic
both on the client, and on the upstream router (upstream to fw1 and
fw2).
On the client, I just see that when wget stops downloading, the
traffic stops in the middle of the TCP flow. The client is just
waiting for the server to send more data.

But on the capture taken on the upstream router, I see that, at the
point when the client hangs, there are many RST packets directed to
the website from which the client was downloading. Obviously, this
resets the connection and the client has a hard time waiting for more
data. I suspect these RST are coming from one of the firewalls. For
example, while there is a failover from fw1 to fw2, but fw2 hasn't
synchronized the full conntrack table yet, data comes back from the
website; fw2 doesn't know anything about it yet, so it sends RSTs.
Could a race like this be possible? I can probably live with that, but
I'd like to understand if my analysis is correct.
If I disable external cache, it still happens, although less frequently.

Thanks!

Re: conntrackd questions

[Pablo Neira Ayuso]

Hi Marco,

On Thu, Feb 21, 2013 at 01:23:30PM +0100, Marco wrote:
> 2013/2/19 Pablo Neira Ayuso <pablo@netfilter.org>:
> 
> > There are several things that you can check to troubleshoot
> > conntrackd:
> 
> Ok, I think I see what's happening. To understand, I captured traffic
> both on the client, and on the upstream router (upstream to fw1 and
> fw2).
> On the client, I just see that when wget stops downloading, the
> traffic stops in the middle of the TCP flow. The client is just
> waiting for the server to send more data.
> 
> But on the capture taken on the upstream router, I see that, at the
> point when the client hangs, there are many RST packets directed to
> the website from which the client was downloading. Obviously, this
> resets the connection and the client has a hard time waiting for more
> data. I suspect these RST are coming from one of the firewalls. For
> example, while there is a failover from fw1 to fw2, but fw2 hasn't
> synchronized the full conntrack table yet, data comes back from the
> website; fw2 doesn't know anything about it yet, so it sends RSTs.

Thanks, that's a much more accurate report, I can help you better with
that.

> Could a race like this be possible? I can probably live with that, but
> I'd like to understand if my analysis is correct.
> If I disable external cache, it still happens, although less frequently.

The firewall sends an RST if it finds no state information / the
traffic is considered to be invalid by the TCP tracking code.

In your previous config, assuming you use a 3.x kernel, I saw you did
not enabled TCPWindowTracking On. That allows the new primary to
recover TCP window tracking from the middle.

Another possibility is to disable disable TCP tracking.

echo 1 > /proc/sys/net/netfilter/nf_conntrack_tcp_be_liberal

I think they are documented in the official docs, let me know if not.

It's also good to have a "sane" stateful rule-set, ie. one that drops
invalid traffic, see the rule-set in:

http://conntrack-tools.netfilter.org/testcase.html

for instance.

Regards.

Re: conntrackd questions

[Marco]

2013/2/21 Pablo Neira Ayuso <pablo@netfilter.org>:

>> But on the capture taken on the upstream router, I see that, at the
>> point when the client hangs, there are many RST packets directed to
>> the website from which the client was downloading. Obviously, this
>> resets the connection and the client has a hard time waiting for more
>> data. I suspect these RST are coming from one of the firewalls. For
>> example, while there is a failover from fw1 to fw2, but fw2 hasn't
>> synchronized the full conntrack table yet, data comes back from the
>> website; fw2 doesn't know anything about it yet, so it sends RSTs.
>
> Thanks, that's a much more accurate report, I can help you better with
> that.
>
>> Could a race like this be possible? I can probably live with that, but
>> I'd like to understand if my analysis is correct.
>> If I disable external cache, it still happens, although less frequently.
>
> The firewall sends an RST if it finds no state information / the
> traffic is considered to be invalid by the TCP tracking code.

From the capture, that's what I suspected too.

> In your previous config, assuming you use a 3.x kernel, I saw you did
> not enabled TCPWindowTracking On. That allows the new primary to
> recover TCP window tracking from the middle.

Unfortunately, the system where this will run has a 2.6.32 kernel, so
this is not an option for the moment.

> Another possibility is to disable disable TCP tracking.
>
> echo 1 > /proc/sys/net/netfilter/nf_conntrack_tcp_be_liberal

I've done this, but I'm still seeing client hangs from time to time.

Perhaps what I need is really the TCPWindowTracking On setting?
according to question 5 in the docs:

5. Does conntrackd support TCP flow-recovery with window tracking enabled?
Yes, but you require a Linux kernel >= 2.6.36 and the conntrack-tools
>= 0.9.15. To enable it, check the TCPWindowTracking clause in the
example configuration files.

(assuming TCP flow-recovery is

> I think they are documented in the official docs, let me know if not.

Well, the docs mention window tracking here and there, but (at least
to me) it's not clear what that does, and that it's (or could be) the
solution to this problem I'm seeing.
Furthermore, I found no documentation or explanation of
nf_conntrack_tcp_be_liberal on google, neither it is in the sysctl.txt
file that documents the /proc/sys/net entries, nor anywhere else.

> It's also good to have a "sane" stateful rule-set, ie. one that drops
> invalid traffic, see the rule-set in:
>
> http://conntrack-tools.netfilter.org/testcase.html

The ruleset I'm using is indeed sane (with added rules to do NAT, as I
explained).

Thanks

Re: conntrackd questions

[Pablo Neira Ayuso]

On Fri, Feb 22, 2013 at 11:12:55AM +0100, Marco wrote:
[...]
> > In your previous config, assuming you use a 3.x kernel, I saw you did
> > not enabled TCPWindowTracking On. That allows the new primary to
> > recover TCP window tracking from the middle.
> 
> Unfortunately, the system where this will run has a 2.6.32 kernel, so
> this is not an option for the moment.

I really recommend you to upgrade to some stable branch of 3.x. Many
relevant updates and fixes went into the ctnetlink code since that
version you're using.

[...]
> Well, the docs mention window tracking here and there, but (at least
> to me) it's not clear what that does, and that it's (or could be) the
> solution to this problem I'm seeing.
> Furthermore, I found no documentation or explanation of
> nf_conntrack_tcp_be_liberal on google, neither it is in the sysctl.txt
> file that documents the /proc/sys/net entries, nor anywhere else.

http://git.kernel.org/?p=linux/kernel/git/davem/net-next.git;a=blob;f=Documentation/networking/nf_conntrack-sysctl.txt;h=70da5086153dbd24a9c9258e73cc16440d247519;hb=HEAD

Regards.

Re: conntrackd questions

[Marco]

2013/2/25 Pablo Neira Ayuso <pablo@netfilter.org>:

> On Fri, Feb 22, 2013 at 11:12:55AM +0100, Marco wrote:
> [...]
>> > In your previous config, assuming you use a 3.x kernel, I saw you did
>> > not enabled TCPWindowTracking On. That allows the new primary to
>> > recover TCP window tracking from the middle.
>>
>> Unfortunately, the system where this will run has a 2.6.32 kernel, so
>> this is not an option for the moment.
>
> I really recommend you to upgrade to some stable branch of 3.x. Many
> relevant updates and fixes went into the ctnetlink code since that
> version you're using.

Ok, I've finally found some time to set this up. Now I'm using kernel
3.7.10, keepalived 1.2.2 and contrack-tools 1.4.0, with
"TCPWindowTracking On". I'm sorry to report that I'm still seeing the
original behavior (ie firewall sends RST to origin server and client
hangs).

> [...]
>> Well, the docs mention window tracking here and there, but (at least
>> to me) it's not clear what that does, and that it's (or could be) the
>> solution to this problem I'm seeing.
>> Furthermore, I found no documentation or explanation of
>> nf_conntrack_tcp_be_liberal on google, neither it is in the sysctl.txt
>> file that documents the /proc/sys/net entries, nor anywhere else.
>
> http://git.kernel.org/?p=linux/kernel/git/davem/net-next.git;a=blob;f=Documentation/networking/nf_conntrack-sysctl.txt;h=70da5086153dbd24a9c9258e73cc16440d247519;hb=HEAD

Thanks! Hopefully at some point it will be published in some place
where search engines can find it (the obvious place seems to be
https://www.kernel.org/doc/Documentation/networking/, where it still
isn't visible)..