conntrackd: failover problems

conntrackd: failover problems

[Simone Zaffalon]

Hi.
I'm trying to set-up an HA firewall with Debian, ucarp and conntrackd
in a testbed.
Debian is version 5.0.7 (stock kernel 2.6.26).

I have two hosts in active/passive configuration. At the moment i
don't have any particular firewall rule in place, only a couple of
iptables statements to nat clients ips and let them connect to
internet:
iptables -t nat -A POSTROUTING -s state --state
NEW,ESTABLISHED,RELATED -p TCP -s $internal_lan -d 0/0 -j SNAT --to
source $ext_fw_ip
iptables -t nat -A POSTROUTING -s state --state
NEW,ESTABLISHED,RELATED -p UDP -s $internal_lan -d 0/0 -j SNAT --to
source $ext_fw_ip

Conntrackd is installed and conntrackd -s report no error in multicast traffic.
Anyway i'm not able to keep the sessions active between failovers.
I can see connections in cache external, but it seems that such
connections are not committed.
[Mon Dec 27 02:01:19 2010] (pid=2032) [notice] initialization completed
[Mon Dec 27 02:01:19 2010] (pid=2041) [notice] -- starting in daemon mode --
[Mon Dec 27 02:08:39 2010] (pid=2481) [notice] committing external cache
[Mon Dec 27 02:08:39 2010] (pid=2481) [notice] Committed 1 new entries
[Mon Dec 27 02:08:39 2010] (pid=2483) [notice] committing external cache
[Mon Dec 27 02:08:39 2010] (pid=2483) [notice] Committed 0 new entries
[Mon Dec 27 02:08:39 2010] (pid=2483) [notice] 1 entries can't be committed
[Mon Dec 27 02:08:39 2010] (pid=2041) [notice] flushing caches
[Mon Dec 27 02:08:39 2010] (pid=2041) [notice] resync with master table
[Mon Dec 27 02:08:39 2010] (pid=2041) [notice] flushing caches
[Mon Dec 27 02:08:39 2010] (pid=2041) [notice] resync with master table

As far as i understood, with this sequence of commands:
in master
conntrackd -n

in backup
conntrackd -c
conntrackd -f
conntrackd -R

i should have the same sessions in master and backup (listed with
conntrack -L) or am i totally wrong?

Is there any way to increment log verbosity to understand what's going on?
I really don't know well the internals of conntrackd: am i missing
something? Kernel parameters? sysctl settings?

Many thanks in advance
Regards
Simone Zaffalon

Re: conntrackd: failover problems

[Pablo Neira Ayuso]

Hi,

On 27/12/10 15:50, Simone Zaffalon wrote:
> Hi.
> I'm trying to set-up an HA firewall with Debian, ucarp and conntrackd
> in a testbed.
> Debian is version 5.0.7 (stock kernel 2.6.26).
> 
> I have two hosts in active/passive configuration. At the moment i
> don't have any particular firewall rule in place, only a couple of
> iptables statements to nat clients ips and let them connect to
> internet:
> iptables -t nat -A POSTROUTING -s state --state
> NEW,ESTABLISHED,RELATED -p TCP -s $internal_lan -d 0/0 -j SNAT --to
> source $ext_fw_ip
> iptables -t nat -A POSTROUTING -s state --state
> NEW,ESTABLISHED,RELATED -p UDP -s $internal_lan -d 0/0 -j SNAT --to
> source $ext_fw_ip
> 
> Conntrackd is installed and conntrackd -s report no error in multicast traffic.
> Anyway i'm not able to keep the sessions active between failovers.
> I can see connections in cache external, but it seems that such
> connections are not committed.
> [Mon Dec 27 02:01:19 2010] (pid=2032) [notice] initialization completed
> [Mon Dec 27 02:01:19 2010] (pid=2041) [notice] -- starting in daemon mode --
> [Mon Dec 27 02:08:39 2010] (pid=2481) [notice] committing external cache
> [Mon Dec 27 02:08:39 2010] (pid=2481) [notice] Committed 1 new entries
> [Mon Dec 27 02:08:39 2010] (pid=2483) [notice] committing external cache
> [Mon Dec 27 02:08:39 2010] (pid=2483) [notice] Committed 0 new entries
> [Mon Dec 27 02:08:39 2010] (pid=2483) [notice] 1 entries can't be committed
> [Mon Dec 27 02:08:39 2010] (pid=2041) [notice] flushing caches
> [Mon Dec 27 02:08:39 2010] (pid=2041) [notice] resync with master table
> [Mon Dec 27 02:08:39 2010] (pid=2041) [notice] flushing caches
> [Mon Dec 27 02:08:39 2010] (pid=2041) [notice] resync with master table
> 
> As far as i understood, with this sequence of commands:
> in master
> conntrackd -n
> 
> in backup
> conntrackd -c
> conntrackd -f
> conntrackd -R

Better use the primary-backup.sh script that is included in the
conntrack-tools package. You can find it under doc/sync. That script
should be called by your HA manager during the failover.

> i should have the same sessions in master and backup (listed with
> conntrack -L) or am i totally wrong?

After the failover, you should see the flow-entries in the new primary
with conntrack -L.

> Is there any way to increment log verbosity to understand what's going on?
> I really don't know well the internals of conntrackd: am i missing
> something? Kernel parameters? sysctl settings?

Reading this helps:
http://conntrack-tools.netfilter.org/manual.html
http://conntrack-tools.netfilter.org/testcase.html

It can help you to get some more background on it and to spot what
you're doing wrong.

Please, have a look at them and let me know if your problems persist.
Include also your software versions in your reports.

Re: conntrackd: failover problems

[Simone Zaffalon]

Hi Pablo.
Thank you for the answer.
>
> Better use the primary-backup.sh script that is included in the
> conntrack-tools package. You can find it under doc/sync. That script
> should be called by your HA manager during the failover.

Already done. I've included
/usr/share/doc/conntrackd/examples/sync/ftfw/script_master.sh in
vip-up.sh (the ucarp script called when failover occours):
-----------vip-up.sh--------------------------
#! /bin/sh
exec 2> /dev/null

/sbin/ip addr add "$2"/24 dev "$1"

# or alternatively:
# /sbin/ifconfig "$1":254 "$2" netmask 255.255.255.0
#!/bin/sh

/usr/sbin/conntrackd -c # commit the cache
/usr/sbin/conntrackd -f # flush the caches
/usr/sbin/conntrackd -R # resync with kernel conntrack table
-----------------------------------------------

and of course i did the same with script_backup.sh in vip-down.sh

Now I'm switching to keepalived (to stick more with provided examples)
and i am doing some other experiment and i let you know how far i'll
go with that.
In the meantime, as i asked before, is there any way to increment log
verbosity to understand what's going on (in /var/log/conntrackd.log)?

Thanks again
Regards
Simone Zaffalon

Re: conntrackd: failover problems

[Pablo Neira Ayuso]

On 29/12/10 12:40, Simone Zaffalon wrote:
> Hi Pablo.
> Thank you for the answer.
>>
>> Better use the primary-backup.sh script that is included in the
>> conntrack-tools package. You can find it under doc/sync. That script
>> should be called by your HA manager during the failover.
> 
> Already done. I've included
> /usr/share/doc/conntrackd/examples/sync/ftfw/script_master.sh in
> vip-up.sh (the ucarp script called when failover occours):
> -----------vip-up.sh--------------------------
> #! /bin/sh
> exec 2> /dev/null
> 
> /sbin/ip addr add "$2"/24 dev "$1"
> 
> # or alternatively:
> # /sbin/ifconfig "$1":254 "$2" netmask 255.255.255.0
> #!/bin/sh
> 
> /usr/sbin/conntrackd -c # commit the cache
> /usr/sbin/conntrackd -f # flush the caches
> /usr/sbin/conntrackd -R # resync with kernel conntrack table
> -----------------------------------------------
> 
> and of course i did the same with script_backup.sh in vip-down.sh

This seems to be an old script version. What conntrack-tools version are
you using?

> Now I'm switching to keepalived (to stick more with provided examples)
> and i am doing some other experiment and i let you know how far i'll
> go with that.
> In the meantime, as i asked before, is there any way to increment log
> verbosity to understand what's going on (in /var/log/conntrackd.log)?

You cannot increase verbosity but you have plenty of conntrackd -s
information to know what is going on (see manpage for -s option parameters).

I still don't know what is your problem, please provide a more detailed
procedure of what you're testing and I'll be happy to help you.

To test conntrackd, it's a good idea to start with something simple:

1) create a TCP connection between two peers that are firewalled.
2) check in the primary firewall that `conntrackd -i' shows the flow
(this is to display the internal cache).
3) check in the backup firewall that `conntrackd -e' show the foreign
flow, that means it has been correctly propagated (-e shows the external
cache).
4) for a simple failover scenario, set a link down in the primary to
trigger the failover.
5) the new primary must contain the entry with `conntrack -L' (to
display the kernel conntrack table).
6) check the logs, they must contain that one flow was injected (committed).

limiting not working for individual IPs

[J Webster]

I have installed and started running the following script but it seems to be 
limiting all the traffic on the network to 1Mb.
Any ideas how I can do it for each individual IP address?
I have not made any changes to iptables.

#!/bin/bash
#
#  tc uses the following units when passed as a parameter.
#  kbps: Kilobytes per second
#  mbps: Megabytes per second
#  kbit: Kilobits per second
#  mbit: Megabits per second
#  bps: Bytes per second
#       Amounts of data can be specified in:
#       kb or k: Kilobytes
#       mb or m: Megabytes
#       mbit: Megabits
#       kbit: Kilobits
#  To get the byte figure from bits, divide the number by 8 bit
#

#
# Name of the traffic control command.
TC=/sbin/tc

# The network interface we're planning on limiting bandwidth.
IF=eth0             # Interface

# Download limit (in mega bits)
DNLD=1mbit          # DOWNLOAD Limit

# Upload limit (in mega bits)
UPLD=1mbit          # UPLOAD Limit

# IP address of the machine we are controlling
IP=216.3.128.12     # Host IP

# Filter options for limiting the intended interface.
U32="$TC filter add dev $IF protocol ip parent 1:0 prio 1 u32"

start() {

# We'll use Hierarchical Token Bucket (HTB) to shape bandwidth.
# For detailed configuration options, please consult Linux man
# page.

$TC qdisc add dev $IF root handle 1: htb default 30
$TC class add dev $IF parent 1: classid 1:1 htb rate $DNLD
$TC class add dev $IF parent 1: classid 1:2 htb rate $UPLD
$U32 match ip dst $IP/32 flowid 1:1
$U32 match ip src $IP/32 flowid 1:2

# The first line creates the root qdisc, and the next two lines
# create two child qdisc that are to be used to shape download
# and upload bandwidth.
#
# The 4th and 5th line creates the filter to match the interface.
# The 'dst' IP address is used to limit download speed, and the
# 'src' IP address is used to limit upload speed.

}

stop() {

# Stop the bandwidth shaping.
$TC qdisc del dev $IF root

}

restart() {

# Self-explanatory.
stop
sleep 1
start

}

show() {

# Display status of traffic control status.
$TC -s qdisc ls dev $IF

}

case "$1" in

start)

echo -n "Starting bandwidth shaping: "
start
echo "done"
;;

stop)

echo -n "Stopping bandwidth shaping: "
stop
echo "done"
;;

restart)

echo -n "Restarting bandwidth shaping: "
restart
echo "done"
;;

show)

echo "Bandwidth shaping status for $IF:"
show
echo ""
;;

*)

pwd=$(pwd)
echo "Usage: tc.bash {start|stop|restart|show}"
;;

esac exit 0 

Re: conntrackd: failover problems

[Simone Zaffalon]

Hi Pablo.
I managed to do some more testing, and maybe i found my problem.
With conntrack running, i took for granted that every tcp connection
would be tracked and resumed.
So, as i said in my first email, i didn't set up any iptables rules
(except one for snat).

With this configuration conntrackd does'nt work. Then i tried some
iptables rules like the ones in
http://conntrack-tools.netfilter.org/testcase.html
and things started working.
Please, to be crystal clear on that, can you confirm that when you say in manual

"
...
3) A well-formed stateful rule-set. Otherwise you are likely to
experience problems during the fail-over. An example of a well-formed
stateful iptables rule-set is available in the conntrack-tools
website.
...
"

you mean : "To ensure that conntrackd work correctly you must have a
set of iptables rules with state tracking enabled" ?
If so that's great, i found my problem.
Many thanks in advance for your help,

regards
Simone Zaffalon

Re: conntrackd: failover problems

[Pablo Neira Ayuso]

On 04/01/11 11:06, Simone Zaffalon wrote:
> Hi Pablo.
> I managed to do some more testing, and maybe i found my problem.
> With conntrack running, i took for granted that every tcp connection
> would be tracked and resumed.
> So, as i said in my first email, i didn't set up any iptables rules
> (except one for snat).
> 
> With this configuration conntrackd does'nt work. Then i tried some
> iptables rules like the ones in
> http://conntrack-tools.netfilter.org/testcase.html
> and things started working.
> Please, to be crystal clear on that, can you confirm that when you say in manual
> 
> "
> ...
> 3) A well-formed stateful rule-set. Otherwise you are likely to
> experience problems during the fail-over. An example of a well-formed
> stateful iptables rule-set is available in the conntrack-tools
> website.
> ...
> "
> 
> you mean : "To ensure that conntrackd work correctly you must have a
> set of iptables rules with state tracking enabled" ?
> If so that's great, i found my problem.

Yes. Otherwise you trigger the following problem that can be reproduced
it if you don't use a well-formed stateful rule-set, with the following
steps:

1) you create the tcp connection between two peers A -> B, given the
following configuration A -> FW -> B. We assume NAT enabled, as in your
case.
2) you trigger the failover
3) the flow entries were not committed yet but B sends a reply packet.
The firewall receives it and it finds no flow information to undo the
NAT handling, thus it considers that it's a packet for itself and it
passes it to its local stack. This results in a tcp rst that the
firewall delivers to B.

If you have the appropriate rule-set, the firewall drops invalid
requests. Thus, if B sends a packet to the firewall and the flow
information is not present yet, then it drops the packet. B will resend
and the firewall will find the flow information to undo the NAT.