conntrackd working, but netfilter not impressed

conntrackd working, but netfilter not impressed

[Dirk H. Schulz]

Hi folks,

I am in the process of setting up a redundant router/firewall cluster. That 
means: 2 routers, connected to 2 upstream routers, using OSPFv2 for 
routing. OS is Centos 5.2.

Both have also firewalls configured. Since using OSPF I cannot avoid 
asynchronous routing perfectly, so I have installed conntrack-tools to 
synchronize the connection tracking tables of the firewalls.

conntrackd seems to work fine - using "conntrackd -e" I can see entries 
having been synchronized over from the other router - and vice versa.

But testing it practically it failes nonetheless. This is what I tested:

1. Ping from a server to the outside (internet).
2. I can see the ping packets leave via router2 and the answer come back in 
via router1.
3. conntrackd -e on router1 shows exactly this connection, so it has been 
replicated from router2 to router1
4. netfilter on router1 blocks the incoming packets as being new (I check 
that via log prefixes)

That looks like conntrackd is sync'ing over the connection table entries, 
but not writing them into the kernel's connection tracking table. I have 
set "CacheWriteThrough" to "on" to achieve that. The logs do not show 
anything at all (just the startup messages).

Googling for "active-active" setup of conntrackd show some hints on special 
configuration this setup needs, but the documentation does not list any.

So I am stuck. Any help or hint is appreciated. Here is my setup:
> Sync {
> 	Mode FTFW {
> 		ResendBufferSize 262144
> 		CommitTimeout 180
> 		ACKWindowSize 20
> 	}
> 	Multicast {
> 		IPv4_address 225.0.0.50
>		IPv4_address ROUTERLINKADDRESS
> 		Interface eth1
> 		Group 3780
> 	}
> 	Checksum on
> 	CacheWriteThrough On
> }
> General {
> 	HashSize 8192
> 	HashLimit 65535
> 	LogFile /var/log/conntrackd.log
> 	Syslog on
> 	LockFile /var/lock/conntrack.lock
> 	UNIX {
> 		Path /tmp/sync.sock
> 		Backlog 20
> 	}
> 	SocketBufferSize 262142
> 	SocketBufferSizeMaxGrown 655355
> }
> IgnoreTrafficFor {
> 	IPv4_address INTERROUTERINTERFACE
> 	IPv4_address EXTERNALINTERFACE
> 	IPv4_address INTERNALINTERFACE1
> 	IPv4_address INTERNALINTERFACE2
> 	IPv4_address INTERNALVIRTUALIP
> }
>
> IgnoreProtocol {
> 	IGMP
> 	VRRP
> }



Dirk

Re: conntrackd working, but netfilter not impressed

[Dirk H. Schulz]

Hi Michael,

--On 11. August 2008 13:22:57 +0200 Michael Schwartzkopff 
<misch@multinet.de> wrote:


> Did you conntrackd tell to import sync'ed tables into the kernel tables?
> Scripts see:
> /usr/share/doc/examples/sync/ftfw/script_master.sh

That is what I missed. I have looked into the example script now - it looks 
like committing the external cache into the kernel tables is something to 
do manually?!?
That means in an active-active setup like mine I would have to commit every 
second - which of course can be done, but does that make sense? I would 
have expected conntrackd to do it automatically or to have an option that 
makes it do it automatically.

Thanks for your help.

Dirk

> or whereever your distribution stores doc for conntrackd.
>
> Greetings,
>
> --
> Dr. Michael Schwartzkopff
> MultiNET Services GmbH
> Addresse: Bretonischer Ring 7; 85630 Grasbrunn; Germany
> Tel: +49 - 89 - 45 69 11 0
> Fax: +49 - 89 - 45 69 11 21
> mob: +49 - 174 - 343 28 75
>
> mail: misch@multinet.de
> web: www.multinet.de
>
> Sitz der Gesellschaft: 85630 Grasbrunn
> Registergericht: Amtsgericht München HRB 114375
> Geschäftsführer: Günter Jurgeneit, Hubert Martens
>
> ---
>
> PGP Fingerprint: F919 3919 FF12 ED5A 2801 DEA6 AA77 57A4 EDD8 979B
> Skype: misch42



--------------------------------------------------------------
Dirk H. Schulz
IT Systems Service
Wiesenweg 12, 85567 Grafing
Tel. 0 80 92/86 25 68
Fax. 0 80 92/86 25 72
--------------------------------------------------------------
Technik vom Feinsten - und das nötige Tuning

Re: conntrackd working, but netfilter not impressed

[Pablo Neira Ayuso]

Dirk H. Schulz wrote:
> <misch@multinet.de> wrote:
>> Did you conntrackd tell to import sync'ed tables into the kernel tables?
>> Scripts see:
>> /usr/share/doc/examples/sync/ftfw/script_master.sh
> 
> That is what I missed. I have looked into the example script now - it
> looks like committing the external cache into the kernel tables is
> something to do manually?!?

No. The scripts are there for for the primary-backup or multi-primary
with flow persistency, ie. when we can guarantee that the same firewall
handles the same subset of flows at any time - symmetric routing.

> That means in an active-active setup like mine I would have to commit
> every second - which of course can be done, but does that make sense? I
> would have expected conntrackd to do it automatically or to have an
> option that makes it do it automatically.

The CacheWriteThrough clause should do that for you but with some
important considerations: higher CPU consumption and possible race
conditions - the time to transmit the state to the other firewall
replica should be smaller than the RTT between the firewall and the
end-peer. This is generally true if your firewall is connected to a DSL
line or whatever that inherently inserts some latency in the communications.

Anyhow, the multi-primary setup with asynchronous routing is really bad
design for stateful firewalls. The key problem is that stateful
firewalling works with at flow-level and OSPF only knows about packets.
The preferred way to go should be the multi-primary with symmetric
routing or simply use primary-backup instead if you cannot guarantee the
previous statement.

I'm finishing some documentation for the upcoming release that should
stop this confusion, that will be out soon.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: conntrackd working, but netfilter not impressed

[Dirk H. Schulz]

Pablo,

--On 12. August 2008 13:40:18 +0200 Pablo Neira Ayuso <pablo@netfilter.org> 
wrote:

- snip -
> The CacheWriteThrough clause should do that for you but with some
> important considerations: higher CPU consumption and possible race
> conditions - the time to transmit the state to the other firewall
> replica should be smaller than the RTT between the firewall and the
> end-peer.

That is why I tested with a long stream of pings and long consecutive http 
requests. The result was not better.

> This is generally true if your firewall is connected to a DSL
> line or whatever that inherently inserts some latency in the
> communications.

In this case it is 100 MBit uplinks with very low latency.

>
> Anyhow, the multi-primary setup with asynchronous routing is really bad
> design for stateful firewalls. The key problem is that stateful
> firewalling works with at flow-level and OSPF only knows about packets.
> The preferred way to go should be the multi-primary with symmetric
> routing

I am not sure if this can be achieved with OSPF - there is many differing 
posts on that out there but I also have an intensive look at that, of 
course.

Thanks for your patience.

Dirk

Re: conntrackd working, but netfilter not impressed

[Pablo Neira Ayuso]

Hi Dirk,

Dirk H. Schulz wrote:
> --On 12. August 2008 13:40:18 +0200 Pablo Neira Ayuso
> <pablo@netfilter.org> wrote:
> 
> - snip -
>> The CacheWriteThrough clause should do that for you but with some
>> important considerations: higher CPU consumption and possible race
>> conditions - the time to transmit the state to the other firewall
>> replica should be smaller than the RTT between the firewall and the
>> end-peer.
> 
> That is why I tested with a long stream of pings and long consecutive
> http requests. The result was not better.

ICMP request/replies are a bad test since, as for now, there's one ICMP
event per packet. Give a run to `conntrack -E -p icmp' and you'll notice.

For TCP connections, you should notice that *some packets* hit invalid.
I don't have the OSPF setup but I'm emulating a multi-primary with
asymmetric routing in my testbed [1] by means of:

ip ro add 192.168.1.0/24 nexthop via 192.168.0.3 dev eth1 weight 1
nexthop via 192.168.0.4 dev eth1 weight 1

in the client A, so that it sends a packet to FW(A) and FW(B) in a
round-robin basis. It's simple but I think that's enough for testing.
Probably, setting up OSPF may be better, I'd appreciate if you can
detail me a more realistic scenario (in private, if you want).

Note that in my asymmetric routing testbed the assumption: "RTT(firewall
to peer) > time to transmit the state-change" does not fulfill. Thus, we
have races due to the asynchronous nature of conntrackd. Making this
whole thing synchronous does not seem feasible as it would incur
performance and latency drops.

In my testbed, if I create a HTTP connection, the TCP ACK packets to
finish the hand-shake between the client and the server sometimes hit
invalid, thus we lost race and the client has to re-send the packet
after the drop, this result is a slow-down in the TCP connection setup.

You can notice the packet drop as invalid by enabling connection
tracking log message reporting to verify this:

echo 255 > /proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid

The log should say something like, nf_ct_tcp: invalid state.

BTW, I'm testing with current conntrack-tools git snapshot, just in case
that you're hitting any already resolved bug or observing a different
problem.

>> This is generally true if your firewall is connected to a DSL
>> line or whatever that inherently inserts some latency in the
>> communications.
> 
> In this case it is 100 MBit uplinks with very low latency.

Then, the assumption: RTT(firewall to peer) > time to transmit the
state-change, does not fulfill.

>> Anyhow, the multi-primary setup with asynchronous routing is really bad
>> design for stateful firewalls. The key problem is that stateful
>> firewalling works with at flow-level and OSPF only knows about packets.
>> The preferred way to go should be the multi-primary with symmetric
>> routing
> 
> I am not sure if this can be achieved with OSPF - there is many
> differing posts on that out there but I also have an intensive look at
> that, of course.

Nor me. I don't know any at the moment for this setup - and I did not
have time to check that yet, my current priorities fixing a couple of
reported bugs in the userspace and kernel code, finish the documentation
and the multi-primary with symmetric routing that is based on hashing.
So please, if you find any, let me know.

In short, this scenario is not recommended. It does not exploit the
design of conntrackd and I even think that it somehow goes against the
design of stateful firewalls.

[1] http://conntrack-tools.netfilter.org/testcase.html

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers