Possibilities and performance of conntrackd, NATing cluster

Possibilities and performance of conntrackd, NATing cluster

[icovnik]

Hello,

I'd like to create high available and high performance router cluster.
Currently I use 1 router performing NAT running on 2.6 kernel. The
router slowly reaches its capacity limit, so I'd like to add another
router (or two) and create a cluster from those routers. I came
accross conntrack-tools which seems to offer some possibilities here -
simply synchronize all router's stacks and distribute traffic to all
routers. Each router would know everything about each connection, so
each of them would "know" what to do witch each packet. I would simply
distribute the traffic to all routers and they would do the job.

I saw this functionality in Checkpoint few years ago. Is it possible
to do this witch linux kernel and conntrackd? Does conntrackd do this
in real-time? With how many routers?

If it is not possible, how would you solve my problem? I need to route
and NAT 500+ mbps in each direction. And the rate is increasing :-)

Thank you

ico

Re: Possibilities and performance of conntrackd, NATing cluster

[Grant Taylor]

On 09/16/08 09:16, icovnik wrote:
> I'd like to create high available and high performance router 
> cluster. Currently I use 1 router performing NAT running on 2.6 
> kernel. The router slowly reaches its capacity limit, so I'd like to 
> add another router (or two) and create a cluster from those routers. 
> I came accross conntrack-tools which seems to offer some 
> possibilities here - simply synchronize all router's stacks and 
> distribute traffic to all routers. Each router would know everything 
> about each connection, so each of them would "know" what to do witch 
> each packet. I would simply distribute the traffic to all routers and 
> they would do the job.
> 
> I saw this functionality in Checkpoint few years ago. Is it possible 
> to do this witch linux kernel and conntrackd? Does conntrackd do this 
> in real-time? With how many routers?

Purportedly this can be done with Linux using the help of conntrackd.

I know that you can do Active / Standby with conntrackd and I believe 
that you can do Active / Active as well.  It is my understanding that 
conntrackd broadcasts connection state on a separate network connection. 
  I believe that the routers participating in the conntrackd failover 
usually have three (or more) network cards on them, one internal and one 
external interface as well as an additional separate interface just for 
connection state information.  I /believe/ that conntrackd works by 
using multicast to advertise it's state changes to other systems that 
then decide what to do with the information.

I'm thinking that you could have three systems set up like this if you 
wanted to.  I'd expect that if you were using Active / Active you could 
have one system doing the inbound traffic and another doing outbound 
traffic with the third as a backup system in case one of the other two 
went down.

Remember that your traffic should (in an ideal world) pass through the 
same router (as far as IP is concerned) going both directions (symmetric 
routing) but is not required to.  With this in mind I'd recommend 
something like VRRP for the internal and external interfaces where one 
router is primary for the internal and outgoing interface and the other 
router is primary for the external and incoming interface.  Using VRRP 
will make things easier for upstream routers as well as down stream 
devices because even if things fail over to the other router the MAC 
address that they are communicating with will stay the same.  As an 
aside I'd recommend that you have an IP per system plus an IP for the 
logical VRRP router its self.  So if you are using three boxen plus the 
VRRP you will need four IPs per subnet to do this.

> If it is not possible, how would you solve my problem? I need to 
> route and NAT 500+ mbps in each direction. And the rate is increasing 
> :-)

Can you separate your routing from your NATing so that there is less load?

Can you do stateless NAT and / or firewalling?



Grant. . . .

Re: Possibilities and performance of conntrackd, NATing cluster

[Pablo Neira Ayuso]

Grant Taylor wrote:
> On 09/16/08 09:16, icovnik wrote:
>> I'd like to create high available and high performance router cluster.
>> Currently I use 1 router performing NAT running on 2.6 kernel. The
>> router slowly reaches its capacity limit, so I'd like to add another
>> router (or two) and create a cluster from those routers. I came
>> accross conntrack-tools which seems to offer some possibilities here -
>> simply synchronize all router's stacks and distribute traffic to all
>> routers. Each router would know everything about each connection, so
>> each of them would "know" what to do witch each packet. I would simply
>> distribute the traffic to all routers and they would do the job.

If you are refering to an asymmetric setup where the packets can be
filtered by whatever node, it's likely that you'll experience problems.
The daemon `conntrackd' is asynchronous, so there are race conditions
between the packets and the state updates, ie. one of the firewall nodes
may receive a packet but its state may not be up-to-date yet.

>> I saw this functionality in Checkpoint few years ago. Is it possible
>> to do this witch linux kernel and conntrackd?

The way to go is a symmetric setup where all nodes receives the packets
and only one firewall node handles them. This can be achieved by means
of hash-based load-sharing. There's some works on that direction.

>> Does conntrackd do this in real-time?

It's soft real-time. conntrackd does its best here. A hard real-time
approach would harm performance in terms of latency and bandwidth.

> With how many routers?

Limit? I don't know yet, I'm still testing with only two nodes, but I
expect to do it with up to four. Moreover, the replication approaches
still require a small change in the code to cleanly support more than
two nodes.

> Purportedly this can be done with Linux using the help of conntrackd.
> 
> I know that you can do Active / Standby with conntrackd and I believe
> that you can do Active / Active as well.  It is my understanding that
> conntrackd broadcasts connection state on a separate network connection.
>  I believe that the routers participating in the conntrackd failover
> usually have three (or more) network cards on them, one internal and one
> external interface as well as an additional separate interface just for
> connection state information.  I /believe/ that conntrackd works by
> using multicast to advertise it's state changes to other systems that
> then decide what to do with the information.

This is right.

> I'm thinking that you could have three systems set up like this if you
> wanted to.  I'd expect that if you were using Active / Active you could
> have one system doing the inbound traffic and another doing outbound
> traffic with the third as a backup system in case one of the other two
> went down.

This is asymmetric multipath, it is not really a good idea and also
you'll waste lots of resources in the replication. Therefore, if your
intention is to improve scalability, this won't help. The way to go is
the symmetric setup.

> Remember that your traffic should (in an ideal world) pass through the
> same router (as far as IP is concerned) going both directions (symmetric
> routing) but is not required to.  With this in mind I'd recommend
> something like VRRP for the internal and external interfaces where one
> router is primary for the internal and outgoing interface and the other
> router is primary for the external and incoming interface.  Using VRRP
> will make things easier for upstream routers as well as down stream
> devices because even if things fail over to the other router the MAC
> address that they are communicating with will stay the same.  As an
> aside I'd recommend that you have an IP per system plus an IP for the
> logical VRRP router its self.  So if you are using three boxen plus the
> VRRP you will need four IPs per subnet to do this.

This is a description for the asymmetric setup, isn't it?

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: Possibilities and performance of conntrackd, NATing cluster

[Grant Taylor]

On 09/17/08 05:34, Pablo Neira Ayuso wrote:
> If you are refering to an asymmetric setup where the packets can be 
> filtered by whatever node, it's likely that you'll experience 
> problems. The daemon `conntrackd' is asynchronous, so there are race 
> conditions between the packets and the state updates, ie. one of the 
> firewall nodes may receive a packet but its state may not be 
> up-to-date yet.

Ah.  Ok.

> The way to go is a symmetric setup where all nodes receives the 
> packets and only one firewall node handles them. This can be achieved 
> by means of hash-based load-sharing. There's some works on that 
> direction.

Interesting.  I can see how this would easily scale beyond two nodes (or 
  two primary and two backup) much easier.

> It's soft real-time. conntrackd does its best here. A hard real-time 
> approach would harm performance in terms of latency and bandwidth.

Ok...  Can you comment on whether or not CheckPoint's is soft or hard 
real-time (or something in between)?  What about any thing else?  In 
other words, is this a Linux / conntrackd shortcoming or just a 
shortcoming of load balancing across firewalls?

> Limit? I don't know yet, I'm still testing with only two nodes, but I 
> expect to do it with up to four. Moreover, the replication approaches 
> still require a small change in the code to cleanly support more than 
> two nodes.

*nod*

> This is right.

:)

> This is asymmetric multipath, it is not really a good idea and also 
> you'll waste lots of resources in the replication. Therefore, if your 
> intention is to improve scalability, this won't help. The way to go 
> is the symmetric setup.

Ok.

> This is a description for the asymmetric setup, isn't it?

Well, it was initially written as asymmetric, but it could easily be 
changed to symmetric by having one node be the primary for both inbound 
and outbound traffic and have the other node be backup.

Considering the hashed based load balancing I'm not quite sure how I 
would apply VRRP.  I think I'd end up using hashing across multiple sets 
of VRRP active / standby nodes.  But that is quite a ways beyond the OPs 
question.



Grant. . . .

Re: Possibilities and performance of conntrackd, NATing cluster

[julien vehent]

That sounds really interesting.
Pablo : do you have more information/articles on what is done around
conntrackd and how to set it up in a test bed environment ?
I saw you published a paper earlier this year about that, but it's not
available online. Is there any way to get it ?

2008/9/17 Pablo Neira Ayuso <pablo@netfilter.org>
>
> Grant Taylor wrote:
> > On 09/16/08 09:16, icovnik wrote:
> >> I'd like to create high available and high performance router cluster.
> >> Currently I use 1 router performing NAT running on 2.6 kernel. The
> >> router slowly reaches its capacity limit, so I'd like to add another
> >> router (or two) and create a cluster from those routers. I came
> >> accross conntrack-tools which seems to offer some possibilities here -
> >> simply synchronize all router's stacks and distribute traffic to all
> >> routers. Each router would know everything about each connection, so
> >> each of them would "know" what to do witch each packet. I would simply
> >> distribute the traffic to all routers and they would do the job.
>
> If you are refering to an asymmetric setup where the packets can be
> filtered by whatever node, it's likely that you'll experience problems.
> The daemon `conntrackd' is asynchronous, so there are race conditions
> between the packets and the state updates, ie. one of the firewall nodes
> may receive a packet but its state may not be up-to-date yet.
>
> >> I saw this functionality in Checkpoint few years ago. Is it possible
> >> to do this witch linux kernel and conntrackd?
>
> The way to go is a symmetric setup where all nodes receives the packets
> and only one firewall node handles them. This can be achieved by means
> of hash-based load-sharing. There's some works on that direction.
>
> >> Does conntrackd do this in real-time?
>
> It's soft real-time. conntrackd does its best here. A hard real-time
> approach would harm performance in terms of latency and bandwidth.
>
> > With how many routers?
>
> Limit? I don't know yet, I'm still testing with only two nodes, but I
> expect to do it with up to four. Moreover, the replication approaches
> still require a small change in the code to cleanly support more than
> two nodes.
>
> > Purportedly this can be done with Linux using the help of conntrackd.
> >
> > I know that you can do Active / Standby with conntrackd and I believe
> > that you can do Active / Active as well.  It is my understanding that
> > conntrackd broadcasts connection state on a separate network connection.
> >  I believe that the routers participating in the conntrackd failover
> > usually have three (or more) network cards on them, one internal and one
> > external interface as well as an additional separate interface just for
> > connection state information.  I /believe/ that conntrackd works by
> > using multicast to advertise it's state changes to other systems that
> > then decide what to do with the information.
>
> This is right.
>
> > I'm thinking that you could have three systems set up like this if you
> > wanted to.  I'd expect that if you were using Active / Active you could
> > have one system doing the inbound traffic and another doing outbound
> > traffic with the third as a backup system in case one of the other two
> > went down.
>
> This is asymmetric multipath, it is not really a good idea and also
> you'll waste lots of resources in the replication. Therefore, if your
> intention is to improve scalability, this won't help. The way to go is
> the symmetric setup.
>
> > Remember that your traffic should (in an ideal world) pass through the
> > same router (as far as IP is concerned) going both directions (symmetric
> > routing) but is not required to.  With this in mind I'd recommend
> > something like VRRP for the internal and external interfaces where one
> > router is primary for the internal and outgoing interface and the other
> > router is primary for the external and incoming interface.  Using VRRP
> > will make things easier for upstream routers as well as down stream
> > devices because even if things fail over to the other router the MAC
> > address that they are communicating with will stay the same.  As an
> > aside I'd recommend that you have an IP per system plus an IP for the
> > logical VRRP router its self.  So if you are using three boxen plus the
> > VRRP you will need four IPs per subnet to do this.
>
> This is a description for the asymmetric setup, isn't it?
>
> --
> "Los honestos son inadaptados sociales" -- Les Luthiers
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Possibilities and performance of conntrackd, NATing cluster

[Pablo Neira Ayuso]

julien vehent wrote:
> That sounds really interesting.
> Pablo : do you have more information/articles on what is done around
> conntrackd and how to set it up in a test bed environment ?

I'm finishing the user manual for the conntrack-tools right now. It will
be available in the upcoming 0.9.8 that should be out soon - once I
finish some QA tests.

But you can check a simple testbed here [1].

> I saw you published a paper earlier this year about that, but it's not
> available online. Is there any way to get it ?

Indeed. There are a couple of them, but they are mostly about design. I
expect to publish some drafts of them as they are copyrighted (those are
university works and the publishers impose important restrictions there,
that sucks but I had no choice).

[1] http://conntrack-tools.netfilter.org/testbed.html

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: Possibilities and performance of conntrackd, NATing cluster

[Matt Zagrabelny]

[-- Attachment #1: Type: text/plain, Size: 482 bytes --]

On Thu, 2008-09-18 at 16:25 +0200, Pablo Neira Ayuso wrote:

> [1] http://conntrack-tools.netfilter.org/testbed.html

I get a HTTP 404 on that.

-- 
Matt Zagrabelny - mzagrabe@d.umn.edu - (218) 726 8844
University of Minnesota Duluth
Information Technology Systems & Services
PGP key 1024D/84E22DA2 2005-11-07
Fingerprint: 78F9 18B3 EF58 56F5 FC85  C5CA 53E7 887F 84E2 2DA2

He is not a fool who gives up what he cannot keep to gain what he cannot
lose.
-Jim Elliot

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Possibilities and performance of conntrackd, NATing cluster

[Pablo Neira Ayuso]

Matt Zagrabelny wrote:
> On Thu, 2008-09-18 at 16:25 +0200, Pablo Neira Ayuso wrote:
> 
>> [1] http://conntrack-tools.netfilter.org/testbed.html
> 
> I get a HTTP 404 on that.

http://conntrack-tools.netfilter.org/testcase.html

Wrong address. This is what happen when you tell things by heart :(

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: Possibilities and performance of conntrackd, NATing cluster

[Michael Schwartzkopff]

Am Donnerstag, 18. September 2008 16:25 schrieben Sie:
> julien vehent wrote:
> > That sounds really interesting.
> > Pablo : do you have more information/articles on what is done around
> > conntrackd and how to set it up in a test bed environment ?
>
> I'm finishing the user manual for the conntrack-tools right now. It will
> be available in the upcoming 0.9.8 that should be out soon - once I
> finish some QA tests.
>
> But you can check a simple testbed here [1].
>
> > I saw you published a paper earlier this year about that, but it's not
> > available online. Is there any way to get it ?
>
> Indeed. There are a couple of them, but they are mostly about design. I
> expect to publish some drafts of them as they are copyrighted (those are
> university works and the publishers impose important restrictions there,
> that sucks but I had no choice).
>
> [1] http://conntrack-tools.netfilter.org/testbed.html

Link shoud be:

http://conntrack-tools.netfilter.org/testcase.html

-- 
Dr. Michael Schwartzkopff
MultiNET Services GmbH
Addresse: Bretonischer Ring 7; 85630 Grasbrunn; Germany
Tel: +49 - 89 - 45 69 11 0
Fax: +49 - 89 - 45 69 11 21
mob: +49 - 174 - 343 28 75

mail: misch@multinet.de
web: www.multinet.de

Sitz der Gesellschaft: 85630 Grasbrunn
Registergericht: Amtsgericht München HRB 114375
Geschäftsführer: Günter Jurgeneit, Hubert Martens

---

PGP Fingerprint: F919 3919 FF12 ED5A 2801 DEA6 AA77 57A4 EDD8 979B
Skype: misch42

Re: Possibilities and performance of conntrackd, NATing cluster

[icovnik]

Now only to clarify that I understand it correctly:

Asymmetric setup: Any router receives any of packets. All routers have
the same information about all connections in cluster, so it doesn't
matter which of them handles which connection.

Symmetric setup: Once the connection is setup on RouterX, the whole
connection should be handled by that very same router.

Is this correct?

On Wed, Sep 17, 2008 at 12:34 PM, Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> The way to go is a symmetric setup where all nodes receives the packets
> and only one firewall node handles them. This can be achieved by means
> of hash-based load-sharing. There's some works on that direction.

How is it possible to have only one firewall to handle packets in
cluster? Is it like in the setup in the testcase
(http://conntrack-tools.netfilter.org/testcase.html)? If I understand
it correctly, it means to have only one active firewall/router and one
passive waiting for failure. How is ti possible to scale to higher
loads?

Can you point me also to some info about hash-based load-sharing?

>> With how many routers?
>
> Limit? I don't know yet, I'm still testing with only two nodes, but I
> expect to do it with up to four. Moreover, the replication approaches
> still require a small change in the code to cleanly support more than
> two nodes.

If the load-sharing works (with more than two nodes maybe) I'd like to
test it. If it proves to work I can test it in real world scenario
with real ISP traffic. We are currently moving to new office so I can
post some results from testing in few weeks.

>> I know that you can do Active / Standby with conntrackd and I believe
>> that you can do Active / Active as well.  It is my understanding that
>> conntrackd broadcasts connection state on a separate network connection.
>>  I believe that the routers participating in the conntrackd failover
>> usually have three (or more) network cards on them, one internal and one

Yes, active/active is what I want.

> This is asymmetric multipath, it is not really a good idea and also
> you'll waste lots of resources in the replication. Therefore, if your
> intention is to improve scalability, this won't help. The way to go is
> the symmetric setup.

Can you write more about this? I'd like to test this setup.

>> routing) but is not required to.  With this in mind I'd recommend
>> something like VRRP for the internal and external interfaces where one
>> router is primary for the internal and outgoing interface and the other
>> router is primary for the external and incoming interface.  Using VRRP

Hm this is interresting - split incoming/outgoing traffic to separate
routers. Maybe the conntrackd can be used in this scenario. I would
test it.

ico

Re: Possibilities and performance of conntrackd, NATing cluster

[Grant Taylor]

On 09/23/08 05:05, icovnik wrote:
> Now only to clarify that I understand it correctly:
> 
> Asymmetric setup: Any router receives any of packets. All routers 
> have the same information about all connections in cluster, so it 
> doesn't matter which of them handles which connection.
> 
> Symmetric setup: Once the connection is setup on RouterX, the whole 
> connection should be handled by that very same router.
> 
> Is this correct?

Eh, close.

Symmetric is where all the traffic passes through the same firewall 
going both inbound and outbound, much like symmetric routes.

Where as asymmetric is where traffic passes through different firewalls 
going inbound and outbound, much like asymmetric routes.

As far as which firewalls know about the connection or not depends on 
how replication is set up.  However the symmetric verses asymmetric 
firewalling still applies.

> How is it possible to have only one firewall to handle packets in 
> cluster? Is it like in the setup in the testcase 
> (http://conntrack-tools.netfilter.org/testcase.html)? If I understand 
> it correctly, it means to have only one active firewall/router and 
> one passive waiting for failure. How is ti possible to scale to 
> higher loads?

Active / passive does not scale. A/P is only meant for redundancy / 
protection against one node failing.

> Hm this is interresting - split incoming/outgoing traffic to separate 
> routers. Maybe the conntrackd can be used in this scenario. I would 
> test it.

According to Pablo's reply to my earlier post, this is apparently not a 
good idea to do.  Though it sounds like it /may/ work, with some likely 
undesired side effects.



Grant. . . .