[LARTC] Re: gateway failover with linux

[LARTC] Re: gateway failover with linux

[Abhijit Menon-Sen]

At 2007-07-19 22:32:51 +0530, ams@toroid.org wrote:
>
> I'm wondering if there's a good way to configure a Linux firewall box
> to failover to a single backup server, while preserving connection
> state.

Looks like this is it:

http://people.netfilter.org/pablo/conntrack-tools/

-- ams
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

[LARTC] Re: gateway failover with linux

[Abhijit Menon-Sen]

Hi Grant.

At 2007-07-19 16:15:01 -0500, gtaylor@riverviewtech.net wrote:
>
> I'm a bit confused, are you wanting a single Linux firewall /
> router to have redundant internet connections, or to route
> traffic to redundant systems behind it and intelligently
> handle the failure of one or more of said redundant systems?

Neither.

I just want a hot standby for a single Linux firewall, such that clients
behind it are not affected by a hardware failure on the firewall. If my
configuration would allow me to someday promote the backup and run both
firewall machines in a load-balancing configuration, so much the better.

The following example looks very much like what I want:

    http://people.netfilter.org/pablo/conntrack-tools/testcase.html

(Can anyone comment on whether I should stick with keepalived as
described above, or try out ucarp?)

> Will you please clarify what you are really wanting to do per
> above and I'll be more than happy to try to point you in the
> right direction.

Thanks, I'd appreciate any advice you can give me.

-- ams
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Re: gateway failover with linux

[Mohan Sundaram]

Abhijit Menon-Sen wrote:
> Hi Grant.
> 
> At 2007-07-19 16:15:01 -0500, gtaylor@riverviewtech.net wrote:
>> I'm a bit confused, are you wanting a single Linux firewall /
>> router to have redundant internet connections, or to route
>> traffic to redundant systems behind it and intelligently
>> handle the failure of one or more of said redundant systems?
> 
> Neither.
> 
> I just want a hot standby for a single Linux firewall, such that clients
> behind it are not affected by a hardware failure on the firewall. If my
> configuration would allow me to someday promote the backup and run both
> firewall machines in a load-balancing configuration, so much the better.
> 
> The following example looks very much like what I want:
> 
>     http://people.netfilter.org/pablo/conntrack-tools/testcase.html
> 
> (Can anyone comment on whether I should stick with keepalived as
> described above, or try out ucarp?)
> 
>> Will you please clarify what you are really wanting to do per
>> above and I'll be more than happy to try to point you in the
>> right direction.
> 
> Thanks, I'd appreciate any advice you can give me.
> 
> -- ams
In case your firewall is a proxy for some service, those connections 
will fail though - unless you can use a virtual interface with the same 
IP as the source for such connections.

I guess you'll use vrrp in conjunction for failover. It would make sense 
to use vrrpd with status tracking of WAN gateway but AFAIK no such 
feature exists as yet.

Mohan
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Re: gateway failover with linux

[Grant Taylor]

On 07/19/07 20:20, Abhijit Menon-Sen wrote:
> I just want a hot standby for a single Linux firewall, such that 
> clients behind it are not affected by a hardware failure on the 
> firewall. If my configuration would allow me to someday promote the 
> backup and run both firewall machines in a load-balancing 
> configuration, so much the better.

Ok, to pull this off I would use a pair of Linux boxen with vrrpd and 
conntrackd with almost identical configurations on each system (save for 
host name and management IPs).

For the sake of discussion I'll presume that you are using a cable modem 
or adsl connection to the internet with a static IP address.  PPPoE / 
DHCP should be possible, but it will get much nastier.  You will really 
want a minimum of two static IP addresses, three if possible, on each 
side of the firewall.  One IP address will be for the routing of traffic 
and the other two will be for management.  The one routing IP address 
will be the virtual IP that is passed back and forth between the systems 
and also the IP address that it looks like all your traffic passes 
through.  Use VRRP to make sure that one of the systems will always have 
the routing IP up and functional.

> The following example looks very much like what I want:
> 
> 	http://people.netfilter.org/pablo/conntrack-tools/testcase.html

*nod*  This is very much what you are wanting to do.

However I'm not sure that the direct connection between the two systems 
is absolutely required.  However having not done this my self I can not 
say for sure.  The last time I looked at this I was going to be trying 
to make redundant routers for a 1 - 3 Mbps connection and thus was 
planing on using VLANs to do Router-on-a-stick across two bonded 100 
BaseT connections.  So, if I needed to I could create a new VLAN for the 
router to router communications.

> (Can anyone comment on whether I should stick with keepalived as 
> described above, or try out ucarp?)

I would use vrrpd over keepalived.  As far as ucarp, I'm not familiar 
with it so I can't comment.  I can say that keepalived is much more 
complex and can do a lot more than what you need to do.  Now if you were 
running a load balancing cluster of servers behind it, keepalived is the 
proper answer.  However for just making sure that a router is available, 
vrrpd is much closer to what you need with out overkilling.  In fact, I 
believe vrrpd only needs command line options and no config file where 
as, last I looked, keepalived has a very complex config file for all 
that it could do.

With regards to "... run both firewall machines in a load-balancing 
configuration ..." I think that conntrackd will allow you to do this. 
At the very least, you can have inbound traffic use one firewall as its 
primary and outbound use the other firewall as its primary where each is 
a failover for the other.  If you want inbound and / or outbound to use 
both routers at the same time, it gets very trick to have the layer 2 
ethernet network know which router to use to send the traffic to.

A quick run down of how VRRP (and Cisco's proprietary HSRP) works for 
those that may not know or want a refresher.  VRRP uses a management IP 
and a virtual routing (that may or may not be one of the management 
IPs).  The virtual routing IP uses a special MAC address 
(00-00-5E-00-01-<virtual router ID>).  This special MAC address allows 
all client workstations using the virtual routing IP to be able to cache 
the MAC address and not have to re-arp for the system that currently has 
the IP.  In the event that the backup router detects that the primary 
router is down, the backup router will claim to be the virtual IP on the 
virtual router MAC address, usually GARPing so that switches now see the 
virtual router MAC address on the new switch port.  Thus when clients 
try to send traffic out through the virtual router, the layer 2 traffic 
will be to the virtual MAC address which the switches now know to be 
elsewhere on the functioning router.  The net result is a very brief 
down time while the backup router is detecting that the primary is not 
functioning and switches over.

(Note:  This is a very far fetch problem, but is possible.)  The only 
real problem with this scenario is if for some reason both routers are 
up and functioning, but for some reason they can not see each other, 
thus they both think the other node is down.  In this case, you may have 
a battle for who is up.  If you want to know what to do in this 
situation read about SONITH (Shoot Other Node In The Head) to make sure 
that there is only one active node at a time.

You can configure VRRP to automatically switch back from the backup to 
the primary when the primary comes back on line, or let the backup 
remain the primary until it fails and then the real primary that is 
acting as the backup will resume its role as the primary.

> Thanks, I'd appreciate any advice you can give me.

Having a functioning understanding of what is going on will help make 
this a success.  What you are wanting to do is completely possible and 
should be able to be achieved with out a lot of problems, but it is out 
side of the scope of any point and click GUI that I have seen for Linux.

Good luck and let me know if there is any thing else that I can do to help.



Grant. . . .
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Re: gateway failover with linux

[Grant Taylor]

On 07/19/07 20:32, Mohan Sundaram wrote:
> It would make sense to use vrrpd with status tracking of WAN gateway 
> but AFAIK no such feature exists as yet.

If I understand what you are saying, you are referring to an external 
T-1 router converting from the WAN circuit to the ethernet going in to 
the two routers right?

If this is the case, no matter what is done, the WAN router is a single 
point of failure and thus can not be avoided.

No matter what, the Linux boxen can be configured such that they will 
try to reach this single point of failure and in the event that they can 
not do so, they (both) will ultimately return an ICMP "no route to host" 
error message back to the client.  However this is out side the scope of 
what VRRP is meant to do.



Grant. . . .
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

[LARTC] Re: gateway failover with linux

[Abhijit Menon-Sen]

At 2007-07-20 10:12:01 -0500, gtaylor@riverviewtech.net wrote:
>
> > I just want a hot standby for a single Linux firewall [...]
> 
> I would use a pair of Linux boxen with vrrpd and conntrackd

OK, great. I didn't know about vrrpd. I'll check it out.

> As far as ucarp, I'm not familiar with it so I can't comment.

If I have the time, I'll try out ucarp and post a summary of my
experiences for the archives.

> If you want to know what to do in this situation read about SONITH
> (Shoot Other Node In The Head) to make sure that there is only one
> active node at a time.

("STONITH", for those asking Google.)

I have one other question. How does conntrackd interact with traffic
shaping? My firewall also uses HTB to impose various bandwidth limits
on clients. From what I've read so far, I have the impression that the
failover may lose some packets that are being delayed in a queue, but
existing connections should recover and be esentially unaffected.

Can anyone confirm that?

-- ams
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Re: gateway failover with linux

[Grant Taylor]

On 7/20/2007 9:03 PM, Mohan Sundaram wrote:
> I think my point was misunderstood. Let us say each of these Linux 
> boxes are connected to a WAN link each. If the WAN gateway/link of a 
> box goes down, vrrp must flag itself down or as secondary. Some 
> similarity to our earlier discussions on redundant gateways. This 
> feature of object tracking is available in CISCO (their patent) but 
> only tracks the interface status and not the gateway reachability. 
> I'd love to have a feature where gateway reachability is tracked.

Each firewall / router / gateway having its own independent wan / 
internet connection makes things a bit different.  First of all, each 
will have its own public IP address for the associated WAN link and as 
such probably have it's own NAT configuration.

I wonder, what type of WAN connection are these?  Could both be hooked 
up to both gateways?  In other words are they ethernet or something that 
gets bridged to ethernet or are they some sort of legacy WAN link, i.e. 
T1, Frame Relay, ATM?  If you could connect the WAN links to both 
systems, you can have even more functionality and you would be back to 
what I was originally thinking except for the fact that instead of one 
WAN connection, you have two to make each router aware of.

At the very least, I think you will need to make each router aware of 
the other one.  This way, if a router's (primary) WAN link is not usable 
it can route the traffic over to the other router and have it route the 
traffic out to the world.  Thus each router would have a primary default 
gateway of the router at the other end of its WAN link *AND* a secondary 
default gateway of the other router that it is connected to.

I also think that you are very close to needing to use some sort of 
monitoring utility / daemon to check the status of the WAN link *AND* to 
make the other router aware of the status.  This may be easily done with 
a small daemon to monitor the link and update the local routing table in 
conjunction with a routing protocol between the two routers to keep each 
aware of the others routing table.

If you have any more information on Cisco's technology I'd be interested 
in doing some reading about it if you would care to toss it my way.



Grant. . . .
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Re: gateway failover with linux

[Grant Taylor]

On 7/20/2007 9:18 PM, Mohan Sundaram wrote:
> In VRRP based gateway pairs, this is normally done by partitioning 
> the LAN to use both gateways by different subnets. i.e G1 is primary 
> for subnet1 with G2 as secondary; G2 is primary for subnet2 with G1 
> as secondary. This is  done by defining multiple vrrp groups. AFAIK, 
> no dynamic balancing methods/ features exist.

Agreed, this would be the easiest to implement active / active with 
traffic flowing out through both routers.  I was trying to state that 
outbound for the network would flow through one router while inbound 
would flow through the other router.  This could also be extended to 
mirror the multiple subnets like you are talking about too, though I 
think load sharing reasoning would be lost at that point as you have 
more traffic on both routers.  At this point in time it would come down 
to statistics of network load to see how you wanted to do it.

I'm sure there are ways that you can have all traffic spread across both 
routers in either direction.  Though I'm not quite sure how to go about 
it.  I'm going to say that VRRP / Load Balancing / Bridging / Layer 2 
Filtering / and other services would be in effect to divide the traffic 
across both active routers.  No matter how you slice it, this is beyond 
the scope of the OPs question.  I was merely stating that things are 
possible.




Grant. . . .
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

[LARTC] Re: gateway failover with linux

[Abhijit Menon-Sen]

At 2007-07-21 05:59:54 +0530, ams@toroid.org wrote:
>
> If I have the time, I'll try out ucarp and post a summary of my
> experiences for the archives.

Not much to report. I set up ucarp as directed in the README, and it
just worked. It was simple and did what I wanted (which was to allow
two machines to share a virtual IP with heartbeat/failover).

-- ams
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc