[LARTC] Iptables, SNAT/MASQ, Multiple gateways

[LARTC] Iptables, SNAT/MASQ, Multiple gateways

[Simon Matthews]

I have a dual-homed firewall. It has 2 Internet connections, provided by
different ISPs (each with an associated IP address). The 2 Internet
connections are connected to the same physical interface. The 2 Internet
connections do NOT have equal bandwidth

How do I configure the SNAT/MASQ and ensure sharing of the gateways with
the correct ratio of usage and with the correct source IP address? 

I know how to use the 'ip' commands to configure gateway sharing according
to my defined ratios and ensure that packets go out of the correct gateway 
according to their source address.

However, I need to configure SNAT-ed or MASQ-ed in such a way that it 
shares the gateways according to my defined ration and so that it sets the 
source address correctly for the gateway. How do I do this?

Can someone please point me in the correct direction for the SNAT/MASQ 
setup with iptables/iproute2?

Thanks!
Simon

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Iptables, SNAT/MASQ, Multiple gateways

[José Luis Domingo López]

On Sunday, 29 September 2002, at 11:24:03 -0700,
Simon Matthews wrote:

> I have a dual-homed firewall. It has 2 Internet connections, provided by
> different ISPs (each with an associated IP address). The 2 Internet
> connections are connected to the same physical interface. The 2 Internet
> connections do NOT have equal bandwidth
> 

> How do I configure the SNAT/MASQ and ensure sharing of the gateways with
> the correct ratio of usage and with the correct source IP address? 
> 
ip route add default nexthop via $CONN1_IP dev $ETHX weight $X \
                     nexthop via $CONN2_IP dev $ETHX weight $Y

weight(s) are the relative bandwidths of the connections. If the first
is a 2 Mbps line and the second a 512 Kbps one, $=4 and $Y=1, for example.
What matters is the ratio, not the actual bandwidth.

> I know how to use the 'ip' commands to configure gateway sharing according
> to my defined ratios and ensure that packets go out of the correct gateway 
> according to their source address.
> 
The above defines routing. And SNAT/MASQ is something done afterwards,
so you could SNAT/MASQ traffic going to each connection differently.
What I don't know right now is how to determine to which Internet
connection are packets going in your setup with only an ethernet card
for both. With a card for each Internet connection it is simple.

Hope this helps.

-- 
Jose Luis Domingo Lopez
Linux Registered User #189436     Debian Linux Woody (Linux 2.4.19-pre6aa1)
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Iptables, SNAT/MASQ, Multiple gateways

[Don Cohen]

 > > I have a dual-homed firewall. It has 2 Internet connections, provided by
 > > different ISPs (each with an associated IP address). The 2 Internet
 > > connections are connected to the same physical interface. The 2 Internet
 > > connections do NOT have equal bandwidth
 > > How do I configure the SNAT/MASQ and ensure sharing of the gateways with
 > > the correct ratio of usage and with the correct source IP address? 
 > > 
 > ip route add default nexthop via $CONN1_IP dev $ETHX weight $X \
 >                      nexthop via $CONN2_IP dev $ETHX weight $Y

Note that this only shapes outgoing traffic and also relies on your
ISPs to NOT do the ingress filtering that they're really supposed to do.

If most of the traffic at your site is going out this might be
reasonable.  For most people, most of the traffic is coming in, e.g.,
behind your firewall are clients who want to download data.  The
bandwidth going out is mostly acks, which is very small compared to
that coming in.

Which interface traffic comes in on depends on which IP address it's
sent to, which depends on which address the original request came
from.  That's related to the nat decision but there's no way to make
the "right" decision.  You can divide the original requests in some
given proportion, but you don't know when you do that how much traffic
each one will generate.  So, for instance, the clients behind your
firewall make 10 http requests.  You nat 8 to the fast interface and 2
to the slow one.  But then it turns out that the 8 you send to the
fast one all result in small amounts of data while one of the two sent
to the slow interface results in a large amount.  You might be better
off just using the fast one, or perhaps using the slow one only for
stuff that you can be sure will fit comfortably in its bandwidth.
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Iptables, SNAT/MASQ, Multiple gateways

[Simon Matthews]

On Sun, 29 Sep 2002, Don Cohen wrote:

>  > > I have a dual-homed firewall. It has 2 Internet connections, provided by
>  > > 
>  > ip route add default nexthop via $CONN1_IP dev $ETHX weight $X \
>  >                      nexthop via $CONN2_IP dev $ETHX weight $Y
> 
> Note that this only shapes outgoing traffic and also relies on your
> ISPs to NOT do the ingress filtering that they're really supposed to do.
> 
> If most of the traffic at your site is going out this might be
> reasonable.  For most people, most of the traffic is coming in, e.g.,
> behind your firewall are clients who want to download data.  The
> bandwidth going out is mostly acks, which is very small compared to
> that coming in.
> 
> off just using the fast one, or perhaps using the slow one only for
> stuff that you can be sure will fit comfortably in its bandwidth.

OK, this may be a reasonable approach, but how do I force it initiate 
connections from the "fast" interface, yet allow it to fail over to the 
slow interface if the sytem removes the route to the fast gateway because 
it has detected that it is not responding? 

Simon

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Iptables, SNAT/MASQ, Multiple gateways

[Don Cohen]

Simon Matthews writes:
 > OK, this may be a reasonable approach, but how do I force it initiate 
 > connections from the "fast" interface, yet allow it to fail over to the 
 > slow interface if the sytem removes the route to the fast gateway because 
 > it has detected that it is not responding? 

Off hand I don't know anything built in for this (I look forward to
hearing an answer from someone who does), but I don't think this is 
really what you want anyway.  It's not as if your link is the only one
that could fail!
If ISP1's upstream link fails then you want to use ISP2 for all
traffic other than that intended for ISP1 itself.  And of course, 
problems further upstream prevent you from reaching certain addresses
but not others, and you don't really know which without a global view
of the routing.

I think the "right" solution involves monitoring the traffic.
There's a wide range of things you could do, the simplest being 
simply detecting that the link is not responding.  You could also
try to detect tcp retransmits, measure RTT, aggregate data to measure
how well individual connections are working, further aggregate data to
determine which addresses blocks are working well and which poorly, etc.
Then use that data to decide which of your links to use for a given
destination.

I actually sent a proposal to this list that I think provides a good
solution to the general problem: an extension to TCP (possibly even
IP) that supports multiple addresses/ports.  This would even allow you
to switch addresses in the middle of a connection.  I think what I
described before applies more to the machine on the other side of your
connection, which now would know both of your addresses.  Whenever it
does a tcp retransmit it switches the address.  It therefore tends to
stay on the one that works most reliably.  (Perhaps this algorithm
could be improved to take speed into account too.)  This discussion
points out that something similar should be done on your end: you
should switch the output interface you use when you retransmit.

Of course this is not yet implemented.  It's on my queue, but not
close to the beginning.  I'd be glad if someone out there could beat
me to it. 
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Iptables, SNAT/MASQ, Multiple gateways

[Michael T. Babcock]

Don Cohen wrote:

>I actually sent a proposal to this list that I think provides a good
>solution to the general problem: an extension to TCP (possibly even
>IP) that supports multiple addresses/ports.  This would even allow you
>to switch addresses in the middle of a connection.  I think what I
>  
>
SCTP actually supports this already; look it up -- its quite a bit 
different from TCP but allows you to do all the same types of things, 
with more options.

That said, a Zebra (routing software) plugin that would run iptables 
scripts would be all you'd need in many cases.

-- 
Michael T. Babcock
C.T.O., FibreSpeed Ltd.
http://www.fibrespeed.net/~mbabcock


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Iptables, SNAT/MASQ, Multiple gateways

[Jose Luis Domingo Lopez]

On Sunday, 29 September 2002, at 22:18:30 -0700,
Don Cohen wrote:

>  > ip route add default nexthop via $CONN1_IP dev $ETHX weight $X \
>  >                      nexthop via $CONN2_IP dev $ETHX weight $Y
> 
> Note that this only shapes outgoing traffic and also relies on your
> ISPs to NOT do the ingress filtering that they're really supposed to do.
> 
Just a note. The above routing doesn't prevent you from applying
SNAT/MASQ to the outgoing traffic, at least not when you have an
ethernet card for each connection (not the case) and you can know
through each one the traffic will go out.

So adding another ethernet card and a couple of "iptables" rules can
avoid problems with ISPs filtering "alien" incoming traffic :)

-- 
Jose Luis Domingo Lopez
Linux Registered User #189436     Debian Linux Woody (Linux 2.4.19-pre6aa1)
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Iptables, SNAT/MASQ, Multiple gateways

[Julian Anastasov]

	Hello,

On Mon, 30 Sep 2002, Michael T. Babcock wrote:

> Don Cohen wrote:
>
> >I actually sent a proposal to this list that I think provides a good
> >solution to the general problem: an extension to TCP (possibly even
> >IP) that supports multiple addresses/ports.  This would even allow you
> >to switch addresses in the middle of a connection.  I think what I

	Yes, we can implement it as separate IP protocol :)
Of course, at the beginning the idea may sound too stupid, we
have to change that. May be there is already solution for that?
A "simple" tunnel without encryption that will support failover
and balancing of the negotiated traffic, ability to negotiate
multiple IPs for each endpoint. Of course, there should be some
problems with the proper tunneling of this traffic in each end,
see how difficult is routed the IPSec traffic. Each endpoint will do
failover detection of all negotiated links and will do balancing (if
desired) over these links, based on relative ratio. This tunnel
should be transparent to the upper layers (TCP/UDP/ICMP/SCTP).

> SCTP actually supports this already; look it up -- its quite a bit
> different from TCP but allows you to do all the same types of things,
> with more options.

	But this feature is only for SCTP. We want the traffic
from one multihomed router to use multiple links when talking
to another router, both understanding this "our new" IP tunneling
protocol.

	I see it in this way: when such packet is received, we
decapsulate it and place it on the expected interface. As
result, the upper layers will see the packet on the right
input interface even if it is received on another input
interface (for example, if it is the only alive).

Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Iptables, SNAT/MASQ, Multiple gateways

[Simon Matthews]

On Mon, 30 Sep 2002, Don Cohen wrote:

> Simon Matthews writes:
>  > OK, this may be a reasonable approach, but how do I force it initiate 
>  > connections from the "fast" interface, yet allow it to fail over to the 
>  > slow interface if the sytem removes the route to the fast gateway because 
>  > it has detected that it is not responding? 
> 
> Off hand I don't know anything built in for this (I look forward to
> hearing an answer from someone who does), but I don't think this is 
> really what you want anyway.  It's not as if your link is the only one
> that could fail!

Don, there are some kernel patches (already installed on my system) that 
support dead gateway detection and static routes. "Static" means that the 
routes are not forgotten when the system removes an interface because the 
gateway is not working. 

But the problem remains: how to handle this in iptables MASQ/SNAT
commands? One can postulate that if the interface is removed because the
gateway is dead, then the MASQ command will use the source related to the 
other gateway.  

However, the question now is: how to force the system to use the source
address related to the "fast" gateway under normal operation while
allowing a failover to the the slow gateway?

Simon

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Iptables, SNAT/MASQ, Multiple gateways

[Simon Matthews]

On Mon, 30 Sep 2002, Michael T. Babcock wrote:

> Don Cohen wrote:
> 
> 
> That said, a Zebra (routing software) plugin that would run iptables 
> scripts would be all you'd need in many cases.

The ISP that provides the "fast" connection won't provide any IGP routing 
information (RIP, OSPF, etc), so I don't think this is possible. 

> 
> 

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

RE: [LARTC] Iptables, SNAT/MASQ, Multiple gateways

[Greg Scott]

 
> ip route add default nexthop via $CONN1_IP dev $ETHX weight $X \
>                     nexthop via $CONN2_IP dev $ETHX weight $Y
>

Would this technique work for more than two gateways?  How many nexthop clauses are allowed?  Is there a limit?

thanks

- Greg Scott
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Iptables, SNAT/MASQ, Multiple gateways

[William L. Thomson Jr.]

Simon,

On Mon, 2002-09-30 at 00:06, Simon Matthews wrote:
> 
> OK, this may be a reasonable approach, but how do I force it initiate 
> connections from the "fast" interface, yet allow it to fail over to the 
> slow interface if the sytem removes the route to the fast gateway because 
> it has detected that it is not responding? 

The way I see things it basically works like this, the out going traffic
will get balanced via the next hop and weights. So based on the weight
values, one will be used more than the other.

Now to get the same thing to happen from the outside world, you need to
load balance your dns. Look into setting up multiple dns records, and
possible weight values as well.

It's pretty safe to say that if the fast connection goes down, it will
be unavailable for the dns system to load balance, and all should divert
to the line that is up. I would assume it to be best if using internal
dns servers, but may work better with outside dns servers if possible?
Now that's from the outside.

You mentioned using dead gateway detection to remove, and hopefully add
back the route when it becomes available again. Which will be needed so
that the route is no longer used from the inside. Causing the route the
is available to be the only one used from the inside.

This switch over can be seamless, but if you are say streaming media, it
will be interrupted, as an example. Other things like http, will most
likely be fine, stateless things won't care, others will.

You real problem is for say if a large download or something is diverted
from a fast line to a slow line. I would imagine it to be somewhat
unlikely since, the fast one may but in during the lookup process. You
may want to look into some sort of traffic control or shaping, which may
be a good idea in case that the fast line is down. You may really want
to control the left over bandwidth for important stuff. Someone else
also mentioned this. TC and QoS is not an area I have much experience
in, so you will have to look to other there.

In the end the over all load balancing logic is very general, and not
extremely smart/advanced. It would be nice if it was. You have the next
hops from the inside, and dns for the outside. Not to much beyond that.

Beyond that you will have to look to other things to really dial things
in, in my opinion. Not sure if this helps or not, but hopefully it will.

Good luck

-- 
Sincerely,
William L. Thomson Jr.
Support Group
Obsidian-Studios Inc.
439 Amber Way
Petaluma, Ca. 94952
Phone  707.766.9509
Fax    707.766.8989
http://www.obsidian-studios.com

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/