[LARTC] Balancing multiple connections and NAT

[LARTC] Balancing multiple connections and NAT

[Raj Mathur]

Hi,

I have a client connected to the 'net through 3 ISP's.  Have set up a
Linux box to do routing and load sharing for the 3 connections.  A
fourth interface is connected to the LAN with private IP addresses.
Am using iptables to SNAT traffic to the appropriate IP depending on
the interface the packet gets routed onto.  The setup looks something
like this:

Interface       IP               Gateway	  Table	  Network
---------       --               -------          -----   -------
intA            ipA              gwA		  tableA  netA
intB            ipB              gwB		  tableB  netB
intC            ipC              gwC		  tableC  netC
[intD is the LAN interface]
intD            ipD (private)    no gateway	  global  netD

This works fine most of the time, except that once in a while (every
5-10 minutes or so) packets going out on (e.g.) intB suddenly start
getting NAT'ed to source address ipA (i.e. the address of another
interface).  Obviously this plays hell with the existing connections
on that link!

The ip commands I'm using are:

/sbin/ip route add netA dev intA src ipA table tableA
/sbin/ip route add netA dev intA src ipA
/sbin/ip route add default via gwA table tableA
/sbin/ip route add netB dev intB src ipB table tableB
/sbin/ip route add netB dev intB src ipB
/sbin/ip route add default via gwB table tableB
/sbin/ip route add netC dev intC src ipC table tableC
/sbin/ip route add netC dev intC src ipC
/sbin/ip route add default via gwC table tableC
/sbin/ip route add default scope global nexthop via gwB dev intB weight 1 nexthop via gwC dev intC weight 2 nexthop via gwA dev intA weight 2
/sbin/ip rule add from ipA table tableA
/sbin/ip rule add from ipB table tableB
/sbin/ip rule add from ipC table tableC

The iptables commands are:

/sbin/iptables -P FORWARD DROP
# Enable full flow on the LAN
/sbin/iptables -I FORWARD -s netD -i intD -j ACCEPT
/sbin/iptables -I FORWARD -d netD -o intD -j ACCEPT
# Allow all packets to go out
/sbin/iptables -I OUTPUT -o intA -j ACCEPT
/sbin/iptables -I OUTPUT -o intB -j ACCEPT
/sbin/iptables -I OUTPUT -o intC -j ACCEPT
/sbin/iptables -I OUTPUT -o intD -j ACCEPT
/sbin/iptables -I INPUT -i intD -j ACCEPT
/sbin/iptables -I INPUT -i lo -j ACCEPT
/sbin/iptables -P INPUT DROP
/sbin/iptables -A INPUT -i ! intD -m state --state RELATED,ESTABLISHED -j ACCEPT
# Hmmm, why is this one there?
/sbin/iptables -A INPUT -i intD -m state --state RELATED,ESTABLISHED -j ACCEPT
# NAT depending on outbound interface
/sbin/iptables -t nat -A POSTROUTING -s netD -o intA -j SNAT --to-source ipA
/sbin/iptables -t nat -A POSTROUTING -s netD -o intB -j SNAT --to-source ipB
/sbin/iptables -t nat -A POSTROUTING -s netD -o intC -j SNAT --to-source ipC

Any idea why connections that are flowing perfectly would suddenly
decide to start getting NAT'ed to the wrong source?  Or some place on
the 'net I can start looking?

Regards,

-- Raju
-- 
Raj Mathur                raju@kandalaya.org      http://kandalaya.org/
       GPG: 78D4 FC67 367F 40E2 0DD5  0FEF C968 D0EF CC68 D17F
                      It is the mind that moves
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Balancing multiple connections and NAT

[Edmundo Carmona]

That's because the route to a host X has been changed. There's a
routing decision to host X every so or so minutes.

Ways to handle that? Boy, that can be tough. I've read of prople who
use a different LAN IP for every public connection. So If you have 3
public connections, you should use three different IPs on the lan.
Then, when you DNAT packets on the way in (from the internet), you
mark those packets with a different fwmark for each internet
connection. On POSTROUTING do a SNAT according to this fwmark to a
different IP each (remember you have three inner IPs to choose from).
Then when packets come back from your servers, they will be sent to
three different IPs and you can tell packets that should go out one
way or the other.

Now, if you don't care about incoming (to your lan) connections but
outgoing.... I guess you are very much out of luck.... unless you use
some policy routing for stateful connections  (I think that's the name
of connections that DO care if you change the IP the connection is
going through) so that they use a single interface all the time.

On 2/23/06, Raj Mathur <raju@linux-delhi.org> wrote:
> Hi,
>
> I have a client connected to the 'net through 3 ISP's.  Have set up a
> Linux box to do routing and load sharing for the 3 connections.  A
> fourth interface is connected to the LAN with private IP addresses.
> Am using iptables to SNAT traffic to the appropriate IP depending on
> the interface the packet gets routed onto.  The setup looks something
> like this:
>
> Interface       IP               Gateway          Table   Network
> ---------       --               -------          -----   -------
> intA            ipA              gwA              tableA  netA
> intB            ipB              gwB              tableB  netB
> intC            ipC              gwC              tableC  netC
> [intD is the LAN interface]
> intD            ipD (private)    no gateway       global  netD
>
> This works fine most of the time, except that once in a while (every
> 5-10 minutes or so) packets going out on (e.g.) intB suddenly start
> getting NAT'ed to source address ipA (i.e. the address of another
> interface).  Obviously this plays hell with the existing connections
> on that link!
>
> The ip commands I'm using are:
>
> /sbin/ip route add netA dev intA src ipA table tableA
> /sbin/ip route add netA dev intA src ipA
> /sbin/ip route add default via gwA table tableA
> /sbin/ip route add netB dev intB src ipB table tableB
> /sbin/ip route add netB dev intB src ipB
> /sbin/ip route add default via gwB table tableB
> /sbin/ip route add netC dev intC src ipC table tableC
> /sbin/ip route add netC dev intC src ipC
> /sbin/ip route add default via gwC table tableC
> /sbin/ip route add default scope global nexthop via gwB dev intB weight 1 nexthop via gwC dev intC weight 2 nexthop via gwA dev intA weight 2
> /sbin/ip rule add from ipA table tableA
> /sbin/ip rule add from ipB table tableB
> /sbin/ip rule add from ipC table tableC
>
> The iptables commands are:
>
> /sbin/iptables -P FORWARD DROP
> # Enable full flow on the LAN
> /sbin/iptables -I FORWARD -s netD -i intD -j ACCEPT
> /sbin/iptables -I FORWARD -d netD -o intD -j ACCEPT
> # Allow all packets to go out
> /sbin/iptables -I OUTPUT -o intA -j ACCEPT
> /sbin/iptables -I OUTPUT -o intB -j ACCEPT
> /sbin/iptables -I OUTPUT -o intC -j ACCEPT
> /sbin/iptables -I OUTPUT -o intD -j ACCEPT
> /sbin/iptables -I INPUT -i intD -j ACCEPT
> /sbin/iptables -I INPUT -i lo -j ACCEPT
> /sbin/iptables -P INPUT DROP
> /sbin/iptables -A INPUT -i ! intD -m state --state RELATED,ESTABLISHED -j ACCEPT
> # Hmmm, why is this one there?
> /sbin/iptables -A INPUT -i intD -m state --state RELATED,ESTABLISHED -j ACCEPT
> # NAT depending on outbound interface
> /sbin/iptables -t nat -A POSTROUTING -s netD -o intA -j SNAT --to-source ipA
> /sbin/iptables -t nat -A POSTROUTING -s netD -o intB -j SNAT --to-source ipB
> /sbin/iptables -t nat -A POSTROUTING -s netD -o intC -j SNAT --to-source ipC
>
> Any idea why connections that are flowing perfectly would suddenly
> decide to start getting NAT'ed to the wrong source?  Or some place on
> the 'net I can start looking?
>
> Regards,
>
> -- Raju
> --
> Raj Mathur                raju@kandalaya.org      http://kandalaya.org/
>        GPG: 78D4 FC67 367F 40E2 0DD5  0FEF C968 D0EF CC68 D17F
>                       It is the mind that moves
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Balancing multiple connections and NAT

[Nataniel Klug]

Raj,

I use something just like you make and for this problem I have patched my
kernel with diff-routes patch (there is a link in LARTC) and I have make a
little script that makes just a ping to an outside address every 5 min (cron
job) and flush the route cache after this ping, like this:

ip route flush cache

So all cache will be lost (off course that conections that still exists will
remain with their track to the destination).

Att,

Nataniel Klug


----- Original Message ----- 
From: "Raj Mathur" <raju@linux-delhi.org>
To: <lartc@mailman.ds9a.nl>
Sent: Thursday, February 23, 2006 4:14 PM
Subject: [LARTC] Balancing multiple connections and NAT


> Hi,
>
> I have a client connected to the 'net through 3 ISP's.  Have set up a
> Linux box to do routing and load sharing for the 3 connections.  A
> fourth interface is connected to the LAN with private IP addresses.
> Am using iptables to SNAT traffic to the appropriate IP depending on
> the interface the packet gets routed onto.  The setup looks something
> like this:
>
> Interface       IP               Gateway   Table   Network
> ---------       --               -------          -----   -------
> intA            ipA              gwA   tableA  netA
> intB            ipB              gwB   tableB  netB
> intC            ipC              gwC   tableC  netC
> [intD is the LAN interface]
> intD            ipD (private)    no gateway   global  netD
>
> This works fine most of the time, except that once in a while (every
> 5-10 minutes or so) packets going out on (e.g.) intB suddenly start
> getting NAT'ed to source address ipA (i.e. the address of another
> interface).  Obviously this plays hell with the existing connections
> on that link!
>
> The ip commands I'm using are:
>
> /sbin/ip route add netA dev intA src ipA table tableA
> /sbin/ip route add netA dev intA src ipA
> /sbin/ip route add default via gwA table tableA
> /sbin/ip route add netB dev intB src ipB table tableB
> /sbin/ip route add netB dev intB src ipB
> /sbin/ip route add default via gwB table tableB
> /sbin/ip route add netC dev intC src ipC table tableC
> /sbin/ip route add netC dev intC src ipC
> /sbin/ip route add default via gwC table tableC
> /sbin/ip route add default scope global nexthop via gwB dev intB weight 1
nexthop via gwC dev intC weight 2 nexthop via gwA dev intA weight 2
> /sbin/ip rule add from ipA table tableA
> /sbin/ip rule add from ipB table tableB
> /sbin/ip rule add from ipC table tableC
>
> The iptables commands are:
>
> /sbin/iptables -P FORWARD DROP
> # Enable full flow on the LAN
> /sbin/iptables -I FORWARD -s netD -i intD -j ACCEPT
> /sbin/iptables -I FORWARD -d netD -o intD -j ACCEPT
> # Allow all packets to go out
> /sbin/iptables -I OUTPUT -o intA -j ACCEPT
> /sbin/iptables -I OUTPUT -o intB -j ACCEPT
> /sbin/iptables -I OUTPUT -o intC -j ACCEPT
> /sbin/iptables -I OUTPUT -o intD -j ACCEPT
> /sbin/iptables -I INPUT -i intD -j ACCEPT
> /sbin/iptables -I INPUT -i lo -j ACCEPT
> /sbin/iptables -P INPUT DROP
> /sbin/iptables -A INPUT -i ! intD -m state --state RELATED,ESTABLISHED -j
ACCEPT
> # Hmmm, why is this one there?
> /sbin/iptables -A INPUT -i intD -m state --state RELATED,ESTABLISHED -j
ACCEPT
> # NAT depending on outbound interface
> /sbin/iptables -t nat -A POSTROUTING -s netD -o intA -j SNAT --to-source
ipA
> /sbin/iptables -t nat -A POSTROUTING -s netD -o intB -j SNAT --to-source
ipB
> /sbin/iptables -t nat -A POSTROUTING -s netD -o intC -j SNAT --to-source
ipC
>
> Any idea why connections that are flowing perfectly would suddenly
> decide to start getting NAT'ed to the wrong source?  Or some place on
> the 'net I can start looking?
>
> Regards,
>
> -- Raju
> -- 
> Raj Mathur                raju@kandalaya.org      http://kandalaya.org/
>        GPG: 78D4 FC67 367F 40E2 0DD5  0FEF C968 D0EF CC68 D17F
>                       It is the mind that moves
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Balancing multiple connections and NAT

[Markus Schulz]

Am Donnerstag, 23. Februar 2006 20:14 schrieb Raj Mathur:
> Hi,
>
> I have a client connected to the 'net through 3 ISP's.  Have set up a
> Linux box to do routing and load sharing for the 3 connections.  A
> fourth interface is connected to the LAN with private IP addresses.
> Am using iptables to SNAT traffic to the appropriate IP depending on
> the interface the packet gets routed onto.  The setup looks something
> like this:
>
> Interface       IP               Gateway	  Table	  Network
> ---------       --               -------          -----   -------
> intA            ipA              gwA		  tableA  netA
> intB            ipB              gwB		  tableB  netB
> intC            ipC              gwC		  tableC  netC
> [intD is the LAN interface]
> intD            ipD (private)    no gateway	  global  netD
>
> This works fine most of the time, except that once in a while (every
> 5-10 minutes or so) packets going out on (e.g.) intB suddenly start
> getting NAT'ed to source address ipA (i.e. the address of another
> interface).  Obviously this plays hell with the existing connections
> on that link!

you need a patch for NAT processing with multiple gateways. this will 
then save the routing information for each connection inside NAT 
structures, so that each packet of an established connection will be 
get routed over the same gateway. you can find the patches here:
http://www.ssi.bg/~ja/#routes
please read the guides (nano howto or dgd-usage) carefully.


-- 
Markus Schulz
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Balancing multiple connections and NAT

[Sebastian Bork]

On Fr, 2006-02-24 at 00:44 +0530, Raj Mathur wrote:
> I have a client connected to the 'net through 3 ISP's.  Have set up a
> Linux box to do routing and load sharing for the 3 connections.  A
> fourth interface is connected to the LAN with private IP addresses.
> Am using iptables to SNAT traffic to the appropriate IP depending on
> the interface the packet gets routed onto.

I use exactly the same setup with a customer's conenction, the only
difference: I use MASQUERADE instead of SNAT. I did not see anything
like the problem you describe. Maybe because MAQUERADE works stateful,
SNAT not? If you do not have a special reason for using SNAT, I think
you should try MASQUERADE. If your problem persits, please tell me, as I
have to look at my customer's setup very closely then, to catch this
before anyone complains.

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Balancing multiple connections and NAT

[Raj Mathur]

>>>>> "Sebastian" = Sebastian Bork <sebi@sebi.org> writes:

    Sebastian> On Fr, 2006-02-24 at 00:44 +0530, Raj Mathur wrote:
    >> I have a client connected to the 'net through 3 ISP's.  Have
    >> set up a Linux box to do routing and load sharing for the 3
    >> connections.  A fourth interface is connected to the LAN with
    >> private IP addresses.  Am using iptables to SNAT traffic to the
    >> appropriate IP depending on the interface the packet gets
    >> routed onto.

    Sebastian> I use exactly the same setup with a customer's
    Sebastian> conenction, the only difference: I use MASQUERADE
    Sebastian> instead of SNAT. I did not see anything like the
    Sebastian> problem you describe. Maybe because MAQUERADE works
    Sebastian> stateful, SNAT not? If you do not have a special reason
    Sebastian> for using SNAT, I think you should try MASQUERADE. If
    Sebastian> your problem persits, please tell me, as I have to look
    Sebastian> at my customer's setup very closely then, to catch this
    Sebastian> before anyone complains.

Well, both MASQUERADE and SNAT are stateful (MASQUERADE is just a
special case of SNAT as far as I remember); however it's worth a shot
if it's working for you.

It's pretty easy to trap the wrong source IP errors -- going back to
my example, just run:

  tcpdump -i intA -q -t -n ! host ipA
  tcpdump -i intB -q -t -n ! host ipB
  tcpdump -i intC -q -t -n ! host ipC

Any IP packets that get displayed will be those with wrong source IPs.
You may need to start some large FTP uploads or similar and watch for
a while -- the problem manifests itself for me when the client is
uploading 10+ MB files to his public FTP server.  Of course, it may be
present in other places also, but outgoing FTP comprises the bulk of
his traffic so it's most patent there.

Digressing a bit, from the responses I've got from this list, it seems
that a kernel patch is required to make the whole load sharing +
iptables NAT work properly.  I'm a bit disappointed that this isn't
part of the mainstream kernel -- any chances of it being rolled in
upstream?

Regards,

-- Raju
-- 
Raj Mathur                raju@kandalaya.org      http://kandalaya.org/
       GPG: 78D4 FC67 367F 40E2 0DD5  0FEF C968 D0EF CC68 D17F
                      It is the mind that moves
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Balancing multiple connections and NAT

[Sebastian Bork]

On Sa, 2006-02-25 at 00:23 +0530, Raj Mathur wrote:
> >>>>> "Sebastian" = Sebastian Bork <sebi@sebi.org> writes:
>     Sebastian> I use exactly the same setup with a customer's
>     Sebastian> conenction, the only difference: I use MASQUERADE
>     Sebastian> instead of SNAT. I did not see anything like the
>     Sebastian> problem you describe. Maybe because MAQUERADE works
>     Sebastian> stateful, SNAT not? If you do not have a special reason
>     Sebastian> for using SNAT, I think you should try MASQUERADE. If
>     Sebastian> your problem persits, please tell me, as I have to look
>     Sebastian> at my customer's setup very closely then, to catch this
>     Sebastian> before anyone complains.
> 
> Well, both MASQUERADE and SNAT are stateful (MASQUERADE is just a
> special case of SNAT as far as I remember); however it's worth a shot
> if it's working for you.
> 
> It's pretty easy to trap the wrong source IP errors -- going back to
> my example, just run:

Done. It happens here, too. But now it gets really strange: the data (I
tried scp) goes out on IF1 with IF2's source address. The ACK packets
come in on IF2. The connection works anyway ... *That's* what I'd call
really cool load-balancing.


_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Balancing multiple connections and NAT

[Sebastian Bork]

On Fr, 2006-02-24 at 22:22 +0100, Sebastian Bork wrote:
> Done. It happens here, too. But now it gets really strange: the data (I
> tried scp) goes out on IF1 with IF2's source address. The ACK packets
> come in on IF2. The connection works anyway ... *That's* what I'd call
> really cool load-balancing.

It obviously only worked with my setup because two of the three routes
end on the same router of the upstream provider. As soon as the other
provider is used for a route, everything breaks down. I'll try those
patches tomorrow.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Balancing multiple connections and NAT

[Andreas Hasenack]

Em Qui 23 Fev 2006 20:41, Markus Schulz escreveu:
> you need a patch for NAT processing with multiple gateways. this will
> then save the routing information for each connection inside NAT
> structures, so that each packet of an established connection will be
> get routed over the same gateway. you can find the patches here:
> http://www.ssi.bg/~ja/#routes
> please read the guides (nano howto or dgd-usage) carefully.

Any idea why these patches are not yet integrated into the upstream kernel?
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Balancing multiple connections and NAT

[Jody Shumaker]

On 3/5/06, Andreas Hasenack <ahasenack@terra.com.br> wrote:
> Em Qui 23 Fev 2006 20:41, Markus Schulz escreveu:
> > you need a patch for NAT processing with multiple gateways. this will
> > then save the routing information for each connection inside NAT
> > structures, so that each packet of an established connection will be
> > get routed over the same gateway. you can find the patches here:
> > http://www.ssi.bg/~ja/#routes
> > please read the guides (nano howto or dgd-usage) carefully.
>
> Any idea why these patches are not yet integrated into the upstream kernel?
>

Possibly disagreement on the features included, or any number of other
reasons.  Personally I had serious problems with the patches in that
they didn't fully function.  Multi-path routing was only choosing one
path, and if too many routes were requested a kernel panic would
occur.

- Jody
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc