load-balancing router: trouble with breaking connections

load-balancing router: trouble with breaking connections

[Lloyd Standish]

Hi All,

I have a load balancing router to distribute traffic from an internal LAN over several small (5 Mbit) uplinks, using NAT.  The router works, but I had a problem which prevents me from doing the balancing the way I would like.  I have spent weeks trying to fix the problem I will describe below.

Here is an ASCII picture stolen from http://lartc.org/howto/lartc.rpdb.multiple-links.html.  This shows the general scheme of my simple network setup (there are 5 uplinks instead of 2).  I have added the interface addresses to refer to below.

                                                                                         ________
                                                                  +------------+        /
                                                                  |    gw1     |       |
                                                    +-------------+ Provider 1 +-------
                                __            192.168.1.7         |            |     /
                            ___/  \_         +------+-------+     +------------+    |
                          _/        \__      |     if1      |                      /
                         /             \     |              |                      |
| Local network -----+ Linux router |                      |     Internet
                         \_           __/    |              |                      |
                           \__     __/       |     if2      |                      \
                              \___/          +------+-------+     +------------+    |
                                              200.91.104.144      |   gw2      |     \
                                                    +-------------+ Provider 2 +-------
                                                                  |            |       |
                                                                  +------------+        \________


I have distinct routing tables for each interface, as described on http://lartc.org/howto/lartc.rpdb.multiple-links.html.

The simple round-robin load-balancing described on this page takes advantage of the _route_cache_ to choose a new connection based on routing for a previous connection.  This (partially, at least) solves the primary problem with connmark-based load-balancing, which is the tendency to break sessions.

Unfortunately I have a bad problem with round-robin balancing that I have not been able to overcome: connections traveling through interfaces having a private IP address (if1 above) are often broken.

Is there a known problem with this sort of load-balancing when there is a private IP on the interface?

I'm quite sure the problem is not in the NAT done by Provider 1, since when this same interface is used with my Linux router doing connmark-based load balancing, connections are not dropped.

http://lartc.org/howto/lartc.rpdb.multiple-links.html explains this simple routing scheme clearly, and I think I have followed it carefully.  Basically, for each interface I execute commands like these (IP numbers and interfaces are replaced by variables from my script, but it should be clear.  CONNMARK<n> are simply chains to put a fwmark on a packet.  This is used only for special cases on this router.):

      ip route flush table $table
      ip route add ${!network} dev ${interface} src ${!wan} table $table

      ip route add ${!lan_net} dev ${lan_if} table $table
      ip route add 127.0.0.0/8 dev lo table $table #ok

# also add route in main routing table for network (see below)
      ip route add ${!network} dev ${interface} src ${!wan}
      ip route add default via ${!gateway} dev ${interface} table $table

# the following rule is supposed to ensure packets are replied to over the interface they came from
# frankly I don't clearly understand this; please comment
      ip rule add from ${!wan} table $table priority $((${#ifaces[@]}*100))

      # masquerade outgoing connections on secondary interfaces
      iptables -t nat -A POSTROUTING -o ${interface} -j SNAT --to-source ${!wan}

# mark new incoming connection, *not* from LAN, so we route back out the right interface
      iptables -t mangle -A PREROUTING -i ${interface} -m state --state NEW -j CONNMARK$((cardnum+1))

This is repeated for each interface, and then round-robin load balancing is done over the interfaces by a command like this (this example if for 3 interfaces):
ip route add default scope global  nexthop via 192.168.1.1 dev eth1 weight 1 nexthop via 192.168.2.1 dev eth2 weight 1 nexthop via 200.91.104.144 dev ppp0 weight 1

Can you see anything wrong with this configuration?  The problem is very serious, since I am forced to use connmark to mark connections for load-balancing, and the session-breaking is very annoying.

Regards,
Lloyd

Re: load-balancing router: trouble with breaking connections

[Brian Austin - Standard Universal]

you need to restore connmarks coming in from the wan so the system can 
send them back out that way

this is the guts of my 4 way rig.

     `iptables -t mangle -X`;

     `iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark`;
     `iptables -t mangle -A PREROUTING -i eth19 -j MARK --set-mark 0x01`;
     `iptables -t mangle -A PREROUTING -i eth19 -j CONNMARK --save-mark`;
     `iptables -t mangle -A PREROUTING -i eth20 -j MARK --set-mark 0x02`;
     `iptables -t mangle -A PREROUTING -i eth20 -j CONNMARK --save-mark`;
     `iptables -t mangle -A PREROUTING -i tun0 -j MARK --set-mark 0x03`;
     `iptables -t mangle -A PREROUTING -i tun0 -j CONNMARK --save-mark`;
     `iptables -t mangle -A PREROUTING -i tun1 -j MARK --set-mark 0x04`;
     `iptables -t mangle -A PREROUTING -i tun1 -j CONNMARK --save-mark`;

     `iptables -t mangle -A INPUT -i eth19  -j MARK --set-mark 0x01`;
     `iptables -t mangle -A INPUT -i eth19  -j CONNMARK --save-mark`;
     `iptables -t mangle -A INPUT -i eth20  -j MARK --set-mark 0x02`;
     `iptables -t mangle -A INPUT -i eth20  -j CONNMARK --save-mark`;
     `iptables -t mangle -A INPUT -i tun0  -j MARK --set-mark 0x03`;
     `iptables -t mangle -A INPUT -i tun0  -j CONNMARK --save-mark`;
     `iptables -t mangle -A INPUT -i tun1  -j MARK --set-mark 0x04`;
     `iptables -t mangle -A INPUT -i tun1  -j CONNMARK --save-mark`;

     `iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark`;
     `iptables -t mangle -A FORWARD -j CONNMARK --restore-mark`;

     `iptables -t mangle -A OUTPUT -s $myeth19ip -j MARK --set-mark 0x01`;
     `iptables -t mangle -A OUTPUT -s $myeth19ip -j CONNMARK --save-mark`;
     `iptables -t mangle -A FORWARD -i eth19 -j MARK --set-mark 0x01`;
     `iptables -t mangle -A FORWARD -i eth19 -j CONNMARK --save-mark`;

     `iptables -t mangle -A OUTPUT -s $myeth20ip -j MARK --set-mark 0x02`;
     `iptables -t mangle -A OUTPUT -s $myeth20ip -j CONNMARK --save-mark`;
     `iptables -t mangle -A FORWARD -i eth20 -j MARK --set-mark 0x02`;
     `iptables -t mangle -A FORWARD -i eth20 -j CONNMARK --save-mark`;

     `iptables -t mangle -A OUTPUT -s 192.168.101.1 -j MARK --set-mark 
0x03`;
     `iptables -t mangle -A OUTPUT -s 192.168.101.1 -j CONNMARK 
--save-mark`;
     `iptables -t mangle -A FORWARD -i tun0 -j MARK --set-mark 0x03`;
     `iptables -t mangle -A FORWARD -i tun0 -j CONNMARK --save-mark`;

     `iptables -t mangle -A OUTPUT -s 192.168.102.1 -j MARK --set-mark 
0x04`;
     `iptables -t mangle -A OUTPUT -s 192.168.102.1 -j CONNMARK 
--save-mark`;
     `iptables -t mangle -A FORWARD -i tun1 -j MARK --set-mark 0x04`;
     `iptables -t mangle -A FORWARD -i tun1 -j CONNMARK --save-mark`;


     `iptables -t mangle -A POSTROUTING -o eth19  -j MARK --set-mark 0x1`;
     `iptables -t mangle -A POSTROUTING -o eth19  -j  CONNMARK --save-mark`;
     `iptables -t mangle -A POSTROUTING -o eth20  -j MARK --set-mark 0x2`;
     `iptables -t mangle -A POSTROUTING -o eth20  -j  CONNMARK --save-mark`;

     `iptables -t mangle -A POSTROUTING -o tun0  -j MARK --set-mark 0x3`;
     `iptables -t mangle -A POSTROUTING -o tun0  -j  CONNMARK --save-mark`;
     `iptables -t mangle -A POSTROUTING -o tun1  -j MARK --set-mark 0x4`;
     `iptables -t mangle -A POSTROUTING -o tun1  -j  CONNMARK --save-mark`;

cheers


On 19/02/2012 9:40 AM, Lloyd Standish wrote:
> Hi All,
>
> I have a load balancing router to distribute traffic from an internal 
> LAN over several small (5 Mbit) uplinks, using NAT.  The router works, 
> but I had a problem which prevents me from doing the balancing the way 
> I would like.  I have spent weeks trying to fix the problem I will 
> describe below.
>
> Here is an ASCII picture stolen from 
> http://lartc.org/howto/lartc.rpdb.multiple-links.html.  This shows the 
> general scheme of my simple network setup (there are 5 uplinks instead 
> of 2).  I have added the interface addresses to refer to below.
>
>                                                                                         
> ________
>                                                                  
> +------------+        /
>                                                                  |    
> gw1     |       |
>                                                    +-------------+ 
> Provider 1 +-------
>                                __            192.168.1.7         
> |            |     /
>                            ___/  \_         +------+-------+     
> +------------+    |
>                          _/        \__      |     if1      
> |                      /
>                         /             \     |              
> |                      |
> | Local network -----+ Linux router |                      |     Internet
>                         \_           __/    |              
> |                      |
>                           \__     __/       |     if2      
> |                      \
>                              \___/          +------+-------+     
> +------------+    |
>                                              200.91.104.144      |   
> gw2      |     \
>                                                    +-------------+ 
> Provider 2 +-------
>                                                                  
> |            |       |
>                                                                  
> +------------+        \________
>
>
> I have distinct routing tables for each interface, as described on 
> http://lartc.org/howto/lartc.rpdb.multiple-links.html.
>
> The simple round-robin load-balancing described on this page takes 
> advantage of the _route_cache_ to choose a new connection based on 
> routing for a previous connection.  This (partially, at least) solves 
> the primary problem with connmark-based load-balancing, which is the 
> tendency to break sessions.
>
> Unfortunately I have a bad problem with round-robin balancing that I 
> have not been able to overcome: connections traveling through 
> interfaces having a private IP address (if1 above) are often broken.
>
> Is there a known problem with this sort of load-balancing when there 
> is a private IP on the interface?
>
> I'm quite sure the problem is not in the NAT done by Provider 1, since 
> when this same interface is used with my Linux router doing 
> connmark-based load balancing, connections are not dropped.
>
> http://lartc.org/howto/lartc.rpdb.multiple-links.html explains this 
> simple routing scheme clearly, and I think I have followed it 
> carefully.  Basically, for each interface I execute commands like 
> these (IP numbers and interfaces are replaced by variables from my 
> script, but it should be clear.  CONNMARK<n> are simply chains to put 
> a fwmark on a packet.  This is used only for special cases on this 
> router.):
>
>      ip route flush table $table
>      ip route add ${!network} dev ${interface} src ${!wan} table $table
>
>      ip route add ${!lan_net} dev ${lan_if} table $table
>      ip route add 127.0.0.0/8 dev lo table $table #ok
>
> # also add route in main routing table for network (see below)
>      ip route add ${!network} dev ${interface} src ${!wan}
>      ip route add default via ${!gateway} dev ${interface} table $table
>
> # the following rule is supposed to ensure packets are replied to over 
> the interface they came from
> # frankly I don't clearly understand this; please comment
>      ip rule add from ${!wan} table $table priority 
> $((${#ifaces[@]}*100))
>
>      # masquerade outgoing connections on secondary interfaces
>      iptables -t nat -A POSTROUTING -o ${interface} -j SNAT 
> --to-source ${!wan}
>
> # mark new incoming connection, *not* from LAN, so we route back out 
> the right interface
>      iptables -t mangle -A PREROUTING -i ${interface} -m state --state 
> NEW -j CONNMARK$((cardnum+1))
>
> This is repeated for each interface, and then round-robin load 
> balancing is done over the interfaces by a command like this (this 
> example if for 3 interfaces):
> ip route add default scope global  nexthop via 192.168.1.1 dev eth1 
> weight 1 nexthop via 192.168.2.1 dev eth2 weight 1 nexthop via 
> 200.91.104.144 dev ppp0 weight 1
>
> Can you see anything wrong with this configuration?  The problem is 
> very serious, since I am forced to use connmark to mark connections 
> for load-balancing, and the session-breaking is very annoying.
>
> Regards,
> Lloyd
> -- 
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: load-balancing router: trouble with breaking connections

[Lloyd Standish]

On Sat, 18 Feb 2012 19:59:00 -0600, Brian Austin - Standard Universal <brian@standarduniversal.com.au> wrote:

> you need to restore connmarks coming in from the wan so the system can send them back out that way
>

Hello Brian,

Thanks for the reply.  The router I described does not use connmark.  It uses a command like this to set up round-robin balancing:
ip route add default scope global  nexthop via 192.168.1.1 dev eth1 weight 1 nexthop via 192.168.2.1 dev eth2 weight 1 nexthop via 200.91.104.144 dev ppp0 weight 1

This is described here:
http://lartc.org/howto/lartc.rpdb.multiple-links.html

The article teaches that this balancing depend on the following rule (one for each interface) to route traffic out the same interface as it was received on:
ip rule add from ${!wan} table $table priority $((${#ifaces[@]}*100))

(Of course, the priority value can be ignored.)

Since this system results in breaking connections, I am forced for the time being to use a connmarks for balancing, and restoration of marks, as you mentioned.

-- 
Lloyd

Re: load-balancing router: trouble with breaking connections

[Brian Austin - Standard Universal]

Hi Lloyd,

after months of bashing at this and I'm onto revision 3 of the rig, 
conntrack is the answer.

also conntrack allows you to connect to both wan IP eg ssh to one and 
smtp to the other.
without conntrack, the route cache will only allow connection to one wan 
port, so if your
ssh into one side, any connection to the other side will mysteriously 
fail, then vica versa.

cheers

On 19/02/2012 2:19 PM, Lloyd Standish wrote:
> On Sat, 18 Feb 2012 19:59:00 -0600, Brian Austin - Standard Universal 
> <brian@standarduniversal.com.au> wrote:
>
>> you need to restore connmarks coming in from the wan so the system 
>> can send them back out that way
>>
>
> Hello Brian,
>
> Thanks for the reply.  The router I described does not use connmark.  
> It uses a command like this to set up round-robin balancing:
> ip route add default scope global  nexthop via 192.168.1.1 dev eth1 
> weight 1 nexthop via 192.168.2.1 dev eth2 weight 1 nexthop via 
> 200.91.104.144 dev ppp0 weight 1
>
> This is described here:
> http://lartc.org/howto/lartc.rpdb.multiple-links.html
>
> The article teaches that this balancing depend on the following rule 
> (one for each interface) to route traffic out the same interface as it 
> was received on:
> ip rule add from ${!wan} table $table priority $((${#ifaces[@]}*100))
>
> (Of course, the priority value can be ignored.)
>
> Since this system results in breaking connections, I am forced for the 
> time being to use a connmarks for balancing, and restoration of marks, 
> as you mentioned.
>

Re: load-balancing router: trouble with breaking connections

[Lloyd Standish]

On Sat, 18 Feb 2012 16:40:24 -0600, Lloyd Standish <lloyd@crnatural.net> wrote:

> Is there a known problem with this sort of load-balancing when there is a private IP on the interface?
> I'm quite sure the problem is not in the NAT done by Provider 1, since when this same interface is used with my Linux router doing connmark-based load balancing, connections are not dropped.
> http://lartc.org/howto/lartc.rpdb.multiple-links.html explains this simple routing scheme clearly, and I think I have followed it carefully.  Basically, for each interface I execute commands like these (IP numbers and interfaces are replaced by variables from my script, but it should be clear.  CONNMARK<n> are simply chains to put a fwmark on a packet.  This is used only for special cases on this router.):

I finally found a solution to this issue, after weeks of frustration and unnecessary effort.  (This problem forced me to code a connmark-based load-balancing router, which produced unacceptable session-breaking.  I will comment on this below.)  I am posting this reply to my own post for the benefit of others who will certainly run into this problem.

I refer to http://lartc.org/howto/lartc.rpdb.multiple-links.html.  According to my experience, that scheme, which does not use connmark to mark packets, DOES NOT work properly when at least one of the uplinks carries a private IP number.  Under these circumstances, connmark-and-friends must be used to avoid having RELATED,ESTABLISHED packets sent out the wrong interface.

Specifically, to fix this problem the following must be done in addition to what is described at http://lartc.org/howto/lartc.rpdb.multiple-links.html:

PREROUTING
1. For RELATED,ESTABLISHED packets entering from the LAN interface, do "--restore-mark"
2. For all packets coming in from an outward-facing interface that has no mark, mark according to the interface

POSTROUTING
3. For NEW connections leaving on an outward-facing interface, set the mark on the packet according to the outbound interface.
4. For all packets leaving router on any interface, "--save-mark"

In addition, rules must be added to send packets out through interfaces that have the corresponding mark, for example:

ip rule add fwmark 1 table T1
ip rule add fwmark 2 table T2
etc.

In sum, the strategy is to allow the route cache and the round-robin interface selection of "nexthop via" to choose the outgoing interface, and connmark is used to keep the packets belonging to a connection on the same connection.

Note that if the route cache is ignored during the process of choosing an outbound connection, *sessions* will be constantly broken, resulting in a completely unacceptable Internet browsing experience for users.

-- 
Lloyd

Re: load-balancing router: trouble with breaking connections

[Brian Austin - Standard Universal]

Hi,
you need to restore marks to packets from the local machine too.. or its 
sessions will be messed up.
first line in mangle output should be

iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark;

I believe conntrack replaces the route cache function entirely for 
session persistence.

cheers



On 22/02/2012 2:07 PM, Lloyd Standish wrote:
> On Sat, 18 Feb 2012 16:40:24 -0600, Lloyd Standish 
> <lloyd@crnatural.net> wrote:
>
>> Is there a known problem with this sort of load-balancing when there 
>> is a private IP on the interface?
>> I'm quite sure the problem is not in the NAT done by Provider 1, 
>> since when this same interface is used with my Linux router doing 
>> connmark-based load balancing, connections are not dropped.
>> http://lartc.org/howto/lartc.rpdb.multiple-links.html explains this 
>> simple routing scheme clearly, and I think I have followed it 
>> carefully.  Basically, for each interface I execute commands like 
>> these (IP numbers and interfaces are replaced by variables from my 
>> script, but it should be clear.  CONNMARK<n> are simply chains to put 
>> a fwmark on a packet.  This is used only for special cases on this 
>> router.):
>
> I finally found a solution to this issue, after weeks of frustration 
> and unnecessary effort.  (This problem forced me to code a 
> connmark-based load-balancing router, which produced unacceptable 
> session-breaking.  I will comment on this below.)  I am posting this 
> reply to my own post for the benefit of others who will certainly run 
> into this problem.
>
> I refer to http://lartc.org/howto/lartc.rpdb.multiple-links.html.  
> According to my experience, that scheme, which does not use connmark 
> to mark packets, DOES NOT work properly when at least one of the 
> uplinks carries a private IP number.  Under these circumstances, 
> connmark-and-friends must be used to avoid having RELATED,ESTABLISHED 
> packets sent out the wrong interface.
>
> Specifically, to fix this problem the following must be done in 
> addition to what is described at 
> http://lartc.org/howto/lartc.rpdb.multiple-links.html:
>
> PREROUTING
> 1. For RELATED,ESTABLISHED packets entering from the LAN interface, do 
> "--restore-mark"
> 2. For all packets coming in from an outward-facing interface that has 
> no mark, mark according to the interface
>
> POSTROUTING
> 3. For NEW connections leaving on an outward-facing interface, set the 
> mark on the packet according to the outbound interface.
> 4. For all packets leaving router on any interface, "--save-mark"
>
> In addition, rules must be added to send packets out through 
> interfaces that have the corresponding mark, for example:
>
> ip rule add fwmark 1 table T1
> ip rule add fwmark 2 table T2
> etc.
>
> In sum, the strategy is to allow the route cache and the round-robin 
> interface selection of "nexthop via" to choose the outgoing interface, 
> and connmark is used to keep the packets belonging to a connection on 
> the same connection.
>
> Note that if the route cache is ignored during the process of choosing 
> an outbound connection, *sessions* will be constantly broken, 
> resulting in a completely unacceptable Internet browsing experience 
> for users.
>

Re: load-balancing router: trouble with breaking connections

[Lloyd Standish]

On Tue, 21 Feb 2012 21:46:40 -0600, Brian Austin - Standard Universal <brian@standarduniversal.com.au> wrote:

> Hi,
> you need to restore marks to packets from the local machine too.. or its
> sessions will be messed up.
> first line in mangle output should be
>
> iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark;
>
> I believe conntrack replaces the route cache function entirely for
> session persistence.
>
> cheers
>

Thanks for your comment.  I do --restore-mark for OUTPUT as well, although I didn't mention it in my post.

The main point of my post was to show how load-balancing can be done using the route cache to choose a route based on previous routing, and use conntrack to keep packets on the same interfaces.

It may be that there is confusion about my use of the word "session."  I am not referring to keeping all packets belonging to the same *connection* on the same interface, but rather to keeping a series of connections by a user to the same destination on the same interface.

In my experience the only practical way to achieve session persistence is to allow the route cache to choose the route (and therefore the outbound interface).  When I ran a load-balancing router that ignored the route cache, using the statistics module in "probability" mode to choose an outbound interface at random, marking packets with connmark, I got beautiful load-balancing, but sessions (not connections) were broken constantly.  That is, websites that expected a logged-in user to keep the same IP number gave endless trouble.  Interestingly, most banking sites don't have a problem with this (although PayPal does).

--
Lloyd

Re: load-balancing router: trouble with breaking connections

[Amos Jeffries]

On 22/02/2012 5:19 p.m., Lloyd Standish wrote:
> On Tue, 21 Feb 2012 21:46:40 -0600, Brian Austin - Standard Universal 
> <brian@standarduniversal.com.au> wrote:
>
>> Hi,
>> you need to restore marks to packets from the local machine too.. or its
>> sessions will be messed up.
>> first line in mangle output should be
>>
>> iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark;
>>
>> I believe conntrack replaces the route cache function entirely for
>> session persistence.
>>
>> cheers
>>
>
> Thanks for your comment.  I do --restore-mark for OUTPUT as well, 
> although I didn't mention it in my post.
>
> The main point of my post was to show how load-balancing can be done 
> using the route cache to choose a route based on previous routing, and 
> use conntrack to keep packets on the same interfaces.
>
> It may be that there is confusion about my use of the word "session."  
> I am not referring to keeping all packets belonging to the same 
> *connection* on the same interface, but rather to keeping a series of 
> connections by a user to the same destination on the same interface.
>
> In my experience the only practical way to achieve session persistence 
> is to allow the route cache to choose the route (and therefore the 
> outbound interface).  When I ran a load-balancing router that ignored 
> the route cache, using the statistics module in "probability" mode to 
> choose an outbound interface at random, marking packets with connmark, 
> I got beautiful load-balancing, but sessions (not connections) were 
> broken constantly.  That is, websites that expected a logged-in user 
> to keep the same IP number gave endless trouble.  Interestingly, most 
> banking sites don't have a problem with this (although PayPal does).

I think the LB setup was suffering more from NAT than from routing 
issues. It is perfectly reasonable to expect that load balancer to work. 
Just as it would be perfectly reasonable to expect a router with an 
intermittent primary uplink to work with the same output style.
Only NAT on the LBs outbound interface or at the ISP level would cause 
the broken behaviour you describe.

AYJ

Re: load-balancing router: trouble with breaking connections

[Lloyd Standish]

On Wed, 22 Feb 2012 01:22:02 -0600, Amos Jeffries <squid3@treenet.co.nz> wrote:

> I think the LB setup was suffering more from NAT than from routing issues. It is perfectly reasonable to expect that load balancer to work. Just as it would be perfectly reasonable to expect a router with an intermittent primary uplink to work with the same output style.
> Only NAT on the LBs outbound interface or at the ISP level would cause the broken behaviour you describe.
>AYJ

I would certainly like to understand WHY I had to use connmarks to keep the packets belonging to a connection on the right interface.  However, I don't believe the problem was NAT, because the only changes I had to make to get this load-balancing router to work (that is, to stop breaking connections) were the ones I mentioned in a previous post.  I did not add or change any NAT rules.  The router is doing NAT the way it was before, set up with a command like this for each interface:

iptables -t nat -A POSTROUTING -o ${interface} -j SNAT --to-source ${!wan}

Furthermore, on this router I was already using connmark to mark and route packets for those destinations and origin IP for which we did not want to have load-balancing.  This by the way worked fine (connections were not broken).  The only thing I added to fix the connection-breaking was marking of NEW packets after netfilter had made the routing decision (based on either the routing cache or round-robin distribution).

I would like to know whether or not anyone has succeeded in doing load-balancing with "nexthop via..." over interfaces with *private* IPs.

-- 
Lloyd

Re: load-balancing router: trouble with breaking connections

[Brian Austin - Standard Universal]

On 23/02/2012 1:53 AM, Lloyd Standish wrote:
> On Wed, 22 Feb 2012 01:22:02 -0600, Amos Jeffries 
> <squid3@treenet.co.nz> wrote:
>
>> I think the LB setup was suffering more from NAT than from routing 
>> issues. It is perfectly reasonable to expect that load balancer to 
>> work. Just as it would be perfectly reasonable to expect a router 
>> with an intermittent primary uplink to work with the same output style.
>> Only NAT on the LBs outbound interface or at the ISP level would 
>> cause the broken behaviour you describe.
>> AYJ
>
> I would certainly like to understand WHY I had to use connmarks to 
> keep the packets belonging to a connection on the right interface.  
> However, I don't believe the problem was NAT, because the only changes 
> I had to make to get this load-balancing router to work (that is, to 
> stop breaking connections) were the ones I mentioned in a previous 
> post.  I did not add or change any NAT rules.  The router is doing NAT 
> the way it was before, set up with a command like this for each 
> interface:
>
> iptables -t nat -A POSTROUTING -o ${interface} -j SNAT --to-source 
> ${!wan}
>
> Furthermore, on this router I was already using connmark to mark and 
> route packets for those destinations and origin IP for which we did 
> not want to have load-balancing.  This by the way worked fine 
> (connections were not broken).  The only thing I added to fix the 
> connection-breaking was marking of NEW packets after netfilter had 
> made the routing decision (based on either the routing cache or 
> round-robin distribution).
>
> I would like to know whether or not anyone has succeeded in doing 
> load-balancing with "nexthop via..." over interfaces with *private* IPs.
>
My set up has nat at the adsl modems, not at the linux box. So my router 
is in private ip space on all interfaces.

  I don't see how NAT could be an issue either, but I'm not a guru at 
this - just enough to get it going.
Without thorough conntrack, it was rubbish.