trouble setting default route for load-balancing routing tables

trouble setting default route for load-balancing routing tables

[Lloyd Standish]

Hi Everyone,

I'm configuring a load-balancing netfilter router.  It does round-robin load balancing over 2 external interfaces, eth0 and eth1.  The natted LAN is on eth4.  eth0 is attached to table T0, and eth1 uses table T1.

I am unable to add default routes to the routing tables T0 and T1.  Here is the output from my bash script that sets up the router:

+ ip route flush table T0
+ ip route add 192.168.100.0/24 dev eth0 src 192.168.100.10 table T0
+ ip route add default via 192.168.100.11 table T0
RTNETLINK answers: No such process

Even with the lack of default route for T0 and T1, the router appears to properly load-balance over the 2 outgoing interfaces (!)  However, it ignores my rules which should direct certain traffic over a particular interface.

After setting up the router, here is table main:

root@debiandesk2:/home/lloyd/data/traffic_shaping# ip route show table main
192.168.100.0/24 dev eth0  scope link  src 192.168.100.10
192.168.50.0/24 dev eth4  scope link  src 192.168.50.1
192.168.90.0/24 dev eth1  scope link  src 192.168.90.10
default
	nexthop via 192.168.100.11  dev eth0 weight 92
	nexthop via 192.168.90.1  dev eth1 weight 7


And here is table T0, devoid of a default route:

root@debiandesk2:/home/lloyd/data/traffic_shaping# ip route show table T0
192.168.100.0/24 dev eth0  scope link  src 192.168.100.10
192.168.50.0/24 dev eth4  scope link
127.0.0.0/8 dev lo  scope link

Let me know if any more info is needed to debug this.

Regards,
Lloyd

Re: trouble setting default route for load-balancing routing tables

[Andrew Beverley]

On Sat, 2011-11-19 at 20:11 -0600, Lloyd Standish wrote:
> Hi Everyone,
> 
> I'm configuring a load-balancing netfilter router.  It does round-robin load balancing over 2 external interfaces, eth0 and eth1.  The natted LAN is on eth4.  eth0 is attached to table T0, and eth1 uses table T1.
> 
> I am unable to add default routes to the routing tables T0 and T1.  Here is the output from my bash script that sets up the router:
> 
> + ip route flush table T0
> + ip route add 192.168.100.0/24 dev eth0 src 192.168.100.10 table T0
> + ip route add default via 192.168.100.11 table T0
> RTNETLINK answers: No such process

I see nothing wrong with your rules. That error implies that the kernel
doesn't know what 192.168.100.11 is, but it's clearly accepted it for
the default route in the main routing table. Has anything changed
between you writing the rules above and writing the rules for the main
table?

Is the IP address for eth0 correctly configured at this point? An
incorrect netmask will cause the error above.

> 
> Even with the lack of default route for T0 and T1, the router appears
>  to properly load-balance over the 2 outgoing interfaces (!)

I would expect that, as the nexthop rules in the main table are doing
the load balancing.

>   However,
>  it ignores my rules which should direct certain traffic over a
>  particular interface.

I assume that you have no need to keep the same connection stream on the
same interface? If you do, you can use netfilter marks (and netfilter
connection tracking) to do the load balancing instead, as described on
this page:

http://www.sysresccd.org/Sysresccd-Networking-EN-Iptables-and-netfilter-load-balancing-using-connmark

> 
> After setting up the router, here is table main:
> 
> root@debiandesk2:/home/lloyd/data/traffic_shaping# ip route show table main
> 192.168.100.0/24 dev eth0  scope link  src 192.168.100.10
> 192.168.50.0/24 dev eth4  scope link  src 192.168.50.1
> 192.168.90.0/24 dev eth1  scope link  src 192.168.90.10
> default
> 	nexthop via 192.168.100.11  dev eth0 weight 92
> 	nexthop via 192.168.90.1  dev eth1 weight 7
> 
> 
> And here is table T0, devoid of a default route:
> 
> root@debiandesk2:/home/lloyd/data/traffic_shaping# ip route show table T0
> 192.168.100.0/24 dev eth0  scope link  src 192.168.100.10
> 192.168.50.0/24 dev eth4  scope link
> 127.0.0.0/8 dev lo  scope link
> 
> Let me know if any more info is needed to debug this.
> 
> Regards,
> Lloyd
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: trouble setting default route for load-balancing routing tables

[Lloyd Standish]

On Sun, 20 Nov 2011 08:16:43 -0600, Andrew Beverley <andy@andybev.com> wrote:

> On Sat, 2011-11-19 at 20:11 -0600, Lloyd Standish wrote:
>> Hi Everyone,
>>
>> I'm configuring a load-balancing netfilter router.  It does round-robin load balancing over 2 external interfaces, eth0 and eth1.  The natted LAN is on eth4.  eth0 is attached to table T0, and eth1 uses table T1.
>>
>> I am unable to add default routes to the routing tables T0 and T1.  Here is the output from my bash script that sets up the router:
>>
>> + ip route flush table T0
>> + ip route add 192.168.100.0/24 dev eth0 src 192.168.100.10 table T0
>> + ip route add default via 192.168.100.11 table T0
>> RTNETLINK answers: No such process
>
> I see nothing wrong with your rules. That error implies that the kernel
> doesn't know what 192.168.100.11 is, but it's clearly accepted it for
> the default route in the main routing table. Has anything changed
> between you writing the rules above and writing the rules for the main
> table?

These are the rules taken straight from my bash script.  I have spent hour after hour searching the Internet for clues on this, and finally gave up at 1:00 AM this morning.

>
> Is the IP address for eth0 correctly configured at this point? An
> incorrect netmask will cause the error above.

Moments after reading your question about eth0 being configured at the point the default gateway for the table is created, I checked that, and saw eth0 was NOT yet configured!  I just changed the order of the commands, and it worked!:

+ ip route flush table T1
+ ip route add 192.168.90.0/24 dev eth1 src 192.168.90.10 table T1
+ ip rule add from 192.168.90.10 table T1
+ ip route add 192.168.50.0/24 dev eth4 table T1
+ ip route add 127.0.0.0/8 dev lo table T1
+ ip route add 192.168.90.0/24 dev eth1 src 192.168.90.10
+ ip route add default via 192.168.90.1 table T1

Thanks a million!  Now I'll check to see if these rules (added later in the script) are working:

+ ip rule add to 200.122.146.32/27 table T1
+ ip rule add to 200.91.79.0/27 table T1

These are supposed to route all traffic to IPs in those subnets through table T1, overriding load-balancing for those destinations.

<snip>

>>   However,
>>  it ignores my rules which should direct certain traffic over a
>>  particular interface.
>
> I assume that you have no need to keep the same connection stream on the
> same interface? If you do, you can use netfilter marks (and netfilter
> connection tracking) to do the load balancing instead, as described on
> this page:

I may have to do that, but from what I've read, rule like the ones I posted above are supposed to be evaluated before the default "nexthopping" route in table main, because of the rules numbering, and that's the way my rules appear:

root@debiandesk2:/home/lloyd/data/traffic_shaping# ip rule show
0:	from all lookup local
32762:	from all to 200.91.79.0/27 lookup T1
32763:	from all to 200.122.146.32/27 lookup T1
32764:	from 192.168.90.10 lookup T1
32765:	from 192.168.100.10 lookup T0
32766:	from all lookup main
32767:	from all lookup default

My script has to reconfigure the firewall when an uplink fails and again when it recovers connectivity.  I used a program called lsm (link status monitor, http://lsm.foobar.fi/) for this.  I think this will be useful to others.  When I get it all working I'll post it on my website and post here.

Re: trouble setting default route for load-balancing routing tables

[Lloyd Standish]

<snip>
> Thanks a million!  Now I'll check to see if these rules (added later in the script) are working:
>
> + ip rule add to 200.122.146.32/27 table T1
> + ip rule add to 200.91.79.0/27 table T1
>
> These are supposed to route all traffic to IPs in those subnets through table T1, overriding load-balancing for those destinations.
>
> <snip>
>
>>>   However,
>>>  it ignores my rules which should direct certain traffic over a
>>>  particular interface.

Now that the default routes are set in tables T1 and T0, the above rules to "override" the load-balancing appear to work perfectly.  The router will ONLY send packets to T1 (eth1) for those subnets, even though that interface is "weighted" to receive a lot less traffic:

+ ip route add default scope global nexthop via 192.168.100.11 dev eth0 weight 92 nexthop via 192.168.90.1 dev eth1 weight 7

Now for tests with multiple connections... :)

Re: trouble setting default route for load-balancing routing tables

[Andrew Beverley]

On Sun, 2011-11-20 at 09:10 -0600, Lloyd Standish wrote:
> Now that the default routes are set in tables T1 and T0, the above
>  rules to "override" the load-balancing appear to work perfectly.  The
>  router will ONLY send packets to T1 (eth1) for those subnets,

Ah, got you. So you're splitting the network by subnets and routing over
the link depending on the subnet. Therefore packets from one client PC
will always go over the same link?

Andy

Re: trouble setting default route for load-balancing routing tables

[Lloyd Standish]

On Sun, 20 Nov 2011 14:34:06 -0600, Andrew Beverley <andy@andybev.com> wrote:

> On Sun, 2011-11-20 at 09:10 -0600, Lloyd Standish wrote:
>> Now that the default routes are set in tables T1 and T0, the above
>>  rules to "override" the load-balancing appear to work perfectly.  The
>>  router will ONLY send packets to T1 (eth1) for those subnets,
>
> Ah, got you. So you're splitting the network by subnets and routing over
> the link depending on the subnet. Therefore packets from one client PC
> will always go over the same link?
>

No, the subnets define destination IPs, not LAN IPs.  The subnets for a Costa Rican bank that chokes if traffic from a user during a "session" changes source IP.  The rules force all connections to those subnets to go out through the same IP, so the bank won't see an change in source IP due to my router's load-balancing.

(A change in IP to a given destination over a short period would not be likely to happen even *without* this rule, since routes are cached, and cached routes are not subject to load-balancing.)

--
Lloyd

Re: trouble setting default route for load-balancing routing tables

[Andrew Beverley]

> > I assume that you have no need to keep the same connection stream on the
> > same interface? If you do, you can use netfilter marks (and netfilter
> > connection tracking) to do the load balancing instead, as described on
> > this page:
> 
> I may have to do that, but from what I've read, rule like the ones I
>  posted above are supposed to be evaluated before the default
>  "nexthopping" route in table main, because of the rules numbering, and
>  that's the way my rules appear:

Correct, they will be evaluated first.

What I meant was that if you have a stream of packets all to do with one
connection (such as an FTP download) and you want all the packets for
that connection to go over the same link, then you will need to use
netfilter connection tracking for your iproute rules.

If your internet links are independent then you will need to do this in
order to not break connection streams.

> My script has to reconfigure the firewall when an uplink fails and
>  again when it recovers connectivity.  I used a program called lsm
>  (link status monitor, http://lsm.foobar.fi/) for this.

Thanks for that. I wish I'd known that earlier, as I have just written
my own basic version!

>   I think this
>  will be useful to others.  When I get it all working I'll post it on
>  my website and post here.

I've been working on a project for something similar. I'll also post it
to this list shortly for the benefit of others.

Andy