Re: semi OT: default route

Re: semi OT: default route

[Lopsch]

I think this is really difficult to solve. Because you will have to bind
a connection to a certain interface and route. For example you start a
connection to a webserver over route A1. The answer will come back over
route A1 but what happens if you answer out again over the route A2 will
the server be able to answer or will it fail? Note the connection
started over A1 so the server will be awaiting packets from the IP of
the interface lying on route A1. So it will be necessary that a
process/connection keeps on one route. On our side where the router is
there is no problem conntrack will keep an eye on it but how will
servers react that get a SYN from an certain adress and then
confirmations on another IP?

RE: semi OT: default route

[Daniel Chemko]

> Quote from LARTC-Howto Chapter 4:
*snip*

Just on this topic, I haven't looked at this for a long time, but with
said configuration, how is the traffic balanced exactly?

Route A1 and Route A2 are main routes for the network interface.

I have a single connection, say A:1111 -> B:80. If I send it through the
firewall, will it always use route A1/A2, or will it use the line's
weight as a differentiator? I must assume that 1. The source IP of the
connection will always return on the same path A1/A2 but not balanced
over both. 2. Not all ISP's will route packets sourced from another ISP.
I'm pretty sure the bandwidth splitter in iproute2 is stateless, so it
won't split up SYN's between each line's source IP. 

So you could have a perfect 50:50 split of outgoing packets and
potentially end up with a horribly unbalanced 100:0 inbound rate if the
source IP ip of each connection isn't equalized based on the weight as
well. My previously stated nth example forces connection based
equalization which I think is a better balancing scheme than a single
direction flow control of a stateless round robbin balance. Man, that's
a long description.

Assuming you have many similar sessions, my stated solution works great.
The worst case is when you have connections that vary from bandwidth
usage. If you have 1 http connection sucking up 800MB, and 30 10kb http
sessions, the balance will place 800.018MB on one line while the other
line only utilizes 150kbits. What we need is a netfilter match rule that
works better than the round robbin 'nth' rule to select the best
outgoing route... (*dreaming*)

RE: semi OT: default route

[Daniel Chemko]

> routed might help here. Right?
> Can anyone with 2 ISP setup please check the load balance thing?
> I vaguely remember seeing something about multiple default routes
> in early chapters of TCP/IP Illustrated Vol. 1 (I am not at all sure).

It is all possible. Since I assume you're using more than a single ISP
and BGP is too costly, I'm assuming you just want to perform single
sided balancing. You can't control the inbound return routes, so all you
can manipulate is the outgoing traffic. If you want to do inbound load
balancing, there's always DNS round-robbin. You can use the ip route
'equalize' keyword, but I can't say if that parameter conntrack's tcp
sessions. For outgoing connections, I use multiple IP ROUTE tables to
define default gateways then use IP RULES to point to each interface,
then in iptables I use any matching that desire to redirect routing
decisions.

Eg.
I have 2 proxy servers on one of my firewalls, one goes out of ppp0 and
the other one goes out of ppp1.
iptables -t mangle -A OUTPUT -m mark ! --mark 0 -j ACCEPT
iptables -t mangle -A OUTPUT -m mark --mark 0 -m owner --uid-owner
squid1 -j MARK --set-mark 0x1
iptables -t mangle -A OUTPUT -m mark --mark 0 -m owner --uid-owner
squid2 -j MARK --set-mark 0x2
iptables -t mangle -A OUTPUT -j CONNMARK --save-mark

I could have both proxy servers use both lines equalized using TCP
round-robbin
iptables -t mangle -A OUTPUT -m mark ! --mark 0 -j ACCEPT
iptables -t mangle -A OUTPUT -m mark --mark 0 -m multiport --dports
80,443 -m nth --every 2 --packet 0 -j MARK --set-mark 0x1
iptables -t mangle -A OUTPUT -m mark --mark 0 -m multiport --dports
80,443 -m nth --every 2 --packet 1 -j MARK --set-mark 0x2
iptables -t mangle -A OUTPUT -j CONNMARK --save-mark

Most people fall apart when I start talking about iproute2 since its
quite different from iptables. I'll describe it from the beginning to
help elaborate. When the kernel wants to look up a route, it looks up
the rule table to find what routing table to use. Here's a default one:
#ip rule show
0:      from all lookup local 
32766:  from all lookup main 
32767:  from all lookup default

The local entry is destined for this machine, the main route table is
where 'normal' routing entries go into. If you use ip route add.. you
are putting them into the main routing table by default. The trick is
that you can add new rules to the rule table to change what routing
table you decide to use. For instance, I want two new routing tables for
my specialized dual WAN firewall. I would create the rules as such:

ip rule add fwmark 1 table 1
ip rule add fwmark 2 table 2

So, if iptables MARKed the packet as 1, then I'd use table 1. What is
table 1? Well right now its blank. We need to populate it with data.

# Clean out that table
ip route flush table 1
# Add every routing entry from the main table BUT the default route
ip route show table main | grep -Ev ^default | while read ROUTE ; do
         ip route add table 1 $ROUTE
      done;
# Add the default route for that network interface (_table_gateway), and
the source address to use when sending the packet out (_table_source).
ip route add table 1 default via ${_table_gateway} src ${_table_source};


Conclussion
I've used this routing behavior because its powerful and doesn't break
any expected behaviours in the system, unlike the ROUTE target built
into netfilter.

Re: semi OT: default route

[Lopsch]

Daniel Chemko schrieb:
>routed might help here. Right?
>Can anyone with 2 ISP setup please check the load balance thing?
>I vaguely remember seeing something about multiple default routes
>in early chapters of TCP/IP Illustrated Vol. 1 (I am not at all sure).

Quote from LARTC-Howto Chapter 4:

4.2.2. Load balancing

The second question is how to balance traffic going out over the two 
providers. This is actually not hard if you already have set up split 
access as above.

Instead of choosing one of the two providers as your default route, you 
now set up the default route to be a multipath route. In the default 
kernel this will balance routes over the two providers. It is done as 
follows (once more building on the example in the section on split-access):

ip route add default scope global nexthop via $P1 dev $IF1 weight 1 \
nexthop via $P2 dev $IF2 weight 1
	

This will balance the routes over both providers. The weight parameters 
can be tweaked to favor one provider over the other.

Note that balancing will not be perfect, as it is route based, and 
routes are cached. This means that routes to often-used sites will 
always be over the same provider.

Furthermore, if you really want to do this, you probably also want to 
look at Julian Anastasov's patches at http://www.ssi.bg/~ja/#routes , 
Julian's route patch page. They will make things nicer to work with.

Explaining the variables:
$Pi ist the IP of the Provider Gateway to which the interface $IFi is 
connected.

I think a combination of IPTables and IPRoute2 will get best effects 
like described before.

semi OT: default route

[Payal Rathod]

Hi,
I have a question which has haunted me for many months. If I have 2 ISP
connections with me, with default gw 1.2.3.4 and 4.5.6.7 and if I add 
both as default routes on my Linux gateway as,
route add -net default gw 1.2.3.4
route add -net default gw 4.5.6.7
and if I send a packet from a windows client to internet, which ISP will it
go through?

Thanks a lot for the answer in advance and waiting eagerly for any
replies.

With warm regards,
-Payal

Re: semi OT: default route

[Jason Opperisano]

On Wed, Oct 20, 2004 at 09:25:51AM -0400, Payal Rathod wrote:
> Hi,
> I have a question which has haunted me for many months. If I have 2 ISP
> connections with me, with default gw 1.2.3.4 and 4.5.6.7 and if I add 
> both as default routes on my Linux gateway as,
> route add -net default gw 1.2.3.4
> route add -net default gw 4.5.6.7
> and if I send a packet from a windows client to internet, which ISP will it
> go through?

this will sound like a stupid answer, but it will probably always use
the route that you added first.  the 'route' command will allow you to
added multiple default routes, but the ones you add after the first one
end up getting ignored.  the 'ip' command won't let you add a default
route once you have one (it uses teq and multipath for this):

	$ ip route list | grep default
	default via 10.2.1.1 dev eth0

	$ sudo ip route add default via 10.2.1.2
	RTNETLINK answers: File exists

if you want to load-balance outbound traffic over multiple ISP links,
you'll want to use something along the lines of the 'nth' patch from POM:

	http://netfilter.org/documentation/HOWTO/netfilter-extensions-HOWTO-3.html#ss3.9

if you want to split outbound traffic over multiple links by source IP
or protocol, etc--you can use the concepts from:

	http://lartc.org/howto/lartc.netfilter.html

-j

-- 
Jason Opperisano <opie@817west.com>

Re: semi OT: default route

[Nick Drage]

On Wed, Oct 20, 2004 at 12:17:25PM -0400, Jason Opperisano wrote:
> On Wed, Oct 20, 2004 at 09:25:51AM -0400, Payal Rathod wrote:
> > Hi,
> > I have a question which has haunted me for many months. If I have 2 ISP
> > connections with me, with default gw 1.2.3.4 and 4.5.6.7 and if I add 
> > both as default routes on my Linux gateway as,
> > route add -net default gw 1.2.3.4
> > route add -net default gw 4.5.6.7
> > and if I send a packet from a windows client to internet, which ISP will it
> > go through?

Looking at this from a slightly different, well, simpler, point of view
than Jason...

> this will sound like a stupid answer, but it will probably always use
> the route that you added first.

On the host I've just tried this on - admittedly just the once, it tried
the route I added *last*.  But this is about thirty seconds worth of
testing :)

> the 'route' command will allow you to added multiple default routes,
> but the ones you add after the first one end up getting ignored.  the
> 'ip' command won't let you add a default route once you have one (it
> uses teq and multipath for this):
> 
> 	$ ip route list | grep default
> 	default via 10.2.1.1 dev eth0
> 
> 	$ sudo ip route add default via 10.2.1.2
> 	RTNETLINK answers: File exists

However if you want to give the routes different metrics....

ip route add default via 1.2.3.4 metric 1
ip route add default via 4.5.6.7 metric 2

"ip" will accept that input.

That should mean if the host can't send the traffic via 1.2.3.4 it will
realise this ( I presume solely if it gets no arp-reply for 1.2.3.4) it
will try and send the traffic via 4.5.6.7 instead.

That seems to be the way it should work, however on a test box my host
is happily trying to arp for 1.2.3.4 continuosly.

Anyone help me finish off this answer ;)

-- 
We are the Willing, led by the Unknowing,
Are doing the Impossible, for the Ungrateful.
We have done so much, for so long, with so little,
We are now qualified to do anything with nothing.

Re: semi OT: default route

[Jason Opperisano]

On Wed, Oct 20, 2004 at 05:37:34PM +0100, Nick Drage wrote:
> However if you want to give the routes different metrics....
> 
> ip route add default via 1.2.3.4 metric 1
> ip route add default via 4.5.6.7 metric 2
> 
> "ip" will accept that input.
> 
> That should mean if the host can't send the traffic via 1.2.3.4 it will
> realise this ( I presume solely if it gets no arp-reply for 1.2.3.4) it
> will try and send the traffic via 4.5.6.7 instead.
> 
> That seems to be the way it should work, however on a test box my host
> is happily trying to arp for 1.2.3.4 continuosly.
> 
> Anyone help me finish off this answer ;)

yes--the linux routing code will do dead gateway detection and fall
back to a lower metric route in the event of failure, and also go back
to the higher metric route upon resurrection.  this can be useful for
an active-standby setup.

as for which route added by 'route' actually gets used--i've never spent
the time to figure out how it picks the one it uses, but it certainly
only appears to ever use one--maybe it picks the one with the lowest
numerical value...  or maybe it uses one of those magic 8-ball things...

-j

-- 
Jason Opperisano <opie@817west.com>

Re: semi OT: default route

[Payal Rathod]

On Wed, Oct 20, 2004 at 12:48:30PM -0400, Jason Opperisano wrote:
> yes--the linux routing code will do dead gateway detection and fall
> back to a lower metric route in the event of failure, and also go back
> to the higher metric route upon resurrection.  this can be useful for
> an active-standby setup.

routed might help here. Right?
Can anyone with 2 ISP setup please check the load balance thing?
I vaguely remember seeing something about multiple default routes
in early chapters of TCP/IP Illustrated Vol. 1 (I am not at all sure).
Thanks.
With warm regards,
-Payal

-- 

Re: semi OT: default route

[Jason Opperisano]

On Wed, Oct 20, 2004 at 12:57:37PM -0400, Payal Rathod wrote:
> On Wed, Oct 20, 2004 at 12:48:30PM -0400, Jason Opperisano wrote:
> > yes--the linux routing code will do dead gateway detection and fall
> > back to a lower metric route in the event of failure, and also go back
> > to the higher metric route upon resurrection.  this can be useful for
> > an active-standby setup.
> 
> routed might help here. Right?

no--routed is a method for learning routes.  you already know what the
routes are.  you're looking for a method to balance traffic over two
physical links.  the solution to that goal is more complicated than just
adding two default gateways.

-j

-- 
Jason Opperisano <opie@817west.com>