* [LARTC] Load balancing on multiple NICs with iproute
@ 2001-03-17 2:06 Scott Russell
2001-03-17 11:45 ` bert hubert
` (4 more replies)
0 siblings, 5 replies; 6+ messages in thread
From: Scott Russell @ 2001-03-17 2:06 UTC (permalink / raw)
To: lartc
All -
I have what I think is a very simple problem but the solution has eluded me.
I have three Token Ring interfaces in one box and I want to load balance
outgoing traffic across the NICs. The system is a RH 6.2 base with 2.2.18
kernel and current RH errata. Based on the Adv-Routing HOWTO I have the
proper stuff configured into the kernel already.
I'm using DNS to do cheap and easy round robin style connections to the box.
[scottrus@linux scottrus]$ host ftp3.linux.ibm.com
ftp3.linux.ibm.com has address 9.37.145.124
ftp3.linux.ibm.com has address 9.37.145.120
ftp3.linux.ibm.com has address 9.37.145.127
ip address show
1: lo: <LOOPBACK,UP> mtu 3924 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:60:94:3f:15:63 brd ff:ff:ff:ff:ff:ff
inet 10.200.1.33/24 brd 10.200.1.255 scope global eth0
3: tr0: <BROADCAST,MULTICAST,UP> mtu 2000 qdisc pfifo_fast qlen 100
link/tr 00:20:35:ff:09:66 brd ff:ff:ff:ff:ff:ff
inet 9.37.145.124/20 brd 9.37.159.255 scope global tr0
4: tr1: <BROADCAST,MULTICAST,UP> mtu 2000 qdisc pfifo_fast qlen 100
link/tr 00:60:94:b2:19:dd brd ff:ff:ff:ff:ff:ff
inet 9.37.145.120/20 brd 9.37.159.255 scope global tr1
5: tr2: <BROADCAST,MULTICAST,UP> mtu 2000 qdisc pfifo_fast qlen 100
link/tr 00:60:94:b2:19:6d brd ff:ff:ff:ff:ff:ff
inet 9.37.145.127/20 brd 9.37.159.255 scope global tr2
All trX interfaces are on the same subnet and use the same default gateway
of 9.37.144.1. The eth0 interface is a private network. Traffic from the
public Token Ring network should never cross onto the private eth0 network
or visversa.
I'm looking for one of two possible solutions:
1) Incoming requests are responded to on the same interface they came in on.
For example an ftp connection coming in on tr1 has all TX / RX packets stick
to tr1.
2) All output packets are load balanced across the tr0, tr1, tr2
interfaces regardless of which tr interface they came in on. (This is my
preferred solution.)
What's currently happening is requests come in on tr0, tr1, tr2 but outgoing
traffic is being sent back only via tr0, the default route.
[scottrus@linux scottrus]$ ip route show
10.200.1.33 dev eth0 scope link
9.37.145.120 dev tr1 scope link
9.37.145.127 dev tr2 scope link
9.37.145.124 dev tr0 scope link
10.200.1.0/24 dev eth0 proto kernel scope link src 10.200.1.33
9.37.144.0/20 dev tr0 proto kernel scope link src 9.37.145.124
9.37.144.0/20 dev tr1 proto kernel scope link src 9.37.145.120
9.37.144.0/20 dev tr2 proto kernel scope link src 9.37.145.127
10.200.0.0/16 via 10.200.1.1 dev eth0
127.0.0.0/8 dev lo scope link
default via 9.37.144.1 dev tr2
default via 9.37.144.1 dev tr1
default via 9.37.144.1 dev tr0
As you can see I've tried simply adding default routes for each token ring
interface and I think this is a step in the right direction but I'm missing
something.
*sigh* I thought I knew it all, how wrong I was. I have to say all this is
rather cool and beyond this initial problem I know I'm going to have play
with it more just for fun. The very thought of being able to twist these
packets into my own demented idea of how they should be handled is just to
tempting to ignore. Baby steps, must remember to take baby steps.
1) Understand what the heck I've been reading all day.
2) Get the routes on ftp3 working so users stop yelling at me :)
3) Read more, play with iproute2, watch mailing list, help out. :) :)
Any help would be great. Examples would be better of course. Thanks much!
--
Regards,
Scott Russell (scottrus@raleigh.ibm.com)
Linux Technology Center, System Admin, RHCE.
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [LARTC] Load balancing on multiple NICs with iproute
2001-03-17 2:06 [LARTC] Load balancing on multiple NICs with iproute Scott Russell
@ 2001-03-17 11:45 ` bert hubert
2001-03-18 22:05 ` Scott Russell
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: bert hubert @ 2001-03-17 11:45 UTC (permalink / raw)
To: lartc
On Fri, Mar 16, 2001 at 09:06:18PM -0500, Scott Russell wrote:
> I'm using DNS to do cheap and easy round robin style connections to the box.
>
> [scottrus@linux scottrus]$ host ftp3.linux.ibm.com
> ftp3.linux.ibm.com has address 9.37.145.124
> ftp3.linux.ibm.com has address 9.37.145.120
> ftp3.linux.ibm.com has address 9.37.145.127
DNS is wildly underestimated as a loadbalancing/distribution device. It
works very well, especially if you set your TTL pretty low.
> 1) Incoming requests are responded to on the same interface they came in on.
> For example an ftp connection coming in on tr1 has all TX / RX packets stick
> to tr1.
This is possible, I think.
> As you can see I've tried simply adding default routes for each token ring
> interface and I think this is a step in the right direction but I'm missing
> something.
Add source routing, as described in
http://ds9a.nl/2.4Routing/HOWTO//cvs/2.4routing/output/2.4routing-4.html#ss4.1
Route packets with a source of 9.37.145.124 to trX, .120 to trY and 127 to
trZ.
> Any help would be great. Examples would be better of course. Thanks much!
# ip rule add from 9.37.145.124 table 124
# ip rule add from 9.37.145.120 table 120
# ip rule add from 9.37.145.127 table 127
# ip route add default via 9.37.145.1 dev tr0 table 124
# ip route add default via 9.37.145.1 dev tr1 table 120
# ip route add default via 9.37.145.1 dev tr2 table 127
# ip route flush cache
Something like that should work. I messed up the ip addresses I think, but
hey :-)
Regards,
bert
--
http://www.PowerDNS.com Versatile DNS Services
Trilab The Technology People
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [LARTC] Load balancing on multiple NICs with iproute
2001-03-17 2:06 [LARTC] Load balancing on multiple NICs with iproute Scott Russell
2001-03-17 11:45 ` bert hubert
@ 2001-03-18 22:05 ` Scott Russell
2001-03-19 12:40 ` Arthur van Leeuwen
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Scott Russell @ 2001-03-18 22:05 UTC (permalink / raw)
To: lartc
Thanks for the quick reply. Based on reading the HOWTO examples I thought
this was what I wanted to do and did some quick tests similar to what you
outlined below. My tests failed so I assumed I was missing the obvious and
posted to the list.
Well, I followed the examples below and ran the following commands in order:
ip rule add from 9.37.145.124 table 124
ip rule add from 9.37.145.120 table 120
ip rule add from 9.37.145.127 table 127
ip route add default via 9.37.144.1 dev tr0 table 124
ip route add default via 9.37.144.1 dev tr1 table 120
ip route add default via 9.37.144.1 dev tr2 table 127
ip route flush cache
A bit of back ground:
tr0 = 9.37.145.124/20
tr1 = 9.37.145.120/20
tr2 = 9.37.145.127/20
default GW = 9.37.144.1
So based on the above rules I would expect connections that originate on tr2
to stay on tr2. To test this I ftped to the system using the IP of tr2 and
started receiving a file. I then shut down interface tr0 with /sbin/ifdown
tr0. When I did so my file xfer stopped as well. When tr0 was brought back
up the xfer resumed. Bringing down tr2 and tr1 had no effect on the transfer
at all. (Expected since most likely the ftpd opened the ftp-data connection
on tr0 instead of tr2 as I wanted.)
Here's what my rules look like after running the above commands:
[scottrus@linux scottrus]$ ip rule show
0: from all lookup local
32763: from 9.37.145.127 lookup 127
32764: from 9.37.145.120 lookup 120
32765: from 9.37.145.124 lookup 124
32766: from all lookup main
32767: from all lookup 253
Since the rules 23763-65 all appear before the main table I would expect
them to take precedence over anything in the main table. Is this correct?
So here's what my main routing table looks like after running the above rules:
[scottrus@linux scottrus]$ ip route
10.200.1.33 dev eth0 scope link
9.37.145.120 dev tr1 scope link
9.37.145.127 dev tr2 scope link
9.37.145.124 dev tr0 scope link
10.200.1.0/24 dev eth0 proto kernel scope link src 10.200.1.33
9.37.144.0/20 dev tr0 proto kernel scope link src 9.37.145.124
9.37.144.0/20 dev tr1 proto kernel scope link src 9.37.145.120
9.37.144.0/20 dev tr2 proto kernel scope link src 9.37.145.127
10.200.0.0/16 via 10.200.1.1 dev eth0
127.0.0.0/8 dev lo scope link
default via 9.37.144.1 dev tr2
default via 9.37.144.1 dev tr1
default via 9.37.144.1 dev tr0
I expect that the rules I inserted are being ignored (and hence the tables
the rules point to are ignored as well). This means that the primary routing
table is being used and seems to explain why things are going out tr0 by
default.
Am I off base on what I'm seeing going on? I think I also have ECMP (Equal
Cost MultiPath) compiled into my kernel, is that affecting things as well?
Again, thanks for the help.
-- Scott
On Sat, Mar 17, 2001 at 12:45:43PM +0100, bert hubert wrote:
> On Fri, Mar 16, 2001 at 09:06:18PM -0500, Scott Russell wrote:
>
> > I'm using DNS to do cheap and easy round robin style connections to the box.
> >
> > [scottrus@linux scottrus]$ host ftp3.linux.ibm.com
> > ftp3.linux.ibm.com has address 9.37.145.124
> > ftp3.linux.ibm.com has address 9.37.145.120
> > ftp3.linux.ibm.com has address 9.37.145.127
>
> DNS is wildly underestimated as a loadbalancing/distribution device. It
> works very well, especially if you set your TTL pretty low.
>
> > 1) Incoming requests are responded to on the same interface they came in on.
> > For example an ftp connection coming in on tr1 has all TX / RX packets stick
> > to tr1.
>
> This is possible, I think.
>
> > As you can see I've tried simply adding default routes for each token ring
> > interface and I think this is a step in the right direction but I'm missing
> > something.
>
> Add source routing, as described in
> http://ds9a.nl/2.4Routing/HOWTO//cvs/2.4routing/output/2.4routing-4.html#ss4.1
>
> Route packets with a source of 9.37.145.124 to trX, .120 to trY and 127 to
> trZ.
>
> > Any help would be great. Examples would be better of course. Thanks much!
>
> # ip rule add from 9.37.145.124 table 124
> # ip rule add from 9.37.145.120 table 120
> # ip rule add from 9.37.145.127 table 127
>
> # ip route add default via 9.37.145.1 dev tr0 table 124
> # ip route add default via 9.37.145.1 dev tr1 table 120
> # ip route add default via 9.37.145.1 dev tr2 table 127
> # ip route flush cache
>
> Something like that should work. I messed up the ip addresses I think, but
> hey :-)
>
> Regards,
>
> bert
>
> --
> http://www.PowerDNS.com Versatile DNS Services
> Trilab The Technology People
> 'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet
>
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/
--
Regards,
Scott Russell (scottrus@raleigh.ibm.com)
Linux Technology Center, System Admin, RHCE.
T/L 441-9289 / External 919-543-9289
http://bzimage.raleigh.ibm.com/webcam
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [LARTC] Load balancing on multiple NICs with iproute
2001-03-17 2:06 [LARTC] Load balancing on multiple NICs with iproute Scott Russell
2001-03-17 11:45 ` bert hubert
2001-03-18 22:05 ` Scott Russell
@ 2001-03-19 12:40 ` Arthur van Leeuwen
2001-03-20 4:43 ` Scott Russell
2001-03-20 9:24 ` Arthur van Leeuwen
4 siblings, 0 replies; 6+ messages in thread
From: Arthur van Leeuwen @ 2001-03-19 12:40 UTC (permalink / raw)
To: lartc
On Sun, 18 Mar 2001, Scott Russell wrote:
> default via 9.37.144.1 dev tr2
> default via 9.37.144.1 dev tr1
> default via 9.37.144.1 dev tr0
Interesting. This doesn't seem to be a multipath route. Did you add the
multipath route with the nexthop commands?
Doei, Arthur.
--
/\ / | arthurvl@sci.kun.nl | Work like you don't need the money
/__\ / | A friend is someone with whom | Love like you have never been hurt
/ \/__ | you can dare to be yourself | Dance like there's nobody watching
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [LARTC] Load balancing on multiple NICs with iproute
2001-03-17 2:06 [LARTC] Load balancing on multiple NICs with iproute Scott Russell
` (2 preceding siblings ...)
2001-03-19 12:40 ` Arthur van Leeuwen
@ 2001-03-20 4:43 ` Scott Russell
2001-03-20 9:24 ` Arthur van Leeuwen
4 siblings, 0 replies; 6+ messages in thread
From: Scott Russell @ 2001-03-20 4:43 UTC (permalink / raw)
To: lartc
Umm, no they're not :) These statements are inserted as straight default
routes into the main routing table. What would a command for multipath route
look like? If it is what I think it is last time I tried to add one I
received an error in the process.
If I add a multipath route I assume i should add it to the main table and
that I should remove the three default routes shown in your last email?
-- Scott
On Mon, Mar 19, 2001 at 01:40:52PM +0100, Arthur van Leeuwen wrote:
> On Sun, 18 Mar 2001, Scott Russell wrote:
>
> > default via 9.37.144.1 dev tr2
> > default via 9.37.144.1 dev tr1
> > default via 9.37.144.1 dev tr0
>
> Interesting. This doesn't seem to be a multipath route. Did you add the
> multipath route with the nexthop commands?
>
> Doei, Arthur.
>
> --
> /\ / | arthurvl@sci.kun.nl | Work like you don't need the money
> /__\ / | A friend is someone with whom | Love like you have never been hurt
> / \/__ | you can dare to be yourself | Dance like there's nobody watching
--
Regards,
Scott Russell (scottrus@raleigh.ibm.com)
Linux Technology Center, System Admin, RHCE.
T/L 441-9289 / External 919-543-9289
http://bzimage.raleigh.ibm.com/webcam
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [LARTC] Load balancing on multiple NICs with iproute
2001-03-17 2:06 [LARTC] Load balancing on multiple NICs with iproute Scott Russell
` (3 preceding siblings ...)
2001-03-20 4:43 ` Scott Russell
@ 2001-03-20 9:24 ` Arthur van Leeuwen
4 siblings, 0 replies; 6+ messages in thread
From: Arthur van Leeuwen @ 2001-03-20 9:24 UTC (permalink / raw)
To: lartc
On Mon, 19 Mar 2001, Scott Russell wrote:
> Umm, no they're not :) These statements are inserted as straight default
> routes into the main routing table. What would a command for multipath route
> look like? If it is what I think it is last time I tried to add one I
> received an error in the process.
ip route add default scope global nexthop via 9.37.144.1 dev tr2 \
nexthop via 9.37.144.1 dev tr1 \
nexthop via 9.37.144.1 dev tr0
> If I add a multipath route I assume i should add it to the main table and
> that I should remove the three default routes shown in your last email?
Yes.
Doei, Arthur.
--
/\ / | arthurvl@sci.kun.nl | Work like you don't need the money
/__\ / | A friend is someone with whom | Love like you have never been hurt
/ \/__ | you can dare to be yourself | Dance like there's nobody watching
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2001-03-20 9:24 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2001-03-17 2:06 [LARTC] Load balancing on multiple NICs with iproute Scott Russell
2001-03-17 11:45 ` bert hubert
2001-03-18 22:05 ` Scott Russell
2001-03-19 12:40 ` Arthur van Leeuwen
2001-03-20 4:43 ` Scott Russell
2001-03-20 9:24 ` Arthur van Leeuwen
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.