[LARTC] many ways to do load balancing (or not?)

[LARTC] many ways to do load balancing (or not?)

[Andreas Hasenack]

I'm a little confused about the many ways I've read that can be used
for traffic load balancing, that is, two or more interfaces to the
outside world being used transparently and efficiently by the internal
machines.

I heard about:
a) netfilter SNAT to more than one IP. If I'm correct, this is only
a round robin, that is, one connection goes here, the other goes there,
then the next goes here again, etc without much thought.

b) multipath default route. Seems to do something similar, but it "caches"
routes. What exactly does this mean in the long run?

c) multipath default route, but with the equalize option. Again, seems to
work, but the best description I could find about it was something along
the lines of "packet randomization", whatever that means. What does it do
in the long run? Is it better/worse than b)?

d) OSPF. I read in the RFC that OSPF can do "load balancing", but I failed
to understand how (no, I didn't read that RFC thoroughly, it's really high
tech for me at this point). Does it use multipath routes to accomplish this?

Any help is appreciated, thanks.

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] many ways to do load balancing (or not?)

[Ashok N N]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

hi,
   since my masters thesis is also concerned with this, i thought of taking 
this opportunity to clarify/get clarified some of my own thougths. please see 
inline.

On Thursday 21 November 2002 07:19 am, Andreas Hasenack wrote:
> I'm a little confused about the many ways I've read that can be used
> for traffic load balancing, that is, two or more interfaces to the
> outside world being used transparently and efficiently by the internal
> machines.
> 
> I heard about:
> a) netfilter SNAT to more than one IP. If I'm correct, this is only
> a round robin, that is, one connection goes here, the other goes there,
> then the next goes here again, etc without much thought.
> 

i believe that SNAT is for connections initiated from outside world towards 
the internal network. and this would be reverse of what is intended here, 
that of using the links to the outside world efficiently. moreover if setup 
properly, the packets will go out back on the interface they came in.

> b) multipath default route. Seems to do something similar, but it "caches"
> routes. What exactly does this mean in the long run?
> 

i'm still looking at the code here. AFAIK, the when you have multipath option 
set, then when looking up for a route to an address, among the multiple equal 
cost paths, one is selected according to some criteria (i read random 
somewhere in the kernel but am not able to locate it now). but once the route 
is found, it is cached in the route cache and all packets will follow that 
route. so it is kind of a route-based load-balancing.

> c) multipath default route, but with the equalize option. Again, seems to
> work, but the best description I could find about it was something along
> the lines of "packet randomization", whatever that means. What does it do
> in the long run? Is it better/worse than b)?
> 

i read in one of the threads that 'equalize' causes per-packet load balancing, 
i.e looks up the route for each packet. but i am doubtful about the 
performance when each packet causes a route-lookup in the FIB.
(http://www.uwsg.iu.edu/hypermail/linux/net/0107.3/0028.html)

> d) OSPF. I read in the RFC that OSPF can do "load balancing", but I failed
> to understand how (no, I didn't read that RFC thoroughly, it's really high
> tech for me at this point). Does it use multipath routes to accomplish this?
> 
> Any help is appreciated, thanks.

ashok
- -----------------------------------------------------------------------------
My public key:
   gpg --recv-keys --keyserver blackhole.pca.dfn.de DCB44F2E
   
http://pgp.mit.edu:11371/pks/lookup?search=N+N+Ashok+%28CSE+MSU%29+%3Cnnashok%40yahoo.com%3E&op=index
- -----------------------------------------------------------------------------
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE93RwTRhXpVty0Ty4RArc/AKCM7VWIK1CygSawQxDtPvcia/nclACeMqMI
w5aTSJ7L1utzv0ePbUWsGxk=oALf
-----END PGP SIGNATURE-----

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] many ways to do load balancing (or not?)

[Andreas Hasenack]

Em Thu, Nov 21, 2002 at 12:46:56PM -0500, Ashok N N escreveu:
> i believe that SNAT is for connections initiated from outside world towards 
> the internal network. and this would be reverse of what is intended here, 

That's DNAT.

> i'm still looking at the code here. AFAIK, the when you have multipath option 
> set, then when looking up for a route to an address, among the multiple equal 
> cost paths, one is selected according to some criteria (i read random 
> somewhere in the kernel but am not able to locate it now). but once the route 

This criteria I would like to understand.

> route. so it is kind of a route-based load-balancing.

Considering that the route cache is 60s by default, this should work I guess.
I'm setting up such a setup for testing.

> i read in one of the threads that 'equalize' causes per-packet load balancing, 
> i.e looks up the route for each packet. but i am doubtful about the 
> performance when each packet causes a route-lookup in the FIB.
> (http://www.uwsg.iu.edu/hypermail/linux/net/0107.3/0028.html)

Interesting, so 'equalize' bypasses the cache, that's about it?

So we only need to know the criteria with which the route is choosed in a multipath
route. Packet count? Round robin?

> > d) OSPF. I read in the RFC that OSPF can do "load balancing", but I failed
> > to understand how (no, I didn't read that RFC thoroughly, it's really high
> > tech for me at this point). Does it use multipath routes to accomplish this?

Any thoughts on how OSPF does this?

Thanks for your input

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] many ways to do load balancing (or not?)

[Ramin Alidousti]

On Thu, Nov 21, 2002 at 05:11:23PM -0200, Andreas Hasenack wrote:

> > > d) OSPF. I read in the RFC that OSPF can do "load balancing", but I failed
> > > to understand how (no, I didn't read that RFC thoroughly, it's really high
> > > tech for me at this point). Does it use multipath routes to accomplish this?
> 
> Any thoughts on how OSPF does this?

All OSPF does is installing equal cost routes in the FIB. The actual
forwarding, making routing decision and/or load balancing has nothing
to do with the OSPF protocol.

Ramin
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] many ways to do load balancing (or not?)

[William L. Thomson Jr.]

On Thu, 2002-11-21 at 11:11, Andreas Hasenack wrote:
> Em Thu, Nov 21, 2002 at 12:46:56PM -0500, Ashok N N escreveu:
> > i believe that SNAT is for connections initiated from outside world towards 
> > the internal network. and this would be reverse of what is intended here, 
> 
> That's DNAT.
>

Correct DNAT is outside in and SNAT is inside out.

> > i'm still looking at the code here. AFAIK, the when you have multipath option 
> > set, then when looking up for a route to an address, among the multiple equal 
> > cost paths, one is selected according to some criteria (i read random 
> > somewhere in the kernel but am not able to locate it now). but once the route 
> 
> This criteria I would like to understand.

From what I understand it's like this. The whole load balancing thing
works in two ways.

load balancing outside in request
and
load balancing inside out request

The multipath feature is there so you can have multiple paths from the
inside out. Although without the equalize option each route would have
to be specified and there really would be no balancing.

Equalize along with the weights is what creates the balancing, from
request coming from the inside going out.

Multipath, equalize, and weights do not effect the incoming requests
from the outside world. That are initiated from the outside. That type
of load balancing typically is done at the DNS level. Balancing outside
to in requests.

As for the algorithm to balance from within using multipath and equalize
its like this.If you have two routes with equal weights each one will be
used half the time.

If you have two routes one with weight 1, and the other with 2, then
more than likely the first will be used 2 out of 3 times, with the
second being used 1 out of 3 times. Or something like that.
 
> > route. so it is kind of a route-based load-balancing.
> 
> Considering that the route cache is 60s by default, this should work I guess.
> I'm setting up such a setup for testing.

I played around with the cache settings for like a week straight a while
back. In the end I reverted to default values and they have worked out
fine.

> > i read in one of the threads that 'equalize' causes per-packet load balancing, 
> > i.e looks up the route for each packet. but i am doubtful about the 
> > performance when each packet causes a route-lookup in the FIB.
> > (http://www.uwsg.iu.edu/hypermail/linux/net/0107.3/0028.html)
> 
> Interesting, so 'equalize' bypasses the cache, that's about it?

No, it will first look to the catch, and then consult the multipath for
the next route/gateway to use when using the equalize option if it does
not already exist in the catch.

> So we only need to know the criteria with which the route is choosed in a multipath
> route. Packet count? Round robin?

Also I do not believe the load balancing is packet based. Usually it's
more "connection" based. Meaning that if you request a file, more than
likely all parts of that file will be transfered using the same route.
If you request it again, it may take the same route or another.

Once again a DNS thing if the request comes from the outside world.

Now if the request was generated from the inside it would still work
some what the same. If I send two emails out at once, the first will use
gw1 and the other will use gw2.

All packets for each will travel via the same route and use the same
gateway from start to finish.

If it was more on a packet level, the other end would be confused.

It would be getting responses from an IP it was not expecting response
from. I would imagine each side to send redirects, and all sorts of
problems. Like it receiving every other packet and dropping the packets
in between.

I believe the cache also has effect on this.

If during a file transfer the route cache is flushed, there is the
possibility of the rest of the packets going out a different interface.
But at that point all subsequent packets would be going out that
interface as well. At some point I would further assume the client to
catch on, and finish the transfer at some point.

SSH and FTP most likely will not. Not to sure about HTTP, probably
protocol dependent 1.0/1.1.

This is also what would happen in case of link failure, so if redundancy
is your main concern you may want to play with the catch settings.

In the end load balancing is not exactly what you might expect it to be.
From the inside world out, you do it via the kernel multipath equalize.
The the outside world in you use DNS.

Neither does it perfectly or with intelligent algorithms. Neither allow
you to use all paths for a single transfer.

So if you have two 1.5 mbs connection, you do not end up with a 3.0 mbs
line. You do have one internal gateway for both, and if one goes down
the other can be used. So you do have redundancy, and both lines can be
used to serve difference requests to different places.

You also have separate paths, so if ISP1 is closer client1, and ISP2 is
closer to client2, each others traffic will not effect the other. Not to
mention better transfers due to less distance.

But that's it. No bonding or true load balancing.

> > > d) OSPF. I read in the RFC that OSPF can do "load balancing", but I failed
> > > to understand how (no, I didn't read that RFC thoroughly, it's really high
> > > tech for me at this point). Does it use multipath routes to accomplish this?
> 
> Any thoughts on how OSPF does this?

Never heard of it, but basically I am only aware of one way. Without
having your own public ip block, and different provider routing for your
own block. I believe then you would be using BGP or something.
 
> Thanks for your input
> 
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
> 
-- 
Sincerely,
William L. Thomson Jr.
Support Group
Obsidian-Studios Inc.
439 Amber Way
Petaluma, Ca. 94952
Phone  707.766.9509
Fax    707.766.8989
http://www.obsidian-studios.com

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] many ways to do load balancing (or not?)

[Christoph Simon]

On 21 Nov 2002 14:20:57 -0800
"William L. Thomson Jr." <support@obsidian-studios.com> wrote:

> Also I do not believe the load balancing is packet based. Usually it's
> more "connection" based. Meaning that if you request a file, more than
> likely all parts of that file will be transfered using the same route.
> If you request it again, it may take the same route or another.

It is packet based. There is no connection concept in IP.

From the IP Command Reference:

	equalize: allow packet by packet randomization on multipath
	routes.

My understanding is, that for equalize to work, all lines must go to
the same point and that must not be the end point. Also, this same
point must implement the equalize very much the same way.

> Now if the request was generated from the inside it would still work
> some what the same. If I send two emails out at once, the first will use
> gw1 and the other will use gw2.
> 
> All packets for each will travel via the same route and use the same
> gateway from start to finish.

Not using equalize.

> If it was more on a packet level, the other end would be confused.

unless there is a routing cache which will hold the path.

-- 
Christoph Simon
ciccio@kiosknet.com.br
---
^X^C
q
quit
:q
^C
end
x
exit
ZZ
^D
?
help
.
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] many ways to do load balancing (or not?)

[Christoph Simon]

On 21 Nov 2002 15:07:11 -0800
"William L. Thomson Jr." <support@obsidian-studios.com> wrote:

> > From the IP Command Reference:
> > 
> > 	equalize: allow packet by packet randomization on multipath
> > 	routes.
> > 
> > My understanding is, that for equalize to work, all lines must go to
> > the same point and that must not be the end point. Also, this same
> > point must implement the equalize very much the same way.
> 
> Not necessarily. I have two lines going in completely different
> directions, different private nets, and then via different ISP/Public
> IPs.

Two lines in different directions using equalize? What's the point of
this?

> > > Now if the request was generated from the inside it would still work
> > > some what the same. If I send two emails out at once, the first will
> > > use gw1 and the other will use gw2.
> > > 
> > > All packets for each will travel via the same route and use the same
> > > gateway from start to finish.
> > 
> > Not using equalize.
> 
> Yes using equalize, because the path will already be known and used from
> catch. Correct that if a further look up occurs the packets will use a
> different interface, but that is rare. Most of the time during a
> transfer the route is already known and kept in catch from the first
> initial lookup.

I didn't play too much with equalize, but using it for load balancing
and redundancy failed for reason by with equalize is defined. As far
as I understood how weighting works: There are several identic routes
so that the probability to pick one of them corresponds to the given
weights. Still on a packet base. With equalize, the balancing should
be perfect, which can never be the case with few connections, even if
they imply massive data transfer. But if there is no NAT later, it
just can't work.

> > unless there is a routing cache which will hold the path.
> 
> That's the problem. The route cache on the Linux load balancing router
> will be correct, but the route cache on the client will not be.

If "client" here is a computer reaching the internet through the linux
router (it can't be anything else, because you said that there are two
independent lines out of your control), and unless that client uses
source routing which might be ignored by the linux router, the
client's view of this situation really doesn't matter.

> The client will be caching the wrong path. So unless a redirect is sent
> the client will site there and expect further communication using the
> route in cache.

The client will cache the route which leads to the Linux router and
not bother with details happening later.

-- 
Christoph Simon
ciccio@kiosknet.com.br
---
^X^C
q
quit
:q
^C
end
x
exit
ZZ
^D
?
help
.
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] many ways to do load balancing (or not?)

[William L. Thomson Jr.]

On Thu, 2002-11-21 at 14:55, Christoph Simon wrote:
> On 21 Nov 2002 14:20:57 -0800
> "William L. Thomson Jr." <support@obsidian-studios.com> wrote:
> 
> > Also I do not believe the load balancing is packet based. Usually it's
> > more "connection" based. Meaning that if you request a file, more than
> > likely all parts of that file will be transfered using the same route.
> > If you request it again, it may take the same route or another.
> 
> It is packet based. There is no connection concept in IP.

That's why I put that word in quotes. I meant more like during a file
transfer than with a connection in the true sense.

> >From the IP Command Reference:
> 
> 	equalize: allow packet by packet randomization on multipath
> 	routes.
> 
> My understanding is, that for equalize to work, all lines must go to
> the same point and that must not be the end point. Also, this same
> point must implement the equalize very much the same way.

Not necessarily. I have two lines going in completely different
directions, different private nets, and then via different ISP/Public
IPs.

Now for true load balancing or bonding yes. Both lines would need to be
coming from and going to the same ISP.

> > Now if the request was generated from the inside it would still work
> > some what the same. If I send two emails out at once, the first will use
> > gw1 and the other will use gw2.
> > 
> > All packets for each will travel via the same route and use the same
> > gateway from start to finish.
> 
> Not using equalize.

Yes using equalize, because the path will already be known and used from
catch. Correct that if a further look up occurs the packets will use a
different interface, but that is rare. Most of the time during a
transfer the route is already known and kept in catch from the first
initial lookup.

I have not run into any problems along those lines.

> > If it was more on a packet level, the other end would be confused.
> 
> unless there is a routing cache which will hold the path.

That's the problem. The route cache on the Linux load balancing router
will be correct, but the route cache on the client will not be.

The client will be caching the wrong path. So unless a redirect is sent
the client will site there and expect further communication using the
route in cache. 

When I was experimenting with route cache timing in the Linux kernel, I
had all kinds of problems on the client side. The client's catch
settings would need to be the same as the linux router. Which is pretty
impossible on the Internet. On a lan it would be a different story.


-- 
Sincerely,
William L. Thomson Jr.
Support Group
Obsidian-Studios Inc.
439 Amber Way
Petaluma, Ca. 94952
Phone  707.766.9509
Fax    707.766.8989
http://www.obsidian-studios.com

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] many ways to do load balancing (or not?)

[William L. Thomson Jr.]

On Thu, 2002-11-21 at 15:41, Christoph Simon wrote:
> On 21 Nov 2002 15:07:11 -0800
> "William L. Thomson Jr." <support@obsidian-studios.com> wrote:
> > 
> > Not necessarily. I have two lines going in completely different
> > directions, different private nets, and then via different ISP/Public
> > IPs.
> 
> Two lines in different directions using equalize? What's the point of
> this?

Once again let me clarify myself. By different directions I mean
different ISPs. Not incoming from one and outgoing on another.

It's more having two separate connections to the Internet with two
separate ISPs. That's what I mean by two different directions. One of my
ISPs is North of me, the other is South. 

So if I were to say send out a mass mailer or god forbid spam, I could
use both lines from the inside to send data out. Same page now? Sorry
bout that.

> > > > Now if the request was generated from the inside it would still work
> > > > some what the same. If I send two emails out at once, the first will
> > > > use gw1 and the other will use gw2.
> > > > 
> > > > All packets for each will travel via the same route and use the same
> > > > gateway from start to finish.
> > > 
> > > Not using equalize.
> > 
> > Yes using equalize, because the path will already be known and used from
> > catch. Correct that if a further look up occurs the packets will use a
> > different interface, but that is rare. Most of the time during a
> > transfer the route is already known and kept in catch from the first
> > initial lookup.
> 
> I didn't play too much with equalize, but using it for load balancing
> and redundancy failed for reason by with equalize is defined.

Yes for true redundancy the routes need to be removed and then added
back or to stop being used. I believe that's were Julian's dead gateway
detection stuff comes in.

> As far
> as I understood how weighting works: There are several identic routes
> so that the probability to pick one of them corresponds to the given
> weights. Still on a packet base. With equalize, the balancing should
> be perfect, which can never be the case with few connections, even if
> they imply massive data transfer. But if there is no NAT later, it
> just can't work.

Correct based on weights using equalize. Because without equalize it
will just travel the first route no matter what unless a route using the
second is already know and existing in cache. From the inside balancing
is basically perfect using equalize. It's code I mean it does what it's
is supposed to do but is not smart. If one line is being used more than
another from the outside, it will still balance things from the inside.
Even thought one line is being bogged down.

Even if you say have two rather large unequal files to send out, it will
send the first file out one, and the second file out the other, and then
keep going back and forth equally if the weights are equal. Even though
the actual load is not equal.

In my case I am doing NAT, and it is part of the reason why things work,
along with Julian's patches.

But I have been informed I believe by Julian and others that the load
balancing, multipath equalize feature can be used even without NAT but
in a different situation that mine?

> > > unless there is a routing cache which will hold the path.
> > 
> > That's the problem. The route cache on the Linux load balancing router
> > will be correct, but the route cache on the client will not be.
> 
> If "client" here is a computer reaching the internet through the linux
> router (it can't be anything else, because you said that there are two
> independent lines out of your control), and unless that client uses
> source routing which might be ignored by the linux router, the
> client's view of this situation really doesn't matter.
 
By client I mean someone in the outside world talking to say a server.
The server is behind the Linux router. Not a client on the local lan
looking out.

> > The client will be caching the wrong path. So unless a redirect is sent
> > the client will site there and expect further communication using the
> > route in cache.
> 
> The client will cache the route which leads to the Linux router and
> not bother with details happening later.

I am talking about the client caching the wrong public ip and
communication being attempted on a separate public ip from the server.
While the client sits there and times out on the first ip, when ever
it's cache expires.


-- 
Sincerely,
William L. Thomson Jr.
Support Group
Obsidian-Studios Inc.
439 Amber Way
Petaluma, Ca. 94952
Phone  707.766.9509
Fax    707.766.8989
http://www.obsidian-studios.com

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] many ways to do load balancing (or not?)

[Ashok N N]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thursday 21 November 2002 05:20 pm, William L. Thomson Jr. wrote:
> On Thu, 2002-11-21 at 11:11, Andreas Hasenack wrote:
> > Em Thu, Nov 21, 2002 at 12:46:56PM -0500, Ashok N N escreveu:
> > > i believe that SNAT is for connections initiated from outside world 
towards 
> > > the internal network. and this would be reverse of what is intended 
here, 
> > 
> > That's DNAT.
> >
> 
> Correct DNAT is outside in and SNAT is inside out.
> 

Yep..my mistake (i seem to always confuse between the two)

ashok
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE93YWoRhXpVty0Ty4RArnnAKCXYk3nVU16mvVEHf/WSgALqBQDYgCggm1A
QGctynuQzP9vOTdtof8bohU=hqBe
-----END PGP SIGNATURE-----

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] many ways to do load balancing (or not?)

[Andreas Hasenack]

Em Thu, Nov 21, 2002 at 02:20:57PM -0800, William L. Thomson Jr. escreveu:
> Also I do not believe the load balancing is packet based. Usually it's
> more "connection" based. Meaning that if you request a file, more than
> likely all parts of that file will be transfered using the same route.
> If you request it again, it may take the same route or another.

If I make many connections from one IP (inside) to a web server (outside),
for example (like many simultaneous downloads, or a complex page), I think
they will all go via the same route, because the originating IP and the
destination are the same. It will hit the cache.
Hmm, not good if your users use a proxy, but then again, the proxy would
cache the page probably.

> Now if the request was generated from the inside it would still work
> some what the same. If I send two emails out at once, the first will use
> gw1 and the other will use gw2.

Unless they are sent to the same MTA in the outside, then it will get a
cache hit (supposing the 60s haven't gone by then). Or not?

> All packets for each will travel via the same route and use the same
> gateway from start to finish.

Agreed.

> If it was more on a packet level, the other end would be confused.

Sure. When I said "packet count" before I was thinking about something
along the lines of real traffic balancing, that is, the router somehow
remembering how many packets it sent to each route and choosing the
less used one.

> It would be getting responses from an IP it was not expecting response
> from. I would imagine each side to send redirects, and all sorts of
> problems. Like it receiving every other packet and dropping the packets
> in between.

And breaking stateful firewalls.

> If during a file transfer the route cache is flushed, there is the
> possibility of the rest of the packets going out a different interface.

Uh oh... It shouldn't be that simple, what about that 60s timeout for
the cache? It's very likely to occur during a file transfer.

> Neither does it perfectly or with intelligent algorithms. Neither allow
> you to use all paths for a single transfer.

Only things like MPPP I guess, for example, or channel bonding, or TQL.

> So if you have two 1.5 mbs connection, you do not end up with a 3.0 mbs
> line. You do have one internal gateway for both, and if one goes down
> the other can be used. So you do have redundancy, and both lines can be
> used to serve difference requests to different places.

So it's more like redundancy/HA with a best effort towards balancing.

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] many ways to do load balancing (or not?)

[Andreas Hasenack]

Em Thu, Nov 21, 2002 at 08:55:05PM -0200, Christoph Simon escreveu:
> My understanding is, that for equalize to work, all lines must go to
> the same point and that must not be the end point. Also, this same
> point must implement the equalize very much the same way.

What is it that you call a "point" here (destination)? The same ISP? The
same network?

I understand that it should be the same ISP because of egress filtering, that
is, one ISP should block packets with a source address that doesn't belong
to the ISP supplying the link.

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] many ways to do load balancing (or not?)

[Andreas Hasenack]

Em Thu, Nov 21, 2002 at 04:24:06PM -0800, William L. Thomson Jr. escreveu:
> But I have been informed I believe by Julian and others that the load
> balancing, multipath equalize feature can be used even without NAT but
> in a different situation that mine?

I'm confused as well. Suppose you have two links to the internet, a DMZ,
and an internal network, SNAT'ed. Suppose you have a public web server
in the DMZ (the DMZ is not SNAT'ed).

How would multipath route (with or without equalize) help here? I mean,
it would only really help if there were connections starting from
the inside (DMZ or SNAT'ed network) to the outside. But:

- the internal network would probably do many downloads, and not uploads

- the web server doesn't originate traffic, it responds to requests from
the outside world, and it will respond using the same link the request
came in (or not?)

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] many ways to do load balancing (or not?)

[Andreas Hasenack]

Em Thu, Nov 21, 2002 at 04:06:26PM -0800, William L. Thomson Jr. escreveu:
> Not necessarily. I have two lines going in completely different
> directions, different private nets, and then via different ISP/Public
> IPs.

Do you have public servers that are accessed from the outside? How do you
respond to requests comming down one link? I suppose via the same link, or
using SNAT, otherwise you would have packets with a source IP from ISP1
traveling through the link to ISP2, right?

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] many ways to do load balancing (or not?)

[Christoph Simon]

On Fri, 22 Nov 2002 10:30:35 -0200
Andreas Hasenack <andreas@conectiva.com.br> wrote:

> Em Thu, Nov 21, 2002 at 08:55:05PM -0200, Christoph Simon escreveu:
> > My understanding is, that for equalize to work, all lines must go to
> > the same point and that must not be the end point. Also, this same
> > point must implement the equalize very much the same way.
> 
> What is it that you call a "point" here (destination)? The same ISP? The
> same network?
> 
> I understand that it should be the same ISP because of egress filtering,
> that is, one ISP should block packets with a source address that doesn't
> belong to the ISP supplying the link.

I was thinking of a configuration like:

	Satelite                 Main-Office

	     -> ISP1 -m-
	LAN1 -> ISP2 -m- ISPx -> linux-router2 -> ISP-BIG ->
	     -> ISP3 -m-

If I have an office with one big Internet tube (e.g., MBs), I can put
"linux-router2" there. Then all satelite offices could use one or more
modems to reach this router. That linux-router2 would NAT everybody
coming from the modems and forward the packets to ISP-BIG. Nobody in
LAN1 would even attempt to reach linux-router2, which is the last
point where their actually source addresses are known. With equalize
this should allow for a packetwise loadbalancing to increase the
effective bandwidth even for a single file transfer between a host in
LAN1 and ISP-BIG. In this configuration, all modem lines go finally to
one point (linux-router2) and that is not the endpoint. In fact, all
internal clients would use linux-router2 as the default gateway. No
cooperation of any ISP is required.

This configuration makes sense if LAN1 and linux-router2 are in
different cities and either LAN1 has no other reliable choice than
modems or wants redundancy. ISP-BIG also needs to be cheap and
reliable enough for this to make sense.

As I said before, I didn't play too much with equalize, but can't see
a essencially different scenario for a packet-wise loadbalancing.

-- 
Christoph Simon
ciccio@kiosknet.com.br
---
^X^C
q
quit
:q
^C
end
x
exit
ZZ
^D
?
help
.
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] many ways to do load balancing (or not?)

[Vincent Jaussaud]

On Fri, 2002-11-22 at 13:39, Andreas Hasenack wrote:
> - the internal network would probably do many downloads, and not uploads
> 
> - the web server doesn't originate traffic, it responds to requests from
> the outside world, and it will respond using the same link the request
> came in (or not?)
> 
No, replies packet will hit the multipath route, and thus may not take
the same link to come back.

But in such situation, the sraddr in the replies packets are likely to
be set to the one on which the original request came on.
In that case, we can add specific routing rules to ensure that the
replies goes back through the right link.

As far as I know, sraddr is likely to be set correctly for TCP servers,
while for UDP servers it may not work as expected.

Also, if our ISPs don't do rp_filtering, then we don't care what link is
beeing used for the replies, as soon as the saddr is correct.

Vincent. 

> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
-- 
Vincent Jaussaud
Kelkoo.com Security Manager 
email: tatooin@kelkoo.com

"The UNIX philosophy is to design small tools that do one thing, and do
it well."

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] many ways to do load balancing (or not?)

[William L. Thomson Jr.]

On Fri, 2002-11-22 at 04:28, Andreas Hasenack wrote:
>
> Sure. When I said "packet count" before I was thinking about something
> along the lines of real traffic balancing, that is, the router somehow
> remembering how many packets it sent to each route and choosing the
> less used one.

That would be one step closer to true load balancing but I do not thing
would work for the other end the client. Instead it just splits
requests/lookups and does not completely load balance traffic or
packets. 

It is very easy to have uneven loads. For this one day I will look into
QoS when the time is right, hoping it may be able to help even out the
load? At least from the inside.

Once again from the outside load balancing is done via DNS.

> > It would be getting responses from an IP it was not expecting response
> > from. I would imagine each side to send redirects, and all sorts of
> > problems. Like it receiving every other packet and dropping the packets
> > in between.
> 
> And breaking stateful firewalls.

Yep

> > If during a file transfer the route cache is flushed, there is the
> > possibility of the rest of the packets going out a different interface.
> 
> Uh oh... It shouldn't be that simple, what about that 60s timeout for
> the cache? It's very likely to occur during a file transfer.

Well that's more a worse case scenario like manually flushing the cache.
I believe the cache gc algorithm takes in consideration routes that are
currently in use. So lookups do not occur during a transfer?

I have not really seen this to be a big problem. Although I do not have
any large files being downloaded, and if I did it could cause problems
there.

But so far I have not had any problems along those lines.

> > Neither does it perfectly or with intelligent algorithms. Neither allow
> > you to use all paths for a single transfer.
> 
> Only things like MPPP I guess, for example, or channel bonding, or TQL.

Yes, but I believe each of those requires special configurations on both
ends, client and ISP.

> > So if you have two 1.5 mbs connection, you do not end up with a 3.0 mbs
> > line. You do have one internal gateway for both, and if one goes down
> > the other can be used. So you do have redundancy, and both lines can be
> > used to serve difference requests to different places.
> 
> So it's more like redundancy/HA with a best effort towards balancing.

Yes, or in other terms. My need was a single gateway for my servers
although I have two ISPs. The amount of load balancing you get it about
the same as the amount of redundancy. You get a partial solution to
both, but not a complete solution.

-- 
Sincerely,
William L. Thomson Jr.
Support Group
Obsidian-Studios Inc.
439 Amber Way
Petaluma, Ca. 94952
Phone  707.766.9509
Fax    707.766.8989
http://www.obsidian-studios.com

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] many ways to do load balancing (or not?)

[William L. Thomson Jr.]

On Fri, 2002-11-22 at 04:41, Andreas Hasenack wrote:
> Em Thu, Nov 21, 2002 at 04:06:26PM -0800, William L. Thomson Jr. escreveu:
> > Not necessarily. I have two lines going in completely different
> > directions, different private nets, and then via different ISP/Public
> > IPs.
> 
> Do you have public servers that are accessed from the outside? 

Yes that's basically all I have. I have three lines, two 1.1mps SDSL and
one ADSL.

My ADSL line is for surfing, and is my main inbound line for normal
office traffic.

My SDSL lines handle all requests to and from my public servers. So most
of the traffic is initiated from the outside. Very little inbound
traffic, mostly allot of outbound.

> How do you
> respond to requests comming down one link? I suppose via the same link, or
> using SNAT, otherwise you would have packets with a source IP from ISP1
> traveling through the link to ISP2, right?

That's where it gets a little crazy. Each SDSL line terminates in a
router that performs PAT. From the two SDSL routers doing PAT, the
requests then make it to the Linux router, which does a second round of
PAT, or DPAT.

Now when inbound request leave then get SPAT as they leave the Linux
router heading for the SDSL routers to be PAT again back to Public IPs.

A little excessive, but having two firewalls back to back on either
connection is nice. Pretty dam hard to get any packet through both
that's is not supposed to be going through them.

Now the hardest part for me to grasp, was how ipchains is able to take a
single internal IP and convert it into two different internal/external
IPs. Internal in my case, since I do not use Public IPs until hitting
and leaving the SDSL routers. There is a particular rule that does this,
but I have not found an equivalent using iptables. Although I am sure
one exists, or hope one does.

Now this is also where Julian's patches came into play. By using NAT and
Julian's patches, the NAT cache is looked at so the choice of what IP to
map the servers IP to is based on the cache. If there is nothing there
in the cache, then it goes to the multipath route.

Sounds simple, but was a pain for me to grasp, and get working. But it
has been working perfectly ever since, or at least as good as can be
expected. Since I have implemented true dead gateway detection, nor have
I dialed things in specifically with cache timeouts, gcs, etc.

-- 
Sincerely,
William L. Thomson Jr.
Support Group
Obsidian-Studios Inc.
439 Amber Way
Petaluma, Ca. 94952
Phone  707.766.9509
Fax    707.766.8989
http://www.obsidian-studios.com

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] many ways to do load balancing (or not?)

[William L. Thomson Jr.]

On Fri, 2002-11-22 at 04:39, Andreas Hasenack wrote:
> Em Thu, Nov 21, 2002 at 04:24:06PM -0800, William L. Thomson Jr. escreveu:
> > But I have been informed I believe by Julian and others that the load
> > balancing, multipath equalize feature can be used even without NAT but
> > in a different situation that mine?
> 
> I'm confused as well. Suppose you have two links to the internet, a DMZ,
> and an internal network, SNAT'ed. Suppose you have a public web server
> in the DMZ (the DMZ is not SNAT'ed).
> 
> How would multipath route (with or without equalize) help here? I mean,
> it would only really help if there were connections starting from
> the inside (DMZ or SNAT'ed network) to the outside. But:

Exactly
 
> - the internal network would probably do many downloads, and not uploads

In this case the multipath will help out. Splitting the downloads over
the different interfaces. This way all lines will be used for downloads.
Although the actual load will not be equal on all lines, as previously
mentioned. But depending on the amount of routes and weights, the
requests/lookups will be properly balanced.

> - the web server doesn't originate traffic, it responds to requests from
> the outside world, and it will respond using the same link the request
> came in (or not?)

This is why I kept saying that you have to load balance from the outside
in and the inside out. 

Inside out was above using multipath.

Outside in is load balanced using DNS. Bind allows you to specify
multiple IPs for a domain, and a similar weight value. So you can load
balance DNS look ups.

In my case where everything is even I have two Public IPs for each
server in my DNS records. Do a dig on my domain.

You will get two IPs, and each time a dns lookup occurs you should get
on or the other.

In theory every other request gets a different IP. So Bind is trying to
load balance things from the outside world. But depending on allot of
different variables, the client could end up doing another lookup or
using an interface out of the load balancing order. So it's not perfect.

But to complete the load balancing scenario it's absolutely necessary.
Otherwise there is no way to load balance from the outside world without
your ISP providing a service or using something like BGP.

So load balancing inside out is done via multipath equalize, and load
balancing from the outside in is done via DNS, in my case BIND.

-- 
Sincerely,
William L. Thomson Jr.
Support Group
Obsidian-Studios Inc.
439 Amber Way
Petaluma, Ca. 94952
Phone  707.766.9509
Fax    707.766.8989
http://www.obsidian-studios.com

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] many ways to do load balancing (or not?)

[William L. Thomson Jr.]

On Fri, 2002-11-22 at 05:26, Vincent Jaussaud wrote:
> On Fri, 2002-11-22 at 13:39, Andreas Hasenack wrote:
> > - the internal network would probably do many downloads, and not uploads
> > 
> > - the web server doesn't originate traffic, it responds to requests from
> > the outside world, and it will respond using the same link the request
> > came in (or not?)
> > 
> No, replies packet will hit the multipath route, and thus may not take
> the same link to come back.

Actually if you create the proper rules, there are gateways defined in a
table prior to the multipath. So if the route is know via cache, it will
take a known gateway.

Otherwise if it is not, it will be compared to each table, and then
finally hit the multipath equalize one and go from there.

> But in such situation, the sraddr in the replies packets are likely to
> be set to the one on which the original request came on.

Which will use the rules if defined. They should be there.

> In that case, we can add specific routing rules to ensure that the
> replies goes back through the right link.

Exactly, but you put them before the multipath route so they are looked
at and compared to first.

> As far as I know, sraddr is likely to be set correctly for TCP servers,
> while for UDP servers it may not work as expected.

Haven't really seen and problems there. So far from my experience my DNS
queries use both TCP and UDP. I have not seen and problems effecting UDP
and not TCP. But DNS lookups are quick and short, as most UDP services,
so there could be some problems there that are not effecting me in my
current scenario.

> Also, if our ISPs don't do rp_filtering, then we don't care what link is
> beeing used for the replies, as soon as the saddr is correct.

Not sure here, I never got into that aspect. I just assumed they did do
rp filtering. I played with turing rp filtering on in the Linux router,
but in the end left it on or what ever the default is.

-- 
Sincerely,
William L. Thomson Jr.
Support Group
Obsidian-Studios Inc.
439 Amber Way
Petaluma, Ca. 94952
Phone  707.766.9509
Fax    707.766.8989
http://www.obsidian-studios.com

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] many ways to do load balancing (or not?)

[Andreas Hasenack]

Em Fri, Nov 22, 2002 at 10:05:25AM -0800, William L. Thomson Jr. escreveu:
> > So it's more like redundancy/HA with a best effort towards balancing.
> 
> Yes, or in other terms. My need was a single gateway for my servers
> although I have two ISPs. The amount of load balancing you get it about
> the same as the amount of redundancy. You get a partial solution to
> both, but not a complete solution.

I just found this patch, has anybody already played with it?

ftp://sliepen.warande.net/pub/eql/patch-2.4.18-2.gz

Excerpt:

Load balancing needed a slight adjustment to the unpatched linux kernel,
because of the route cache. Multipath is an option already found in the old
2.1.x kernels. However, once a packet arrives, and it matches a multipath
route, a (quasi random) device out of the list of nexthops is taken for its
destination. That's okay, but after that the kernel puts everything into a
hash table, and the next time a packet with the same source/dest/tos arrives,
it finds it is in the hash table, and routes it via the same device as last
time. The adjustment I made is as follows: If the kernel sees that the route
to be taken has got the 'equalize' flag set, it not only selects the random
device, but also tags the packet with the RTCF_EQUALIZE flag. If another
packet of the same kind arrives, it is looked up in the hash table. It then
checks if our flag is set, and if so, it deletes the entry in the cache and
has to recalculate the destination again.

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] many ways to do load balancing (or not?)

[Vincent Jaussaud]

On Fri, 2002-11-22 at 19:47, William L. Thomson Jr. wrote:

> Actually if you create the proper rules, there are gateways defined in a
> table prior to the multipath. So if the route is know via cache, it will
> take a known gateway.
> 
Yes.

> Otherwise if it is not, it will be compared to each table, and then
> finally hit the multipath equalize one and go from there.
> 
> > But in such situation, the sraddr in the replies packets are likely to
> > be set to the one on which the original request came on.
> 
> Which will use the rules if defined. They should be there.
Yes.

> Haven't really seen and problems there. So far from my experience my DNS
> queries use both TCP and UDP. I have not seen and problems effecting UDP
> and not TCP. But DNS lookups are quick and short, as most UDP services,
> so there could be some problems there that are not effecting me in my
> current scenario.
> 
That's good news. Normally, TCP port 53 is only used for zone
transferts, while UDP port is used for DNS queries. If you don't see
anything strange in your logs, such as "connection reset by peer", then
it means it works fine. Well, you probably already figured this out :-)

Did you tried with other UDP services ? (like timed for instance)

If it works for simple UDP servers like timed, then I would definitly
consider linux as the best OS in the world ;-)

> > Also, if our ISPs don't do rp_filtering, then we don't care what link is
> > beeing used for the replies, as soon as the saddr is correct.
> 
> Not sure here, I never got into that aspect. I just assumed they did do
> rp filtering. I played with turing rp filtering on in the Linux router,
> but in the end left it on or what ever the default is.

Anyway, if the do, it should be simple to ask them to disable
rp_filtering. I had some talks with my ISP some days ago, and according
to them, most ISPs don't bother with rp_filtering, except for private
netblocks trying to 'escape' in case of a broken NAT configuration.


-- 
Vincent Jaussaud
Kelkoo.com Security Manager 
email: tatooin@kelkoo.com

"The UNIX philosophy is to design small tools that do one thing, and do
it well."

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/