Re: Poor Performance for ethernet bonding

Re: Poor Performance for ethernet bonding

[Andi Kleen]

Bernhard Busch <bbusch@biochem.mpg.de> writes:

> Hi
> 
> 
> I have tried to use ethernet  network interfaces bonding to increase
> peformance.
> 
> Bonding is working fine, but the performance is rather poor.
> FTP between 2 machines ( kernel 2.4.4 and 4 port DLink 100Mbit ethernet
> card)
> results in a transfer rate of 3MB/s).
> 
> Any Hints?

Bonding reorders packets, which causes frequent retransmits and stalls in TCP.
One setup that doesn't is multipath routing (ip route .. with multiple 
nexthops over different interfaces). It'll only load balance (srcip, dstip,tos)
tuples though, not individual flows, but then it has the advantage that
it actually works.

-Andi

Re: Poor Performance for ethernet bonding

[Ben Greear]

Andi Kleen wrote:
> 
> Bernhard Busch <bbusch@biochem.mpg.de> writes:
> 
> > Hi
> >
> >
> > I have tried to use ethernet  network interfaces bonding to increase
> > peformance.
> >
> > Bonding is working fine, but the performance is rather poor.
> > FTP between 2 machines ( kernel 2.4.4 and 4 port DLink 100Mbit ethernet
> > card)
> > results in a transfer rate of 3MB/s).
> >
> > Any Hints?
> 
> Bonding reorders packets, which causes frequent retransmits and stalls in TCP.
> One setup that doesn't is multipath routing (ip route .. with multiple
> nexthops over different interfaces). It'll only load balance (srcip, dstip,tos)
> tuples though, not individual flows, but then it has the advantage that
> it actually works.

Couldn't the bonding code be made to distribute pkts to one interface or
another based on a hash of the sending IP port or something?  Seems like that
would fix the reordering problem for IP packets....  It wouldn't help for
a single stream, but I'm guessing the real world problem involves many streams,
which on average should hash such that the load is balanced...

Ben

> 
> -Andi
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
Ben Greear <greearb@candelatech.com>          <Ben_Greear@excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear

Re: Poor Performance for ethernet bonding

[Andi Kleen]

On Fri, Aug 24, 2001 at 08:19:50AM -0700, Ben Greear wrote:
> Couldn't the bonding code be made to distribute pkts to one interface or
> another based on a hash of the sending IP port or something?  Seems like that
> would fix the reordering problem for IP packets....  It wouldn't help for
> a single stream, but I'm guessing the real world problem involves many streams,
> which on average should hash such that the load is balanced...

It could, but then it is already implemented in a better way in multipath 
routing and I see no reason to duplicate the functionality.

-Andi

Re: Poor Performance for ethernet bonding

[Ben Greear]

Andi Kleen wrote:
> 
> On Fri, Aug 24, 2001 at 08:19:50AM -0700, Ben Greear wrote:
> > Couldn't the bonding code be made to distribute pkts to one interface or
> > another based on a hash of the sending IP port or something?  Seems like that
> > would fix the reordering problem for IP packets....  It wouldn't help for
> > a single stream, but I'm guessing the real world problem involves many streams,
> > which on average should hash such that the load is balanced...
> 
> It could, but then it is already implemented in a better way in multipath
> routing and I see no reason to duplicate the functionality.
> 

On the surface, multi-path routing sounds complicated to me, while
layer-2 bonding seems relatively trivial to set up/administer.  Since we do
support bonding, if it's a simple fix to make it better, we
might as well do that, eh?

I haven't used either, so this is just idle supposition on my part...

Ben

-- 
Ben Greear <greearb@candelatech.com>          <Ben_Greear@excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear

Re: Poor Performance for ethernet bonding

[Andi Kleen]

On Fri, Aug 24, 2001 at 08:45:33AM -0700, Ben Greear wrote:
> On the surface, multi-path routing sounds complicated to me, while
> layer-2 bonding seems relatively trivial to set up/administer.  Since we do
> support bonding, if it's a simple fix to make it better, we
> might as well do that, eh?

multipath routing is really not complicated; I don't know why it "sounds"
complicated to you. Of course you could always add new features to the kernel
because the existing ones which do the same thing in a better way
"sound complicated" to someone; I doubt it is a good use of developer time 
however. 

BTW when you would teach bonding about flows it wouldn't be layer-2 anymore.

To kill the "sounds complicated" myth: 

ip route add 10.0.0.0/8 nexthop dev eth0 nexthop dev eth1 

gives you a multipath route with eth0 and eth1 with the same weight for 
10.0.0.0 netmask 255.0.0.0. If you replace 10.0.0.0/8 with default it'll
be your default route. The kernel does the rest.


-Andi

Re: Poor Performance for ethernet bonding

[Henning P. Schmiedehausen]

Andi Kleen <ak@suse.de> writes:

>On Fri, Aug 24, 2001 at 08:45:33AM -0700, Ben Greear wrote:
>> On the surface, multi-path routing sounds complicated to me, while
>> layer-2 bonding seems relatively trivial to set up/administer.  Since we do
>> support bonding, if it's a simple fix to make it better, we
>> might as well do that, eh?

>multipath routing is really not complicated; I don't know why it "sounds"
>complicated to you. Of course you could always add new features to the kernel
>because the existing ones which do the same thing in a better way
>"sound complicated" to someone; I doubt it is a good use of developer time 
>however. 

But it is still the wrong layer.

Ethernet Bonding ist "below" IP. Very handy if you want to run
something "not ip".

	Regards
		Henning


-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     hps@intermeta.de

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   info@intermeta.de
D-91054 Buckenhof     Fax.: 09131 / 50654-20   

Re: Poor Performance for ethernet bonding

[Thomas Davis]

Ben Greear wrote:
> 
> Couldn't the bonding code be made to distribute pkts to one interface or
> another based on a hash of the sending IP port or something?  Seems like that
> would fix the reordering problem for IP packets....  It wouldn't help for
> a single stream, but I'm guessing the real world problem involves many streams,
> which on average should hash such that the load is balanced...
> 

Cisco etherchannel does this, by XOR'ing the dest address with the
source address, AND'ing with # of interfaces (limiting you to a power of
2), and then using the number to determine what channel to use.

Now, you end up in a 4 way Etherchannel, all the traffic going down one
channel, and the none going down the other three.  Does that sound like
a balanced solution?

Most bonding problems are either the card driver is busted, the card is
busted (el-cheapo NE2000 PCI clones are really bad..) or the switch
can't handle it.  Most cheap, dumb switches break big time when using
Bonding with them.


-- 
------------------------+--------------------------------------------------
Thomas Davis		| ASG Cluster guy
tadavis@lbl.gov		| 
(510) 486-4524		| "80 nodes and chugging Captain!"

Re: Poor Performance for ethernet bonding

[Ben Greear]

Thomas Davis wrote:
> 
> Ben Greear wrote:
> >
> > Couldn't the bonding code be made to distribute pkts to one interface or
> > another based on a hash of the sending IP port or something?  Seems like that
> > would fix the reordering problem for IP packets....  It wouldn't help for
> > a single stream, but I'm guessing the real world problem involves many streams,
> > which on average should hash such that the load is balanced...
> >
> 
> Cisco etherchannel does this, by XOR'ing the dest address with the
> source address, AND'ing with # of interfaces (limiting you to a power of
> 2), and then using the number to determine what channel to use.
> 
> Now, you end up in a 4 way Etherchannel, all the traffic going down one
> channel, and the none going down the other three.  Does that sound like
> a balanced solution?

If Cisco can't do a balanced hash, that doesn't mean the idea is bad,
it just means they implemented it poorly.  I would think that
something as simple as: src_ip_port % num_devices would give a
fairly even spread.  I'm sure you could get fancier and take other
fields (dst_ip_port) into account in your hash.
 

-- 
Ben Greear <greearb@candelatech.com>          <Ben_Greear@excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear

Re: Poor Performance for ethernet bonding

[Willy Tarreau]

Hi Ben,

> Couldn't the bonding code be made to distribute pkts to one interface or
> another based on a hash of the sending IP port or something?  Seems like that
> would fix the reordering problem for IP packets....  It wouldn't help for
> a single stream, but I'm guessing the real world problem involves many streams,
> which on average should hash such that the load is balanced...

You may take a look at this for enhancements to the bonding driver :

  http://sourceforge.net/projects/bonding/

The XOR method has been implemented to hash the flows based on the SRC/DST
MAC addresses.

Regards,
Willy

Re: Poor Performance for ethernet bonding

[Ben Greear]

Willy Tarreau wrote:
> 
> Hi Ben,
> 
> > Couldn't the bonding code be made to distribute pkts to one interface or
> > another based on a hash of the sending IP port or something?  Seems like that
> > would fix the reordering problem for IP packets....  It wouldn't help for
> > a single stream, but I'm guessing the real world problem involves many streams,
> > which on average should hash such that the load is balanced...
> 
> You may take a look at this for enhancements to the bonding driver :
> 
>   http://sourceforge.net/projects/bonding/
> 
> The XOR method has been implemented to hash the flows based on the SRC/DST
> MAC addresses.

If you are only hashing on MAC addresses, then you would become highly
un-optimized in the case where a machine is communicating with it's
default gateway primarily.  Perhaps the XOR hash is really more sophisticated??

Ben


> 
> Regards,
> Willy

-- 
Ben Greear <greearb@candelatech.com>          <Ben_Greear@excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear

Re: Poor Performance for ethernet bonding

[willy tarreau]

> > The XOR method has been implemented to hash the
> flows based on the SRC/DST
> > MAC addresses.
> 
> If you are only hashing on MAC addresses, then you
> would become highly un-optimized in the case where a
> machine is communicating with it's default gateway
> primarily.

absolutely right, but I think that when it was
designed
(sun? cisco?), it was primarily intended as many hosts
to one (server, router or switch). A file server, a
router or even a firewall can easily balance their
load
when they are connected to several tens of
workstations. If your concern is really for peer to
peer, then Andy's method is better as long as other
hashing methods have not been implemented in the
bonding code.

> Perhaps the XOR hash is really more sophisticated??

it may be one day, but at the moment, it's designed to
be compatible with other equipments which use it too.

Willy


___________________________________________________________
Do You Yahoo!? -- Un e-mail gratuit @yahoo.fr !
Yahoo! Courrier : http://fr.mail.yahoo.com

Poor Performance for ethernet bonding

[Bernhard Busch]

Hi


I have tried to use ethernet  network interfaces bonding to increase
peformance.

Bonding is working fine, but the performance is rather poor.
FTP between 2 machines ( kernel 2.4.4 and 4 port DLink 100Mbit ethernet
card)
results in a transfer rate of 3MB/s).

Any Hints?


here is my setup for the channel bonding

insmod bonding
insmod tulip
ifconfig bond0 192.168.100.2 netmask 255.255.255.0 broadcast
192.168.100.255
ifconfig bond0 down
./ifenslave  -v bond0 eth2 eth3 eth4 eth5
ifconfig bond0 up
route add -host 192.168.100.1 dev bond0


Many thanks for every help

Bernhard

--
Dr. Bernhard Busch
Max-Planck-Institut für Biochemie
Am Klopferspitz 18a
D-82152 Martinsried
Tel: +48(89)8578-2582
Fax: +49(89)8578-2479
Email bbusch@biochem.mpg.de

Re: Poor Performance for ethernet bonding

[Sergey Ostrovsky]

Bernhard Busch wrote:

> Hi
>
> I have tried to use ethernet  network interfaces bonding to increase
> peformance.
>
> Bonding is working fine, but the performance is rather poor.
> FTP between 2 machines ( kernel 2.4.4 and 4 port DLink 100Mbit ethernet
> card)
> results in a transfer rate of 3MB/s).

Slightly OT, but :
I had to return my 4 port DLink 10/100Mbit _switch_,
because this piece of junk dropped 50-70% of packets sent.
I'd suggest first to ping for some time and see statistics.

Sergey Ostrovsky.

Re: Poor Performance for ethernet bonding

[Martin Josefsson]

On Fri, 24 Aug 2001, Bernhard Busch wrote:

> Hi
> 
> 
> I have tried to use ethernet  network interfaces bonding to increase
> peformance.
> 
> Bonding is working fine, but the performance is rather poor.
> FTP between 2 machines ( kernel 2.4.4 and 4 port DLink 100Mbit ethernet
> card)
> results in a transfer rate of 3MB/s).
> 
> Any Hints?

I've seen this too, it doesn't have anything to do with bonding, it's the
fact that you are sending packets out several interfaces at once that's
the cause. Same thing happend to me when transmitting at high speeds on
two interfaces at once without bonding. And you are using 4 interfaces so
I can imagine that it will be even worse than I saw.

And bonding on my two eepro100 in another machine works perfectly,
no problem maxing out at aproximatly 200Mbit.

/Martin

> here is my setup for the channel bonding
> 
> insmod bonding
> insmod tulip
> ifconfig bond0 192.168.100.2 netmask 255.255.255.0 broadcast
> 192.168.100.255
> ifconfig bond0 down
> ./ifenslave  -v bond0 eth2 eth3 eth4 eth5
> ifconfig bond0 up
> route add -host 192.168.100.1 dev bond0
> 
> 
> Many thanks for every help
> 
> Bernhard
> 
> --
> Dr. Bernhard Busch
> Max-Planck-Institut für Biochemie
> Am Klopferspitz 18a
> D-82152 Martinsried
> Tel: +48(89)8578-2582
> Fax: +49(89)8578-2479
> Email bbusch@biochem.mpg.de
> 
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Linux hackers are funny people: They count the time in patchlevels.

Re: Poor Performance for ethernet bonding

[Ben Greear]

Martin Josefsson wrote:
> 
> On Fri, 24 Aug 2001, Bernhard Busch wrote:
> 
> > Hi
> >
> >
> > I have tried to use ethernet  network interfaces bonding to increase
> > peformance.
> >
> > Bonding is working fine, but the performance is rather poor.
> > FTP between 2 machines ( kernel 2.4.4 and 4 port DLink 100Mbit ethernet
> > card)
> > results in a transfer rate of 3MB/s).
> >
> > Any Hints?
> 
> I've seen this too, it doesn't have anything to do with bonding, it's the
> fact that you are sending packets out several interfaces at once that's
> the cause. Same thing happend to me when transmitting at high speeds on
> two interfaces at once without bonding. And you are using 4 interfaces so
> I can imagine that it will be even worse than I saw.
> 
> And bonding on my two eepro100 in another machine works perfectly,
> no problem maxing out at aproximatly 200Mbit.

I've run a sustained 10Mbps tx & rx on 8 ports (2 of the Dlink 4-port NICs)
for over 24 hours on a 2.4.7 kernel.  I was not using bonding though.  I'm guessing you're
seeing lots of carrier errors, as the tulip driver used to be very bad
at...uh..working.  Try a 2.4.6 kernel or better and I bet your problem
goes away...

Ben

-- 
Ben Greear <greearb@candelatech.com>          <Ben_Greear@excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear