Re: bonding vs 802.3ad/Cisco EtherChannel link agregation

[Boris Protopopov <borisp@mc.com>]

Hi, Scott, David, thanks for your replies, 
I agree that it seems that I bonding does not help with a single conection (btw,
is this a single TCP conneciton, or an ordered pair of IP addresses, or an
ordered pair of MAC addresses ?). My question is: why ? 

Scott, I am not sure I understand your analogy. It seems that you suggest that I
cannot saturate GigE with a single hose (a single Netpipe connection, I am
guessing). My experiments, however, indicate that I can - I get ~900Mbits/sec
out of 1000Mbits/sec with 20% CPU utilization when I use a single Netpipe
connection. I am guessing, the ~10% underutilization is caused by the protocol
overheads (headers/trailers). I added another GigE link, bonded it with the
first one, and repeated the experiment. I saw the ethernet frames evenly
distributed between the two links (ifconfig statistics), which, in my opinion,
indicated that a single Netpipe connection was in fact using both links. The CPU
utilization rose a bit, but it was nowhere near even 50%. The memory subsystem
seems to be able to handle over 1000Mbytes/sec, the peripheral bus is 100Mhz
64-bit PCI-X (handles 800Mbytes/sec). So, obviously, there is a bottneck
somewhere, but I don't understand where. I think my CPU/memory/I/O hose is big
enough to fill up two GigE links (~200Mbytes/sec), however, this is not
happening. My question is: why ?

I know why this is happens when I use teaming: because, on each side, one GigE
link is dedicated to sending, and the other one - to receiving (again, I use
ifconfig statistics to see what's happening). Acording to what I was told, this
is because 802.3ad and EtherChannel insist that all GigE frames that belong to a
"conversation" (I was unable to find a technical definition of the
"conversation" so far) are delivered in order. I would like to understand what a
"conversation" is, and why TCP/IP could not handle out-of-order delivery in it's
usual manner. I am guessing, because it would be too slow or would incur too
much CPU overhead. I was wondering if someone knew for sure :)

Thanks again, hope to get more opinions (it seems I am not alone :)),
Boris.


"Feldman, Scott" wrote:
> 
> > Bonding does not help with single stream performance.
> > You have to have multiple apps generating multiple streams
> > of data before you'll realize any improvement.
> > Therefore netpipe is a bad test for what you're doing.
> 
> The analogy that I like is to imagine a culvert under your driveway and you
> want to fill up the ditch on the other side, so you stick your garden hose
> in the culvert.  The rate of water flow is good, but you're just not
> utilizing the volume (bandwidth) of the culvert.  So you stick your
> neighbors garden hose in the culvert.  And so on.  Now the ditch is filling
> up.
> 
> So stick a bunch of garden hoses (streams) into that culvert (gigabit) and
> flood it to the point of saturation, and now measure the efficiency of the
> system (CPU %)  How much CPU is left to do other useful work?  The lower the
> CPU utilization, the higher the efficiency of the system.
> 
> Ok, the analogy is corny, and it doesn't have anything to do with bonding,
> but you'd be surprise how often this question comes up.
> 
> -scott
> -
> To unsubscribe from this list: send the line "unsubscribe linux-net" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html