Use of 802.3ad bonding for increasing link throughput

Use of 802.3ad bonding for increasing link throughput

[Tom Brown]

[couldn't thread with '802.3ad bonding brain damaged', as I've just 
signed up]

So, under what circumstances would a user actually use 802.3ad mode to 
"increase" link throughput, rather than just for redundancy? Are there 
any circumstances in which a single file, for example, could be 
transferred at multiple-NIC speed? The 3 hashing options are:

- layer 2: presumably this always puts traffic on the same NIC, even in 
a LAG with multiple NICs? Should layer 2 ever be used?

- layer2+3: can't be used for a single file, since it still hashes to 
the same NIC, and can't be used for load-balancing, since different IP 
endpoints go unintelligently to different NICs

- layer3+4: seems to have exactly the same issue as layer2+3, as well as 
being non-compliant

I guess my problem is in understanding whether the 802.3/802.1AX spec 
has any use at all beyond redundancy. Given the requirement to maintain 
frame order at the distributor, I can't immediately see how having a 
bonded group of, say, 3 NICs is any better than having 3 separate NICs. 
Have I missed something obvious?

And, having said that, the redundancy features seem limited. For hot 
standby, when the main link fails, you have to wait for both ends to 
timeout, and re-negotiate via LACP, and hopefully pick up the same 
lower-priority NIC, and then rely on a higher layer to request 
retransmission of the missing frame. Do any of you have any experience 
of using 802.1AX for anything useful and non-trivial?

So, to get multiple-NIC speed, are we stuck with balance-rr? But 
presumably this only works if the other end of the link is also running 
the bonding driver?

Thanks -

Tom

Re: Use of 802.3ad bonding for increasing link throughput

[Chris Adams]

Once upon a time, Tom Brown <sa212+glibc@cyconix.com> said:
> So, under what circumstances would a user actually use 802.3ad mode to 
> "increase" link throughput, rather than just for redundancy? Are there 
> any circumstances in which a single file, for example, could be 
> transferred at multiple-NIC speed? The 3 hashing options are:

It isn't going to increase the rate for a single stream.  However, few
setups have only a single TCP stream going across a segment, so this is
still quite useful for real-world setups to increase the total
throughput.

-- 
Chris Adams <cmadams@hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

Re: Use of 802.3ad bonding for increasing link throughput

[Simon Farnsworth]

Tom Brown wrote:

> [couldn't thread with '802.3ad bonding brain damaged', as I've just
> signed up]
> 
> So, under what circumstances would a user actually use 802.3ad mode to
> "increase" link throughput, rather than just for redundancy? Are there
> any circumstances in which a single file, for example, could be
> transferred at multiple-NIC speed? The 3 hashing options are:
> 
As an example, from my server room here; I have an install server (TFTP, FTP 
and HTTP) connected by a 2x1G LACP bond to the switch. When I have multiple 
clients installing simultaneously, the layer2 hash distributes the load 
nicely across both NICs - I can reach saturation on both NICs together.

If I had routers between my clients and the install server, I'd need 
layer2+3 hashing to spread the clients over the links, but I'd still be able 
to push over a gigabit per second to the clients, despite being limited to 
1GBit/s to each individual client by the packet distribution.

I'm sure that you can think of lots of other situations in which you have 
multiple conversations sharing a link - those are the situations that gain 
speed from 802.3ad.
-- 
Simon Farnsworth

Re: Use of 802.3ad bonding for increasing link throughput

[Jay Vosburgh]

Tom Brown <sa212+glibc@cyconix.com> wrote:

>[couldn't thread with '802.3ad bonding brain damaged', as I've just signed
>up]
>
>So, under what circumstances would a user actually use 802.3ad mode to
>"increase" link throughput, rather than just for redundancy? Are there any
>circumstances in which a single file, for example, could be transferred at
>multiple-NIC speed? 

	Network load balancing, by and large, increases throughput in
aggregate, not for individual connections.

[...] The 3 hashing options are:
>
>- layer 2: presumably this always puts traffic on the same NIC, even in a
>LAG with multiple NICs? Should layer 2 ever be used?

	Perhaps the network is such that the destinations are not
bonded, and can't handle more than 1 interface's worth of throughput.
Having the "server" end bonded still permits the clients deal with a
single IP address, handle failures of devices on the server, etc.

>- layer2+3: can't be used for a single file, since it still hashes to the
>same NIC, and can't be used for load-balancing, since different IP
>endpoints go unintelligently to different NICs
>
>- layer3+4: seems to have exactly the same issue as layer2+3, as well as
>being non-compliant
>
>I guess my problem is in understanding whether the 802.3/802.1AX spec has
>any use at all beyond redundancy. Given the requirement to maintain frame
>order at the distributor, I can't immediately see how having a bonded
>group of, say, 3 NICs is any better than having 3 separate NICs. Have I
>missed something obvious?

	Others have answered this part already (that it permits larger
aggregate throughput to/from the host, but not single-stream throughput
greater than one interface's worth).  This is by design, to prevent out
of order delivery of packets.

	An aggregate of N devices can be better than 3 individual
devices in that it will gracefully handle failure of one of the devices
in the aggregate, and permits sharing of the bandwidth in aggregate
without the peers having to be hard-coded to specific destinations.

>And, having said that, the redundancy features seem limited. For hot
>standby, when the main link fails, you have to wait for both ends to
>timeout, and re-negotiate via LACP, and hopefully pick up the same
>lower-priority NIC, and then rely on a higher layer to request
>retransmission of the missing frame. Do any of you have any experience of
>using 802.1AX for anything useful and non-trivial?

	In the linux implementation, as soon as the link goes down, that
port is removed from the aggregator and a new aggregator is selected
(which may be the same aggregator, depending on the option and
configuration).  Language in 802.1AX section 5.3.13 permits us to
immediately remove a failed port from an aggregator without waiting for
LACP to time out.

>So, to get multiple-NIC speed, are we stuck with balance-rr? But
>presumably this only works if the other end of the link is also running
>the bonding driver?

	Striping a single connection across multiple network interfaces
is very difficult to do without causing packets to be delivered out of
order.

	Now, that said, if you want to have one TCP connection utilize
more than one interface's worth of throughput, then yes, balance-rr is
the only mode that may do that.  The other end doesn't have to run
bonding, but it must have sufficient aggregate bandwidth to accomodate
the aggregate rate (e.g., N slower devices feeding into one faster
device).

	Running balance-rr itself can be tricky to configure.  An
unmanaged switch may not handle multiple ports with the same MAC address
very well (e.g., sending everything to one port, or sending everything
to all the ports).  A managed switch must have the relevant ports
configured for etherchannel ("static link aggregation" in some
documentation), and the switch may balance the traffic when it leaves
the switch using its transmit algorithm.  I'm not aware of any switches
that have a round-robin balance policy, so the switch may end up hashing
your traffic anyway (which will probably drop some of your packets,
because you're feeding them in faster than the switch can send them out
after they're hashed to one switch port).

	It's possible to play games on managed switches and, e.g., put
each pair of ports (one at each end) into a separate VLAN, but schemes
like that will fail badly if a link goes down somewhere.

	If each member of the bond goes through a different unmanaged
and not interconnected switch, that may avoid those issues (and this was
a common configuration back in the 10 Mb/sec days; it's described in
bonding.txt in more detail).  That configuration still has issues if a
link fails.  Connecting systems directly, back-to-back, should also
avoid those issues.

	Lastly, balance-rr will deliver traffic out of order.  Even the
best case, N slow links feeding one faster link, delivers some small
percentage out of order (in the low single digits).

	On linux, the tcp_reordering sysctl value can be raised to
compensate, but it will still result in increased packet overhead, and
is not likely to be very efficient, and doesn't help with anything
that's not TCP/IP.  I have not tested balance-rr in a few years now, but
my recollection is that, as a best case, throughput of one TCP
connection could reach about 1.5x with 2 slaves, or about 2.5x with 4
slaves (where the multipliers are in units of "bandwidth of one slave").

	-J

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com

Re: Use of 802.3ad bonding for increasing link throughput

[Simon Horman]

On Wed, Aug 10, 2011 at 10:46:12AM -0700, Jay Vosburgh wrote:

[snip]

> 	On linux, the tcp_reordering sysctl value can be raised to
> compensate, but it will still result in increased packet overhead, and
> is not likely to be very efficient, and doesn't help with anything
> that's not TCP/IP.  I have not tested balance-rr in a few years now, but
> my recollection is that, as a best case, throughput of one TCP
> connection could reach about 1.5x with 2 slaves, or about 2.5x with 4
> slaves (where the multipliers are in units of "bandwidth of one slave").

Hi Jay,

for what it is worth I would like to chip in with the results of some
testing I did using ballance-rr and 3 gigabit NICs late last year.  The
link was three direct ("cross-over") cables to a machine that was also
using balance-rr.


I found that by increasing both rx-usecs (from 3 to 45) and enabling GRO
and TSO I was able to push 2.7*10^9 bits/s.

Local CPU utilisation was 30% and remote CPU utilisation was 10%.
Local service demand was 1.7 us/KB and remote service demand was 2.2us/KB.

The MTU was 1500 bytes.

In this configuration, with the tuning options described above, increasing
tcp_reordering (to 127) did not have a noticable effect on throughput but
did increase local CPU utilisation to about 50% and local service demand to
3.0 us/KB.  There was also increased remote CPU utilisation and service
demand, although not as significant.


By using an 9000 byte MTU I was able to get close to 3*10^9 bits/s
with other parameters at their default values.

Local CPU utilisation was 15% and remote CPU utilisation was 5%.
Local service demand was 0.8us/KB and remote service demand was 1.1us/KB.


Increasing rx-usecs was suggested to me by Eric Dumazet on this list.

I no longer have access to the systems that I used to run these tests but I
do have other results that I have omitted from this email for the sake of
brevity.


Anecdotally my opinion after running these and other tests is that if you
want to push more than a  gigabit/s over a single TCP stream then you would
be well advised to get a faster link rather than bond gigabit devices.  I
believe you stated something similar earlier on in this thread.