bonding and SR-IOV -- do we need arp_validation for loadbalancing too?

bonding and SR-IOV -- do we need arp_validation for loadbalancing too?

[Chris Friesen]

Hi all,

We've been starting to look at bonding VFs from separate physical 
devices in a guest, but we've run into a problem.

The host is bonding the corresponding PFs, and it uses arp monitoring. 
  What we have found is that any broadcast traffic from the guest (if 
they enable arp monitoring, for example) will be seen by the internal L2 
switch of the NIC and sent up into the host, where the bonding driver 
will count it as incoming packets and use it to mark the link as good.

The only solutions I've been able to come up with are:
1) add arp validation for load balancing modes as well as active-backup.
2) put all the VMs in VLANs

Anyone have any better ideas?

Chris


-- 

Chris Friesen
Software Designer

3500 Carling Avenue
Ottawa, Ontario K2H 8E9
www.genband.com

Re: bonding and SR-IOV -- do we need arp_validation for loadbalancing too?

[Jiri Pirko]

Tue, Jul 24, 2012 at 05:57:03PM CEST, chris.friesen@genband.com wrote:
>Hi all,
>
>We've been starting to look at bonding VFs from separate physical
>devices in a guest, but we've run into a problem.
>
>The host is bonding the corresponding PFs, and it uses arp
>monitoring.  What we have found is that any broadcast traffic from
>the guest (if they enable arp monitoring, for example) will be seen
>by the internal L2 switch of the NIC and sent up into the host, where
>the bonding driver will count it as incoming packets and use it to
>mark the link as good.
>
>The only solutions I've been able to come up with are:
>1) add arp validation for load balancing modes as well as active-backup.

This is my favourite.... No reason to not to turn arp validation on.
TEAM device (teamd arpping linkwatch) does arp or NSNA validation
always.

>2) put all the VMs in VLANs
>
>Anyone have any better ideas?
>
>Chris
>
>
>-- 
>
>Chris Friesen
>Software Designer
>
>3500 Carling Avenue
>Ottawa, Ontario K2H 8E9
>www.genband.com
>
>--
>To unsubscribe from this list: send the line "unsubscribe netdev" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bonding and SR-IOV -- do we need arp_validation for loadbalancing too?

[Jay Vosburgh]

Jiri Pirko <jiri@resnulli.us> wrote:

>Tue, Jul 24, 2012 at 05:57:03PM CEST, chris.friesen@genband.com wrote:
>>Hi all,
>>
>>We've been starting to look at bonding VFs from separate physical
>>devices in a guest, but we've run into a problem.
>>
>>The host is bonding the corresponding PFs, and it uses arp
>>monitoring.  What we have found is that any broadcast traffic from
>>the guest (if they enable arp monitoring, for example) will be seen
>>by the internal L2 switch of the NIC and sent up into the host, where
>>the bonding driver will count it as incoming packets and use it to
>>mark the link as good.
>>
>>The only solutions I've been able to come up with are:
>>1) add arp validation for load balancing modes as well as active-backup.
>
>This is my favourite.... No reason to not to turn arp validation on.
>TEAM device (teamd arpping linkwatch) does arp or NSNA validation
>always.

	How does that operate for a load balancing mode?

	For arp validate to function (as it's implemented in bonding),
the arp requests (broadcasts) or the arp replies (unicasts) must be seen
by each slave at regular intervals.  Most load balance systems
(etherchannel or 802.3ad, for example) don't flood the broadcast
requests to all members of a channel group, and the unicast replies only
go to one member.

	This generally results in either only one slave staying up, or
slaves going up and down at odd intervals.  The arp monitor for the load
balance modes is already dependent upon there being a steady stream of
traffic to all slaves, and can be unreliable in low traffic conditions
(because not all slaves receive traffic with sufficient frequency).

	-J

>>2) put all the VMs in VLANs
>>
>>Anyone have any better ideas?

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com

Re: bonding and SR-IOV -- do we need arp_validation for loadbalancing too?

[Chris Friesen]

On 07/24/2012 12:13 PM, Jay Vosburgh wrote:
> Jiri Pirko<jiri@resnulli.us>  wrote:
>
>> Tue, Jul 24, 2012 at 05:57:03PM CEST, chris.friesen@genband.com wrote:
>>> Hi all,
>>>
>>> We've been starting to look at bonding VFs from separate physical
>>> devices in a guest, but we've run into a problem.
>>>
>>> The host is bonding the corresponding PFs, and it uses arp
>>> monitoring.  What we have found is that any broadcast traffic from
>>> the guest (if they enable arp monitoring, for example) will be seen
>>> by the internal L2 switch of the NIC and sent up into the host, where
>>> the bonding driver will count it as incoming packets and use it to
>>> mark the link as good.
>>>
>>> The only solutions I've been able to come up with are:
>>> 1) add arp validation for load balancing modes as well as active-backup.
>> This is my favourite.... No reason to not to turn arp validation on.
>> TEAM device (teamd arpping linkwatch) does arp or NSNA validation
>> always.
> 	How does that operate for a load balancing mode?
>
> 	For arp validate to function (as it's implemented in bonding),
> the arp requests (broadcasts) or the arp replies (unicasts) must be seen
> by each slave at regular intervals.  Most load balance systems
> (etherchannel or 802.3ad, for example) don't flood the broadcast
> requests to all members of a channel group, and the unicast replies only
> go to one member.
>
> 	This generally results in either only one slave staying up, or
> slaves going up and down at odd intervals.  The arp monitor for the load
> balance modes is already dependent upon there being a steady stream of
> traffic to all slaves, and can be unreliable in low traffic conditions
> (because not all slaves receive traffic with sufficient frequency).

In loadbalance mode wouldn't it just work similar to active-backup?  If 
it's a reply then verify that it came from the arp target, if it's a 
request then check to see if it came from one of the other slaves.

In our case we have control over the L2 switches involved so we ensure 
that the broadcast arp request is sent to all the other slaves, while 
the reply comes back to the sender.  I think we still have a window 
where you could have a device with a faulty tx but functional rx and 
never detect the problem in the monitor.

A more general solution might be to have the device driver also track 
the time of the last incoming packet that came from the external network 
(rather than a VF) and having the bond driver ignore those packets for 
the purpose of link health.  Doing this efficiently would likely require 
some kind of hardware support though--as an example the 82599 seems to 
support this with the "LB" bit in the rx descriptor.

Chris

Re: bonding and SR-IOV -- do we need arp_validation for loadbalancing too?

[Chris Friesen]

On 07/24/2012 02:18 PM, Chris Friesen wrote:
>
> A more general solution might be to have the device driver also track 
> the time of the last incoming packet that came from the external 
> network (rather than a VF) and having the bond driver ignore those 
> packets for the purpose of link health.  Doing this efficiently would 
> likely require some kind of hardware support though--as an example the 
> 82599 seems to support this with the "LB" bit in the rx descriptor.

That should of course be reversed.  We want the bond driver to only use 
the packets from the external network for the purpose of link health.

Does anyone other than bonding actually care about dev->last_rx?  If not 
then we could just change the drivers to only set it for external packets.

Chris

Re: bonding and SR-IOV -- do we need arp_validation for loadbalancing too?

[Jay Vosburgh]

Chris Friesen <chris.friesen@genband.com> wrote:

>On 07/24/2012 12:13 PM, Jay Vosburgh wrote:
>> Jiri Pirko<jiri@resnulli.us>  wrote:
>>
>>> Tue, Jul 24, 2012 at 05:57:03PM CEST, chris.friesen@genband.com wrote:
>>>> Hi all,
>>>>
>>>> We've been starting to look at bonding VFs from separate physical
>>>> devices in a guest, but we've run into a problem.
>>>>
>>>> The host is bonding the corresponding PFs, and it uses arp
>>>> monitoring.  What we have found is that any broadcast traffic from
>>>> the guest (if they enable arp monitoring, for example) will be seen
>>>> by the internal L2 switch of the NIC and sent up into the host, where
>>>> the bonding driver will count it as incoming packets and use it to
>>>> mark the link as good.
>>>>
>>>> The only solutions I've been able to come up with are:
>>>> 1) add arp validation for load balancing modes as well as active-backup.
>>> This is my favourite.... No reason to not to turn arp validation on.
>>> TEAM device (teamd arpping linkwatch) does arp or NSNA validation
>>> always.
>> 	How does that operate for a load balancing mode?
>>
>> 	For arp validate to function (as it's implemented in bonding),
>> the arp requests (broadcasts) or the arp replies (unicasts) must be seen
>> by each slave at regular intervals.  Most load balance systems
>> (etherchannel or 802.3ad, for example) don't flood the broadcast
>> requests to all members of a channel group, and the unicast replies only
>> go to one member.
>>
>> 	This generally results in either only one slave staying up, or
>> slaves going up and down at odd intervals.  The arp monitor for the load
>> balance modes is already dependent upon there being a steady stream of
>> traffic to all slaves, and can be unreliable in low traffic conditions
>> (because not all slaves receive traffic with sufficient frequency).
>
>In loadbalance mode wouldn't it just work similar to active-backup?  If
>it's a reply then verify that it came from the arp target, if it's a
>request then check to see if it came from one of the other slaves.

	The problem isn't verifying the requests or replies, it's that
the ARP packets are not distributed across all slaves (because the
switch ports are in a channel group / aggregator), so some slaves do not
receive any ARPs.

	The bond sends the ARP request as a broadcast.  For
active-backup, this ends up at the inactive slaves because the switch
sends the broadcast to all ports.  For a loadbalance mode, the switch
won't send the broadcast ARP to the other slaves, because all the slaves
are in a channel group or lacp aggregator, which is treated by the
switch as effectively a single switch port for this case.

	Similarly, the ARP replies are unicast, and the switch will send
those unicast replies to only one member of the channel group or
aggregator.  The choice there is usually a hash of some kind, so
generally only one slave will receive the replies.

>In our case we have control over the L2 switches involved so we ensure
>that the broadcast arp request is sent to all the other slaves, while the
>reply comes back to the sender.  I think we still have a window where you
>could have a device with a faulty tx but functional rx and never detect
>the problem in the monitor.

	You can set up -xor or -rr mode against a switch without setting
up a channel group on the switch, but that has the down side that any
incoming broadcast or multicast packet may be received multiple times
(one copy per slave).  Some switches will also disable ports (due to MAC
flapping) or complain about seeing the same MAC address on multiple
ports for this case.  This also will not load balance incoming traffic
to the bond very well.

>On 07/24/2012 02:18 PM, Chris Friesen wrote:
>> A more general solution might be to have the device driver also track
>> the time of the last incoming packet that came from the external network
>> (rather than a VF) and having the bond driver ignore those packets for
>> the purpose of link health.  Doing this efficiently would likely require
>> some kind of hardware support though--as an example the 82599 seems to
>> support this with the "LB" bit in the rx descriptor.
>
>That should of course be reversed.  We want the bond driver to only use
>the packets from the external network for the purpose of link health.
>
>Does anyone other than bonding actually care about dev->last_rx?  If not
>then we could just change the drivers to only set it for external packets.

	I believe bonding is the main user of last_rx (a search shows a
couple of drivers using it internally).  For bonding use, in current
mainline last_rx is set by bonding itself, not in the network device
driver.

	-J

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com

Re: bonding and SR-IOV -- do we need arp_validation for loadbalancing too?

[Nicolas de Pesloüan]

Le 24/07/2012 22:49, Jay Vosburgh a écrit :
[...]
>> In loadbalance mode wouldn't it just work similar to active-backup?  If
>> it's a reply then verify that it came from the arp target, if it's a
>> request then check to see if it came from one of the other slaves.
>
> 	The problem isn't verifying the requests or replies, it's that
> the ARP packets are not distributed across all slaves (because the
> switch ports are in a channel group / aggregator), so some slaves do not
> receive any ARPs.
>
> 	The bond sends the ARP request as a broadcast.  For
> active-backup, this ends up at the inactive slaves because the switch
> sends the broadcast to all ports.  For a loadbalance mode, the switch
> won't send the broadcast ARP to the other slaves, because all the slaves
> are in a channel group or lacp aggregator, which is treated by the
> switch as effectively a single switch port for this case.
>
> 	Similarly, the ARP replies are unicast, and the switch will send
> those unicast replies to only one member of the channel group or
> aggregator.  The choice there is usually a hash of some kind, so
> generally only one slave will receive the replies.

I assume team should suffer the exact same problem, because most of this is on the switch side and 
out of the control of the host. Jiri, can you confirm?

[...]

> 	I believe bonding is the main user of last_rx (a search shows a
> couple of drivers using it internally).  For bonding use, in current
> mainline last_rx is set by bonding itself, not in the network device
> driver.

If last_rx is set and used internally by bonding and mostly unused elsewhere, can't we remove it 
from net_device and move it into private data for the slaves in bonding?

A comment in netdevice.h even recommends not to set it into drivers:

         unsigned long           last_rx;        /* Time of last Rx
                                                  * This should not be set in
                                                  * drivers, unless really needed,
                                                  * because network stack (bonding)
                                                  * use it if/when necessary, to
                                                  * avoid dirtying this cache line.
                                                  */

	Nicolas.

Re: bonding and SR-IOV -- do we need arp_validation for loadbalancing too?

[Chris Friesen]

On 07/24/2012 02:49 PM, Jay Vosburgh wrote:
> Chris Friesen<chris.friesen@genband.com>  wrote:
>
>> In loadbalance mode wouldn't it just work similar to active-backup?  If
>> it's a reply then verify that it came from the arp target, if it's a
>> request then check to see if it came from one of the other slaves.
>
> 	The problem isn't verifying the requests or replies, it's that
> the ARP packets are not distributed across all slaves (because the
> switch ports are in a channel group / aggregator), so some slaves do not
> receive any ARPs.

Yeah, okay.  And if we turn on arp validation then we ignore all the 
other packets and so they looks dead.  Got it.

In our environment (ATCA shelf) the switches have been customized to 
handle some of this stuff so arpmon does work reliably with xor.

In the general case it sounds like the "PF bonding ignores packets from 
VFs" is a better bet then.


>> On 07/24/2012 02:18 PM, Chris Friesen wrote:
>>> A more general solution might be to have the device driver also track
>>> the time of the last incoming packet that came from the external network
>>> (rather than a VF) and having the bond driver ignore those packets for
>>> the purpose of link health.  Doing this efficiently would likely require
>>> some kind of hardware support though--as an example the 82599 seems to
>>> support this with the "LB" bit in the rx descriptor.
>>
>> That should of course be reversed.  We want the bond driver to only use
>> the packets from the external network for the purpose of link health.
>>
>> Does anyone other than bonding actually care about dev->last_rx?  If not
>> then we could just change the drivers to only set it for external packets.
> That should of course be reversed.  We want the bond driver to only use
>> the packets from the external network for the purpose of link health.
>>
> 	I believe bonding is the main user of last_rx (a search shows a
> couple of drivers using it internally).  For bonding use, in current
> mainline last_rx is set by bonding itself, not in the network device
> driver.

Right, I was looking at older code.  In that case presumably the driver 
could set an skb flag (external vs VF loopback) that the bonding code 
could check?

Chris

Re: bonding and SR-IOV -- do we need arp_validation for loadbalancing too?

[Andy Gospodarek]

On Tue, Jul 24, 2012 at 03:38:11PM -0600, Chris Friesen wrote:
> On 07/24/2012 02:49 PM, Jay Vosburgh wrote:
> >Chris Friesen<chris.friesen@genband.com>  wrote:
> >
> >>In loadbalance mode wouldn't it just work similar to active-backup?  If
> >>it's a reply then verify that it came from the arp target, if it's a
> >>request then check to see if it came from one of the other slaves.
> >
> >	The problem isn't verifying the requests or replies, it's that
> >the ARP packets are not distributed across all slaves (because the
> >switch ports are in a channel group / aggregator), so some slaves do not
> >receive any ARPs.
> 
> Yeah, okay.  And if we turn on arp validation then we ignore all the
> other packets and so they looks dead.  Got it.
If you wanted to make this work, I think it could pretty easily be done
if it isn't working right now.  Did you try arp_validate=all in your
setup?

> In our environment (ATCA shelf) the switches have been customized to
> handle some of this stuff so arpmon does work reliably with xor.
Good.

> In the general case it sounds like the "PF bonding ignores packets
> from VFs" is a better bet then.
It really might be.  There are some registers in the 82599 datasheets
that are not used by the ixgbe driver, but might help you in this area.

If you take a look at PFVML2FLT and PFUTA and their current status on
your system you might be able to put something together that gives you
what you want.

It would likely mean you have to run a custom ixgbe-driver, but that
doesn't sound like much of an issue.

> 
> 
> >>On 07/24/2012 02:18 PM, Chris Friesen wrote:
> >>>A more general solution might be to have the device driver also track
> >>>the time of the last incoming packet that came from the external network
> >>>(rather than a VF) and having the bond driver ignore those packets for
> >>>the purpose of link health.  Doing this efficiently would likely require
> >>>some kind of hardware support though--as an example the 82599 seems to
> >>>support this with the "LB" bit in the rx descriptor.
> >>
> >>That should of course be reversed.  We want the bond driver to only use
> >>the packets from the external network for the purpose of link health.
> >>
> >>Does anyone other than bonding actually care about dev->last_rx?  If not
> >>then we could just change the drivers to only set it for external packets.
> >That should of course be reversed.  We want the bond driver to only use
> >>the packets from the external network for the purpose of link health.
> >>
> >	I believe bonding is the main user of last_rx (a search shows a
> >couple of drivers using it internally).  For bonding use, in current
> >mainline last_rx is set by bonding itself, not in the network device
> >driver.
> 
> Right, I was looking at older code.  In that case presumably the
> driver could set an skb flag (external vs VF loopback) that the
> bonding code could check?
> 
> Chris
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bonding and SR-IOV -- do we need arp_validation for loadbalancing too?

[Chris Friesen]

On 07/27/2012 08:55 AM, Andy Gospodarek wrote:
> On Tue, Jul 24, 2012 at 03:38:11PM -0600, Chris Friesen wrote:

>> In our environment (ATCA shelf) the switches have been customized to
>> handle some of this stuff so arpmon does work reliably with xor.
> Good.
>
>> In the general case it sounds like the "PF bonding ignores packets
>> from VFs" is a better bet then.
> It really might be.  There are some registers in the 82599 datasheets
> that are not used by the ixgbe driver, but might help you in this area.
>
> If you take a look at PFVML2FLT and PFUTA and their current status on
> your system you might be able to put something together that gives you
> what you want.

I think it's simpler than that.  By my reading of the 82599 datasheet, 
the LB bit in the status field of the rx descriptor will indicate if the 
packet came from a VM.

> It would likely mean you have to run a custom ixgbe-driver, but that
> doesn't sound like much of an issue.


While I'd like to avoid it if possible, we've already had to tweak the 
driver for other things.

The complication is that we've got a few different types of hardware and 
they're not all running 82599.  I'm trying to figure out if the other 
hardware can do something similar.

Chris

Re: bonding and SR-IOV -- do we need arp_validation for loadbalancing too?

[Andy Gospodarek]

On Fri, Jul 27, 2012 at 10:15:45AM -0600, Chris Friesen wrote:
> On 07/27/2012 08:55 AM, Andy Gospodarek wrote:
> >On Tue, Jul 24, 2012 at 03:38:11PM -0600, Chris Friesen wrote:
> 
> >>In our environment (ATCA shelf) the switches have been customized to
> >>handle some of this stuff so arpmon does work reliably with xor.
> >Good.
> >
> >>In the general case it sounds like the "PF bonding ignores packets
> >>from VFs" is a better bet then.
> >It really might be.  There are some registers in the 82599 datasheets
> >that are not used by the ixgbe driver, but might help you in this area.
> >
> >If you take a look at PFVML2FLT and PFUTA and their current status on
> >your system you might be able to put something together that gives you
> >what you want.
> 
> I think it's simpler than that.  By my reading of the 82599
> datasheet, the LB bit in the status field of the rx descriptor will
> indicate if the packet came from a VM.
> 
Ah, you are correct.  I'm curious if this bit is also set on VF<->VF
traffic.

> >It would likely mean you have to run a custom ixgbe-driver, but that
> >doesn't sound like much of an issue.
> 
> 
> While I'd like to avoid it if possible, we've already had to tweak
> the driver for other things.
As long as driver resets are not needed for some of these changes to
take effect, there is a chance you can write some of these bits from a
userspace program.  I haven't done it myself, but I should try it.  That
might allow you to run a stock driver from a distro and still get what
you want.  

> The complication is that we've got a few different types of hardware
> and they're not all running 82599.  I'm trying to figure out if the
> other hardware can do something similar.
That would indicate that this would be something that the stack as well
as some of the stacked modules bonding, vlans, bridges, etc. may need to
have some special code to handle.  I'm not sure I'm a big fan right now,
but I'll hold off judgement until we see how this shakes out.