bonding questions: carrier based link monitoring / slave device state flags

bonding questions: carrier based link monitoring / slave device state flags

[Or Gerlitz]

First, i'd like to verify what is the parameter setting to have the
bonding driver use netif_carrier_ok(slave_device) as the means for
link detection. Is setting use_carrier = 1 enough or one needs to set
miimon to non-zero as well??? (where the value of miimon translates
to the link monitoring frequency).

Second, I understand that an enslaved device must **not** be UP, so
when enslaving a device the bonding driver calls dev_open(slave_device),
and make sure the device is UP, correct?

What i want to better understand here, is whether for the bonding driver
to declare a slave as "being able to carry traffic" it assumes the slave
will move from UP to RUNNING state (and later netif_carrier_ok would
return TRUE) without an IP address being set for the slave device ???

Is (per the bonding driver) the **time** it should take the slave to get
from UP to (RUNNING && carrier_ok) state limited and/or controlled?

Or.

Re: bonding questions: carrier based link monitoring / slave device state flags

[Jay Vosburgh]

Or Gerlitz <ogerlitz@voltaire.com> wrote:

>First, i'd like to verify what is the parameter setting to have the
>bonding driver use netif_carrier_ok(slave_device) as the means for
>link detection. Is setting use_carrier = 1 enough or one needs to set
>miimon to non-zero as well??? (where the value of miimon translates
>to the link monitoring frequency).

	I'll just note that all of the bonding parameters and such are
documented in the Documentation/networking/bonding.txt file that comes
with the kernel source code.  You can also download the documentation
from sourceforge.net/projects/bonding.

	To answer your question, you'd have to specify an miimon=X
interval (where X is some nonzero value, e.g., 100).  The default for
the use_carrier parameter is 1, which is what you want, so there is no
need to specify it.  The purpose of the use_carrier parameter is to
permit the old method to be used for network drivers that don't support
netif_carrier (which are pretty rare these days).

>Second, I understand that an enslaved device must **not** be UP, so
>when enslaving a device the bonding driver calls dev_open(slave_device),
>and make sure the device is UP, correct?

	Yes, the bonding driver does need to have the slave down during
part of the enslavement (to set the MAC address), but if the slave isn't
up, it will be set down.  Don't worry about it.

>What i want to better understand here, is whether for the bonding driver
>to declare a slave as "being able to carry traffic" it assumes the slave
>will move from UP to RUNNING state (and later netif_carrier_ok would
>return TRUE) without an IP address being set for the slave device ???

	The RUNNING state is set (indirectly) by the device driver when
it calls netif_carrier_on(); this is totally unrelated to the IP address
information attached to an interface.  Whatever IP address is set for a
slave is not used during operation.

>Is (per the bonding driver) the **time** it should take the slave to get
>from UP to (RUNNING && carrier_ok) state limited and/or controlled?

	I'm not exactly sure what you're asking here, but when miimon is
enabled, if the slave is not netif_carrier_ok() during any of the
periodic checks, it will be disabled by the bonding driver.

	There are also updelay and downdelay parameters to handle cases
where it takes some amount of real time for a carrier state transition
to occur (and, thus, state changes should be ignored for a period of
time).  E.g., some switches will assert link up on a port before they
are actually able to send and receive traffic; in that case, bonding
could use the updelay parameter to ignore link up changes for a period
of time.

	-J

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com

Re: bonding questions: carrier based link monitoring / slave device state flags

[Or Gerlitz]

Jay Vosburgh wrote:
> Or Gerlitz <ogerlitz@voltaire.com> wrote:

>         I'll just note that all of the bonding parameters and such are
> documented in the Documentation/networking/bonding.txt file that comes
> with the kernel source code.  You can also download the documentation
> from sourceforge.net/projects/bonding.

Just to be clear, i have looked in Documentation/networking/bonding.txt 
before sending this question.

>         To answer your question, you'd have to specify an miimon=X
> interval (where X is some nonzero value, e.g., 100).  The default for
> the use_carrier parameter is 1, which is what you want, so there is no
> need to specify it.  The purpose of the use_carrier parameter is to
> permit the old method to be used for network drivers that don't support
> netif_carrier (which are pretty rare these days).

With old method you refer to the bonding driver using MII ioctls etc vs 
just calling netif_carrier_ok().

>  >What i want to better understand here, is whether for the bonding driver
>  >to declare a slave as "being able to carry traffic" it assumes the slave
>  >will move from UP to RUNNING state (and later netif_carrier_ok would
>  >return TRUE) without an IP address being set for the slave device ???
> 
>         The RUNNING state is set (indirectly) by the device driver when
> it calls netif_carrier_on(); this is totally unrelated to the IP address
> information attached to an interface.  Whatever IP address is set for a
> slave is not used during operation.

OK, thanks for clarifying this.

>  >Is (per the bonding driver) the **time** it should take the slave to get
>  >from UP to (RUNNING && carrier_ok) state limited and/or controlled?
> 
>         I'm not exactly sure what you're asking here, but when miimon is
> enabled, if the slave is not netif_carrier_ok() during any of the
> periodic checks, it will be disabled by the bonding driver.

The transition from being UP to having carrier might involve third party 
and hence take some time (eg in IP over Infiniband the broadcast traffic 
is carried over a dedicated IB multicast group and joining this group is 
carried by communication with the Infiniband SA). But if the bonding 
code does periodic checks, this is fine.

>         There are also updelay and downdelay parameters to handle cases
> where it takes some amount of real time for a carrier state transition
> to occur (and, thus, state changes should be ignored for a period of
> time).  E.g., some switches will assert link up on a port before they
> are actually able to send and receive traffic; in that case, bonding
> could use the updelay parameter to ignore link up changes for a period
> of time.

OK understood.

Or.

bonding questions: "replaying" call to set_multicast_list and sending IGMP doing Fail-Over

[Or Gerlitz]

Another question that bothers me is the bonding code multicast related 
behavior when it does fail-over.

 From what I see in bond_mc_swap(), set_multicast_list() is well 
handled: dev_mc_delete() is called for the old slave (so if in the 
future the old slave has a link, it will leave the multicast group) and 
dev_mc_add() is called for the active slave (so the active slave joins 
the multicast group).

As for sending IGMP, in bond_xmit_activebackup() i see the following 
comment: "Xmit IGMP frames on all slaves to ensure rapid fail-over
for multicast traffic on snooping switches".

As i don't see any buffering of the IGMP packets, i understand there's 
no "reply" of sending them during fail-over and this means that only 
when the router would do IGMP query on this node it will learn on the 
fail-over. Is it indeed what's going on? if i understand correct it 
would take some meaningful time for the fail-over to be externally 
visible in this respect.

Also assuming it does exactly what the comment says, another issue i see 
here is that in the case of not only the active_slave being UP, the code 
would TX the IGMP packets over > 1 slave, and hence multicast packets 
would be sent by the switch also to ports "connected to" non-active 
slaves, something which will hurt the system performance!?

Or.

Re: bonding questions: "replaying" call to set_multicast_list and sending IGMP doing Fail-Over

[Jay Vosburgh]

Or Gerlitz <ogerlitz@voltaire.com> wrote:
[...]
>As i don't see any buffering of the IGMP packets, i understand there's no
>"reply" of sending them during fail-over and this means that only when the
>router would do IGMP query on this node it will learn on the fail-over. Is
>it indeed what's going on? if i understand correct it would take some
>meaningful time for the fail-over to be externally visible in this respect.

	As I recall, the intent is to keep switches up to date all the
time, so that there is no delay during a failover.  I.e., at every IGMP
query, the switch is answered on all possible ports.

>Also assuming it does exactly what the comment says, another issue i see
>here is that in the case of not only the active_slave being UP, the code
>would TX the IGMP packets over > 1 slave, and hence multicast packets
>would be sent by the switch also to ports "connected to" non-active
>slaves, something which will hurt the system performance!?

	I haven't studied the effects of having large amounts of
multicast traffic coming in under this situation.

	However, I would suspect that the MAC filters found on
sufficiently modern network adapters would drop the incoming multicast
traffic on the backup slaves, as only the active slave in active-backup
mode has its multicast list set.  That information is sent to a slave
when it becomes the active slave; see the call to bond_mc_swap() made by
bond_change_active_slave().

	-J

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com

Re: bonding questions: "replaying" call to set_multicast_list and sending IGMP doing Fail-Over

[Or Gerlitz]

Jay Vosburgh wrote:
> 	I haven't studied the effects of having large amounts of
> multicast traffic coming in under this situation.
> 
> 	However, I would suspect that the MAC filters found on
> sufficiently modern network adapters would drop the incoming multicast
> traffic on the backup slaves, as only the active slave in active-backup
> mode has its multicast list set.  That information is sent to a slave
> when it becomes the active slave; see the call to bond_mc_swap() made by
> bond_change_active_slave().

OK, i agree the MAC filter would drop the incoming traffic on the backup 
slaves b/c bond_mc_swap() calls bond_mc_delete() on the slave which 
becomes a backup one. But as you have noted there might be some impact 
on the switch.

Or.