[PATCH net 0/2] net/bonding: fixes related to neigh

[PATCH net 0/2] net/bonding: fixes related to neigh

[Or Gerlitz]

From: Shlomo Pongratz <shlomop@mellanox.com>

Dave, 

This small patch set fixes some issues we stepped on when preparing 
for some IPoIB changes. 

With the 1st patch, the kernel networking code of arp_netdev_event calls 
neigh_changeaddr which further flashes the boding neighbours in some more 
cases which weren't covered by commit 7d26bb103 "bonding: emit event when 
bonding changes MAC"

The 2nd patch fixes a bug where under bonding, the IPoIB neigh cleanup
function wasn't called.

Or.


Shlomo Pongratz (2):
  net/bonding: emit address change event in more places
  net/bonding: correctly proxy slave neigh param setup ndo function

 drivers/net/bonding/bond_main.c |   56 +++++++++++++++++++++++++++++++++-----
 1 files changed, 48 insertions(+), 8 deletions(-)

[PATCH net 1/2] net/bonding: emit address change event in more places

[Or Gerlitz]

From: Shlomo Pongratz <shlomop@mellanox.com>

commit 7d26bb103c4 "bonding: emit event when bonding changes MAC" didn't
take care to emit the NETDEV_CHANGEADDR event in other places where bonding
actually changes the mac address (to all zeroes), in bond_release, and
bond_release_all. As a result the neighbours aren't deleted by the core
networking code (which does so upon getting that event).

Signed-off-by: Shlomo Pongratz <shlomop@mellanox.com>
---
 drivers/net/bonding/bond_main.c |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)

diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
index a20b585..b0a278d 100644
--- a/drivers/net/bonding/bond_main.c
+++ b/drivers/net/bonding/bond_main.c
@@ -2035,6 +2035,9 @@ int bond_release(struct net_device *bond_dev, struct net_device *slave_dev)
 	write_unlock_bh(&bond->lock);
 	unblock_netpoll_tx();
 
+	if (bond->slave_cnt == 0)
+		call_netdevice_notifiers(NETDEV_CHANGEADDR, bond->dev);
+
 	bond_compute_features(bond);
 	if (!(bond_dev->features & NETIF_F_VLAN_CHALLENGED) &&
 	    (old_features & NETIF_F_VLAN_CHALLENGED))
@@ -2218,6 +2221,8 @@ static int bond_release_all(struct net_device *bond_dev)
 out:
 	write_unlock_bh(&bond->lock);
 
+	call_netdevice_notifiers(NETDEV_CHANGEADDR, bond->dev);
+
 	bond_compute_features(bond);
 
 	return 0;
-- 
1.7.1

Re: [PATCH net 1/2] net/bonding: emit address change event in more places

[Jay Vosburgh]

Or Gerlitz <ogerlitz@mellanox.com> wrote:

>From: Shlomo Pongratz <shlomop@mellanox.com>
>
>commit 7d26bb103c4 "bonding: emit event when bonding changes MAC" didn't
>take care to emit the NETDEV_CHANGEADDR event in other places where bonding
>actually changes the mac address (to all zeroes), in bond_release, and
>bond_release_all. As a result the neighbours aren't deleted by the core
>networking code (which does so upon getting that event).

>Signed-off-by: Shlomo Pongratz <shlomop@mellanox.com>

	Is the bond_release_all notifier call actually necessary in your
testing?

	The bond itself is destroyed immediately after bond_release_all
returns, so I would expect the neighbours to be deleted then.  For that
path we probably don't need to zero the mac at all (or compute features,
for that matter).

	-J


>---
> drivers/net/bonding/bond_main.c |    5 +++++
> 1 files changed, 5 insertions(+), 0 deletions(-)
>
>diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
>index a20b585..b0a278d 100644
>--- a/drivers/net/bonding/bond_main.c
>+++ b/drivers/net/bonding/bond_main.c
>@@ -2035,6 +2035,9 @@ int bond_release(struct net_device *bond_dev, struct net_device *slave_dev)
> 	write_unlock_bh(&bond->lock);
> 	unblock_netpoll_tx();
>
>+	if (bond->slave_cnt == 0)
>+		call_netdevice_notifiers(NETDEV_CHANGEADDR, bond->dev);
>+
> 	bond_compute_features(bond);
> 	if (!(bond_dev->features & NETIF_F_VLAN_CHALLENGED) &&
> 	    (old_features & NETIF_F_VLAN_CHALLENGED))
>@@ -2218,6 +2221,8 @@ static int bond_release_all(struct net_device *bond_dev)
> out:
> 	write_unlock_bh(&bond->lock);
>
>+	call_netdevice_notifiers(NETDEV_CHANGEADDR, bond->dev);
>+
> 	bond_compute_features(bond);
>
> 	return 0;
>-- 
>1.7.1

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com

Re: [PATCH net 1/2] net/bonding: emit address change event in more places

[Or Gerlitz]

On 4/4/2012 1:28 AM, Jay Vosburgh wrote:
> Is the bond_release_all notifier call actually necessary in your 
> testing? The bond itself is destroyed immediately after 
> bond_release_all returns, so I would expect the neighbours to be 
> deleted then. For that path we probably don't need to zero the mac at 
> all (or compute features, for that matter).

yep, you're right, will remove this extra emitting of event and resend, 
thanks.

Or.

[PATCH net 2/2] net/bonding: correctly proxy slave neigh param setup ndo function

[Or Gerlitz]

From: Shlomo Pongratz <shlomop@mellanox.com>

The current implemenation was buggy for slaves who use ndo_neigh_setup,
since the networking stack invokes the bonding device ndo entry (from
neigh_params_alloc) before any devices are enslaved, and the bonding
driver can't further delegate the call at that point in time. As a
result when bonding IPoIB devices, the neigh_cleanup hasn't been called.

Fix that by deferring the actual call into the slave ndo_neigh_setup
from the time the bonding neigh_setup is called.

Signed-off-by: Shlomo Pongratz <shlomop@mellanox.com>
---
 drivers/net/bonding/bond_main.c |   51 ++++++++++++++++++++++++++++++++------
 1 files changed, 43 insertions(+), 8 deletions(-)

diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
index b0a278d..2eed155 100644
--- a/drivers/net/bonding/bond_main.c
+++ b/drivers/net/bonding/bond_main.c
@@ -3707,17 +3707,52 @@ static void bond_set_multicast_list(struct net_device *bond_dev)
 	read_unlock(&bond->lock);
 }
 
-static int bond_neigh_setup(struct net_device *dev, struct neigh_parms *parms)
+static int bond_neigh_init(struct neighbour *n)
 {
-	struct bonding *bond = netdev_priv(dev);
+	struct bonding *bond = netdev_priv(n->dev);
 	struct slave *slave = bond->first_slave;
+	const struct net_device_ops *slave_ops;
+	struct neigh_parms parms;
+	int ret;
+
+	if (!slave)
+		return 0;
+
+	slave_ops = slave->dev->netdev_ops;
+
+	if (!slave_ops->ndo_neigh_setup)
+		return 0;
+
+	parms.neigh_setup = NULL;
+	parms.neigh_cleanup = NULL;
+	ret = slave_ops->ndo_neigh_setup(slave->dev, &parms);
+	if (ret)
+		return ret;
+
+	/*
+	 * must bind here to the slave clenaup. Since when last slave is removed
+	 * there will be no slave device to dereference in a bonding
+	 * neigh_cleanup function that we have could add.
+	 */
+	n->parms->neigh_cleanup = parms.neigh_cleanup;
+
+	/* Does slave implement neigh_setup ? */
+	if (!parms.neigh_setup)
+		return 0;
+
+	return parms.neigh_setup(n);
+}
+
+/*
+ * The bonding ndo_neigh_setup is called at init time beofre any
+ * slave exists. So we must declare proxy setup function which will
+ * be used at run time to resolve the actual slave neigh param setup.
+ */
+static int bond_neigh_setup(struct net_device *dev,
+			    struct neigh_parms *parms)
+{
+	parms->neigh_setup   = bond_neigh_init;
 
-	if (slave) {
-		const struct net_device_ops *slave_ops
-			= slave->dev->netdev_ops;
-		if (slave_ops->ndo_neigh_setup)
-			return slave_ops->ndo_neigh_setup(slave->dev, parms);
-	}
 	return 0;
 }
 
-- 
1.7.1

Re: [PATCH net 2/2] net/bonding: correctly proxy slave neigh param setup ndo function

[Jay Vosburgh]

Or Gerlitz <ogerlitz@mellanox.com> wrote:

>From: Shlomo Pongratz <shlomop@mellanox.com>
>
>The current implemenation was buggy for slaves who use ndo_neigh_setup,
>since the networking stack invokes the bonding device ndo entry (from
>neigh_params_alloc) before any devices are enslaved, and the bonding
>driver can't further delegate the call at that point in time. As a
>result when bonding IPoIB devices, the neigh_cleanup hasn't been called.
>
>Fix that by deferring the actual call into the slave ndo_neigh_setup
>from the time the bonding neigh_setup is called.
>
>Signed-off-by: Shlomo Pongratz <shlomop@mellanox.com>
>---
> drivers/net/bonding/bond_main.c |   51 ++++++++++++++++++++++++++++++++------
> 1 files changed, 43 insertions(+), 8 deletions(-)
>
>diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
>index b0a278d..2eed155 100644
>--- a/drivers/net/bonding/bond_main.c
>+++ b/drivers/net/bonding/bond_main.c
>@@ -3707,17 +3707,52 @@ static void bond_set_multicast_list(struct net_device *bond_dev)
> 	read_unlock(&bond->lock);
> }
>
>-static int bond_neigh_setup(struct net_device *dev, struct neigh_parms *parms)
>+static int bond_neigh_init(struct neighbour *n)
> {
>-	struct bonding *bond = netdev_priv(dev);
>+	struct bonding *bond = netdev_priv(n->dev);
> 	struct slave *slave = bond->first_slave;
>+	const struct net_device_ops *slave_ops;
>+	struct neigh_parms parms;
>+	int ret;
>+
>+	if (!slave)
>+		return 0;
>+
>+	slave_ops = slave->dev->netdev_ops;
>+
>+	if (!slave_ops->ndo_neigh_setup)
>+		return 0;
>+
>+	parms.neigh_setup = NULL;
>+	parms.neigh_cleanup = NULL;
>+	ret = slave_ops->ndo_neigh_setup(slave->dev, &parms);
>+	if (ret)
>+		return ret;
>+
>+	/*
>+	 * must bind here to the slave clenaup. Since when last slave is removed
>+	 * there will be no slave device to dereference in a bonding
>+	 * neigh_cleanup function that we have could add.
>+	 */
>+	n->parms->neigh_cleanup = parms.neigh_cleanup;

	I'd write this comment as:

	/* Assign slave's neigh_cleanup to neighbour in case cleanup is
	 * called after bond has been destroyed.  Assumes that all slaves
	 * utilize the same neigh_cleanup (true at this writing as only user
	 * is ipoib).
	 */

	I.e., this logic works only because there cannot currently be a
situation wherein two slaves have different neigh_cleanup functions
(including one slave with a neigh_cleanup, and another without).

>+	/* Does slave implement neigh_setup ? */
>+	if (!parms.neigh_setup)
>+		return 0;

	I don't think this comment is necessary.

	-J

>+
>+	return parms.neigh_setup(n);
>+}
>+
>+/*
>+ * The bonding ndo_neigh_setup is called at init time beofre any
>+ * slave exists. So we must declare proxy setup function which will
>+ * be used at run time to resolve the actual slave neigh param setup.
>+ */
>+static int bond_neigh_setup(struct net_device *dev,
>+			    struct neigh_parms *parms)
>+{
>+	parms->neigh_setup   = bond_neigh_init;
>
>-	if (slave) {
>-		const struct net_device_ops *slave_ops
>-			= slave->dev->netdev_ops;
>-		if (slave_ops->ndo_neigh_setup)
>-			return slave_ops->ndo_neigh_setup(slave->dev, parms);
>-	}
> 	return 0;
> }

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com

Re: [PATCH net 2/2] net/bonding: correctly proxy slave neigh param setup ndo function

[Or Gerlitz]

On 4/4/2012 1:53 AM, Jay Vosburgh wrote:
> Or Gerlitz<ogerlitz@mellanox.com>  wrote:
>
>> From: Shlomo Pongratz<shlomop@mellanox.com>
>>
>> The current implemenation was buggy for slaves who use ndo_neigh_setup,
>> since the networking stack invokes the bonding device ndo entry (from
>> neigh_params_alloc) before any devices are enslaved, and the bonding
>> driver can't further delegate the call at that point in time. As a
>> result when bonding IPoIB devices, the neigh_cleanup hasn't been called.
>>
>> Fix that by deferring the actual call into the slave ndo_neigh_setup
> >from the time the bonding neigh_setup is called.
>>
>> Signed-off-by: Shlomo Pongratz<shlomop@mellanox.com>
>> ---
>> drivers/net/bonding/bond_main.c |   51 ++++++++++++++++++++++++++++++++------
>> 1 files changed, 43 insertions(+), 8 deletions(-)
>>
>> diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
>> index b0a278d..2eed155 100644
>> --- a/drivers/net/bonding/bond_main.c
>> +++ b/drivers/net/bonding/bond_main.c
>> @@ -3707,17 +3707,52 @@ static void bond_set_multicast_list(struct net_device *bond_dev)
>> 	read_unlock(&bond->lock);
>> }
>>
>> -static int bond_neigh_setup(struct net_device *dev, struct neigh_parms *parms)
>> +static int bond_neigh_init(struct neighbour *n)
>> {
>> -	struct bonding *bond = netdev_priv(dev);
>> +	struct bonding *bond = netdev_priv(n->dev);
>> 	struct slave *slave = bond->first_slave;
>> +	const struct net_device_ops *slave_ops;
>> +	struct neigh_parms parms;
>> +	int ret;
>> +
>> +	if (!slave)
>> +		return 0;
>> +
>> +	slave_ops = slave->dev->netdev_ops;
>> +
>> +	if (!slave_ops->ndo_neigh_setup)
>> +		return 0;
>> +
>> +	parms.neigh_setup = NULL;
>> +	parms.neigh_cleanup = NULL;
>> +	ret = slave_ops->ndo_neigh_setup(slave->dev,&parms);
>> +	if (ret)
>> +		return ret;
>> +
>> +	/*
>> +	 * must bind here to the slave clenaup. Since when last slave is removed
>> +	 * there will be no slave device to dereference in a bonding
>> +	 * neigh_cleanup function that we have could add.
>> +	 */
>> +	n->parms->neigh_cleanup = parms.neigh_cleanup;
>
> 	I'd write this comment as:
>
> 	/* Assign slave's neigh_cleanup to neighbour in case cleanup is
> 	 * called after bond has been destroyed.  Assumes that all slaves
> 	 * utilize the same neigh_cleanup (true at this writing as only user
> 	 * is ipoib).
> 	 */
>
> 	I.e., this logic works only because there cannot currently be a
> situation wherein two slaves have different neigh_cleanup functions
> (including one slave with a neigh_cleanup, and another without).

Jay, we do need that proxy-ing for the specific case of deleting the 
last slave, since in bond_release
the address change and the event emission happen --after-- calling 
bond_detach_slave. Still, will pick
your phrasing for the comment and replace "after bond has been 
destroyed" with "after last slave has been detached"

>
> +	/* Does slave implement neigh_setup ? */
> +	if (!parms.neigh_setup)
> +		return 0;
>
> 	I don't think this comment is necessary.

okay, will remove

Or.

Re: [PATCH net 2/2] net/bonding: correctly proxy slave neigh param setup ndo function

[Jay Vosburgh]

Or Gerlitz <ogerlitz@mellanox.com> wrote:

>On 4/4/2012 1:53 AM, Jay Vosburgh wrote:
>> Or Gerlitz<ogerlitz@mellanox.com>  wrote:
>>
>>> From: Shlomo Pongratz<shlomop@mellanox.com>
>>>
>>> The current implemenation was buggy for slaves who use ndo_neigh_setup,
>>> since the networking stack invokes the bonding device ndo entry (from
>>> neigh_params_alloc) before any devices are enslaved, and the bonding
>>> driver can't further delegate the call at that point in time. As a
>>> result when bonding IPoIB devices, the neigh_cleanup hasn't been called.
>>>
>>> Fix that by deferring the actual call into the slave ndo_neigh_setup
>> >from the time the bonding neigh_setup is called.
>>>
>>> Signed-off-by: Shlomo Pongratz<shlomop@mellanox.com>
>>> ---
>>> drivers/net/bonding/bond_main.c |   51 ++++++++++++++++++++++++++++++++------
>>> 1 files changed, 43 insertions(+), 8 deletions(-)
>>>
>>> diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
>>> index b0a278d..2eed155 100644
>>> --- a/drivers/net/bonding/bond_main.c
>>> +++ b/drivers/net/bonding/bond_main.c
>>> @@ -3707,17 +3707,52 @@ static void bond_set_multicast_list(struct net_device *bond_dev)
>>> 	read_unlock(&bond->lock);
>>> }
>>>
>>> -static int bond_neigh_setup(struct net_device *dev, struct neigh_parms *parms)
>>> +static int bond_neigh_init(struct neighbour *n)
>>> {
>>> -	struct bonding *bond = netdev_priv(dev);
>>> +	struct bonding *bond = netdev_priv(n->dev);
>>> 	struct slave *slave = bond->first_slave;
>>> +	const struct net_device_ops *slave_ops;
>>> +	struct neigh_parms parms;
>>> +	int ret;
>>> +
>>> +	if (!slave)
>>> +		return 0;
>>> +
>>> +	slave_ops = slave->dev->netdev_ops;
>>> +
>>> +	if (!slave_ops->ndo_neigh_setup)
>>> +		return 0;
>>> +
>>> +	parms.neigh_setup = NULL;
>>> +	parms.neigh_cleanup = NULL;
>>> +	ret = slave_ops->ndo_neigh_setup(slave->dev,&parms);
>>> +	if (ret)
>>> +		return ret;
>>> +
>>> +	/*
>>> +	 * must bind here to the slave clenaup. Since when last slave is removed
>>> +	 * there will be no slave device to dereference in a bonding
>>> +	 * neigh_cleanup function that we have could add.
>>> +	 */
>>> +	n->parms->neigh_cleanup = parms.neigh_cleanup;
>>
>> 	I'd write this comment as:
>>
>> 	/* Assign slave's neigh_cleanup to neighbour in case cleanup is
>> 	 * called after bond has been destroyed.  Assumes that all slaves
>> 	 * utilize the same neigh_cleanup (true at this writing as only user
>> 	 * is ipoib).
>> 	 */
>>
>> 	I.e., this logic works only because there cannot currently be a
>> situation wherein two slaves have different neigh_cleanup functions
>> (including one slave with a neigh_cleanup, and another without).
>
>Jay, we do need that proxy-ing for the specific case of deleting the last
>slave, since in bond_release
>the address change and the event emission happen --after-- calling
>bond_detach_slave. Still, will pick
>your phrasing for the comment and replace "after bond has been destroyed"
>with "after last slave has been detached"

	Yes, I understand that the proxying is needed; the point of the
comment is that if there's ever a situation in the future that two
slaves have different neigh_cleanup functions, this methodology will not
work.  There is no guarantee that the slave on which ndo_neigh_setup is
called on will also be the last slave to be removed.

	The change to the comment is ok; I was thinking about ipoib
always destroying the bond itself immediately after releasing the final
slave, so for ipoib, the two events always happen together.

	-J

>>
>> +	/* Does slave implement neigh_setup ? */
>> +	if (!parms.neigh_setup)
>> +		return 0;
>>
>> 	I don't think this comment is necessary.
>
>okay, will remove
>
>Or.
>

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com

Re: [PATCH net 0/2] net/bonding: fixes related to neigh

[Or Gerlitz]

On Mon, Apr 2, 2012 at 7:17 PM, Or Gerlitz <ogerlitz@mellanox.com> wrote:
> This small patch set fixes some issues we stepped on when preparing
> for some IPoIB changes.
>
> With the 1st patch, the kernel networking code of arp_netdev_event calls
> neigh_changeaddr which further flashes the boding neighbours in some more
> cases which weren't covered by commit 7d26bb103 "bonding: emit event when
> bonding changes MAC"
>
> The 2nd patch fixes a bug where under bonding, the IPoIB neigh cleanup
> function wasn't called.

Hi Jay,

Could you have a look on these patches, its important for us to know
if we're on the right direction.

Or.