[PATCH rdma-next 0/3] Fixes to set_netdev/get_netdev flow

[PATCH rdma-next 0/3] Fixes to set_netdev/get_netdev flow

[Leon Romanovsky]

This series fixes the set_netdev/get_netdev flow in mlx5 drive around LAG.

Thanks

Chiara Meiohas (3):
  RDMA/mlx5: Call dev_put() after the blocking notifier
  RDMA/core: Implement RoCE GID port rescan and export delete function
  RDMA/mlx5: Ensure active slave attachment to the bond IB device

 drivers/infiniband/core/roce_gid_mgmt.c       | 30 ++++++++++++++++---
 drivers/infiniband/hw/mlx5/main.c             | 29 +++++++++++-------
 .../net/ethernet/mellanox/mlx5/core/lag/lag.c | 12 ++++++++
 include/rdma/ib_verbs.h                       |  3 ++
 4 files changed, 59 insertions(+), 15 deletions(-)

-- 
2.46.2

[PATCH mlx5-next 1/3] RDMA/mlx5: Call dev_put() after the blocking notifier

[Leon Romanovsky]

From: Chiara Meiohas <cmeiohas@nvidia.com>

Move dev_put() call to occur directly after the blocking
notifier, instead of within the event handler.

Fixes: 8d159eb2117b ("RDMA/mlx5: Use IB set_netdev and get_netdev functions")
Signed-off-by: Chiara Meiohas <cmeiohas@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
 drivers/infiniband/hw/mlx5/main.c                 | 1 -
 drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c | 1 +
 2 files changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index b4476df96ed5..5f7fe32b9051 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -3242,7 +3242,6 @@ static int lag_event(struct notifier_block *nb, unsigned long event, void *data)
 			}
 			err = ib_device_set_netdev(&dev->ib_dev, ndev,
 						   portnum + 1);
-			dev_put(ndev);
 			if (err)
 				return err;
 			/* Rescan gids after new netdev assignment */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
index 8577db3308cc..d661267d98ff 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
@@ -516,6 +516,7 @@ void mlx5_modify_lag(struct mlx5_lag *ldev,
 		blocking_notifier_call_chain(&dev0->priv.lag_nh,
 					     MLX5_DRIVER_EVENT_ACTIVE_BACKUP_LAG_CHANGE_LOWERSTATE,
 					     ndev);
+		dev_put(ndev);
 	}
 }
 
-- 
2.46.2

[PATCH mlx5-next 3/3] RDMA/mlx5: Ensure active slave attachment to the bond IB device

[Leon Romanovsky]

From: Chiara Meiohas <cmeiohas@nvidia.com>

Fix a race condition when creating a lag bond in active backup
mode where after the bond creation the backup slave was
attached to the IB device, instead of the active slave.
This caused stale entries in the GID table, as the gid updating
mechanism relies on ib_device_get_netdev(), which would return
the backup slave.

Send an MLX5_DRIVER_EVENT_ACTIVE_BACKUP_LAG_CHANGE_LOWERSTATE
event when activating the lag, additionally to when modifying
the lag. This ensures that eventually the active netdevice is
stored in the bond IB device.
When handling this event remove the GIDs of the previously
attached netdevice in this port and rescan the GIDs of the
newly attached netdevice.

This ensures that eventually the active slave netdevice is
correctly stored in the IB device port. While there might be
a brief moment where the backup slave GIDs appear in the GID
table, it will eventually stabilize with the correct GIDs
(of the bond and the active slave).

Fixes: 8d159eb2117b ("RDMA/mlx5: Use IB set_netdev and get_netdev functions")
Signed-off-by: Chiara Meiohas <cmeiohas@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
 drivers/infiniband/hw/mlx5/main.c             | 28 ++++++++++++-------
 .../net/ethernet/mellanox/mlx5/core/lag/lag.c | 11 ++++++++
 2 files changed, 29 insertions(+), 10 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index 5f7fe32b9051..5038c52b79aa 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -3219,12 +3219,14 @@ static int lag_event(struct notifier_block *nb, unsigned long event, void *data)
 	struct mlx5_ib_dev *dev = container_of(nb, struct mlx5_ib_dev,
 					       lag_events);
 	struct mlx5_core_dev *mdev = dev->mdev;
+	struct ib_device *ibdev = &dev->ib_dev;
+	struct net_device *old_ndev = NULL;
 	struct mlx5_ib_port *port;
 	struct net_device *ndev;
-	int  i, err;
-	int portnum;
+	u32 portnum = 0;
+	int ret = 0;
+	int i;
 
-	portnum = 0;
 	switch (event) {
 	case MLX5_DRIVER_EVENT_ACTIVE_BACKUP_LAG_CHANGE_LOWERSTATE:
 		ndev = data;
@@ -3240,18 +3242,24 @@ static int lag_event(struct notifier_block *nb, unsigned long event, void *data)
 					}
 				}
 			}
-			err = ib_device_set_netdev(&dev->ib_dev, ndev,
-						   portnum + 1);
-			if (err)
-				return err;
-			/* Rescan gids after new netdev assignment */
-			rdma_roce_rescan_device(&dev->ib_dev);
+			old_ndev = ib_device_get_netdev(ibdev, portnum + 1);
+			ret = ib_device_set_netdev(ibdev, ndev, portnum + 1);
+			if (ret)
+				goto out;
+
+			if (old_ndev)
+				roce_del_all_netdev_gids(ibdev, portnum + 1,
+							 old_ndev);
+			rdma_roce_rescan_port(ibdev, portnum + 1);
 		}
 		break;
 	default:
 		return NOTIFY_DONE;
 	}
-	return NOTIFY_OK;
+
+out:
+	dev_put(old_ndev);
+	return notifier_from_errno(ret);
 }
 
 static void mlx5e_lag_event_register(struct mlx5_ib_dev *dev)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
index d661267d98ff..7f68468c2e75 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
@@ -919,6 +919,7 @@ static void mlx5_do_bond(struct mlx5_lag *ldev)
 {
 	struct mlx5_core_dev *dev0 = ldev->pf[MLX5_LAG_P1].dev;
 	struct lag_tracker tracker = { };
+	struct net_device *ndev;
 	bool do_bond, roce_lag;
 	int err;
 	int i;
@@ -982,6 +983,16 @@ static void mlx5_do_bond(struct mlx5_lag *ldev)
 				return;
 			}
 		}
+		if (tracker.tx_type == NETDEV_LAG_TX_TYPE_ACTIVEBACKUP) {
+			ndev = mlx5_lag_active_backup_get_netdev(dev0);
+			/** Only sriov and roce lag should have tracker->TX_type
+			 *  set so no need to check the mode
+			 */
+			blocking_notifier_call_chain(&dev0->priv.lag_nh,
+						     MLX5_DRIVER_EVENT_ACTIVE_BACKUP_LAG_CHANGE_LOWERSTATE,
+						     ndev);
+			dev_put(ndev);
+		}
 	} else if (mlx5_lag_should_modify_lag(ldev, do_bond)) {
 		mlx5_modify_lag(ldev, &tracker);
 	} else if (mlx5_lag_should_disable_lag(ldev, do_bond)) {
-- 
2.46.2

Re: [PATCH rdma-next 0/3] Fixes to set_netdev/get_netdev flow

[Leon Romanovsky]

On Thu, 31 Oct 2024 15:36:49 +0200, Leon Romanovsky wrote:
> This series fixes the set_netdev/get_netdev flow in mlx5 drive around LAG.
> 
> Thanks
> 
> Chiara Meiohas (3):
>   RDMA/mlx5: Call dev_put() after the blocking notifier
>   RDMA/core: Implement RoCE GID port rescan and export delete function
>   RDMA/mlx5: Ensure active slave attachment to the bond IB device
> 
> [...]

Applied, thanks!

[1/3] RDMA/mlx5: Call dev_put() after the blocking notifier
      https://git.kernel.org/rdma/rdma/c/c4967b6dc20ab1
[2/3] RDMA/core: Implement RoCE GID port rescan and export delete function
      https://git.kernel.org/rdma/rdma/c/ffe3c55b16ddee
[3/3] RDMA/mlx5: Ensure active slave attachment to the bond IB device
      https://git.kernel.org/rdma/rdma/c/bf15c4dc0fbb16

Best regards,
-- 
Leon Romanovsky <leon@kernel.org>