[PATCH net-next 08/10] net/mlx5e: Recreate netdev channels on data direct device unbind

[Tariq Toukan <tariqt@nvidia.com>]

From: Dragos Tatulea <dtatulea@nvidia.com>

To prevent queues from using data direct resources, disable the data
direct private flag and trigger channel re-creation for each netdev that
is registered to a data direct device in the unbind callback.

Channels will be re-created without the data direct flag.

Note that this does not tear down dmabuf bindings attached to the data
direct device; the recreated channels will still pick up the binding via
rxq->mp_params and post DMA addresses from the data direct IOMMU domain
to the PF, causing IOMMU faults. A devmem revoke hook is needed to close
those bindings before the switch.

Signed-off-by: Dragos Tatulea <dtatulea@nvidia.com>
Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
---
 .../net/ethernet/mellanox/mlx5/core/en_main.c | 23 +++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 6476baf71bde..fb3e0bd00f04 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -6144,9 +6144,28 @@ static int mlx5e_data_direct_event(struct notifier_block *nb,
 {
 	struct mlx5e_priv *priv = container_of(nb, struct mlx5e_priv,
 					       data_direct_nb);
+	struct mlx5e_params new_params;
+
+	if (action != MLX5_DATA_DIRECT_UNBIND)
+		return NOTIFY_DONE;
+
+	rtnl_lock();
+	netdev_lock(priv->netdev);
+	mutex_lock(&priv->state_lock);
 
-	if (action == MLX5_DATA_DIRECT_UNBIND)
-		mlx5_core_dbg(priv->mdev, "data direct device unbound\n");
+	if (!MLX5E_GET_PFLAG(&priv->channels.params, MLX5E_PFLAG_DATA_DIRECT))
+		goto out;
+
+	new_params = priv->channels.params;
+	MLX5E_SET_PFLAG(&new_params, MLX5E_PFLAG_DATA_DIRECT, false);
+	mlx5e_safe_switch_params(priv, &new_params, NULL, NULL, true);
+	netdev_info(priv->netdev,
+		    "data_direct device unbound, disabled data_direct\n");
+
+out:
+	mutex_unlock(&priv->state_lock);
+	netdev_unlock(priv->netdev);
+	rtnl_unlock();
 
 	return NOTIFY_OK;
 }
-- 
2.44.0