[PATCH rdma-next 0/2] Two small fixes to MR cache series

[PATCH rdma-next 0/2] Two small fixes to MR cache series

[Leon Romanovsky]

From: Leon Romanovsky <leonro@nvidia.com>

Hi,

There are two small patches with fix and cleanup to previously accepted
MR cache series from Michael.

Thanks

Leon Romanovsky (2):
  RDMA/mlx5: Fix MR cache debugfs in switchdev mode
  RDMA/mlx5: Remove impossible check of mkey cache cleanup failure

 drivers/infiniband/hw/mlx5/main.c    |  7 +------
 drivers/infiniband/hw/mlx5/mlx5_ib.h |  2 +-
 drivers/infiniband/hw/mlx5/mr.c      | 10 ++++++----
 3 files changed, 8 insertions(+), 11 deletions(-)

-- 
2.39.1

[PATCH rdma-next 1/2] RDMA/mlx5: Fix MR cache debugfs in switchdev mode

[Leon Romanovsky]

From: Leon Romanovsky <leonro@nvidia.com>

Block MR cache debugfs creation while in switchdev mode and add missing
debugfs cleanup in error path.

This change fixes the following debugfs errors:

 bond0: (slave enp8s0f1): Enslaving as a backup interface with an up link
 mlx5_core 0000:08:00.0: lag map: port 1:1 port 2:1
 mlx5_core 0000:08:00.0: shared_fdb:1 mode:queue_affinity
 mlx5_core 0000:08:00.0: Operation mode is single FDB
 debugfs: Directory '2' with parent '/' already present!
...
 debugfs: Directory '22' with parent '/' already present!

Fixes: 73d09b2fe833 ("RDMA/mlx5: Introduce mlx5r_cache_rb_key")
Signed-off-by: Michael Guralnik <michaelgur@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
 drivers/infiniband/hw/mlx5/mr.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
index 3a9a787184fc..3f410eef58e4 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -848,6 +848,9 @@ static void mlx5_mkey_cache_debugfs_add_ent(struct mlx5_ib_dev *dev,
 	int order = order_base_2(ent->rb_key.ndescs);
 	struct dentry *dir;
 
+	if (!mlx5_debugfs_root || dev->is_rep)
+		return;
+
 	if (ent->rb_key.access_mode == MLX5_MKC_ACCESS_MODE_KSM)
 		order = MLX5_IMR_KSM_CACHE_ENTRY + 2;
 
@@ -1006,6 +1009,7 @@ int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev)
 
 err:
 	mutex_unlock(&cache->rb_lock);
+	mlx5_mkey_cache_debugfs_cleanup(dev);
 	mlx5_ib_warn(dev, "failed to create mkey cache entry\n");
 	return ret;
 }
-- 
2.39.1

Re: [PATCH rdma-next 1/2] RDMA/mlx5: Fix MR cache debugfs in switchdev mode

[Jason Gunthorpe]

On Thu, Feb 02, 2023 at 11:03:06AM +0200, Leon Romanovsky wrote:
> From: Leon Romanovsky <leonro@nvidia.com>
> 
> Block MR cache debugfs creation while in switchdev mode and add missing
> debugfs cleanup in error path.

Why does switchdev have anything to do with this?

Jason

Re: [PATCH rdma-next 1/2] RDMA/mlx5: Fix MR cache debugfs in switchdev mode

[Leon Romanovsky]

On Thu, Feb 02, 2023 at 09:56:51AM -0400, Jason Gunthorpe wrote:
> On Thu, Feb 02, 2023 at 11:03:06AM +0200, Leon Romanovsky wrote:
> > From: Leon Romanovsky <leonro@nvidia.com>
> > 
> > Block MR cache debugfs creation while in switchdev mode and add missing
> > debugfs cleanup in error path.
> 
> Why does switchdev have anything to do with this?
> 

Because we always had the following code in cleanup:
   697 static void mlx5_mkey_cache_debugfs_cleanup(struct mlx5_ib_dev *dev)
   698 {
   699         if (!mlx5_debugfs_root || dev->is_rep)
   700                 return;
   701

MR cache shouldn't be used at all for IB representors, and more
comprehensive patch will take more work than this simple solution.

Thanks

> Jason

Re: [PATCH rdma-next 1/2] RDMA/mlx5: Fix MR cache debugfs in switchdev mode

[Jason Gunthorpe]

On Thu, Feb 02, 2023 at 04:05:18PM +0200, Leon Romanovsky wrote:
> On Thu, Feb 02, 2023 at 09:56:51AM -0400, Jason Gunthorpe wrote:
> > On Thu, Feb 02, 2023 at 11:03:06AM +0200, Leon Romanovsky wrote:
> > > From: Leon Romanovsky <leonro@nvidia.com>
> > > 
> > > Block MR cache debugfs creation while in switchdev mode and add missing
> > > debugfs cleanup in error path.
> > 
> > Why does switchdev have anything to do with this?
> > 
> 
> Because we always had the following code in cleanup:
>    697 static void mlx5_mkey_cache_debugfs_cleanup(struct mlx5_ib_dev *dev)
>    698 {
>    699         if (!mlx5_debugfs_root || dev->is_rep)
>    700                 return;
>    701
> 
> MR cache shouldn't be used at all for IB representors, and more
> comprehensive patch will take more work than this simple solution.

That make sense, the commit message should explain it..

Jason

Re: [PATCH rdma-next 1/2] RDMA/mlx5: Fix MR cache debugfs in switchdev mode

[Leon Romanovsky]

On Thu, Feb 02, 2023 at 11:07:54AM -0400, Jason Gunthorpe wrote:
> On Thu, Feb 02, 2023 at 04:05:18PM +0200, Leon Romanovsky wrote:
> > On Thu, Feb 02, 2023 at 09:56:51AM -0400, Jason Gunthorpe wrote:
> > > On Thu, Feb 02, 2023 at 11:03:06AM +0200, Leon Romanovsky wrote:
> > > > From: Leon Romanovsky <leonro@nvidia.com>
> > > > 
> > > > Block MR cache debugfs creation while in switchdev mode and add missing
> > > > debugfs cleanup in error path.
> > > 
> > > Why does switchdev have anything to do with this?
> > > 
> > 
> > Because we always had the following code in cleanup:
> >    697 static void mlx5_mkey_cache_debugfs_cleanup(struct mlx5_ib_dev *dev)
> >    698 {
> >    699         if (!mlx5_debugfs_root || dev->is_rep)
> >    700                 return;
> >    701
> > 
> > MR cache shouldn't be used at all for IB representors, and more
> > comprehensive patch will take more work than this simple solution.
> 
> That make sense, the commit message should explain it..

OK, I added the following to the commit message.


    RDMA/mlx5: Fix MR cache debugfs error in IB representors mode

    Block MR cache debugfs creation for IB representor flow as MR cache shouldn't be used
    at all in that mode. As part of this change, add missing debugfs cleanup in error path
    too.


    RDMA/mlx5: Fix MR cache debugfs error in IB representors mode

    Block MR cache debugfs creation for IB representor flow as MR cache shouldn't be used
    at all in that mode. As part of this change, add missing debugfs cleanup in error path
    too.

Thanks

> 
> Jason

[PATCH rdma-next 2/2] RDMA/mlx5: Remove impossible check of mkey cache cleanup failure

[Leon Romanovsky]

From: Leon Romanovsky <leonro@nvidia.com>

mlx5_mkey_cache_cleanup() can't fail and can be changed to be void.

Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
 drivers/infiniband/hw/mlx5/main.c    | 7 +------
 drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 +-
 drivers/infiniband/hw/mlx5/mr.c      | 6 ++----
 3 files changed, 4 insertions(+), 11 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index 10f12e9a4dc3..e252ba138690 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -4005,12 +4005,7 @@ static int mlx5_ib_stage_ib_reg_init(struct mlx5_ib_dev *dev)
 
 static void mlx5_ib_stage_pre_ib_reg_umr_cleanup(struct mlx5_ib_dev *dev)
 {
-	int err;
-
-	err = mlx5_mkey_cache_cleanup(dev);
-	if (err)
-		mlx5_ib_warn(dev, "mr cache cleanup failed\n");
-
+	mlx5_mkey_cache_cleanup(dev);
 	mlx5r_umr_resource_cleanup(dev);
 }
 
diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index 58d02e778c4b..93420b73bf20 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -1337,7 +1337,7 @@ void mlx5_ib_populate_pas(struct ib_umem *umem, size_t page_size, __be64 *pas,
 void mlx5_ib_copy_pas(u64 *old, u64 *new, int step, int num);
 int mlx5_ib_get_cqe_size(struct ib_cq *ibcq);
 int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev);
-int mlx5_mkey_cache_cleanup(struct mlx5_ib_dev *dev);
+void mlx5_mkey_cache_cleanup(struct mlx5_ib_dev *dev);
 struct mlx5_cache_ent *
 mlx5r_cache_create_ent_locked(struct mlx5_ib_dev *dev,
 			      struct mlx5r_cache_rb_key rb_key,
diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
index 3f410eef58e4..68102297b9cb 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -1014,14 +1014,14 @@ int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev)
 	return ret;
 }
 
-int mlx5_mkey_cache_cleanup(struct mlx5_ib_dev *dev)
+void mlx5_mkey_cache_cleanup(struct mlx5_ib_dev *dev)
 {
 	struct rb_root *root = &dev->cache.rb_root;
 	struct mlx5_cache_ent *ent;
 	struct rb_node *node;
 
 	if (!dev->cache.wq)
-		return 0;
+		return;
 
 	cancel_delayed_work_sync(&dev->cache.remove_ent_dwork);
 	mutex_lock(&dev->cache.rb_lock);
@@ -1048,8 +1048,6 @@ int mlx5_mkey_cache_cleanup(struct mlx5_ib_dev *dev)
 
 	destroy_workqueue(dev->cache.wq);
 	del_timer_sync(&dev->delay_timer);
-
-	return 0;
 }
 
 struct ib_mr *mlx5_ib_get_dma_mr(struct ib_pd *pd, int acc)
-- 
2.39.1

Re: [PATCH rdma-next 0/2] Two small fixes to MR cache series

[Leon Romanovsky]

On Thu, 02 Feb 2023 11:03:05 +0200, Leon Romanovsky wrote:
> From: Leon Romanovsky <leonro@nvidia.com>
> 
> Hi,
> 
> There are two small patches with fix and cleanup to previously accepted
> MR cache series from Michael.
> 
> [...]

Applied, thanks!

[1/2] RDMA/mlx5: Fix MR cache debugfs in switchdev mode
      https://git.kernel.org/rdma/rdma/c/828cf5936bea24
[2/2] RDMA/mlx5: Remove impossible check of mkey cache cleanup failure
      https://git.kernel.org/rdma/rdma/c/85f9e38a5ac7d3

Best regards,
-- 
Leon Romanovsky <leon@kernel.org>