Re: [PATCH mlx5-next v1] RDMA/mlx5: Cleanup the synchronize_srcu() from the ODP flow

[Jason Gunthorpe <jgg@nvidia.com>]

On Sun, Jan 31, 2021 at 03:24:50PM +0200, Yishai Hadas wrote:
> On 1/29/2021 2:23 PM, Saeed Mahameed wrote:
> > > diff --git a/drivers/net/ethernet/mellanox/mlx5/core/mr.c
> > > b/drivers/net/ethernet/mellanox/mlx5/core/mr.c
> > > index 9eb51f06d3ae..50af84e76fb6 100644
> > > +++ b/drivers/net/ethernet/mellanox/mlx5/core/mr.c
> > > @@ -56,6 +56,7 @@ int mlx5_core_create_mkey(struct mlx5_core_dev
> > > *dev,
> > >          mkey->size = MLX5_GET64(mkc, mkc, len);
> > >          mkey->key |= mlx5_idx_to_mkey(mkey_index);
> > >          mkey->pd = MLX5_GET(mkc, mkc, pd);
> > > +       init_waitqueue_head(&mkey->wait);
> > > 
> > >          mlx5_core_dbg(dev, "out 0x%x, mkey 0x%x\n", mkey_index, mkey-
> > > > key);
> > >          return 0;
> > > diff --git a/include/linux/mlx5/driver.h
> > > b/include/linux/mlx5/driver.h
> > > index 4901b4fadabb..f9e7036ae5a5 100644
> > > +++ b/include/linux/mlx5/driver.h
> > > @@ -373,6 +373,8 @@ struct mlx5_core_mkey {
> > >          u32                     key;
> > >          u32                     pd;
> > >          u32                     type;
> > > +       struct wait_queue_head wait;
> > > +       refcount_t usecount;
> > mlx5_core_mkey is used everywhere in mlx5_core and we don't care about
> > odp complexity, i would like to keep the core simple and primitive as
> > it is today.
> > please keep the layer separation and find a way to manage refcount and
> > wait queue in mlx5_ib driver..
> > 
> The alternative could really be to come with some wrapped mlx5_ib structure
> that will hold 'mlx5_core_mkey' and will add those two fields.

Yes

struct mlx5_ib_mkey
{
   struct mlx5_core_mkey mkey;
   struct wait_queue_head wait;
   refcount_t usecount;
}

struct mlx5_ib_mr/mw/devx
{
    struct mlx5_ib_mkey mkey;
}

The odp_mkeys XA will store pointers to mlx5_ib_mkey (not core_mkey)
and container_of() to go back to the MW, devx or MR.

> Having it per object (e.g. mlx5_ib_mr, mlx5_ib_mw, mlx5_ib_devx_mr)
> increasing locking scope and branches on data path to find the refcount
> field per its 'type'.  (see mlx5_core_mkey->type).

It is free, just use container_of, no change to locking/etc

Jason