Re: [PATCH for-next v8 8/8] RDMA/rxe: Add wait for completion to obj destruct

[Jason Gunthorpe <jgg@nvidia.com>]

On Thu, Dec 16, 2021 at 05:32:02PM -0600, Bob Pearson wrote:
> This patch adds code to wait until pending activity on RDMA objects has
> completed before freeing or returning to rdma-core where the object may
> be freed.
> 
> Reported-by: kernel test robot <lkp@intel.com>
> Signed-off-by: Bob Pearson <rpearsonhpe@gmail.com>
>  drivers/infiniband/sw/rxe/rxe_mcast.c | 10 +++++
>  drivers/infiniband/sw/rxe/rxe_mr.c    |  2 +
>  drivers/infiniband/sw/rxe/rxe_mw.c    |  3 +-
>  drivers/infiniband/sw/rxe/rxe_pool.c  | 31 +++++++++++++-
>  drivers/infiniband/sw/rxe/rxe_pool.h  |  4 ++
>  drivers/infiniband/sw/rxe/rxe_verbs.c | 60 ++++++++++++++++++---------
>  6 files changed, 89 insertions(+), 21 deletions(-)
> 
> diff --git a/drivers/infiniband/sw/rxe/rxe_mcast.c b/drivers/infiniband/sw/rxe/rxe_mcast.c
> index b935634f86cd..d91c2e30665a 100644
> +++ b/drivers/infiniband/sw/rxe/rxe_mcast.c
> @@ -122,6 +122,11 @@ int rxe_mcast_drop_grp_elem(struct rxe_dev *rxe, struct rxe_qp *qp,
>  
>  out_drop_ref:
>  	rxe_drop_ref(grp);			/* ref from get_key */
> +	/* when grp ref count drops to zero
> +	 * go ahead and free it
> +	 */
> +	if (grp->elem.complete.done)
> +		rxe_fini(grp);

When using the completion pattern with refcounts there has to be a
designated freer that *always* frees the memory. This is how all the
other users of rxe_fini() are working.

You can't mix the completion pattern with multiple users that can
free, it just becomes racy, eg two threads run the above concurrently,
one will free the grp and the other will use-after-free checking done.

In this case I think you want to just run the kfree from the kref
release.

Jason