Re: [PATCH mlx5-next v2] RDMA/mlx5: Cleanup the synchronize_srcu() from the ODP flow

[Jason Gunthorpe <jgg@nvidia.com>]

On Tue, Feb 02, 2021 at 09:13:09AM +0200, Leon Romanovsky wrote:
> From: Yishai Hadas <yishaih@nvidia.com>
> 
> Cleanup the synchronize_srcu() from the ODP flow as it was found to be a
> very heavy time consumer as part of dereg_mr.
> 
> For example de-registration of 10000 ODP MRs each with size of 2M
> hugepage took 19.6 sec comparing de-registration of same number of non
> ODP MRs that took 172 ms.
> 
> The new locking scheme uses the wait_event() mechanism which follows the
> use count of the MR instead of using synchronize_srcu().
> 
> By that change, the time required for the above test took 95 ms which is
> even better than the non ODP flow.
> 
> Once fully dropped the srcu usage, had to come with a lock to protect
> the XA access.
> 
> As part of using the above mechanism we could also clean the
> num_deferred_work stuff and follow the use count instead.
> 
> Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
> Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
> ---
> Changelog:
> v2:
>  * Add checks of xa_erase() result as an outcome of memory error injection tests.
>  * Found extra place where we can change open-coded logic to use mlx5r_store_odp_mkey().
> v1: https://lore.kernel.org/linux-rdma/20210128064812.1921519-1-leon@kernel.org
>  * Deleted not-relevant comment implicit_get_child_mr(), I have no idea
>    why wrong version of this patch was sent as v0.
>  * Deleted two new break lines added by me to make code more uniformly
>    in before "return ..." (sometimes it has new line, sometimes doesn't).
> v0: https://lore.kernel.org/linux-rdma/20210127143051.1873866-1-leon@kernel.org
> ---
>  drivers/infiniband/hw/mlx5/devx.c            |  13 +-
>  drivers/infiniband/hw/mlx5/main.c            |   5 -
>  drivers/infiniband/hw/mlx5/mlx5_ib.h         |  31 ++-
>  drivers/infiniband/hw/mlx5/mr.c              |  26 +--
>  drivers/infiniband/hw/mlx5/odp.c             | 224 +++++++------------
>  drivers/net/ethernet/mellanox/mlx5/core/mr.c |   1 +
>  include/linux/mlx5/driver.h                  |   2 +
>  7 files changed, 127 insertions(+), 175 deletions(-)

Applied to for-next, thanks

Jason