Re: [PATCH for-next] RDMA/hns: Use the reserved loopback QPs to free MR before destroying MPT

[Leon Romanovsky <leon@kernel.org>]

On Wed, Mar 02, 2022 at 08:44:48PM +0800, Wenpeng Liang wrote:
> On 2022/2/28 20:01, Leon Romanovsky wrote:
> > On Fri, Feb 25, 2022 at 05:56:54PM +0800, Wenpeng Liang wrote:
> >> From: Yixing Liu <liuyixing1@huawei.com>
> >>
> >> Before destroying MPT, the reserved loopback QPs send loopback IOs (one
> >> write operation per SL). Completing these loopback IOs represents that
> >> there isn't any outstanding request in MPT, then it's safe to destroy MPT.
> >>
> >> Signed-off-by: Yixing Liu <liuyixing1@huawei.com>
> >> Signed-off-by: Wenpeng Liang <liangwenpeng@huawei.com>
> >> ---
> >>  drivers/infiniband/hw/hns/hns_roce_device.h |   2 +
> >>  drivers/infiniband/hw/hns/hns_roce_hw_v2.c  | 334 +++++++++++++++++++-
> >>  drivers/infiniband/hw/hns/hns_roce_hw_v2.h  |  20 ++
> >>  drivers/infiniband/hw/hns/hns_roce_mr.c     |   6 +-
> >>  4 files changed, 358 insertions(+), 4 deletions(-)
> >>
> >> diff --git a/drivers/infiniband/hw/hns/hns_roce_device.h b/drivers/infiniband/hw/hns/hns_roce_device.h
> >> index 1e0bae136997..da0b4b310aab 100644
> >> --- a/drivers/infiniband/hw/hns/hns_roce_device.h
> >> +++ b/drivers/infiniband/hw/hns/hns_roce_device.h
> >> @@ -624,6 +624,7 @@ struct hns_roce_qp {
> >>  	u32			next_sge;
> >>  	enum ib_mtu		path_mtu;
> >>  	u32			max_inline_data;
> >> +	u8			free_mr_en;
> >>  
> >>  	/* 0: flush needed, 1: unneeded */
> >>  	unsigned long		flush_flag;
> >> @@ -882,6 +883,7 @@ struct hns_roce_hw {
> >>  			 enum ib_qp_state new_state);
> >>  	int (*qp_flow_control_init)(struct hns_roce_dev *hr_dev,
> >>  			 struct hns_roce_qp *hr_qp);
> >> +	void (*dereg_mr)(struct hns_roce_dev *hr_dev);
> >>  	int (*init_eq)(struct hns_roce_dev *hr_dev);
> >>  	void (*cleanup_eq)(struct hns_roce_dev *hr_dev);
> >>  	int (*write_srqc)(struct hns_roce_srq *srq, void *mb_buf);
> >> diff --git a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
> >> index b33e948fd060..62ee9c0bba74 100644
> >> --- a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
> >> +++ b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
> >> @@ -2664,6 +2664,217 @@ static void free_dip_list(struct hns_roce_dev *hr_dev)
> >>  	spin_unlock_irqrestore(&hr_dev->dip_list_lock, flags);
> >>  }
> >>  
> >> +static int free_mr_alloc_pd(struct hns_roce_dev *hr_dev,
> >> +			    struct hns_roce_v2_free_mr *free_mr)
> >> +{
> > 
> > You chose very non-intuitive name "free_mr...", but I don't have anything
> > concrete to suggest.
> > 
> 
> Thank you for your advice. There are two alternative names for this event,
> which are DRAIN_RESIDUAL_WR or DRAIN_WR. It is hard to decide which one is
> better. Could you give me some suggestions for the naming?

mlx5 called to such objects device resource - devr, see mlx5_ib_dev_res_init().
I personally would create something similar to that, one function
without separation to multiple free_mr_alloc_* functions.

Up-to you.

Thanks