Re: [PATCH v1 05/11] xprtrdma: Allocate Protection Domain in rpcrdma_ep_create()

Tom Talpey
2020-03-01T18:11:46Z

On 2/21/2020 2:00 PM, Chuck Lever wrote:
> Make a Protection Domain (PD) a per-connection resource rather than
> a per-transport resource. In other words, when the connection
> terminates, the PD is destroyed.
>
> Thus there is one less HW resource that remains allocated to a
> transport after a connection is closed.

That's a good goal, but there are cases where the upper layer may
want the PD to be shared. For example, in a multichannel configuration,
where RDMA may not be constrained to use a single connection.

How would this approach support such a requirement? Can a PD be
provided to a new connection?

Tom.

> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
> ---
> net/sunrpc/xprtrdma/verbs.c | 26 ++++++++++----------------
> 1 file changed, 10 insertions(+), 16 deletions(-)
>
> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
> index f361213a8157..36fe7baea014 100644
> --- a/net/sunrpc/xprtrdma/verbs.c
> +++ b/net/sunrpc/xprtrdma/verbs.c
> @@ -363,14 +363,6 @@ static void rpcrdma_update_cm_private(struct rpcrdma_xprt *r_xprt,
> rc = PTR_ERR(ia->ri_id);
> goto out_err;
> }
> -
> - ia->ri_pd = ib_alloc_pd(ia->ri_id->device, 0);
> - if (IS_ERR(ia->ri_pd)) {
> - rc = PTR_ERR(ia->ri_pd);
> - pr_err("rpcrdma: ib_alloc_pd() returned %d\n", rc);
> - goto out_err;
> - }
> -
> return 0;
>
> out_err:
> @@ -403,9 +395,6 @@ static void rpcrdma_update_cm_private(struct rpcrdma_xprt *r_xprt,
>
> rpcrdma_ep_destroy(r_xprt);
>
> - ib_dealloc_pd(ia->ri_pd);
> - ia->ri_pd = NULL;
> -
> /* Allow waiters to continue */
> complete(&ia->ri_remove_done);
>
> @@ -423,11 +412,6 @@ static void rpcrdma_update_cm_private(struct rpcrdma_xprt *r_xprt,
> if (ia->ri_id && !IS_ERR(ia->ri_id))
> rdma_destroy_id(ia->ri_id);
> ia->ri_id = NULL;
> -
> - /* If the pd is still busy, xprtrdma missed freeing a resource */
> - if (ia->ri_pd && !IS_ERR(ia->ri_pd))
> - ib_dealloc_pd(ia->ri_pd);
> - ia->ri_pd = NULL;
> }
>
> static int rpcrdma_ep_create(struct rpcrdma_xprt *r_xprt,
> @@ -514,6 +498,12 @@ static int rpcrdma_ep_create(struct rpcrdma_xprt *r_xprt,
> ep->rep_remote_cma.flow_control = 0;
> ep->rep_remote_cma.rnr_retry_count = 0;
>
> + ia->ri_pd = ib_alloc_pd(id->device, 0);
> + if (IS_ERR(ia->ri_pd)) {
> + rc = PTR_ERR(ia->ri_pd);
> + goto out_destroy;
> + }
> +
> rc = rdma_create_qp(id, ia->ri_pd, &ep->rep_attr);
> if (rc)
> goto out_destroy;
> @@ -540,6 +530,10 @@ static void rpcrdma_ep_destroy(struct rpcrdma_xprt *r_xprt)
> if (ep->rep_attr.send_cq)
> ib_free_cq(ep->rep_attr.send_cq);
> ep->rep_attr.send_cq = NULL;
> +
> + if (ia->ri_pd)
> + ib_dealloc_pd(ia->ri_pd);
> + ia->ri_pd = NULL;
> }
>
> /* Re-establish a connection after a device removal event.
>
>
>