[SPDK] RDMA QP leak on spdk nvmf target

[SPDK] RDMA QP leak on spdk nvmf target

[Potnuri Bharat Teja]

[-- Attachment #1: Type: text/plain, Size: 1940 bytes --]

Hi All,
With recent spdk code, RDMA QPs are not destroyed on the nvmf target after each 
spdk perf run until target is cleared/stopped.

SPDK nvmf target is using drain WR completion logic to destroy the RDMA QP. But 
even after send/recv drain completions are received RDMA QP is not destroyed as 
the rqpair->refcnt is 0 in spdk_nvmf_rdma_qpair_destroy().

I believe rqpair refcnt needs to be incremented before spdk_nvmf_rdma_qpair_destroy().
Here is my experimental patch that fixes the issue, Please let me know if this 
qualifies for a patch.
--- a/lib/nvmf/rdma.c
+++ b/lib/nvmf/rdma.c
@@ -2586,9 +2595,10 @@ spdk_nvmf_rdma_poller_poll(struct spdk_nvmf_rdma_transport *rtransport,
	case RDMA_WR_TYPE_DRAIN_RECV:
		rqpair = SPDK_CONTAINEROF(rdma_wr, struct spdk_nvmf_rdma_qpair, drain_recv_wr);
		assert(rqpair->disconnect_flags & RDMA_QP_DISCONNECTING);
		SPDK_DEBUGLOG(SPDK_LOG_RDMA, "Drained QP RECV %u (%p)\n", rqpair->qpair.qid, rqpair);
		              rqpair->disconnect_flags |= RDMA_QP_RECV_DRAINED;
		if (rqpair->disconnect_flags & RDMA_QP_SEND_DRAINED) {
+			spdk_nvmf_rdma_qpair_inc_refcnt(rqpair);
			spdk_nvmf_rdma_qpair_destroy(rqpair);
		}
		/* Continue so that this does not trigger the disconnect path below. */
@@ -2596,9 +2606,10 @@ spdk_nvmf_rdma_poller_poll(struct spdk_nvmf_rdma_transport *rtransport,
	case RDMA_WR_TYPE_DRAIN_SEND:
		rqpair = SPDK_CONTAINEROF(rdma_wr, struct spdk_nvmf_rdma_qpair, drain_send_wr);
		assert(rqpair->disconnect_flags & RDMA_QP_DISCONNECTING);
		SPDK_DEBUGLOG(SPDK_LOG_RDMA, "Drained QP SEND %u (%p)\n", rqpair->qpair.qid, rqpair);
		              rqpair->disconnect_flags |= RDMA_QP_SEND_DRAINED;
		if (rqpair->disconnect_flags & RDMA_QP_RECV_DRAINED) {
+			spdk_nvmf_rdma_qpair_inc_refcnt(rqpair);
			spdk_nvmf_rdma_qpair_destroy(rqpair);
		}
		/* Continue so that this does not trigger the disconnect path below. */
----

Thanks,
Bharat.

Re: [SPDK] RDMA QP leak on spdk nvmf target

[Sasha Kotchubievsky]

[-- Attachment #1: Type: text/plain, Size: 2384 bytes --]

Hi,

I believe, 
https://github.com/spdk/spdk/commit/7da9f8faba8712e655fc75240b2a2dc6660e0e61 
recently merged into "master" should fix the problem.

Best regards

Sasha

On 12/17/2018 9:02 PM, Potnuri Bharat Teja wrote:
> Hi All,
> With recent spdk code, RDMA QPs are not destroyed on the nvmf target after each
> spdk perf run until target is cleared/stopped.
>
> SPDK nvmf target is using drain WR completion logic to destroy the RDMA QP. But
> even after send/recv drain completions are received RDMA QP is not destroyed as
> the rqpair->refcnt is 0 in spdk_nvmf_rdma_qpair_destroy().
>
> I believe rqpair refcnt needs to be incremented before spdk_nvmf_rdma_qpair_destroy().
> Here is my experimental patch that fixes the issue, Please let me know if this
> qualifies for a patch.
> --- a/lib/nvmf/rdma.c
> +++ b/lib/nvmf/rdma.c
> @@ -2586,9 +2595,10 @@ spdk_nvmf_rdma_poller_poll(struct spdk_nvmf_rdma_transport *rtransport,
> 	case RDMA_WR_TYPE_DRAIN_RECV:
> 		rqpair = SPDK_CONTAINEROF(rdma_wr, struct spdk_nvmf_rdma_qpair, drain_recv_wr);
> 		assert(rqpair->disconnect_flags & RDMA_QP_DISCONNECTING);
> 		SPDK_DEBUGLOG(SPDK_LOG_RDMA, "Drained QP RECV %u (%p)\n", rqpair->qpair.qid, rqpair);
> 		              rqpair->disconnect_flags |= RDMA_QP_RECV_DRAINED;
> 		if (rqpair->disconnect_flags & RDMA_QP_SEND_DRAINED) {
> +			spdk_nvmf_rdma_qpair_inc_refcnt(rqpair);
> 			spdk_nvmf_rdma_qpair_destroy(rqpair);
> 		}
> 		/* Continue so that this does not trigger the disconnect path below. */
> @@ -2596,9 +2606,10 @@ spdk_nvmf_rdma_poller_poll(struct spdk_nvmf_rdma_transport *rtransport,
> 	case RDMA_WR_TYPE_DRAIN_SEND:
> 		rqpair = SPDK_CONTAINEROF(rdma_wr, struct spdk_nvmf_rdma_qpair, drain_send_wr);
> 		assert(rqpair->disconnect_flags & RDMA_QP_DISCONNECTING);
> 		SPDK_DEBUGLOG(SPDK_LOG_RDMA, "Drained QP SEND %u (%p)\n", rqpair->qpair.qid, rqpair);
> 		              rqpair->disconnect_flags |= RDMA_QP_SEND_DRAINED;
> 		if (rqpair->disconnect_flags & RDMA_QP_RECV_DRAINED) {
> +			spdk_nvmf_rdma_qpair_inc_refcnt(rqpair);
> 			spdk_nvmf_rdma_qpair_destroy(rqpair);
> 		}
> 		/* Continue so that this does not trigger the disconnect path below. */
> ----
>
> Thanks,
> Bharat.
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] RDMA QP leak on spdk nvmf target

[Potnuri Bharat Teja]

[-- Attachment #1: Type: text/plain, Size: 2570 bytes --]

Yes. That should fix.
Thanks!

From: Sasha Kotchubievsky
Sent: Tuesday, December 18, 12:51 AM
Subject: Re: [SPDK] RDMA QP leak on spdk nvmf target
To: spdk(a)lists.01.org


Hi, I believe, https://github.com/spdk/spdk/commit/7da9f8faba8712e655fc75240b2a2dc6660e0e61 recently merged into "master" should fix the problem. Best regards Sasha On 12/17/2018 9:02 PM, Potnuri Bharat Teja wrote: > Hi All, > With recent spdk code, RDMA QPs are not destroyed on the nvmf target after each > spdk perf run until target is cleared/stopped. > > SPDK nvmf target is using drain WR completion logic to destroy the RDMA QP. But > even after send/recv drain completions are received RDMA QP is not destroyed as > the rqpair->refcnt is 0 in spdk_nvmf_rdma_qpair_destroy(). > > I believe rqpair refcnt needs to be incremented before spdk_nvmf_rdma_qpair_destroy(). > Here is my experimental patch that fixes the issue, Please let me know if this > qualifies for a patch. > --- a/lib/nvmf/rdma.c > +++ b/lib/nvmf/rdma.c > @@ -2586,9 +2595,10 @@ spdk_nvmf_rdma_poller_poll(struct spdk_nvmf_rdma_transport *rtransport, > case RDMA_WR_TYPE_DRAIN_RECV: > rqpair = SPDK_CONTAINEROF(rdma_wr, struct spdk_nvmf_rdma_qpair, drain_recv_wr); > assert(rqpair->disconnect_flags & RDMA_QP_DISCONNECTING); > SPDK_DEBUGLOG(SPDK_LOG_RDMA, "Drained QP RECV %u (%p)\n", rqpair->qpair.qid, rqpair); > rqpair->disconnect_flags |= RDMA_QP_RECV_DRAINED; > if (rqpair->disconnect_flags & RDMA_QP_SEND_DRAINED) { > + spdk_nvmf_rdma_qpair_inc_refcnt(rqpair); > spdk_nvmf_rdma_qpair_destroy(rqpair); > } > /* Continue so that this does not trigger the disconnect path below. */ > @@ -2596,9 +2606,10 @@ spdk_nvmf_rdma_poller_poll(struct spdk_nvmf_rdma_transport *rtransport, > case RDMA_WR_TYPE_DRAIN_SEND: > rqpair = SPDK_CONTAINEROF(rdma_wr, struct spdk_nvmf_rdma_qpair, drain_send_wr); > assert(rqpair->disconnect_flags & RDMA_QP_DISCONNECTING); > SPDK_DEBUGLOG(SPDK_LOG_RDMA, "Drained QP SEND %u (%p)\n", rqpair->qpair.qid, rqpair); > rqpair->disconnect_flags |= RDMA_QP_SEND_DRAINED; > if (rqpair->disconnect_flags & RDMA_QP_RECV_DRAINED) { > + spdk_nvmf_rdma_qpair_inc_refcnt(rqpair); > spdk_nvmf_rdma_qpair_destroy(rqpair); > } > /* Continue so that this does not trigger the disconnect path below. */ > ---- > > Thanks, > Bharat. > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk