From mboxrd@z Thu Jan  1 00:00:00 1970
Content-Type: multipart/mixed; boundary="===============2765718355838521428=="
MIME-Version: 1.0
From: Potnuri Bharat Teja <bharat at chelsio.com>
Subject: [SPDK] RDMA QP leak on spdk nvmf target
Date: Tue, 18 Dec 2018 00:32:49 +0530
Message-ID: <20181217190248.GA20210@chelsio.com>
List-ID: <spdk@lists.01.org>
To: spdk@lists.01.org

--===============2765718355838521428==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable

Hi All,
With recent spdk code, RDMA QPs are not destroyed on the nvmf target after =
each =

spdk perf run until target is cleared/stopped.

SPDK nvmf target is using drain WR completion logic to destroy the RDMA QP.=
 But =

even after send/recv drain completions are received RDMA QP is not destroye=
d as =

the rqpair->refcnt is 0 in spdk_nvmf_rdma_qpair_destroy().

I believe rqpair refcnt needs to be incremented before spdk_nvmf_rdma_qpair=
_destroy().
Here is my experimental patch that fixes the issue, Please let me know if t=
his =

qualifies for a patch.
--- a/lib/nvmf/rdma.c
+++ b/lib/nvmf/rdma.c
@@ -2586,9 +2595,10 @@ spdk_nvmf_rdma_poller_poll(struct spdk_nvmf_rdma_tra=
nsport *rtransport,
	case RDMA_WR_TYPE_DRAIN_RECV:
		rqpair =3D SPDK_CONTAINEROF(rdma_wr, struct spdk_nvmf_rdma_qpair, drain_r=
ecv_wr);
		assert(rqpair->disconnect_flags & RDMA_QP_DISCONNECTING);
		SPDK_DEBUGLOG(SPDK_LOG_RDMA, "Drained QP RECV %u (%p)\n", rqpair->qpair.q=
id, rqpair);
		              rqpair->disconnect_flags |=3D RDMA_QP_RECV_DRAINED;
		if (rqpair->disconnect_flags & RDMA_QP_SEND_DRAINED) {
+			spdk_nvmf_rdma_qpair_inc_refcnt(rqpair);
			spdk_nvmf_rdma_qpair_destroy(rqpair);
		}
		/* Continue so that this does not trigger the disconnect path below. */
@@ -2596,9 +2606,10 @@ spdk_nvmf_rdma_poller_poll(struct spdk_nvmf_rdma_tra=
nsport *rtransport,
	case RDMA_WR_TYPE_DRAIN_SEND:
		rqpair =3D SPDK_CONTAINEROF(rdma_wr, struct spdk_nvmf_rdma_qpair, drain_s=
end_wr);
		assert(rqpair->disconnect_flags & RDMA_QP_DISCONNECTING);
		SPDK_DEBUGLOG(SPDK_LOG_RDMA, "Drained QP SEND %u (%p)\n", rqpair->qpair.q=
id, rqpair);
		              rqpair->disconnect_flags |=3D RDMA_QP_SEND_DRAINED;
		if (rqpair->disconnect_flags & RDMA_QP_RECV_DRAINED) {
+			spdk_nvmf_rdma_qpair_inc_refcnt(rqpair);
			spdk_nvmf_rdma_qpair_destroy(rqpair);
		}
		/* Continue so that this does not trigger the disconnect path below. */
----

Thanks,
Bharat.

--===============2765718355838521428==--