Hi All, With recent spdk code, RDMA QPs are not destroyed on the nvmf target after each spdk perf run until target is cleared/stopped. SPDK nvmf target is using drain WR completion logic to destroy the RDMA QP. But even after send/recv drain completions are received RDMA QP is not destroyed as the rqpair->refcnt is 0 in spdk_nvmf_rdma_qpair_destroy(). I believe rqpair refcnt needs to be incremented before spdk_nvmf_rdma_qpair_destroy(). Here is my experimental patch that fixes the issue, Please let me know if this qualifies for a patch. --- a/lib/nvmf/rdma.c +++ b/lib/nvmf/rdma.c @@ -2586,9 +2595,10 @@ spdk_nvmf_rdma_poller_poll(struct spdk_nvmf_rdma_transport *rtransport, case RDMA_WR_TYPE_DRAIN_RECV: rqpair = SPDK_CONTAINEROF(rdma_wr, struct spdk_nvmf_rdma_qpair, drain_recv_wr); assert(rqpair->disconnect_flags & RDMA_QP_DISCONNECTING); SPDK_DEBUGLOG(SPDK_LOG_RDMA, "Drained QP RECV %u (%p)\n", rqpair->qpair.qid, rqpair); rqpair->disconnect_flags |= RDMA_QP_RECV_DRAINED; if (rqpair->disconnect_flags & RDMA_QP_SEND_DRAINED) { + spdk_nvmf_rdma_qpair_inc_refcnt(rqpair); spdk_nvmf_rdma_qpair_destroy(rqpair); } /* Continue so that this does not trigger the disconnect path below. */ @@ -2596,9 +2606,10 @@ spdk_nvmf_rdma_poller_poll(struct spdk_nvmf_rdma_transport *rtransport, case RDMA_WR_TYPE_DRAIN_SEND: rqpair = SPDK_CONTAINEROF(rdma_wr, struct spdk_nvmf_rdma_qpair, drain_send_wr); assert(rqpair->disconnect_flags & RDMA_QP_DISCONNECTING); SPDK_DEBUGLOG(SPDK_LOG_RDMA, "Drained QP SEND %u (%p)\n", rqpair->qpair.qid, rqpair); rqpair->disconnect_flags |= RDMA_QP_SEND_DRAINED; if (rqpair->disconnect_flags & RDMA_QP_RECV_DRAINED) { + spdk_nvmf_rdma_qpair_inc_refcnt(rqpair); spdk_nvmf_rdma_qpair_destroy(rqpair); } /* Continue so that this does not trigger the disconnect path below. */ ---- Thanks, Bharat.