Re: [PATCH 3/9] IB: add a helper to safely drain a QP

Sagi Grimberg
2015-11-23T10:29:09Z

> That won't work for iWARP. Is this code new? I didn't see any errors that would result from this code when I tested iSER over
> cxgb4 with the old iwarp support patches.

Steve,

I think I figured out why this works with iWARP.

For iWARP, rdma_disconnect() calls iw_cm_disconnect() with abrupt=0
which would make iw_cm_disconnect() move the QP into SQ_DRAIN state"

int iw_cm_disconnect(struct iw_cm_id *cm_id, int abrupt)
{
...

if (qp) {
if (abrupt)
ret = iwcm_modify_qp_err(qp);
else
ret = iwcm_modify_qp_sqd(qp);

/*
* If both sides are disconnecting the QP could
* already be in ERR or SQD states
*/
ret = 0;
}
}

IFAIK, SQD state allows the ULP to post work requests on the send
queue and expect these work requests to FLUSH.

So Maybe we should have:
void ib_drain_qp(struct ib_qp *qp)
{
struct ib_qp_attr attr = { };
struct ib_stop_cqe rstop, sstop;
struct ib_recv_wr rwr = {}, *bad_rwr;
struct ib_send_wr swr = {}, *bad_swr;
enum ib_qp_state state;
int ret;

if rdma_cap_ib_cm(id->device, id->port_num) {
state = IB_QPS_ERR;
else if rdma_cap_iw_cm(id->device, id->port_num)
state = IB_QPS_SQD;
else
return;

rwr.wr_cqe = &rstop.cqe;
rstop.cqe.done = ib_stop_done;
init_completion(&rstop.done);

swr.wr_cqe = &sstop.cqe;
sstop.cqe.done = ib_stop_done;
swr.send_flags = IB_SEND_SIGNALED;
init_completion(&sstop.done);

attr.qp_state = state;
ret = ib_modify_qp(qp, &attr, IB_QP_STATE);
if (ret) {
WARN_ONCE(ret, "failed to drain QP: %d\n", ret);
return;
}

ret = ib_post_recv(qp, &rwr, &bad_rwr);
if (ret) {
WARN_ONCE(ret, "failed to drain recv queue: %d\n", ret);
return;
}

ret = ib_post_send(qp, &swr, &bad_swr);
if (ret) {
WARN_ONCE(ret, "failed to drain send queue: %d\n", ret);
return;
}

wait_for_completion(&rstop.done);
wait_for_completion(&sstop.done);
}

Thoughts?