[PATCH] nvme-rdma: Always signal fabrics private commands

[swise@opengridcomputing.com (Steve Wise)]

> 
> 
> >> Think of a case where we posted unsignaled send, got a successful reply
> >> from the peer, now we drain the qp, and the send which belongs to a
> >> transaction that we already completed is flush with error. Does that
> >> sound like a correct behavior?
> >
> > Well, from the specification, yes.    From
> > https://tools.ietf.org/html/draft-hilland-rddp-verbs-00#section-8.1.3.1 :
> >
> > ----
> > An Unsignaled WR is defined as completed successfully when all of
> >     the following rules are met:
> >
> >
> >     *   A Work Completion is retrieved from the CQ associated with the
> >         SQ where the unsignaled Work Request was posted,
> >
> >     *   that Work Completion corresponds to a subsequent Work Request on
> >         the same Send Queue as the unsignaled Work Request, and
> >
> >     *   the subsequent Work Request is ordered after the unsignaled Work
> >         Request as per the ordering rules. Depending on the Work Request
> >         used, this may require using the Local Fence indicator in order
> >         to guarantee ordering.
> > ---
> 
> OK, thanks for educating me :)
>

No problem. :)  By the way, IB Verbs has the same rules.  From 1.3 of the IBTA
spec:
----
10.8.6 UNSIGNALED COMPLETIONS

An unsignaled Work Request that completed successfully is confirmed
when all of the following rules are met:

. A Work Completion is retrieved from the same CQ that is associated
with the Send Queue to which the unsignaled Work Request
was submitted.

. That Work Completion corresponds to a subsequent Work Request
on the same Send Queue as the unsignaled Work Request.

C10-108: The CI shall not access buffers associated with an Unsignaled
Work Request once a Work Completion has been retrieved that corresponds
to a subsequent Work Request on the same Send Queue.
----

 
> > So in your example, even though the application knows the SEND made it
because
> > the peer replied and genereated an RQ completion, the iwarp provider does
not
> > know the SEND made it...
> 
> So we have two options here:
> 
> 1. always make sure not to free anything related to SQEs until we
> destroy the QP (hopefully won't bite us again, which is not a good
> bet given that the sequence is not trivial).
> 
> 2. always signal sends for iWARP (yukk...)
> 
> I pick poison 1 (for now...)
>

I agree.