[PATCH] nvme-rdma: Always signal fabrics private commands

[sagi@grimberg.me (Sagi Grimberg)]

>> Think of a case where we posted unsignaled send, got a successful reply
>> from the peer, now we drain the qp, and the send which belongs to a
>> transaction that we already completed is flush with error. Does that
>> sound like a correct behavior?
>
> Well, from the specification, yes.    From
> https://tools.ietf.org/html/draft-hilland-rddp-verbs-00#section-8.1.3.1 :
>
> ----
> An Unsignaled WR is defined as completed successfully when all of
>     the following rules are met:
>
>
>     *   A Work Completion is retrieved from the CQ associated with the
>         SQ where the unsignaled Work Request was posted,
>
>     *   that Work Completion corresponds to a subsequent Work Request on
>         the same Send Queue as the unsignaled Work Request, and
>
>     *   the subsequent Work Request is ordered after the unsignaled Work
>         Request as per the ordering rules. Depending on the Work Request
>         used, this may require using the Local Fence indicator in order
>         to guarantee ordering.
> ---

OK, thanks for educating me :)

> So in your example, even though the application knows the SEND made it because
> the peer replied and genereated an RQ completion, the iwarp provider does not
> know the SEND made it...

So we have two options here:

1. always make sure not to free anything related to SQEs until we
destroy the QP (hopefully won't bite us again, which is not a good
bet given that the sequence is not trivial).

2. always signal sends for iWARP (yukk...)

I pick poison 1 (for now...)