From mboxrd@z Thu Jan  1 00:00:00 1970
From: sagi@grimberg.me (Sagi Grimberg)
Date: Wed, 13 Jul 2016 18:02:26 +0300
Subject: [PATCH] nvme-rdma: Always signal fabrics private commands
In-Reply-To: <005501d1dd16$1a32f060$4e98d120$@opengridcomputing.com>
References: <1466698104-32521-1-git-send-email-sagi@grimberg.me>
 <20160624070740.GB4252@infradead.org> <577005C3.4000802@grimberg.me>
 <20160628084105.GA13533@lst.de>
 <005201d1d148$32c33740$9849a5c0$@opengridcomputing.com>
 <010501d1d216$86c0f0c0$9442d240$@opengridcomputing.com>
 <20160630063637.GB5191@lst.de>
 <002c01d1d2d5$777d9180$6678b480$@opengridcomputing.com>
 <57861339.7090705@grimberg.me> <578613C8.7030005@grimberg.me>
 <001f01d1dd12$c5fb8500$51f28f00$@opengridcomputing.com>
 <57865479.1080707@grimberg.me>
 <005501d1dd16$1a32f060$4e98d120$@opengridcomputing.com>
Message-ID: <57865802.1030408@grimberg.me>


>> Think of a case where we posted unsignaled send, got a successful reply
>> from the peer, now we drain the qp, and the send which belongs to a
>> transaction that we already completed is flush with error. Does that
>> sound like a correct behavior?
>
> Well, from the specification, yes.    From
> https://tools.ietf.org/html/draft-hilland-rddp-verbs-00#section-8.1.3.1 :
>
> ----
> An Unsignaled WR is defined as completed successfully when all of
>     the following rules are met:
>
>
>     *   A Work Completion is retrieved from the CQ associated with the
>         SQ where the unsignaled Work Request was posted,
>
>     *   that Work Completion corresponds to a subsequent Work Request on
>         the same Send Queue as the unsignaled Work Request, and
>
>     *   the subsequent Work Request is ordered after the unsignaled Work
>         Request as per the ordering rules. Depending on the Work Request
>         used, this may require using the Local Fence indicator in order
>         to guarantee ordering.
> ---

OK, thanks for educating me :)

> So in your example, even though the application knows the SEND made it because
> the peer replied and genereated an RQ completion, the iwarp provider does not
> know the SEND made it...

So we have two options here:

1. always make sure not to free anything related to SQEs until we
destroy the QP (hopefully won't bite us again, which is not a good
bet given that the sequence is not trivial).

2. always signal sends for iWARP (yukk...)

I pick poison 1 (for now...)