Reliable IB connections (RC) and event ordering

Reliable IB connections (RC) and event ordering

[Bart Van Assche]

Hello,

While working on the SRP target implementation I noticed the following
(the SRP protocol requires that an SRP initiator sets up an RC
connection to the SRP target during login):
* Sometimes (about 25% of SRP logins) the completion queue callback
function reports data arrival to the SRP target before the RTU event
has been delivered.
* Sometimes the completion queue callback function reports data
arrival after the DREQ message has been sent.
* According to the IBTA, message order is preserved over RC
connections and the system that sent the RTU message may only start
sending data after the RTU message has been sent (section 12.2).

Is this known behavior ? Is this the intended behavior of the Linux
InfiniBand stack ?

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Reliable IB connections (RC) and event ordering

[Roland Dreier]

 > While working on the SRP target implementation I noticed the following
 > (the SRP protocol requires that an SRP initiator sets up an RC
 > connection to the SRP target during login):
 > * Sometimes (about 25% of SRP logins) the completion queue callback
 > function reports data arrival to the SRP target before the RTU event
 > has been delivered.
 > * Sometimes the completion queue callback function reports data
 > arrival after the DREQ message has been sent.
 > * According to the IBTA, message order is preserved over RC
 > connections and the system that sent the RTU message may only start
 > sending data after the RTU message has been sent (section 12.2).
 > 
 > Is this known behavior ? Is this the intended behavior of the Linux
 > InfiniBand stack ?

Yes, this is known and I don't think there is any other sane way to
implement things.  Recall that CM messages (RTU, DREQ, etc) are sent on
a separate QP than the actual connection being established, and that
there is no ordering between work requests on separate work queues.

If you want to refer to the IBA, you could look at figure 132 in section
12.9.6 -- an RC connection transitions to established when either the
RTU is received, or a message is received on the connection.  The IBA
takes into account this lack of ordering in multiple places -- defining
"communication established" async events, etc.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Reliable IB connections (RC) and event ordering

[Or Gerlitz]

Roland Dreier wrote:
> The IBA takes into account this lack of ordering in multiple places -- defining
> "communication established" async events, etc.

same goes for the IB stack... e.g take a look on the ib_cm_notify and rdma_notify APIs

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html