[PATCH] libibverbs/doc: clarify selective signaling (Re: using IBV_SEND_SIGNALED)

[PATCH] libibverbs/doc: clarify selective signaling (Re: using IBV_SEND_SIGNALED)

[Or Gerlitz]

Steve Wise wrote:
> On 11/29/2010 11:34 AM, Robert D. Russell wrote:
> See section 8.2.1 of the iWARP Verbs draft at:
> http://tools.ietf.org/html/draft-hilland-rddp-verbs-00#section-8.2.1.
> The last big paragraph in 8.2.1 spells it all out.
> There should be a similar requirement in the IBTA IB Verbs specification...

Yes, I think the below patch to the man page can help people understand how to work 
with the thing

>>> You must signal one WR for every SQ-depth worth of WRs posted.  IE:
>>> If your SQ depth is 16, you must signal at least one out of every 16
>>> This ensures proper flow control for HW resources.


clarify selective signaling usage

Signed-off-by: Or Gerlitz <ogerlitz-smomgflXvOZWk0Htik3J/w@public.gmane.org>

---

 man/ibv_post_send.3 |    6 ++++++
 1 file changed, 6 insertions(+)

Index: libibverbs/man/ibv_post_send.3
===================================================================
--- libibverbs.orig/man/ibv_post_send.3
+++ libibverbs/man/ibv_post_send.3
@@ -112,6 +112,12 @@ request is fully executed and a work com
 from the corresponding completion queue (CQ). However, if the
 IBV_SEND_INLINE flag was set, the buffer can be reused immediately
 after the call returns.
+.PP
+Applications which use selective signaling should make sure to post
+a signaled WR (i.e one for which the IBV_SEND_SIGNALED flag is set) from
+time to time - depending on the requirements of the underlying rdma device.
+Typically the number of consecutive unsignaled WRs should not exceed the
+size of the send queue specified when the QP was created.
 .SH "SEE ALSO"
 .BR ibv_create_qp (3),
 .BR ibv_create_ah (3),
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] libibverbs/doc: clarify selective signaling (Re: using IBV_SEND_SIGNALED)

[Jason Gunthorpe]

On Tue, Nov 30, 2010 at 01:56:13PM +0200, Or Gerlitz wrote:
> Steve Wise wrote:
> > On 11/29/2010 11:34 AM, Robert D. Russell wrote:
> > See section 8.2.1 of the iWARP Verbs draft at:
> > http://tools.ietf.org/html/draft-hilland-rddp-verbs-00#section-8.2.1.
> > The last big paragraph in 8.2.1 spells it all out.
> > There should be a similar requirement in the IBTA IB Verbs specification...
> 
> Yes, I think the below patch to the man page can help people
> understand how to work with the thing

I think the subtly here is that remote completion of a local WR does
not indiciate local completion. This is the only operational case I
can think of that might break the rules.

For instance a protocol that had remote acks for all SENDs might be
tempted to not ever use signaled WR's on the local side - since
receiving the remote ACK SEND must indicate the local SEND WR is
complete. This is the incorrect assumption.

So, I'd suggest this language:

Applications which use selective signaling can only assume that
unsignaled WRs are complete once a completion for a later signaled WR
is received. In practice this means that a signaled WR must be
used periodically, and that the send queue should never be filled with
unsignaled WRs.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] libibverbs/doc: clarify selective signaling (Re: using IBV_SEND_SIGNALED)

[Or Gerlitz]

On 11/30/2010 8:57 PM, Jason Gunthorpe wrote:
> I think the subtly here is that remote completion of a local WR does 
> not indiciate local completion. This is the only operational case I 
> can think of that might break the rules.

The point you're bringing is sure valid and actually I can change the 
patch to address both issues, the one raised on the original thread (see 
"using IBV_SEND_SIGNALED in ewg from this week),  where Robert says 
"However, as soon as I call ibv_post_send() more than max_send_wr 
times,  it fails" so 1st we need to clarify that without posting a 
signaled WR the HCA will fail, and 2nd that the application can't 
consume back the resources associated with a send WR until a "later" WR 
completes.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] libibverbs/doc: clarify selective signaling (Re: using IBV_SEND_SIGNALED)

[Or Gerlitz]

Jason Gunthorpe wrote:
> Applications which use selective signaling can only assume that
> unsignaled WRs are complete once a completion for a later signaled WR
> is received. In practice this means that a signaled WR must be
> used periodically, and that the send queue should never be filled with unsignaled WRs.

clarify selective signaling usage

Signed-off-by: Or Gerlitz <ogerlitz-smomgflXvOZWk0Htik3J/w@public.gmane.org>

---

Jason, how's this?

 man/ibv_post_send.3 |    9 +++++++++
 1 file changed, 9 insertions(+)

Index: libibverbs/man/ibv_post_send.3
===================================================================
--- libibverbs.orig/man/ibv_post_send.3
+++ libibverbs/man/ibv_post_send.3
@@ -112,6 +112,15 @@ request is fully executed and a work com
 from the corresponding completion queue (CQ). However, if the
 IBV_SEND_INLINE flag was set, the buffer can be reused immediately
 after the call returns.
+.PP
+Applications which use selective signaling can only assume that
+unsignaled WRs are complete once a completion for a later signaled WR
+is received. In practice this means that a signaled WR must be
+used periodically, and that the send queue should never be filled with
+unsignaled WRs. Such periodic signaling is also a requirement of most rdma
+devices. In the presence of completion with error, WCs are generated
+also for unsignaled WRs, so applications should make sure to provide a valid
+id (cookie) for all posted WRs.
 .SH "SEE ALSO"
 .BR ibv_create_qp (3),
 .BR ibv_create_ah (3),
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] libibverbs/doc: clarify selective signaling (Re: using IBV_SEND_SIGNALED)

[Jason Gunthorpe]

On Wed, Dec 01, 2010 at 11:01:04AM +0200, Or Gerlitz wrote:
> Jason Gunthorpe wrote:
> > Applications which use selective signaling can only assume that
> > unsignaled WRs are complete once a completion for a later signaled WR
> > is received. In practice this means that a signaled WR must be
> > used periodically, and that the send queue should never be filled with unsignaled WRs.
> 
> clarify selective signaling usage
> 
> Signed-off-by: Or Gerlitz <ogerlitz-smomgflXvOZWk0Htik3J/w@public.gmane.org>
 
> Jason, how's this?

Sure, but reviewing the original thread makes me think there is a
greater mis-understanding at work about how resource ownership works
in verbs than simply related to the unsignaled WRs.

Maybe this would be a better approach:

RESOURCE OWNERSHIP
 ibv_post_* transfers ownership of the described buffer(s) 
 to the device. While the device is processing the WR the buffer(s)
 should not be changed and a queue entry is
 consumed to hold this WR while it is being processed.

 Callers are required to track how many entries in the queue are
 owned by the device and how many are free for new WRs.
 Attempting to post more WRs that there are free queue entries results
 in E???.

 Ownership of the buffer and queue entry is only passed back to the
 caller once a completion on the associated CQ is retrieved.
 A completion indicates all prior WR are completed, including
 unsignaled WRs.

 Note that receiving a completion for the WR through a CQ is the only
 way to know that the queue entry and WR is passed back to the
 caller. Other implicit means, such as receiving a recv completion,
 observing data change in memory, or the like do not reliably indicate
 that the device is finished with the queue entry and WR.

 When using unsignaled WRs it is necessary to periodically use
 a signaled WR and collect completions so that the available free
 space in the queue can be tracked.

Also, I like your sentence about errors for unsignaled WCs, that
should be kept in the IBV_SEND_SIGNALED section..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html