WRITE_WITH_IMMEDIATE target semantic

WRITE_WITH_IMMEDIATE target semantic

[Bernard Metzler]

Hello,

while currently only supported by the InfiniBand transport,
the libibverbs define a IBV_WR_RDMA_WRITE_WITH_IMM work request.
Could someone please explain what the action at the peer side would be
if a WRITE with immediate data comes in? One possible action would be
to:

(1) fetch the next receive queue / shared receive queue entry and
    store the value of the immediate data (32 bits at the API)
    along with that RQE.

(2) place data according to the inbound WRITE hdr but ignore the
    SGL of the RQE.

(3) after completion processing the WRITE:
    (1) create a work completion of type IBV_WC_RECV_RDMA_WITH_IMM,
    (2) transfer the immediate data value into the 'imm_data' field,
    (3) set the WC's 'byte_len' to the received WRITE payload,
    (4) set IBV_WC_WITH_IMM at 'wc_flags', and
    (5) append the WC to the associated CQ.

With that, the receiver would have to carefully inspect the operation
type of the work completion, since a receive queue entry may turn into
a IBV_WC_RECV or IBV_WC_RECV_RDMA_WITH_IMM. In the second case, the data
are placed according to the senders opinion, which may not be the
result a non-aware receiver would expect.

correct?


Many thanks for clarification!
Bernard.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: WRITE_WITH_IMMEDIATE target semantic

[Jason Gunthorpe]

On Mon, Sep 20, 2010 at 06:32:05PM +0200, Bernard Metzler wrote:

> while currently only supported by the InfiniBand transport,
> the libibverbs define a IBV_WR_RDMA_WRITE_WITH_IMM work request.
> Could someone please explain what the action at the peer side would be
> if a WRITE with immediate data comes in? One possible action would be
> to:

I've used this feature and according to my notes this is broadly
correct. A RWQE is consumed, the HCA doesn't use it for anything, and
the WC has the imm_data set, and the wr_id corrisponds to the RWQE.
The IBA notes the RWQE could be zero length.

The app needs to check for:
        if (wc.opcode & IBV_WC_RECV)
To see if the completion consumed a RWQE.
 
>     (3) set the WC's 'byte_len' to the received WRITE payload,

Hmm, I'm surprised by this, but, yep! pg 632 of the IBA.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: WRITE_WITH_IMMEDIATE target semantic

[Hefty, Sean]

> (1) fetch the next receive queue / shared receive queue entry and
>     store the value of the immediate data (32 bits at the API)
>     along with that RQE.
> 
> (2) place data according to the inbound WRITE hdr but ignore the
>     SGL of the RQE.
> 
> (3) after completion processing the WRITE:
>     (1) create a work completion of type IBV_WC_RECV_RDMA_WITH_IMM,
>     (2) transfer the immediate data value into the 'imm_data' field,
>     (3) set the WC's 'byte_len' to the received WRITE payload,
>     (4) set IBV_WC_WITH_IMM at 'wc_flags', and
>     (5) append the WC to the associated CQ.
> 
> With that, the receiver would have to carefully inspect the operation
> type of the work completion, since a receive queue entry may turn into
> a IBV_WC_RECV or IBV_WC_RECV_RDMA_WITH_IMM. In the second case, the data
> are placed according to the senders opinion, which may not be the
> result a non-aware receiver would expect.
> 
> correct?

This looks correct to me, but I didn't trace through the spec.  The only other thing I can think of is to check the solicited bit.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html