ibv_poll_cq() and wc->byte_len

ibv_poll_cq() and wc->byte_len

[Yann Droneaud]

Hi,

I was trying to use byte_len field from struct ibv_wc to report
the total number of bytes actually sent after a ibv_post_send() with
opcode IBV_WR_SEND.

After polling the completion queue (CQ) and ensuring status field of
struct ibv_wc has value IBV_WC_SUCCESS, the field byte_len of the
structure seems to be invalid.

Indeed, according to libmlx4 sources and mlx_poll_one() function, byte_len
is not always set for a send operation. It's set only for
IBV_WC_RDMA_READ, IBV_WC_COMP_SWAP and IBV_WC_FETCH_ADD, and not set for
IBV_WC_RDMA_WRITE and IBV_WC_SEND (and also not for IBV_WC_BIND_MW). This
is the same thing for libmthca.

I haven't checked for QLogic/Intel libipathverbs since it does a direct
memcpy() to the destination struct ibv_wc, looking at the library code
doesn't help me a lot.

Note: Hopefully, for receive operation, byte_len is always valid.

ibv_poll_cq(3) man page is not really helpful to know which field will be
valid depending on the opcode field of struct ibv_wc, exception when
status is not IBV_WC_SUCCESS.

Is libmlx4, libmthca behavior correct ?

In a parallel universe, struct ibv_wc would have a bitmap field indicating
which others fields are valid. In this part of the multiverse, a more
complete documentation would be welcome. If libmlx4/libmthca behavior is
the compliant one, I can provide an updated man page.

Regards.

-- 
Yann Droneaud
OPTEYA


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: ibv_poll_cq() and wc->byte_len

[Hefty, Sean]

> In a parallel universe, struct ibv_wc would have a bitmap field indicating
> which others fields are valid. In this part of the multiverse, a more
> complete documentation would be welcome. If libmlx4/libmthca behavior is
> the compliant one, I can provide an updated man page.

The best documentation available regarding expected verbs behavior is the IB spec (chapter 11).  For this specific field, see section 11.4.2:

The number of bytes transferred.
The number of bytes transferred is returned in Work Completions
for Receive Work Requests for incoming Sends and
RDMA Writes with Immediate Data. This does not include the
length of any immediate data.
The number of bytes transferred is returned in Work Completions
for Send Work Requests for RDMA Read and Atomic Operations.
For the RQ of a UD QP that is not associated with an SRQ or
for an SRQ that is associated with a UD QP, the number of
bytes transferred is the payload of the message plus the 40
bytes reserved for the GRH. For the RQ of a UD QP that is not
associated with an SRQ or for an SRQ that is associated with
a UD QP, the 40 bytes is always included, whether or not the
GRH is present.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: ibv_poll_cq() and wc->byte_len

[Yann Droneaud]

Hi,

>> In a parallel universe, struct ibv_wc would have a bitmap field
>> indicating
>> which others fields are valid. In this part of the multiverse, a more
>> complete documentation would be welcome. If libmlx4/libmthca behavior is
>> the compliant one, I can provide an updated man page.
>
> The best documentation available regarding expected verbs behavior is the
> IB spec (chapter 11).  For this specific field, see section 11.4.2:
>

Thanks a lot for the link.
Once again, I forgot to go through the IB spec.

Regards

-- 
Yann Droneaud
OPTEYA


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html