Re: [RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ

Zhu Yanjun
2021-01-29T06:29:11Z

On Fri, Jan 29, 2021 at 10:16 AM Xiao Yang <yangx.jy@cn.fujitsu.com> wrote:
>
> On 2021/1/28 20:54, Leon Romanovsky wrote:
> > On Thu, Jan 28, 2021 at 02:29:43PM +0800, Xiao Yang wrote:
> >> On 2021/1/27 20:04, Leon Romanovsky wrote:
> >>> On Wed, Jan 27, 2021 at 04:24:31PM +0800, Xiao Yang wrote:
> >>>> Even if we enable sq_sig_all or IBV_SEND_SIGNALED, current rxe
> >>>> module cannot set imm_data in WC when the related WR with imm_data
> >>>> finished on SQ.
> >>>>
> >>>> Signed-off-by: Xiao Yang<yangx.jy@cn.fujitsu.com>
> >>>> ---
> >>>>
> >>>> Current rxe module and other rdma modules(e.g. mlx5) only set
> >>>> imm_data in WC when the related WR with imm_data finished on RQ.
> >>>> I am not sure if it is a expected behavior.
> >>> This is IBTA behavior.
> >>>
> >>> 5.2.11 IMMEDIATE DATA EXTENDED TRANSPORT HEADER (ImmDt) - 4 BYTES
> >>> "Immediate Data (ImmDt) contains data that is placed in the receive
> >>> Completion Queue Element (CQE). The ImmDt is only allowed in SEND or
> >>> RDMA WRITE packets with Immediate Data."
> >>>
> >>> If I understand the spec, you shouldn't set imm_data in SQ.
> >> Hi Leon,
> >>
> >> About the behavior, I have another question:
> >> For send operation with imm_data, we can verify if the delivered imm_data is
> >> correct by CQE on RQ.
> >> For rdma write operation with imm_data, how to verify if the delivered
> >> imm_data is correct? :-)
> > Probably that I didn't understand the question, but the RDMA WRITE is
> > marked with special opcode in the BTH that indicates imm_data.
> Hi Leon,
>
> The quesion is about how to get the imm_data in applications(programs in
> user space)

Any steps or simple method to reproduce this problem?
I want to delve into this problem.

Thanks a lot.
Zhu Yanjun

> 1) If client program does send operation with imm_data, server program
> can get the delivered imm_data by calling ibv_poll_cq(&wc)
> 2) If client program does rdma write operation with imm_data, server
> program cannot get the delivered imm_data by calling ibv_poll_cq(&wc).
> In this case, how does server program get the delivered imm_data?
>
> Best Regards,
> Xiao Yang
> > Thanks
> >
> >> Best Regards,
> >> Xiao Yang
> >>> Thanks
> >>>
> >>>
> >>> .
> >>>
> >>
> >>
> >
> > .
> >
>
>
>