Re: [RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ

Zhu Yanjun
2021-02-03T07:37:40Z

On Wed, Feb 3, 2021 at 1:25 PM Xiao Yang <yangx.jy@cn.fujitsu.com> wrote:
>
> On 2021/1/29 14:28, Zhu Yanjun wrote:
> > On Fri, Jan 29, 2021 at 10:16 AM Xiao Yang<yangx.jy@cn.fujitsu.com> wrote:
> >> On 2021/1/28 20:54, Leon Romanovsky wrote:
> >>> On Thu, Jan 28, 2021 at 02:29:43PM +0800, Xiao Yang wrote:
> >>>> On 2021/1/27 20:04, Leon Romanovsky wrote:
> >>>>> On Wed, Jan 27, 2021 at 04:24:31PM +0800, Xiao Yang wrote:
> >>>>>> Even if we enable sq_sig_all or IBV_SEND_SIGNALED, current rxe
> >>>>>> module cannot set imm_data in WC when the related WR with imm_data
> >>>>>> finished on SQ.
> >>>>>>
> >>>>>> Signed-off-by: Xiao Yang<yangx.jy@cn.fujitsu.com>
> >>>>>> ---
> >>>>>>
> >>>>>> Current rxe module and other rdma modules(e.g. mlx5) only set
> >>>>>> imm_data in WC when the related WR with imm_data finished on RQ.
> >>>>>> I am not sure if it is a expected behavior.
> >>>>> This is IBTA behavior.
> >>>>>
> >>>>> 5.2.11 IMMEDIATE DATA EXTENDED TRANSPORT HEADER (ImmDt) - 4 BYTES
> >>>>> "Immediate Data (ImmDt) contains data that is placed in the receive
> >>>>> Completion Queue Element (CQE). The ImmDt is only allowed in SEND or
> >>>>> RDMA WRITE packets with Immediate Data."
> >>>>>
> >>>>> If I understand the spec, you shouldn't set imm_data in SQ.
> >>>> Hi Leon,
> >>>>
> >>>> About the behavior, I have another question:
> >>>> For send operation with imm_data, we can verify if the delivered imm_data is
> >>>> correct by CQE on RQ.
> >>>> For rdma write operation with imm_data, how to verify if the delivered
> >>>> imm_data is correct? :-)
> >>> Probably that I didn't understand the question, but the RDMA WRITE is
> >>> marked with special opcode in the BTH that indicates imm_data.
> >> Hi Leon,
> >>
> >> The quesion is about how to get the imm_data in applications(programs in
> >> user space)
> > Any steps or simple method to reproduce this problem?
> > I want to delve into this problem.
> Hi Yanjun,
>
> Sorry for the wrong question. :-)
> For rdma write operation with imm_data, I can get the delivered imm_data
> by CQE on RQ.

Wonderful.

Zhu Yanjun
>
> Best Regards,
> Xiao Yang
> > Thanks a lot.
> > Zhu Yanjun
> >
> >> 1) If client program does send operation with imm_data, server program
> >> can get the delivered imm_data by calling ibv_poll_cq(&wc)
> >> 2) If client program does rdma write operation with imm_data, server
> >> program cannot get the delivered imm_data by calling ibv_poll_cq(&wc).
> >> In this case, how does server program get the delivered imm_data?
> >>
> >> Best Regards,
> >> Xiao Yang
> >>> Thanks
> >>>
> >>>> Best Regards,
> >>>> Xiao Yang
> >>>>> Thanks
> >>>>>
> >>>>>
> >>>>> .
> >>>>>
> >>>>
> >>> .
> >>>
> >>
> >>
> >
> > .
> >
>
>
>