Re: [RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ

[Xiao Yang <yangx.jy@cn.fujitsu.com>]

On 2021/1/29 14:28, Zhu Yanjun wrote:
> On Fri, Jan 29, 2021 at 10:16 AM Xiao Yang<yangx.jy@cn.fujitsu.com>  wrote:
>> On 2021/1/28 20:54, Leon Romanovsky wrote:
>>> On Thu, Jan 28, 2021 at 02:29:43PM +0800, Xiao Yang wrote:
>>>> On 2021/1/27 20:04, Leon Romanovsky wrote:
>>>>> On Wed, Jan 27, 2021 at 04:24:31PM +0800, Xiao Yang wrote:
>>>>>> Even if we enable sq_sig_all or IBV_SEND_SIGNALED, current rxe
>>>>>> module cannot set imm_data in WC when the related WR with imm_data
>>>>>> finished on SQ.
>>>>>>
>>>>>> Signed-off-by: Xiao Yang<yangx.jy@cn.fujitsu.com>
>>>>>> ---
>>>>>>
>>>>>> Current rxe module and other rdma modules(e.g. mlx5) only set
>>>>>> imm_data in WC when the related WR with imm_data finished on RQ.
>>>>>> I am not sure if it is a expected behavior.
>>>>> This is IBTA behavior.
>>>>>
>>>>> 5.2.11 IMMEDIATE DATA EXTENDED TRANSPORT HEADER (ImmDt) - 4 BYTES
>>>>> "Immediate Data (ImmDt) contains data that is placed in the receive
>>>>>     Completion Queue Element (CQE). The ImmDt is only allowed in SEND or
>>>>>     RDMA WRITE packets with Immediate Data."
>>>>>
>>>>> If I understand the spec, you shouldn't set imm_data in SQ.
>>>> Hi Leon,
>>>>
>>>> About the behavior, I have another question:
>>>> For send operation with imm_data, we can verify if the delivered imm_data is
>>>> correct by CQE on RQ.
>>>> For rdma write operation with imm_data, how to verify if the delivered
>>>> imm_data is correct? :-)
>>> Probably that I didn't understand the question, but the RDMA WRITE is
>>> marked with special opcode in the BTH that indicates imm_data.
>> Hi Leon,
>>
>> The quesion is about how to get the imm_data in applications(programs in
>> user space)
> Any steps or simple method to reproduce this problem?
> I want to delve into this problem.
Hi Yanjun,

Sorry for the wrong question. :-)
For rdma write operation with imm_data, I can get the delivered imm_data 
by CQE on RQ.

Best Regards,
Xiao Yang
> Thanks a lot.
> Zhu Yanjun
>
>> 1) If client program does send operation with imm_data, server program
>> can get the delivered imm_data by calling ibv_poll_cq(&wc)
>> 2) If client program does rdma write operation with imm_data, server
>> program cannot get the delivered imm_data by calling ibv_poll_cq(&wc).
>>       In this case, how does server program get the delivered imm_data?
>>
>> Best Regards,
>> Xiao Yang
>>> Thanks
>>>
>>>> Best Regards,
>>>> Xiao Yang
>>>>> Thanks
>>>>>
>>>>>
>>>>> .
>>>>>
>>>>
>>> .
>>>
>>
>>
>
> .
>