From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.4 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,MAILING_LIST_MULTI, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DF58C433E0 for ; Wed, 3 Feb 2021 05:25:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D4F4A64F68 for ; Wed, 3 Feb 2021 05:25:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229615AbhBCFZN (ORCPT ); Wed, 3 Feb 2021 00:25:13 -0500 Received: from mail.cn.fujitsu.com ([183.91.158.132]:52108 "EHLO heian.cn.fujitsu.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S229539AbhBCFYx (ORCPT ); Wed, 3 Feb 2021 00:24:53 -0500 X-IronPort-AV: E=Sophos;i="5.79,397,1602518400"; d="scan'208";a="104132017" Received: from unknown (HELO cn.fujitsu.com) ([10.167.33.5]) by heian.cn.fujitsu.com with ESMTP; 03 Feb 2021 13:24:05 +0800 Received: from G08CNEXMBPEKD06.g08.fujitsu.local (unknown [10.167.33.206]) by cn.fujitsu.com (Postfix) with ESMTP id C94144CE1A08; Wed, 3 Feb 2021 13:24:00 +0800 (CST) Received: from [10.167.220.69] (10.167.220.69) by G08CNEXMBPEKD06.g08.fujitsu.local (10.167.33.206) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 3 Feb 2021 13:24:01 +0800 Message-ID: <601A336E.9050006@cn.fujitsu.com> Date: Wed, 3 Feb 2021 13:23:58 +0800 From: Xiao Yang User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.2; zh-CN; rv:1.9.2.18) Gecko/20110616 Thunderbird/3.1.11 MIME-Version: 1.0 To: Zhu Yanjun CC: Leon Romanovsky , RDMA mailing list , Jason Gunthorpe Subject: Re: [RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ References: <20210127082431.2637863-1-yangx.jy@cn.fujitsu.com> <20210127120427.GJ1053290@unreal> <601259D7.1040207@cn.fujitsu.com> <20210128125421.GC5097@unreal> <60136F89.4070402@cn.fujitsu.com> In-Reply-To: Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [10.167.220.69] X-ClientProxiedBy: G08CNEXCHPEKD06.g08.fujitsu.local (10.167.33.205) To G08CNEXMBPEKD06.g08.fujitsu.local (10.167.33.206) X-yoursite-MailScanner-ID: C94144CE1A08.A8A22 X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: yangx.jy@cn.fujitsu.com Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On 2021/1/29 14:28, Zhu Yanjun wrote: > On Fri, Jan 29, 2021 at 10:16 AM Xiao Yang wrote: >> On 2021/1/28 20:54, Leon Romanovsky wrote: >>> On Thu, Jan 28, 2021 at 02:29:43PM +0800, Xiao Yang wrote: >>>> On 2021/1/27 20:04, Leon Romanovsky wrote: >>>>> On Wed, Jan 27, 2021 at 04:24:31PM +0800, Xiao Yang wrote: >>>>>> Even if we enable sq_sig_all or IBV_SEND_SIGNALED, current rxe >>>>>> module cannot set imm_data in WC when the related WR with imm_data >>>>>> finished on SQ. >>>>>> >>>>>> Signed-off-by: Xiao Yang >>>>>> --- >>>>>> >>>>>> Current rxe module and other rdma modules(e.g. mlx5) only set >>>>>> imm_data in WC when the related WR with imm_data finished on RQ. >>>>>> I am not sure if it is a expected behavior. >>>>> This is IBTA behavior. >>>>> >>>>> 5.2.11 IMMEDIATE DATA EXTENDED TRANSPORT HEADER (ImmDt) - 4 BYTES >>>>> "Immediate Data (ImmDt) contains data that is placed in the receive >>>>> Completion Queue Element (CQE). The ImmDt is only allowed in SEND or >>>>> RDMA WRITE packets with Immediate Data." >>>>> >>>>> If I understand the spec, you shouldn't set imm_data in SQ. >>>> Hi Leon, >>>> >>>> About the behavior, I have another question: >>>> For send operation with imm_data, we can verify if the delivered imm_data is >>>> correct by CQE on RQ. >>>> For rdma write operation with imm_data, how to verify if the delivered >>>> imm_data is correct? :-) >>> Probably that I didn't understand the question, but the RDMA WRITE is >>> marked with special opcode in the BTH that indicates imm_data. >> Hi Leon, >> >> The quesion is about how to get the imm_data in applications(programs in >> user space) > Any steps or simple method to reproduce this problem? > I want to delve into this problem. Hi Yanjun, Sorry for the wrong question. :-) For rdma write operation with imm_data, I can get the delivered imm_data by CQE on RQ. Best Regards, Xiao Yang > Thanks a lot. > Zhu Yanjun > >> 1) If client program does send operation with imm_data, server program >> can get the delivered imm_data by calling ibv_poll_cq(&wc) >> 2) If client program does rdma write operation with imm_data, server >> program cannot get the delivered imm_data by calling ibv_poll_cq(&wc). >> In this case, how does server program get the delivered imm_data? >> >> Best Regards, >> Xiao Yang >>> Thanks >>> >>>> Best Regards, >>>> Xiao Yang >>>>> Thanks >>>>> >>>>> >>>>> . >>>>> >>>> >>> . >>> >> >> > > . >