From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E37A3C25B75 for ; Wed, 29 May 2024 09:45:17 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sCFr2-0004QW-Fq; Wed, 29 May 2024 05:44:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sCFr1-0004QF-BT for qemu-devel@nongnu.org; Wed, 29 May 2024 05:44:19 -0400 Received: from mail-ed1-x52a.google.com ([2a00:1450:4864:20::52a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sCFqy-0000Aw-5q for qemu-devel@nongnu.org; Wed, 29 May 2024 05:44:19 -0400 Received: by mail-ed1-x52a.google.com with SMTP id 4fb4d7f45d1cf-57864327f6eso1180407a12.1 for ; Wed, 29 May 2024 02:44:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1716975854; x=1717580654; darn=nongnu.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=jb1IjI18UL4aUBD1jLfaFAS6MnMgb6ZWdkJngXHDUCc=; b=DfBjDWpZ5h1oVJySdEwcqKLNQ4VmsNC0vRA7S/Y9POQIrJ9u2pniLLhmU0xgU07nGX B+aSgkWhQOGH0YpbYqZ4txMWoj+aeJJzvxVIvKmQLdmu9dXd/YNG/Ah7LB4MB8wu2+Zv FbFopSF9Qj66ADKMuL+Ki9v8hqIQgeQgieTkgpd4KyDCNLjkxCRf7ykxg1q5w2+hCH+c 2Xh+z4Ptjt/aSmI6ROKAxhaSaOpzkf2VGOkU4HrASsEqVxrVBPAJWaEKBWT8blWz/VBP j5IlB43AZPhCWMhmkQYTmLfCmjJbabrGw4x+mULqUwSBjRDRqPNnh7dSXTVYpMwRKTJJ 6gmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716975854; x=1717580654; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=jb1IjI18UL4aUBD1jLfaFAS6MnMgb6ZWdkJngXHDUCc=; b=QdoHPrbEZvnNM5UtHgUbhp0+JOUM2xpNqMzeALDVWrO3UBW+ic+gl5+L26kEclYlHn kHpz+sD30T40qpmNpjHwlRAle/KX8yXHhyG23XsqfJTc+k22E3sxHcGHzRpDBuSQn3O4 bH6HWnPjmUufAd6PK339BiV6FqNV3Jm6X/fNehaubMIbLh1TNb5JuR8qQgJfsXk5EMaR hEvsSg0Aco2WURjVe1ZmOGMBE+z1Vx0WswnTbRiYFpzqzgo3d1wR1k7V0qlUBPVMmAgH ZGRxeRkhZLP+NA3Z+ANNf8uaLTIuqLSXGk3wWYZJKJm+4WmyxLIQgzz4LKM3wF/5KJqb JnMg== X-Forwarded-Encrypted: i=1; AJvYcCV2JznUvcMWX0DNojefLm3Ba0Hu1xQTiMJ/90ZOXthd2hkdn9yN1bLk2Y26fZqPY9I51xMR07BIn1rF359z20pfuoErJ9U= X-Gm-Message-State: AOJu0YztrNkslHjx+/lGqgVZZVXuuiCxVz4tWx+52KwA1xSmBufh+ME/ yQga+vrCuKUp+NXZRLnrQs/BVTHdwlnO2pWX9Gb9Tm5sS7xcVtCS8WIPanRNBa4u2NTDiGo9XnS AGIIggWoKDOAiHoJ5hA4RLGNkAV3eLfoZqcWEOg== X-Google-Smtp-Source: AGHT+IEdmDgVPXBx1LhIoIFcbdoXla4D69wXqjpgiHoQlAfMpkF9MXDei8YACNh/cDtcHndeDGel571T3dSvYZH87/E= X-Received: by 2002:a50:8d4e:0:b0:579:c37c:1b36 with SMTP id 4fb4d7f45d1cf-57a04013b81mr1223315a12.17.1716975853938; Wed, 29 May 2024 02:44:13 -0700 (PDT) MIME-Version: 1.0 References: <13ce4f9e-1e7c-24a9-0dc9-c40962979663@huawei.com> <04769507-ac37-495d-a797-e05084d73e64@akamai.com> <2308a8b894244123b638038e40a33990@huawei.com> <7bf81ccee4bd4b0e81e3893ef43502a8@huawei.com> <3b1b7dfa099c4f1db2e869ced1f6f19a@huawei.com> In-Reply-To: <3b1b7dfa099c4f1db2e869ced1f6f19a@huawei.com> From: Jinpu Wang Date: Wed, 29 May 2024 11:44:02 +0200 Message-ID: Subject: Re: [PATCH-for-9.1 v2 2/3] migration: Remove RDMA protocol handling To: "Gonglei (Arei)" Cc: Greg Sword , Peter Xu , Yu Zhang , Michael Galaxy , Elmar Gerdes , zhengchuan , =?UTF-8?Q?Daniel_P=2E_Berrang=C3=A9?= , Markus Armbruster , "Zhijian Li (Fujitsu)" , "qemu-devel@nongnu.org" , Yuval Shaia , Kevin Wolf , Prasanna Kumar Kalever , Cornelia Huck , Michael Roth , Prasanna Kumar Kalever , Paolo Bonzini , "qemu-block@nongnu.org" , "devel@lists.libvirt.org" , Hanna Reitz , "Michael S. Tsirkin" , Thomas Huth , Eric Blake , Song Gao , =?UTF-8?B?TWFyYy1BbmRyw6kgTHVyZWF1?= , =?UTF-8?B?QWxleCBCZW5uw6ll?= , Wainer dos Santos Moschetta , Beraldo Leal , Pannengyuan , Xiexiangyou , Fabiano Rosas , RDMA mailing list , "shefty@nvidia.com" , Haris Iqbal Content-Type: multipart/alternative; boundary="000000000000dccac30619949735" Received-SPF: permerror client-ip=2a00:1450:4864:20::52a; envelope-from=jinpu.wang@ionos.com; helo=mail-ed1-x52a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, T_SCC_BODY_TEXT_LINE=-0.01, T_SPF_PERMERROR=0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org --000000000000dccac30619949735 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, May 29, 2024 at 11:35=E2=80=AFAM Gonglei (Arei) wrote: > > > > -----Original Message----- > > From: Jinpu Wang [mailto:jinpu.wang@ionos.com] > > Sent: Wednesday, May 29, 2024 5:18 PM > > To: Gonglei (Arei) > > Cc: Greg Sword ; Peter Xu ; > > Yu Zhang ; Michael Galaxy ; > > Elmar Gerdes ; zhengchuan > > ; Daniel P. Berrang=C3=A9 ; > > Markus Armbruster ; Zhijian Li (Fujitsu) > > ; qemu-devel@nongnu.org; Yuval Shaia > > ; Kevin Wolf ; Prasanna > > Kumar Kalever ; Cornelia Huck > > ; Michael Roth ; Prasanna > > Kumar Kalever ; Paolo Bonzini > > ; qemu-block@nongnu.org; devel@lists.libvirt.org; > > Hanna Reitz ; Michael S. Tsirkin ; > > Thomas Huth ; Eric Blake ; Song > > Gao ; Marc-Andr=C3=A9 Lureau > > ; Alex Benn=C3=A9e ; > > Wainer dos Santos Moschetta ; Beraldo Leal > > ; Pannengyuan ; > > Xiexiangyou ; Fabiano Rosas ; > > RDMA mailing list ; shefty@nvidia.com; Hari= s > > Iqbal > > Subject: Re: [PATCH-for-9.1 v2 2/3] migration: Remove RDMA protocol > handling > > > > Hi Gonglei, > > > > On Wed, May 29, 2024 at 10:31=E2=80=AFAM Gonglei (Arei) > > > wrote: > > > > > > > > > > > > > -----Original Message----- > > > > From: Greg Sword [mailto:gregsword0@gmail.com] > > > > Sent: Wednesday, May 29, 2024 2:06 PM > > > > To: Jinpu Wang > > > > Subject: Re: [PATCH-for-9.1 v2 2/3] migration: Remove RDMA protocol > > > > handling > > > > > > > > On Wed, May 29, 2024 at 12:33=E2=80=AFPM Jinpu Wang > > > > wrote: > > > > > > > > > > On Wed, May 29, 2024 at 4:43=E2=80=AFAM Gonglei (Arei) > > > > > > > > > wrote: > > > > > > > > > > > > Hi, > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Peter Xu [mailto:peterx@redhat.com] > > > > > > > Sent: Tuesday, May 28, 2024 11:55 PM > > > > > > > > > > Exactly, not so compelling, as I did it first only on > > > > > > > > > > servers widely used for production in our data center. > > > > > > > > > > The network adapters are > > > > > > > > > > > > > > > > > > > > Ethernet controller: Broadcom Inc. and subsidiaries > > > > > > > > > > NetXtreme > > > > > > > > > > BCM5720 2-port Gigabit Ethernet PCIe > > > > > > > > > > > > > > > > > > Hmm... I definitely thinks Jinpu's Mellanox ConnectX-6 > > > > > > > > > looks more > > > > > > > reasonable. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://lore.kernel.org/qemu-devel/CAMGffEn-DKpMZ4tA71MJYdyemg0Zda > > > > > > > 15 > > > > > > > > > wVAqk81vXtKzx-LfJQ@mail.gmail.com/ > > > > > > > > > > > > > > > > > > Appreciate a lot for everyone helping on the testings. > > > > > > > > > > > > > > > > > > > InfiniBand controller: Mellanox Technologies MT27800 > > > > > > > > > > Family [ConnectX-5] > > > > > > > > > > > > > > > > > > > > which doesn't meet our purpose. I can choose RDMA or TC= P > > > > > > > > > > for VM migration. RDMA traffic is through InfiniBand an= d > > > > > > > > > > TCP through Ethernet on these two hosts. One is standby > > > > > > > > > > while the other > > > > is active. > > > > > > > > > > > > > > > > > > > > Now I'll try on a server with more recent Ethernet and > > > > > > > > > > InfiniBand network adapters. One of them has: > > > > > > > > > > BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller > > > > > > > > > > (rev > > > > > > > > > > 01) > > > > > > > > > > > > > > > > > > > > The comparison between RDMA and TCP on the same NIC > > > > > > > > > > could make more > > > > > > > > > sense. > > > > > > > > > > > > > > > > > > It looks to me NICs are powerful now, but again as I > > > > > > > > > mentioned I don't think it's a reason we need to deprecat= e > > > > > > > > > rdma, especially if QEMU's rdma migration has the chance > > > > > > > > > to be refactored > > > > using rsocket. > > > > > > > > > > > > > > > > > > Is there anyone who started looking into that direction? > > > > > > > > > Would it make sense we start some PoC now? > > > > > > > > > > > > > > > > > > > > > > > > > My team has finished the PoC refactoring which works well. > > > > > > > > > > > > > > > > Progress: > > > > > > > > 1. Implement io/channel-rdma.c, 2. Add unit test > > > > > > > > tests/unit/test-io-channel-rdma.c and verifying it is > > > > > > > > successful, 3. Remove the original code from > migration/rdma.c, 4. > > > > > > > > Rewrite the rdma_start_outgoing_migration and > > > > > > > > rdma_start_incoming_migration logic, 5. Remove all rdma_xx= x > > > > > > > > functions from migration/ram.c. (to prevent RDMA live > > > > > > > > migration from polluting the > > > > > > > core logic of live migration), 6. The soft-RoCE implemented > > > > > > > by software is used to test the RDMA live migration. It's > successful. > > > > > > > > > > > > > > > > We will be submit the patchset later. > > > > > > > > > > > > > > That's great news, thank you! > > > > > > > > > > > > > > -- > > > > > > > Peter Xu > > > > > > > > > > > > For rdma programming, the current mainstream implementation is > > > > > > to use > > > > rdma_cm to establish a connection, and then use verbs to transmit > data. > > > > > > > > > > > > rdma_cm and ibverbs create two FDs respectively. The two FDs > > > > > > have different responsibilities. rdma_cm fd is used to notify > > > > > > connection establishment events, and verbs fd is used to notify > > > > > > new CQEs. When > > > > poll/epoll monitoring is directly performed on the rdma_cm fd, only > > > > a pollin event can be monitored, which means that an rdma_cm event > > > > occurs. When the verbs fd is directly polled/epolled, only the > > > > pollin event can be listened, which indicates that a new CQE is > generated. > > > > > > > > > > > > Rsocket is a sub-module attached to the rdma_cm library and > > > > > > provides rdma calls that are completely similar to socket > interfaces. > > > > > > However, this library returns only the rdma_cm fd for listening > > > > > > to link > > > > setup-related events and does not expose the verbs fd (readable and > > > > writable events for listening to data). Only the rpoll interface > > > > provided by the RSocket can be used to listen to related events. > > > > However, QEMU uses the ppoll interface to listen to the rdma_cm fd > > (gotten by raccept API). > > > > > > And cannot listen to the verbs fd event. > > I'm confused, the rs_poll_arm > > : > https://github.com/linux-rdma/rdma-core/blob/master/librdmacm/rsocket.c# > > L3290 > > For STREAM, rpoll setup fd for both cq fd and cm fd. > > > > > > > > > > > > > > Do you guys have any ideas? Thanks. > > > > > +cc linux-rdma > > > > > > > > Why include rdma community? > > > > > > > > > > Can rdma/rsocket provide an API to expose the verbs fd? > > Why do we need verbs fd? looks rsocket during rsend/rrecv is handling > the new > > completion if any via rs_get_comp > > > Actually I said the reason in the previous mail. Listing some header in > librdmacm. > > /* verbs.h */ > struct ibv_comp_channel { > struct ibv_context *context; > int fd; > int refcnt; > }; > > /* rdma_cma.h */ > struct rdma_event_channel { > int fd; > }; > > /* rdma_cma.h */ > struct rdma_cm_id { > struct ibv_context *verbs; > struct rdma_event_channel *channel; //=3D=3D> it can be gotten = by > rsocket.h > void *context; > struct ibv_qp *qp; > struct rdma_route route; > enum rdma_port_space ps; > uint8_t port_num; > struct rdma_cm_event *event; > struct ibv_comp_channel *send_cq_channel; // =3D=3D> can't be go= tten > so that Qemu can't read the CQE dat > ok, but the send_cq_channel is set the same as recv_cq_channel: https://github.com/linux-rdma/rdma-core/blob/master/librdmacm/rsocket.c#L85= 5 and also use the same recv_cq as send_cq. > struct ibv_cq *send_cq; > struct ibv_comp_channel *recv_cq_channel; > struct ibv_cq *recv_cq; > struct ibv_srq *srq; > struct ibv_pd *pd; > enum ibv_qp_type qp_type; > }; > > /* rsocket.h */ > int raccept(int socket, struct sockaddr *addr, socklen_t *addrlen); > int rpoll(struct pollfd *fds, nfds_t nfds, int timeout); > > > > Another question to my mind is Daniel suggested a bit different way of > using > > rsocket: https://lore.kernel.org/qemu-devel/ZjtOreamN8xF9FDE@redhat.com= / > > Have you considered that? > > > We do use 'rsocket' APIs to refactor the RDMA code in QEMU and encounter > the issue. > > > Regards, > -Gonglei > > --000000000000dccac30619949735 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Wed, May 29, 2024 at 11:35=E2=80= =AFAM Gonglei (Arei) <arei.go= nglei@huawei.com> wrote:


> -----Original Message-----
> From: Jinpu Wang [mailto:jinpu.wang@ionos.com]
> Sent: Wednesday, May 29, 2024 5:18 PM
> To: Gonglei (Arei) <arei.gonglei@huawei.com>
> Cc: Greg Sword <gregsword0@gmail.com>; Peter Xu <peterx@redhat.com>;
> Yu Zhang <y= u.zhang@ionos.com>; Michael Galaxy <mgalaxy@akamai.com>;
> Elmar Gerdes <elmar.gerdes@ionos.com>; zhengchuan
> <zhengch= uan@huawei.com>; Daniel P. Berrang=C3=A9 <berrange@redhat.com>;
> Markus Armbruster <armbru@redhat.com>; Zhijian Li (Fujitsu)
> <lizhiji= an@fujitsu.com>; qemu-devel@nongnu.org; Yuval Shaia
> <yuva= l.shaia.ml@gmail.com>; Kevin Wolf <kwolf@redhat.com>; Prasanna
> Kumar Kalever <prasanna.kalever@redhat.com>; Cornelia Huck
> <cohuck@redh= at.com>; Michael Roth <michael.roth@amd.com>; Prasanna
> Kumar Kalever <prasanna4324@gmail.com>; Paolo Bonzini
> <pbonzini@= redhat.com>; qemu-block@nongnu.org; devel@lists.libvirt.org;
> Hanna Reitz <hreitz@redhat.com>; Michael S. Tsirkin <mst@redhat.com>;
> Thomas Huth <= thuth@redhat.com>; Eric Blake <eblake@redhat.com>; Song
> Gao <gaoso= ng@loongson.cn>; Marc-Andr=C3=A9 Lureau
> <m= arcandre.lureau@redhat.com>; Alex Benn=C3=A9e <alex.bennee@linaro.org>; > Wainer dos Santos Moschetta <wainersm@redhat.com>; Beraldo Leal
> <bleal@redhat= .com>; Pannengyuan <pannengyuan@huawei.com>;
> Xiexiangyou <xiexiangyou@huawei.com>; Fabiano Rosas <farosas@suse.de>;
> RDMA mailing list <linux-rdma@vger.kernel.org>; shefty@nvidia.com; Haris
> Iqbal <h= aris.iqbal@ionos.com>
> Subject: Re: [PATCH-for-9.1 v2 2/3] migration: Remove RDMA protocol ha= ndling
>
> Hi Gonglei,
>
> On Wed, May 29, 2024 at 10:31=E2=80=AFAM Gonglei (Arei) <arei.gonglei@huawei.com<= /a>>
> wrote:
> >
> >
> >
> > > -----Original Message-----
> > > From: Greg Sword [mailto:gregsword0@gmail.com]
> > > Sent: Wednesday, May 29, 2024 2:06 PM
> > > To: Jinpu Wang <jinpu.wang@ionos.com>
> > > Subject: Re: [PATCH-for-9.1 v2 2/3] migration: Remove RDMA p= rotocol
> > > handling
> > >
> > > On Wed, May 29, 2024 at 12:33=E2=80=AFPM Jinpu Wang <jinpu.wang@ionos.com<= /a>>
> > > wrote:
> > > >
> > > > On Wed, May 29, 2024 at 4:43=E2=80=AFAM Gonglei (Arei)<= br> > > > > <arei.gonglei@huawei.com>
> > > wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Peter Xu [mailto:peterx@redhat.com]
> > > > > > Sent: Tuesday, May 28, 2024 11:55 PM
> > > > > > > > > Exactly, not so compelling, as= I did it first only on
> > > > > > > > > servers widely used for produc= tion in our data center.
> > > > > > > > > The network adapters are
> > > > > > > > >
> > > > > > > > > Ethernet controller: Broadcom = Inc. and subsidiaries
> > > > > > > > > NetXtreme
> > > > > > > > > BCM5720 2-port Gigabit Etherne= t PCIe
> > > > > > > >
> > > > > > > > Hmm... I definitely thinks Jinpu= 9;s Mellanox ConnectX-6
> > > > > > > > looks more
> > > > > > reasonable.
> > > > > > > >
> > > > > > > >
> > > > > >
> > >
> https://lore.kernel.org/qemu-= devel/CAMGffEn-DKpMZ4tA71MJYdyemg0Zda
> > > > > > 15
> > > > > > > > wVAqk81vXtKzx-LfJQ@= mail.gmail.com/
> > > > > > > >
> > > > > > > > Appreciate a lot for everyone helpi= ng on the testings.
> > > > > > > >
> > > > > > > > > InfiniBand controller: Mellano= x Technologies MT27800
> > > > > > > > > Family [ConnectX-5]
> > > > > > > > >
> > > > > > > > > which doesn't meet our pur= pose. I can choose RDMA or TCP
> > > > > > > > > for VM migration. RDMA traffic= is through InfiniBand and
> > > > > > > > > TCP through Ethernet on these = two hosts. One is standby
> > > > > > > > > while the other
> > > is active.
> > > > > > > > >
> > > > > > > > > Now I'll try on a server w= ith more recent Ethernet and
> > > > > > > > > InfiniBand network adapters. O= ne of them has:
> > > > > > > > > BCM57414 NetXtreme-E 10Gb/25Gb= RDMA Ethernet Controller
> > > > > > > > > (rev
> > > > > > > > > 01)
> > > > > > > > >
> > > > > > > > > The comparison between RDMA an= d TCP on the same NIC
> > > > > > > > > could make more
> > > > > > > > sense.
> > > > > > > >
> > > > > > > > It looks to me NICs are powerful no= w, but again as I
> > > > > > > > mentioned I don't think it'= s a reason we need to deprecate
> > > > > > > > rdma, especially if QEMU's rdma= migration has the chance
> > > > > > > > to be refactored
> > > using rsocket.
> > > > > > > >
> > > > > > > > Is there anyone who started looking= into that direction?
> > > > > > > > Would it make sense we start some P= oC now?
> > > > > > > >
> > > > > > >
> > > > > > > My team has finished the PoC refactoring= which works well.
> > > > > > >
> > > > > > > Progress:
> > > > > > > 1.=C2=A0 Implement io/channel-rdma.c, 2.= =C2=A0 Add unit test
> > > > > > > tests/unit/test-io-channel-rdma.c and ve= rifying it is
> > > > > > > successful, 3.=C2=A0 Remove the original= code from migration/rdma.c, 4.
> > > > > > > Rewrite the rdma_start_outgoing_migratio= n and
> > > > > > > rdma_start_incoming_migration logic, 5.= =C2=A0 Remove all rdma_xxx
> > > > > > > functions from migration/ram.c. (to prev= ent RDMA live
> > > > > > > migration from polluting the
> > > > > > core logic of live migration), 6.=C2=A0 The s= oft-RoCE implemented
> > > > > > by software is used to test the RDMA live mig= ration. It's successful.
> > > > > > >
> > > > > > > We will be submit the patchset later. > > > > > >
> > > > > > That's great news, thank you!
> > > > > >
> > > > > > --
> > > > > > Peter Xu
> > > > >
> > > > > For rdma programming, the current mainstream imple= mentation is
> > > > > to use
> > > rdma_cm to establish a connection, and then use verbs to tra= nsmit data.
> > > > >
> > > > > rdma_cm and ibverbs create two FDs respectively. T= he two FDs
> > > > > have different responsibilities. rdma_cm fd is use= d to notify
> > > > > connection establishment events, and verbs fd is u= sed to notify
> > > > > new CQEs. When
> > > poll/epoll monitoring is directly performed on the rdma_cm f= d, only
> > > a pollin event can be monitored, which means that an rdma_cm= event
> > > occurs. When the verbs fd is directly polled/epolled, only t= he
> > > pollin event can be listened, which indicates that a new CQE= is generated.
> > > > >
> > > > > Rsocket is a sub-module attached to the rdma_cm li= brary and
> > > > > provides rdma calls that are completely similar to= socket interfaces.
> > > > > However, this library returns only the rdma_cm fd = for listening
> > > > > to link
> > > setup-related events and does not expose the verbs fd (reada= ble and
> > > writable events for listening to data). Only the rpoll inter= face
> > > provided by the RSocket can be used to listen to related eve= nts.
> > > However, QEMU uses the ppoll interface to listen to the rdma= _cm fd
> (gotten by raccept API).
> > > > > And cannot listen to the verbs fd event.
> I'm confused, the rs_poll_arm
> :https://github.com/lin= ux-rdma/rdma-core/blob/master/librdmacm/rsocket.c#
> L3290
> For STREAM, rpoll setup fd for both cq fd and cm fd.
>
> > > > >
> > > > > Do you guys have any ideas? Thanks.
> > > > +cc linux-rdma
> > >
> > > Why include rdma community?
> > >
> >
> > Can rdma/rsocket provide an API to expose the verbs fd?
> Why do we need verbs fd? looks rsocket during rsend/rrecv is handling = the new
> completion if any via rs_get_comp
>
Actually I said the reason in the previous mail. Listing some header in lib= rdmacm.

/* verbs.h */
struct ibv_comp_channel {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 struct ibv_context=C2=A0 =C2=A0 =C2=A0*context;=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 int=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0fd;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 int=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0refcnt;
};

/* rdma_cma.h */
struct rdma_event_channel {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 int=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0fd;
};

/* rdma_cma.h */
struct rdma_cm_id {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 struct ibv_context=C2=A0 =C2=A0 =C2=A0 *verbs;<= br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 struct rdma_event_channel *channel;=C2=A0 =C2= =A0//=3D=3D> it can be gotten by rsocket.h
=C2=A0 =C2=A0 =C2=A0 =C2=A0 void=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 *context;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 struct ibv_qp=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0*qp;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 struct rdma_route=C2=A0 =C2=A0 =C2=A0 =C2=A0 ro= ute;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 enum rdma_port_space=C2=A0 =C2=A0 =C2=A0ps;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 uint8_t=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 port_num;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 struct rdma_cm_event=C2=A0 =C2=A0 *event;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 struct ibv_comp_channel *send_cq_channel;=C2=A0= // =3D=3D> can't be gotten so that Qemu can't read the CQE dat<= br>
ok, but the send_cq_channel is set the same as recv_cq= _channel:=C2=A0https://github.com/linux-rdma/rdma-core/blob/m= aster/librdmacm/rsocket.c#L855
and also use the same recv_cq = as send_cq.
=C2=A0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 struct ibv_cq=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0*send_cq;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 struct ibv_comp_channel *recv_cq_channel;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 struct ibv_cq=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0*recv_cq;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 struct ibv_srq=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 *srq;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 struct ibv_pd=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0*pd;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 enum ibv_qp_type=C2=A0 =C2=A0 =C2=A0 =C2=A0 qp_= type;
};

/* rsocket.h */
int raccept(int socket, struct sockaddr *addr, socklen_t *addrlen);
int rpoll(struct pollfd *fds, nfds_t nfds, int timeout);


> Another question to my mind is Daniel suggested a bit different way of= using
> rsocket: https://lore.kernel.org= /qemu-devel/ZjtOreamN8xF9FDE@redhat.com/
> Have you considered that?
>
We do use 'rsocket' APIs to refactor the RDMA code in QEMU and enco= unter the issue.


Regards,
-Gonglei

--000000000000dccac30619949735--