From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5D0ACC64ED6 for ; Wed, 1 Mar 2023 16:39:06 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pXPTc-00008G-Lw; Wed, 01 Mar 2023 11:38:48 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pXPTZ-00004v-MR for qemu-devel@nongnu.org; Wed, 01 Mar 2023 11:38:46 -0500 Received: from mail-wm1-x335.google.com ([2a00:1450:4864:20::335]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pXPTX-0003WJ-GG for qemu-devel@nongnu.org; Wed, 01 Mar 2023 11:38:45 -0500 Received: by mail-wm1-x335.google.com with SMTP id j19-20020a05600c191300b003eb3e1eb0caso7664526wmq.1 for ; Wed, 01 Mar 2023 08:38:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:message-id:in-reply-to:date :subject:cc:to:from:user-agent:references:from:to:cc:subject:date :message-id:reply-to; bh=MLJB0AywjKErpano3hVxQBKZ1gllESkhAW0hnMNDsWQ=; b=Ny6yujlRJ9bR/RWpfN7TJPhmn6Y2SpV5i143jJu4HQ5TYnz2MIBVVthziDHIsWZ3Xz dyQNWzOw3oZiOeU41/aa0VOR/rWnsNKAA8EzD402f/R/euqJ5KakgeTyzP5MuxdU3qbe k0+/GepenSV33Q7DbOHctuPIVpcPVd/8ErpCfyhtmOzRNF6eqV7Zfz9NYdV2QHWNrfIT 5y8XBwFdDUzAmg5Qt306eP3gYVlbSrgZr+of0bm1G+y/0Ax6OBtN7+gXd3pz0K+K7x4d 4gauoX9mKkXQc2eZ7HQ6uqXoIwmz3jiHPdKBDG/HYwnpzPtqM/SxCsmp7u8KgADBDrj5 mNbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:in-reply-to:date :subject:cc:to:from:user-agent:references:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=MLJB0AywjKErpano3hVxQBKZ1gllESkhAW0hnMNDsWQ=; b=Hm+v6xCeemWAdSFrRXzsVsFU44M68GEtme21mtkfrgbW4iGZMUmN4cY/onQjsBxgS1 M8mS65ld1MGCOCEpj7z1qiar7fP9t1E7huoWHF8rUFtPsvatpuNtVisvDk0rBvh8pNBK EZ7WxCP4zfgteMXewDWmPrjfiF/gbSXNSYSvAQmbWbh5Qy19ZWYcoJ51pYYIaf4nCNT6 S342Pt5fGYbatINlhCpa91WALxLDMtBWeHPlAlEnSRZJofHr5EJooN7DK/Nal9khVxQY gKqtwxqX/fM1GK0gZZdvu62JO108l7+YOuD6TNdQDq6e5af7nVs6Ey+TTAOJ3v9j/u1x /NPg== X-Gm-Message-State: AO0yUKWRazln1Cnpt5En6XfTnZbq0RSprC0fgEplQk/9QiKDnlia+7gd C/9FK+xX5pcMgpKWmAW/QgTf7Q== X-Google-Smtp-Source: AK7set8WBgfC36G3IFqFpRa7gBd2pVVtyWUHN64EG9LFoiuEgjP0tf5cL7IIFNkQ8zEijLa0yqq7mg== X-Received: by 2002:a05:600c:44d4:b0:3eb:29fe:7baa with SMTP id f20-20020a05600c44d400b003eb29fe7baamr5520403wmo.34.1677688721652; Wed, 01 Mar 2023 08:38:41 -0800 (PST) Received: from zen.linaroharston ([85.9.250.243]) by smtp.gmail.com with ESMTPSA id q14-20020a05600c46ce00b003daffc2ecdesm101200wmo.13.2023.03.01.08.38.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Mar 2023 08:38:41 -0800 (PST) Received: from zen (localhost [127.0.0.1]) by zen.linaroharston (Postfix) with ESMTP id DAA201FFB7; Wed, 1 Mar 2023 16:38:40 +0000 (GMT) References: User-agent: mu4e 1.9.21; emacs 29.0.60 From: Alex =?utf-8?Q?Benn=C3=A9e?= To: Stefan Hajnoczi Cc: Viresh Kumar , qemu-devel@nongnu.org, "Michael S. Tsirkin" , Vincent Guittot , stratos-dev@op-lists.linaro.org, Oleksandr Tyshchenko , xen-devel@lists.xen.org, Andrew Cooper , Juergen Gross , Sebastien Boeuf , Liu Jiang , Mathieu Poirier , virtio-dev@lists.oasis-open.org Subject: Re: [virtio-dev] [RFC QEMU] docs: vhost-user: Add custom memory mapping support Date: Wed, 01 Mar 2023 16:31:41 +0000 In-reply-to: Message-ID: <877cw0ctpr.fsf@linaro.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=2a00:1450:4864:20::335; envelope-from=alex.bennee@linaro.org; helo=mail-wm1-x335.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Stefan Hajnoczi writes: > [[PGP Signed Part:Undecided]] > Resend - for some reason my email didn't make it out. > > From: Stefan Hajnoczi > Subject: Re: [virtio-dev] [RFC QEMU] docs: vhost-user: Add custom memory > mapping support > To: Viresh Kumar > Cc: qemu-devel@nongnu.org, virtio-dev@lists.oasis-open.org, "Michael S. > Tsirkin" , Vincent Guittot , > Alex Benn=C3=A9e , > stratos-dev@op-lists.linaro.org, Oleksandr Tyshchenko > , xen-devel@lists.xen.org, Andrew Cooper > , Juergen Gross , Sebastien > Boeuf > , Liu Jiang , Mathieu > Poirier > Date: Tue, 21 Feb 2023 10:17:01 -0500 (1 week, 1 day, 1 hour ago) > Flags: seen, signed, personal > > On Tue, Feb 21, 2023 at 03:20:41PM +0530, Viresh Kumar wrote: >> The current model of memory mapping at the back-end works fine with >> Qemu, where a standard call to mmap() for the respective file >> descriptor, passed from front-end, is generally all we need to do before >> the front-end can start accessing the guest memory. >>=20 >> There are other complex cases though, where we need more information at >> the backend and need to do more than just an mmap() call. For example, >> Xen, a type-1 hypervisor, currently supports memory mapping via two >> different methods, foreign-mapping (via /dev/privcmd) and grant-dev (via >> /dev/gntdev). In both these cases, the back-end needs to call mmap() >> followed by an ioctl() (or vice-versa), and need to pass extra >> information via the ioctl(), like the Xen domain-id of the guest whose >> memory we are trying to map. >>=20 >> Add a new protocol feature, 'VHOST_USER_PROTOCOL_F_CUSTOM_MMAP', which >> lets the back-end know about the additional memory mapping requirements. >> When this feature is negotiated, the front-end can send the >> 'VHOST_USER_CUSTOM_MMAP' message type to provide the additional >> information to the back-end. >>=20 >> Signed-off-by: Viresh Kumar >> --- >> docs/interop/vhost-user.rst | 32 ++++++++++++++++++++++++++++++++ >> 1 file changed, 32 insertions(+) > > The alternative to an in-band approach is to configure these details > out-of-band. For example, via command-line options to the vhost-user > back-end: > > $ my-xen-device --mapping-type=3Dforeign-mapping --domain-id=3D123 > > I was thinking about both approaches and don't see an obvious reason to > choose one or the other. What do you think? In-band has the nice property of being dynamic and not having to have some other thing construct command lines. We are also trying to keep the daemons from being Xen specific and keep the type of mmap as an implementation detail that is mostly elided by the rust-vmm memory traits. > >> diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst >> index 3f18ab424eb0..f2b1d705593a 100644 >> --- a/docs/interop/vhost-user.rst >> +++ b/docs/interop/vhost-user.rst >> @@ -258,6 +258,23 @@ Inflight description >>=20=20 >> :queue size: a 16-bit size of virtqueues >>=20=20 >> +Custom mmap description >> +^^^^^^^^^^^^^^^^^^^^^^^ >> + >> ++-------+-------+ >> +| flags | value | >> ++-------+-------+ >> + >> +:flags: 64-bit bit field >> + >> +- Bit 0 is Xen foreign memory access flag - needs Xen foreign memory ma= pping. >> +- Bit 1 is Xen grant memory access flag - needs Xen grant memory mappin= g. >> + >> +:value: a 64-bit hypervisor specific value. >> + >> +- For Xen foreign or grant memory access, this is set with guest's xen = domain >> + id. > > This is highly Xen-specific. How about naming the feature XEN_MMAP > instead of CUSTOM_MMAP? If someone needs to add other mmap data later, > they should define their own struct instead of trying to squeeze into > the same fields as Xen. We hope to support additional mmap mechanisms in the future - one proposal is to use the hypervisor specific interface to support an ioctl() that creates a domain specific device which can then be treated more generically. Could we not declare the message as flag + n bytes of domain specific message as defined my msglen? > There is an assumption in this design that a single > VHOST_USER_CUSTOM_MMAP message provides the information necessary for > all mmaps. Are you sure the limitation that every mmap belongs to the > same domain will be workable in the future? Like a daemon servicing multiple VMs? Won't those still be separate vhost-user control channels though? > >> + >> C structure >> ----------- >>=20=20 >> @@ -867,6 +884,7 @@ Protocol features >> #define VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS 14 >> #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS 15 >> #define VHOST_USER_PROTOCOL_F_STATUS 16 >> + #define VHOST_USER_PROTOCOL_F_CUSTOM_MMAP 17 >>=20=20 >> Front-end message types >> ----------------------- >> @@ -1422,6 +1440,20 @@ Front-end message types >> query the back-end for its device status as defined in the Virtio >> specification. >>=20=20 >> +``VHOST_USER_CUSTOM_MMAP`` > > Most vhost-user protocol messages have a verb like > get/set/close/add/listen/etc. I suggest renaming this to > VHOST_USER_SET_XEN_MMAP_INFO. VHOST_USER_CONFIG_MMAP_QUIRKS? VHOST_USER_CONFIG_MMAP_TYPE? > >> + :id: 41 >> + :equivalent ioctl: N/A >> + :request payload: Custom mmap description >> + :reply payload: N/A >> + >> + When the ``VHOST_USER_PROTOCOL_F_CUSTOM_MMAP`` protocol feature has b= een >> + successfully negotiated, this message is submitted by the front-end to >> + notify the back-end of the special memory mapping requirements, that = the >> + back-end needs to take care of, while mapping any memory regions sent >> + over by the front-end. The front-end must send this message before >> + any memory-regions are sent to the back-end via ``VHOST_USER_SET_MEM_= TABLE`` >> + or ``VHOST_USER_ADD_MEM_REG`` message types. >> + >>=20=20 >> Back-end message types >> ---------------------- >> --=20 >> 2.31.1.272.g89b43f80a514 >>=20 >>=20 >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org >> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org >>=20 > > > > ---------- > > [[End of PGP Signed Part]] --=20 Alex Benn=C3=A9e Virtualisation Tech Lead @ Linaro