From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 20500C02190 for ; Sun, 2 Feb 2025 05:09:36 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.880313.1290405 (Exim 4.92) (envelope-from ) id 1teSE9-0003RD-Jm; Sun, 02 Feb 2025 05:09:01 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 880313.1290405; Sun, 02 Feb 2025 05:09:01 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1teSE9-0003R5-FX; Sun, 02 Feb 2025 05:09:01 +0000 Received: by outflank-mailman (input) for mailman id 880313; Sun, 02 Feb 2025 05:09:00 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1teSE8-0003Qj-2U for xen-devel@lists.xenproject.org; Sun, 02 Feb 2025 05:09:00 +0000 Received: from fout-a2-smtp.messagingengine.com (fout-a2-smtp.messagingengine.com [103.168.172.145]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id cd757400-e123-11ef-a0e7-8be0dac302b0; Sun, 02 Feb 2025 06:08:56 +0100 (CET) Received: from phl-compute-01.internal (phl-compute-01.phl.internal [10.202.2.41]) by mailfout.phl.internal (Postfix) with ESMTP id 36F2A13800F9; Sun, 2 Feb 2025 00:08:55 -0500 (EST) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-01.internal (MEProxy); Sun, 02 Feb 2025 00:08:55 -0500 Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 2 Feb 2025 00:08:53 -0500 (EST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: cd757400-e123-11ef-a0e7-8be0dac302b0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= invisiblethingslab.com; h=cc:content-type:content-type:date:date :from:from:in-reply-to:message-id:mime-version:reply-to:subject :subject:to:to; s=fm3; t=1738472935; x=1738559335; bh=rr1/byr+Wd oUVFGph7xFQwvycFnOeNWMbFMI/glDuBM=; b=Q/05GUTxnXCkG8TjV+QgIg1zk1 D/IkxXDnEnNB+b/eGZ4VPclHUXxhCffBsUy3E9xpn5RMZWVMoc756E7D3fYY922m mZo6ZMN55wNaCf+VRQcVvhPhOaiYAWjOTmpKCvnMnM2nwcEaMpjsphYdLFZBVd2n b5E4W1mC/WurGSry/LHKxDCFS8uwUWORWz6RtDJg1S5/9qPJSFRONLMUvkzQ/316 0bxM2WBrEy4RJBfN/mD7kQoUrtgnySdAVYYAojmFtJLNce0/IzfwYZZG+d51+lOU PeGXuKMFj5benmCjpel4tbi6o3uS3tUx8z87XqUgmibPdV2LWdkUTpu55dbg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:message-id :mime-version:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1738472935; x= 1738559335; bh=rr1/byr+WdoUVFGph7xFQwvycFnOeNWMbFMI/glDuBM=; b=k yc8YgL35qKHFKQnQUCEkTYrk+DPiURcRoH+S/pce3F+MBT+rwmR0EjhPHwQ8dFBn r9Ah5src4J3+3cC9xt9Ta1INVDhhWu7Un/TLkTBWvVAX7uo1xGTc+x5kgQcGMQ07 utHslzjSGpGo8tsMOCOoug9fGcd4dz4pCeYgMqOtzZqJimmA82b/q1ADW8JaWrmH 3tE7+Qhr4rH3GPhe7HRQAYkkdC7XK7h6ON9aFLdTu+mwq1t/kzmt8j/O5WdEHUl3 ppjh/ONpns7tfuO4B2RBpzJp2NdXSSaQQSgm+yLgUe548KKy7yMbq+1OP1RVGX8u i+GvVWjX85Xf33pqmFYpQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgddufeejhecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpeffhffvuffkgggtugesghdtreertddtvdenucfh rhhomhepffgvmhhiucforghrihgvucfqsggvnhhouhhruceouggvmhhisehinhhvihhsih gslhgvthhhihhnghhslhgrsgdrtghomheqnecuggftrfgrthhtvghrnhepudejgfejgeeg lefhveekledtkedvuefhteeiffefhfekhefhveehhfekgfdugeeknecuvehluhhsthgvrh fuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepuggvmhhisehinhhvihhsihgs lhgvthhhihhnghhslhgrsgdrtghomhdpnhgspghrtghpthhtohepudeipdhmohguvgepsh hmthhpohhuthdprhgtphhtthhopehhohhnghhlvghiuddrhhhurghnghesrghmugdrtgho mhdprhgtphhtthhopehrrgihrdhhuhgrnhhgsegrmhgurdgtohhmpdhrtghpthhtohepug hmihhtrhihrdhoshhiphgvnhhkohestgholhhlrggsohhrrgdrtghomhdprhgtphhtthho pegurhhiqdguvghvvghlsehlihhsthhsrdhfrhgvvgguvghskhhtohhprdhorhhgpdhrtg hpthhtoheprghirhhlihgvugesrhgvughhrghtrdgtohhmpdhrtghpthhtohepkhhrrgig vghlsehrvgguhhgrthdrtghomhdprhgtphhtthhopehguhhrtghhvghtrghnshhinhhghh estghhrhhomhhiuhhmrdhorhhgpdhrtghpthhtohepohhlvhgrfhhfvgesghhmrghilhdr tghomhdprhgtphhtthhopegrkhhihhhikhhordhouggrkhhisegurgihnhhigidrtghomh X-ME-Proxy: Feedback-ID: iac594737:Fastmail Date: Sun, 2 Feb 2025 00:08:46 -0500 From: Demi Marie Obenour To: "Huang, Honglei1" , Huang Rui , Dmitry Osipenko , dri-devel@lists.freedesktop.org, David Airlie , Gerd Hoffmann , Gurchetan Singh , Chia-I Wu , Akihiko Odaki , Lingshan Zhu , Xen developer discussion , Marek =?us-ascii?B?PT91dGYtOD9RP01hcmN6eWtvd3NraS1HPUMzPUIzcmVja2k/?= =?us-ascii?Q?=3D?= , Xenia Ragiadakou , Stefano Stabellini , Andrew Cooper , Roger Pau =?utf-8?B?TW9ubsOp?= Subject: Xen memory management primitives for GPU virtualization Message-ID: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="xfrIvn0OgIq9mZNB" Content-Disposition: inline --xfrIvn0OgIq9mZNB Content-Type: text/plain; protected-headers=v1; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Date: Sun, 2 Feb 2025 00:08:46 -0500 From: Demi Marie Obenour To: "Huang, Honglei1" , Huang Rui , Dmitry Osipenko , dri-devel@lists.freedesktop.org, David Airlie , Gerd Hoffmann , Gurchetan Singh , Chia-I Wu , Akihiko Odaki , Lingshan Zhu , Xen developer discussion , Marek =?us-ascii?B?PT91dGYtOD9RP01hcmN6eWtvd3NraS1HPUMzPUIzcmVja2k/?= =?us-ascii?Q?=3D?= , Xenia Ragiadakou , Stefano Stabellini , Andrew Cooper , Roger Pau =?utf-8?B?TW9ubsOp?= Subject: Xen memory management primitives for GPU virtualization Cc:=20 Bcc:=20 Subject: Xen requirements for GPU virtualization via virtio-GPU Reply-To:=20 X-Mutt-Fcc: =3DINBOX,=3Dxen-devel,=3DSent X-Mutt-PGP: S Recently, AMD submitted patches to the dri-devel mailing list to support using application-provided buffers in virtio-GPU. This feature is called Shared Virtual Memory (SVM) and it is implemented via an API called User Pointer (userptr). This lead to some discussion on dri-devel@lists.freedesktop.org and dri-devel IRC, from which I concluded that Xen is missing critical primitives for GPU-accelerated graphics and compute. The missing primitives for graphics are the ones discussed at Xen Project Summit 2024, but it turns out that additional primitives are needed for compute workloads. As discussed at Xen Project Summit 2024, GPU acceleration via virtio-GPU requires that an IOREQ server have access to the following primitives: 1. Map: Map a backend-provided buffer into the frontend. The buffer might point to system memory or to a PCIe BAR. The frontend is _not_ allowed to use these buffers in hypercalls or grant them to other domains. Accessing the pages using hypercalls directed at the frontend fails as if the frontend did not have the pages. The only exception is that the frontend _may_ be allowed to use the buffer in a Map operation, provided that Revoke (below) is transitive. 2. Revoke: Revoke access to a buffer provided by the backend. Once access is revoked, no operation on or in the frontend domain can access or modify the pages, and the backend can safely reuse the backing memory for other purposes. Furthermore, revocation is not allowed to fail unless the backend or hypervisor is buggy, and if it does fail for any reason, the backend will panic. Once access is revoked, further accesses by the frontend will cause a fault that the backend can intercept. Map can be handled by userspace, but Revoke must be handled entirely in-kernel. This is because Revoke happens from a Linux MMU notifier callback, and those are not allowed to block, fail, or involve userspace in any way. Since MMU notifier callbacks are called before freeing memory, failure means that some other part of the system still has access to freed memory that might be reused for other purposes, which is a security vulnerability. It turns out that compute has additional requirements. Graphics APIs use DMA buffers (dmabufs), which only support a subset of operations. In particular, direct I/O doesn't work. Compute APIs allow users to make malloc'd memory accessible to the GPU. This memory can be used in Linux kernel direct I/O and in other operations that do not work with dmabufs. However, such memory starts out as frontend-owned pages, so it must be converted to backend pages before it can be used by the GPU. Linux supports migration of userspace pages, but this is too unreliable to be used for this purpose. Instead, it will need to be done by Xen and the backend. This requires two additional primitives: 3. Steal: Convert frontend-owned pages to backend-owned pages and provide the backend with a mapping of the page. After a successful Steal operation, the pages are in the same state as if they had been provided via Map. Steal fails if the pages are currently being used in a hypercall, are MMIO (as opposed to system memory), were provided by another domain via Map or grant tables, are currently foreign mapped, are currently granted to another domain, or more generally are accessible to any domain other than the target domain. The frontend's quota is decreased by the number of pages stolen, and the backend's quota is increased by the same amount. A successful Steal operation means that Revoke and Map can be used to operate on the pages. 4. Return: Convert a backend-owned page to a frontend-owned page. After a successful call to Return, the backend is no lonter able to use Revoke or Map. The returned page ceases to count against backend quota and now counts against frontend quota. Are these operations ones that Xen is interested in providing? There may be other primitives that are sufficient to implement the above four, but I believe that any solution that allows virtio-GPU to work must allow the above four operations to be implemented. Without the first two, virtio-GPU will not be able to support Vulkan or native contexts, and without the second two also being present, shared virtual memory and compute APIs that require it will not work. --=20 Sincerely, Demi Marie Obenour (she/her/hers) Invisible Things Lab --xfrIvn0OgIq9mZNB Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEopQtqVJW1aeuo9/sszaHOrMp8lMFAmee/d4ACgkQszaHOrMp 8lPaQQ/9Ern0ko7OPnEl4/TRQ+LI7iawb45bEXoOoOmj2AeHl3hlvJK6eGQ3Q520 71VAeRYl3h5MZm5LQRjTUm1iz+q+3vQJhZuUs6m/mD0sla2XHIAS2ILJWe8bNILb q5Pq6bsSlEZMyVJOzrpp5Ym/IWf0oYYvgT6epkggI5lmND3wE8/2ns0RupZW5jQW 0pzNhPYVe9WHMtCZDPQWqzTgrXxbabAw1QO2fV9Epf/jojqbsBhnWoijqHoTKfjd pi7J+QG+v4KyRM1Oql9f/JFhwJ5rP+te/dn08hsKuuwTCortSq3gRjyGup1tHDvU l3roIHfT3nppVUh5cDiLTDybYtD16unhRcoEPO+VtH8kQ15Q9vY3u326k563f5xg WRRVCJUJX+druNkQg1YA1XRGE6N3LvvD189+GWnIsZHuIf4/LD5lC/VlNjxscAIl NRIV6TPP/cgiCLLX6zT1AoCS1vum4jRIopjhOTXZUlcrI9Pn+ZM9cEsazAxE9g4/ xHr3idiUkKksj5D2TjqTHfNCQn3WvMKYphzbxxOLKFLMnQBQRGJ6wSDqY+PGJAIX 1GMQ+pbsJO5XkWj+wJm7j7nDAYas+/qz+A3S+Ko1FM0OJ8/2HYcMzyf+wg219k6j JkKMA0HmxUfgc5tcmyRd0jZZkAQ301ChEatIwzEHTXb2gxv1eKk= =pzYf -----END PGP SIGNATURE----- --xfrIvn0OgIq9mZNB--