From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 368ADE77188 for ; Thu, 9 Jan 2025 03:13:58 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.867789.1279342 (Exim 4.92) (envelope-from ) id 1tVizA-000083-22; Thu, 09 Jan 2025 03:13:28 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 867789.1279342; Thu, 09 Jan 2025 03:13:28 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tViz9-00007r-VW; Thu, 09 Jan 2025 03:13:27 +0000 Received: by outflank-mailman (input) for mailman id 867789; Thu, 09 Jan 2025 03:13:26 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tViz8-00006B-Ar for xen-devel@lists.xenproject.org; Thu, 09 Jan 2025 03:13:26 +0000 Received: from fout-b2-smtp.messagingengine.com (fout-b2-smtp.messagingengine.com [202.12.124.145]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id aeb9fdde-ce37-11ef-99a4-01e77a169b0f; Thu, 09 Jan 2025 04:13:23 +0100 (CET) Received: from phl-compute-08.internal (phl-compute-08.phl.internal [10.202.2.48]) by mailfout.stl.internal (Postfix) with ESMTP id 76D671140115; Wed, 8 Jan 2025 22:13:21 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-08.internal (MEProxy); Wed, 08 Jan 2025 22:13:21 -0500 Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 8 Jan 2025 22:13:18 -0500 (EST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: aeb9fdde-ce37-11ef-99a4-01e77a169b0f DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= invisiblethingslab.com; h=cc:cc:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm2; t=1736392401; x=1736478801; bh=cR6ccP5DFYB7kn+Faf0luNzEduPpGXKTNSRDekWDv3c=; b= NwIEOslt+RQZPmtHMJY/DRbg4wBDysINA5Yn+oGTQ4aCtaxCt4tCyukBse+0lTqW QOjXQjC9fTeVbnWX226KkO1vHaVwp+/mCr11kXCvJdlCIY78xV+MmKMzs8H9tAMm OvEGDEHAUHZvQdpr1m+SxOY47Kn1s9aXKmZ4awRaGZhdgF3GwmHGbHykot9fKW+P 5pJIs+2PY8CD8N00KYDP5JUZlsOnipDoZh+P9tVwd5R09iGBGX9yhM/dOe5lyKWx cgXS86q09J3/T8d0utHYOTdUXfC+gwnchv/9uEGCoblwza6ePQQMVzbFCnLDfLCT U5fa1BYO5tECsWnHzJZBIg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1736392401; x=1736478801; bh=cR6ccP5DFYB7kn+Faf0luNzEduPpGXKTNSR DekWDv3c=; b=CLxVdFPH3wllyngzUCs+it0uNrlH1ZQQhs/1Fo7JrWHD9PEiXFV IEy+vG4QWlcnWAQcy6aKrpnZU21GELTZcIOxVjWLQvgoBw45Su5xYPWKX+9ZUhxz ruWr+Pz10o+abmhHkBOKAgbCXM07hFi/6Zbkr1NjFkcrQ5pc39vlO9xflyoOnvqd Ah1D2v8gdjTOKik2whd9QSqkCJqtPAg8J9URFWTbBxcx2VpCvqb4KUYwvwoamUNs ExRuxN63HAMeDiYfkeFHYf7NbFcGX9J+glfd7vSBrVZpzgwb7Xb1Y3vb4JqEMMCe TGF7Wm6xKJPaFudCNHdl5WFo+Z5pZVMzIfw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefuddrudeghedgheeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnh htshculddquddttddmnecujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtjeen ucfhrhhomhepofgrrhgvkhcuofgrrhgtiiihkhhofihskhhiqdfikphrvggtkhhiuceomh grrhhmrghrvghksehinhhvihhsihgslhgvthhhihhnghhslhgrsgdrtghomheqnecuggft rfgrthhtvghrnhepueekteetgefggfekudehteegieeljeejieeihfejgeevhfetgffgte euteetueetnecuffhomhgrihhnpehgihhthhhusgdrtghomhenucevlhhushhtvghrufhi iigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehmrghrmhgrrhgvkhesihhnvhhish hisghlvghthhhinhhgshhlrggsrdgtohhmpdhnsggprhgtphhtthhopedutddpmhhouggv pehsmhhtphhouhhtpdhrtghpthhtohepthgvugguhidrrghsthhivgesvhgrthgvshdrth gvtghhpdhrtghpthhtohepgigvnhdquggvvhgvlheslhhishhtshdrgigvnhhprhhojhgv tghtrdhorhhgpdhrtghpthhtoheprghnughrvgifrdgtohhophgvrhefsegtihhtrhhigi drtghomhdprhgtphhtthhopehjsggvuhhlihgthhesshhushgvrdgtohhmpdhrtghpthht ohepjhhulhhivghnseigvghnrdhorhhgpdhrtghpthhtohepshhsthgrsggvlhhlihhnih eskhgvrhhnvghlrdhorhhgpdhrtghpthhtoheprhhoghgvrhdrphgruhestghithhrihig rdgtohhmpdhrtghpthhtoheplhhukhgrshiisehhrgifrhihlhhkohdrphhlpdhrtghpth htohepughpshhmihhthhesrghpvghrthhushhsohhluhhtihhonhhsrdgtohhm X-ME-Proxy: Feedback-ID: i1568416f:Fastmail Date: Thu, 9 Jan 2025 04:13:15 +0100 From: Marek =?utf-8?Q?Marczykowski-G=C3=B3recki?= To: Teddy Astie Cc: xen-devel@lists.xenproject.org, Andrew Cooper , Jan Beulich , Julien Grall , Stefano Stabellini , Roger Pau =?utf-8?B?TW9ubsOp?= , Lukasz Hawrylko , "Daniel P. Smith" , Mateusz =?utf-8?B?TcOzd2th?= Subject: Re: [XEN RFC PATCH v4 0/5] IOMMU subsystem redesign and PV-IOMMU interface Message-ID: References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="2eaxd8hbGHv9qt/u" Content-Disposition: inline In-Reply-To: --2eaxd8hbGHv9qt/u Content-Type: text/plain; protected-headers=v1; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Date: Thu, 9 Jan 2025 04:13:15 +0100 From: Marek =?utf-8?Q?Marczykowski-G=C3=B3recki?= To: Teddy Astie Cc: xen-devel@lists.xenproject.org, Andrew Cooper , Jan Beulich , Julien Grall , Stefano Stabellini , Roger Pau =?utf-8?B?TW9ubsOp?= , Lukasz Hawrylko , "Daniel P. Smith" , Mateusz =?utf-8?B?TcOzd2th?= Subject: Re: [XEN RFC PATCH v4 0/5] IOMMU subsystem redesign and PV-IOMMU interface On Mon, Nov 04, 2024 at 02:28:38PM +0000, Teddy Astie wrote: > This work has been presented at Xen Summit 2024 during the > IOMMU paravirtualization and Xen IOMMU subsystem rework > design session. >=20 > Operating systems may want to have access to a IOMMU in order to do DMA > protection or implement certain features (e.g VFIO on Linux). >=20 > VFIO support is mandatory for framework such as SPDK, which can be useful= to > implement an alternative storage backend for virtual machines [1]. >=20 > In this patch series, we introduce in Xen the ability to manage several > contexts per domain and provide a new hypercall interface to allow guests > to manage IOMMU contexts. >=20 > The VT-d driver is updated to support these new features. >=20 > [1] Using SPDK with the Xen hypervisor - FOSDEM 2023 > --- > Changed in v2 : > * fixed Xen crash when dumping IOMMU contexts (using X debug key) > with DomUs without IOMMU > * s/dettach/detach/ > * removed some unused includes > * fix dangling devices in contexts with detach >=20 > Changed in v3 : > * lock entirely map/unmap in hypercall > * prevent IOMMU operations on dying contexts (fix race condition) > * iommu_check_context+iommu_get_context -> iommu_get_context and check fo= r NULL >=20 > Changed in v4 : > * Part of initialization logic is moved to domain or toolstack (IOMMU_ini= t) > + domain/toolstack now decides on "context count" and "pagetable pool s= ize" > + for now, all domains are able to initialize PV-IOMMU > * introduce "dom0-iommu=3Dno-dma" to make default context block all DMA > (disables HAP and sync-pt), enforcing usage of PV-IOMMU for DMA > Can be used to expose properly "Pre-boot DMA protection" > * redesigned locking logic for contexts > + contexts are accessed using iommu_get_context and released with iommu= _put_context >=20 > TODO: > * add stub implementations for bissecting needs and non-ported IOMMU impl= ementations > * fix some issues with no-dma+PV and grants > * complete "no-dma" mode (expose to toolstack, add documentation, ...) > * properly define nested mode and PASID support Hi, I finally got time to try this revision (sorry it took so long!). My goal was to test it this time with some HVM domU too. I didn't get very far... Issues I hit: 1. AMD IOMMU driver is not converted (fails to build), for now disabled CONFIG_AMD_IOMMU. 2. PV shim build fails (linker fails to find p2m_add_identity_entry symbol referenced from iommu.c) 3. Xen complains on boot about missing endbr64 (surprisingly, it didn't exploded): (XEN) alt table ffff82d0404234d8 -> ffff82d040432d82 (XEN) altcall iommu_get_max_iova+0x11/0x30 dest iommu.c#intel_iommu_get= _max_iova has no endbr64 (XEN) altcall context.c#iommu_reattach_phantom+0x30/0x50 dest iommu.c#i= ntel_iommu_add_devfn has no endbr64 (XEN) altcall context.c#iommu_detach_phantom+0x25/0x40 dest iommu.c#int= el_iommu_remove_devfn has no endbr64 (XEN) altcall iommu_context_init+0x27/0x40 dest iommu.c#intel_iommu_con= text_init has no endbr64 (XEN) altcall iommu_attach_context+0x3c/0xd0 dest iommu.c#intel_iommu_a= ttach has no endbr64 (XEN) altcall context.c#iommu_attach_context.cold+0x1d/0x53 dest iommu.= c#intel_iommu_detach has no endbr64 (XEN) altcall iommu_detach_context+0x37/0xa0 dest iommu.c#intel_iommu_d= etach has no endbr64 (XEN) altcall iommu_reattach_context+0x95/0x240 dest iommu.c#intel_iomm= u_reattach has no endbr64 (XEN) altcall context.c#iommu_reattach_context.cold+0x29/0x110 dest iom= mu.c#intel_iommu_reattach has no endbr64 (XEN) altcall iommu_context_teardown+0x3f/0xa0 dest iommu.c#intel_iommu= _context_teardown has no endbr64 (XEN) altcall pci.c#deassign_device+0x99/0x270 dest iommu.c#intel_iommu= _add_devfn has no endbr64 4. Starting a HVM domU with PCI device fails with: libxl: libxl_pci.c:1552:pci_add_dm_done: Domain 1:xc_assign_device fail= ed: No space left on device libxl: libxl_pci.c:1875:device_pci_add_done: Domain 1:libxl__device_pci= _add failed for PCI device 0:aa:0.0 (rc -3) libxl: libxl_create.c:2061:domcreate_attach_devices: Domain 1:unable to= add pci devices I didn't change anything in the toolstack - maybe default context needs to be initialized somehow? But the docs suggest the default context should work out of the box. On the other hand, changelog for v4 says some parts are moved to the toolstack, but I don't see any changes in tools/ in this series... FWIW The exact version I tried is this (this series, on top of staging + qubes patches): https://github.com/QubesOS/qubes-vmm-xen/pull/200 At this stage, dom0 kernel didn't have PV-IOMMU driver included yet. Full Xen log, with some debug info collected: https://gist.github.com/marmarek/e7ac2571df033c7181bf03f21aa5f9ab --=20 Best Regards, Marek Marczykowski-G=C3=B3recki Invisible Things Lab --2eaxd8hbGHv9qt/u Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAmd/PssACgkQ24/THMrX 1ywr9wf/Qv4RzieRjyc+zxLq8Klw4GrZRFE7/iQyidtypTBufGCju0g3JVwROHpC p1C3+NL08uNdzwzxbqWu9gfj0J6pHWIZRlkSEMTi7rGkAXt0t6oU5LEAPMQTMc+g tM5koEKqemkMrdknlmr2wKEfLEZaok73yjj2KEMCG4Fq6wx0oc/TpsM+JcpcY8jt wofvIJQccAoRi2P9agLySXE0JlVWyssnCI2kkVhezc0U8wmYnPXOKZIbqNNWh9FH EYmvRsysKdA/WuWsc0kM+81lfUvqA1F36DL0NiA2HoWBz9mwAwPdNJhKWlcFACpb U0evuzJtKE0OBc9+FRvY1EkE4lmbMw== =1XLc -----END PGP SIGNATURE----- --2eaxd8hbGHv9qt/u--