Kernel KVM virtualization development
 help / color / mirror / Atom feed
From: Chao Gao <chao.gao@intel.com>
To: Lu Baolu <baolu.lu@linux.intel.com>
Cc: <iommu@lists.linux.dev>, <x86@kernel.org>,
	<linux-coco@lists.linux.dev>, <kvm@vger.kernel.org>,
	Jason Gunthorpe <jgg@ziepe.ca>, Kevin Tian <kevin.tian@intel.com>,
	Kiryl Shutsemau <kas@kernel.org>,
	Rick Edgecombe <rick.p.edgecombe@intel.com>,
	<yilun.xu@linux.intel.com>, <xiaoyao.li@intel.com>,
	Binbin Wu <binbin.wu@linux.intel.com>,
	<linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v3 9/9] coco/tdx-host: Enable trusted DMA on IOMMUs on device connect
Date: Wed, 18 Nov 2026 16:11:27 +0800	[thread overview]
Message-ID: <av1dr3IGMqWA1-7r@intel.com> (raw)
In-Reply-To: <20261112043054.2664857-10-baolu.lu@linux.intel.com>

On Thu, Nov 12, 2026 at 12:30:54PM +0800, Lu Baolu wrote:
>Assigning a device to a TD for private memory access requires that DMA
>from that device be translated by structures the TDX module controls.
>An Intel IOMMU that implements the trusted DMA extension provides a
>parallel set of translation and invalidation resources for this purpose.
>They become usable once TDH.IOMMU.SETUP puts the IOMMU into Secure TDX
>Mode and the TDX module initializes them. The host keeps the regular
>translation path.
>
>Issue TDH.IOMMU.SETUP on demand, when the first device behind an IOMMU
>is connected through the TSM, rather than for every capable IOMMU at
>probe time. Secure TDX Mode costs host resources: the TDX module's
>metadata pages and half of the IOMMU's domain ID space. That cost should
>only be paid for IOMMUs that actually serve such devices.
>
>tdx_iommu_get() and tdx_iommu_put() keep a per-IOMMU refcount. The first
>get() issues TDH.IOMMU.SETUP and the last put() issues TDH.IOMMU.CLEAR.
>A single mutex covers both the refcount and the SEAMCALL that follows,
>so a device cannot join an IOMMU that is being cleared or whose setup
>has not finished. The refcount lives here rather than in the VT-d
>driver because tdx-host already owns the per-IOMMU state that must
>outlive TDH.IOMMU.SETUP: the IOMMU_MT pages and the TDX IOMMU ID.
>
>The IOMMU_MT parameter page lists two contiguous buffers for the
>trusted invalidation queue, followed by the metadata pages. The TDX
>module reports how many metadata pages it needs in its global metadata.
>
>With Dynamic PAMT, every 4K page given to the TDX module needs PAMT
>backing first. Take it for each page of the invalidation queue buffers
>and the metadata pages before TDH.IOMMU.SETUP, and drop it if setup
>fails or once TDH.IOMMU.CLEAR succeeds. The parameter page itself only
>carries the page list into the SEAMCALL and needs no backing.
>
>If TDH.IOMMU.CLEAR fails, the TDX module may still reference the
>IOMMU_MT pages, so they are leaked together with their PAMT backing,
>and the IOMMU is marked unusable. Later get() calls for that IOMMU fail
>with -EBUSY rather than reuse an IOMMU in an unknown state. tdx-host
>pins itself while such an IOMMU exists, so unloading can't lose track
>of it. If the failure happens during unload itself, only the tracking
>entry is freed; the VT-d driver still has the IOMMU acquired and keeps
>rejecting it.

If the pin can fail anyway, what's the point of taking it at all?

If tdx-host is reloaded, we can rely on the VT-d driver or the TDX module
to reject transitioning the IOMMU to secure mode.

>
>Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
>---
> drivers/virt/coco/tdx-host/Kconfig   |   5 +
> drivers/virt/coco/tdx-host/Makefile  |   1 +
> drivers/virt/coco/tdx-host/private.h |  21 ++
> drivers/virt/coco/tdx-host/main.c    |  11 +
> drivers/virt/coco/tdx-host/viommu.c  | 401 +++++++++++++++++++++++++++
> 5 files changed, 439 insertions(+)
> create mode 100644 drivers/virt/coco/tdx-host/private.h
> create mode 100644 drivers/virt/coco/tdx-host/viommu.c
>
>diff --git a/drivers/virt/coco/tdx-host/Kconfig b/drivers/virt/coco/tdx-host/Kconfig
>index f8c8cc29d46c..1395929f05bb 100644
>--- a/drivers/virt/coco/tdx-host/Kconfig
>+++ b/drivers/virt/coco/tdx-host/Kconfig
>@@ -6,3 +6,8 @@ config TDX_HOST_SERVICES
> 	select FW_UPLOAD
> 	select PCI_TSM
> 	default m
>+
>+config TDX_CONNECT
>+	def_bool y
>+	depends on TDX_HOST_SERVICES
>+	depends on INTEL_IOMMU

Do we need this TDX_CONNECT config? We can just add "depends on
INTEL_IOMMU" to TDX_HOST_SERVICES, and then drop the stub functions
in private.h. Patch 7 already makes TDX_HOST_SERVICES depend on PCI
in the same way.

The cost is that without INTEL_IOMMU, none of the tdx-host services
are available, including TDX module updates. But I doubt anyone would
want to launch CoCo VMs while compiling out INTEL_IOMMU entirely.

We can always add this option back if such a configuration turns out
to be practical.

>+/*
>+ * Serializes the refcounts together with TDH.IOMMU.SETUP and
>+ * TDH.IOMMU.CLEAR, so a device can't join an IOMMU that is being
>+ * cleared, or one whose setup hasn't finished.
>+ */
>+static DEFINE_MUTEX(tdx_iommu_lock);
>+static LIST_HEAD(tdx_iommu_list);
>+
>+static void *tdx_iommu_alloc_pages(int node, size_t size)
>+{
>+	struct folio *folio;
>+
>+	if (node == NUMA_NO_NODE)
>+		node = numa_mem_id();
>+
>+	folio = __folio_alloc_node(GFP_KERNEL | __GFP_ZERO, get_order(size), node);

Would kzalloc_node() work here? kzalloc_node() handles NUMA_NO_NODE
itself, and there's no need to convert between folios and void *.


>+	if (!folio)
>+		return NULL;
>+
>+	return folio_address(folio);
>+}
>+
>+static void tdx_iommu_free_pages(void *virt)
>+{
>+	if (!virt)
>+		return;
>+
>+	folio_put(virt_to_folio(virt));
>+}
>+
>+/* Number of 4K pages backing IOMMU_MT entry @i. */
>+static unsigned int mt_entry_nr_pages(unsigned int i)
>+{
>+	return i < TDX_IOMMU_MT_IQ_NR ? IQ_BUFFER_SIZE >> PAGE_SHIFT : 1;
>+}
>+
>+/*
>+ * Drop the PAMT backing of the first @nr 4K pages handed to the TDX
>+ * module, in the order mt_pamt_get() took it. Pass UINT_MAX for all.
>+ */
>+static void mt_pamt_put(struct tdx_iommu_pages *mt, unsigned int nr)
>+{
>+	unsigned int i, j;
>+
>+	for (i = 0; i < mt->nr_entries && nr; i++) {
>+		unsigned long pfn = PHYS_PFN(__pa(mt->pages[i]));
>+
>+		for (j = 0; j < mt_entry_nr_pages(i) && nr; j++, nr--)
>+			tdx_pamt_put(pfn + j);
>+	}
>+}
>+
>+/*
>+ * With Dynamic PAMT, every 4K page handed to the TDX module needs PAMT
>+ * backing first. This covers the IQ buffers and the metadata pages, not
>+ * the root page, which only carries the list of pages into the
>+ * SEAMCALL. tdx_pamt_get() is a no-op without Dynamic PAMT.
>+ */
>+static int mt_pamt_get(struct tdx_iommu_pages *mt)
>+{
>+	unsigned int i, j, nr = 0;
>+	int ret;
>+
>+	for (i = 0; i < mt->nr_entries; i++) {
>+		unsigned long pfn = PHYS_PFN(__pa(mt->pages[i]));
>+
>+		for (j = 0; j < mt_entry_nr_pages(i); j++) {
>+			ret = tdx_pamt_get(pfn + j, NULL);
>+			if (ret) {
>+				mt_pamt_put(mt, nr);
>+				return ret;
>+			}
>+			nr++;
>+		}
>+	}
>+
>+	return 0;
>+}
>+
>+static void free_mt_pages(struct tdx_iommu_pages *array)
>+{
>+	unsigned int i;
>+
>+	if (IS_ERR_OR_NULL(array))
>+		return;
>+
>+	mt_pamt_put(array, UINT_MAX);
>+
>+	for (i = 0; i < array->nr_entries; i++)
>+		tdx_iommu_free_pages(array->pages[i]);
>+
>+	tdx_iommu_free_pages(array->root);
>+	kfree(array->pages);
>+	kfree(array);
>+}
>+
>+DEFINE_FREE(free_mt_pages, struct tdx_iommu_pages *, free_mt_pages(_T))
>+
>+static void **alloc_mt_pages(unsigned int nr_entries, int node)
>+{
>+	unsigned int i;
>+	void **pages;
>+	void *vaddr;
>+
>+	pages = kzalloc_objs(*pages, nr_entries);
>+	if (!pages)
>+		return NULL;
>+
>+	/* Allocate two contiguous buffers for the invalidation queue. */
>+	pages[0] = tdx_iommu_alloc_pages(node, IQ_BUFFER_SIZE);
>+	if (!pages[0])
>+		goto free_pages;
>+
>+	pages[1] = tdx_iommu_alloc_pages(node, IQ_BUFFER_SIZE);
>+	if (!pages[1])
>+		goto free_pages;

You can use mt_entry_nr_pages() to fold the two IQ buffer allocations
into the loop below.

>+
>+	/* Allocate the required number of pages for the IOMMU metadata. */
>+	for (i = TDX_IOMMU_MT_IQ_NR; i < nr_entries; i++) {
>+		vaddr = tdx_iommu_alloc_pages(node, SZ_4K);
>+		if (!vaddr)
>+			goto free_pages;
>+		pages[i] = vaddr;
>+	}
>+
>+	return pages;
>+
>+free_pages:
>+	for (i = 0; i < nr_entries; i++)
>+		tdx_iommu_free_pages(pages[i]);
>+	kfree(pages);
>+
>+	return NULL;
>+}

      reply	other threads:[~2026-11-18  8:11 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-12  4:30 [PATCH v3 0/9] coco/tdx-host: Introduce trusted DMA initialization support Lu Baolu
2026-11-12  4:30 ` [PATCH v3 1/9] x86/virt/tdx: Add SEAMCALL wrappers for IOMMU setup/clear Lu Baolu
2026-11-12  4:30 ` [PATCH v3 2/9] x86/virt/tdx: Read global metadata for trusted IOMMU Lu Baolu
2026-11-12  4:30 ` [PATCH v3 3/9] x86/virt/tdx: Export Dynamic PAMT helpers to tdx-host Lu Baolu
2026-11-12  4:30 ` [PATCH v3 4/9] iommu/vt-d: Add interface for tdx-host when enabling trusted DMA Lu Baolu
2026-11-12  4:30 ` [PATCH v3 5/9] iommu/vt-d: Block GCMD and ECMD while acquired by the TDX module Lu Baolu
2026-11-12  4:30 ` [PATCH v3 6/9] iommu/vt-d: Refuse system suspend " Lu Baolu
2026-11-12  4:30 ` [PATCH v3 7/9] coco/tdx-host: Add skeleton to support PCI link encryption Lu Baolu
2026-11-18  7:16   ` Chao Gao
2026-11-12  4:30 ` [PATCH v3 8/9] coco/tdx-host: Rename tdx-host.c to main.c Lu Baolu
2026-11-12  4:30 ` [PATCH v3 9/9] coco/tdx-host: Enable trusted DMA on IOMMUs on device connect Lu Baolu
2026-11-18  8:11   ` Chao Gao [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=av1dr3IGMqWA1-7r@intel.com \
    --to=chao.gao@intel.com \
    --cc=baolu.lu@linux.intel.com \
    --cc=binbin.wu@linux.intel.com \
    --cc=iommu@lists.linux.dev \
    --cc=jgg@ziepe.ca \
    --cc=kas@kernel.org \
    --cc=kevin.tian@intel.com \
    --cc=kvm@vger.kernel.org \
    --cc=linux-coco@lists.linux.dev \
    --cc=linux-kernel@vger.kernel.org \
    --cc=rick.p.edgecombe@intel.com \
    --cc=x86@kernel.org \
    --cc=xiaoyao.li@intel.com \
    --cc=yilun.xu@linux.intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox