All of lore.kernel.org
 help / color / mirror / Atom feed
From: Chao Gao <chao.gao@intel.com>
To: Lu Baolu <baolu.lu@linux.intel.com>
Cc: <iommu@lists.linux.dev>, <x86@kernel.org>,
	<linux-coco@lists.linux.dev>, <kvm@vger.kernel.org>,
	Jason Gunthorpe <jgg@ziepe.ca>, Kevin Tian <kevin.tian@intel.com>,
	Kiryl Shutsemau <kas@kernel.org>,
	Rick Edgecombe <rick.p.edgecombe@intel.com>,
	<yilun.xu@linux.intel.com>, <xiaoyao.li@intel.com>,
	Binbin Wu <binbin.wu@linux.intel.com>,
	<linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v3 9/9] coco/tdx-host: Enable trusted DMA on IOMMUs on device connect
Date: Wed, 18 Nov 2026 16:11:27 +0800	[thread overview]
Message-ID: <av1dr3IGMqWA1-7r@intel.com> (raw)
In-Reply-To: <20261112043054.2664857-10-baolu.lu@linux.intel.com>

On Thu, Nov 12, 2026 at 12:30:54PM +0800, Lu Baolu wrote:
>Assigning a device to a TD for private memory access requires that DMA
>from that device be translated by structures the TDX module controls.
>An Intel IOMMU that implements the trusted DMA extension provides a
>parallel set of translation and invalidation resources for this purpose.
>They become usable once TDH.IOMMU.SETUP puts the IOMMU into Secure TDX
>Mode and the TDX module initializes them. The host keeps the regular
>translation path.
>
>Issue TDH.IOMMU.SETUP on demand, when the first device behind an IOMMU
>is connected through the TSM, rather than for every capable IOMMU at
>probe time. Secure TDX Mode costs host resources: the TDX module's
>metadata pages and half of the IOMMU's domain ID space. That cost should
>only be paid for IOMMUs that actually serve such devices.
>
>tdx_iommu_get() and tdx_iommu_put() keep a per-IOMMU refcount. The first
>get() issues TDH.IOMMU.SETUP and the last put() issues TDH.IOMMU.CLEAR.
>A single mutex covers both the refcount and the SEAMCALL that follows,
>so a device cannot join an IOMMU that is being cleared or whose setup
>has not finished. The refcount lives here rather than in the VT-d
>driver because tdx-host already owns the per-IOMMU state that must
>outlive TDH.IOMMU.SETUP: the IOMMU_MT pages and the TDX IOMMU ID.
>
>The IOMMU_MT parameter page lists two contiguous buffers for the
>trusted invalidation queue, followed by the metadata pages. The TDX
>module reports how many metadata pages it needs in its global metadata.
>
>With Dynamic PAMT, every 4K page given to the TDX module needs PAMT
>backing first. Take it for each page of the invalidation queue buffers
>and the metadata pages before TDH.IOMMU.SETUP, and drop it if setup
>fails or once TDH.IOMMU.CLEAR succeeds. The parameter page itself only
>carries the page list into the SEAMCALL and needs no backing.
>
>If TDH.IOMMU.CLEAR fails, the TDX module may still reference the
>IOMMU_MT pages, so they are leaked together with their PAMT backing,
>and the IOMMU is marked unusable. Later get() calls for that IOMMU fail
>with -EBUSY rather than reuse an IOMMU in an unknown state. tdx-host
>pins itself while such an IOMMU exists, so unloading can't lose track
>of it. If the failure happens during unload itself, only the tracking
>entry is freed; the VT-d driver still has the IOMMU acquired and keeps
>rejecting it.

If the pin can fail anyway, what's the point of taking it at all?

If tdx-host is reloaded, we can rely on the VT-d driver or the TDX module
to reject transitioning the IOMMU to secure mode.

>
>Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
>---
> drivers/virt/coco/tdx-host/Kconfig   |   5 +
> drivers/virt/coco/tdx-host/Makefile  |   1 +
> drivers/virt/coco/tdx-host/private.h |  21 ++
> drivers/virt/coco/tdx-host/main.c    |  11 +
> drivers/virt/coco/tdx-host/viommu.c  | 401 +++++++++++++++++++++++++++
> 5 files changed, 439 insertions(+)
> create mode 100644 drivers/virt/coco/tdx-host/private.h
> create mode 100644 drivers/virt/coco/tdx-host/viommu.c
>
>diff --git a/drivers/virt/coco/tdx-host/Kconfig b/drivers/virt/coco/tdx-host/Kconfig
>index f8c8cc29d46c..1395929f05bb 100644
>--- a/drivers/virt/coco/tdx-host/Kconfig
>+++ b/drivers/virt/coco/tdx-host/Kconfig
>@@ -6,3 +6,8 @@ config TDX_HOST_SERVICES
> 	select FW_UPLOAD
> 	select PCI_TSM
> 	default m
>+
>+config TDX_CONNECT
>+	def_bool y
>+	depends on TDX_HOST_SERVICES
>+	depends on INTEL_IOMMU

Do we need this TDX_CONNECT config? We can just add "depends on
INTEL_IOMMU" to TDX_HOST_SERVICES, and then drop the stub functions
in private.h. Patch 7 already makes TDX_HOST_SERVICES depend on PCI
in the same way.

The cost is that without INTEL_IOMMU, none of the tdx-host services
are available, including TDX module updates. But I doubt anyone would
want to launch CoCo VMs while compiling out INTEL_IOMMU entirely.

We can always add this option back if such a configuration turns out
to be practical.

>+/*
>+ * Serializes the refcounts together with TDH.IOMMU.SETUP and
>+ * TDH.IOMMU.CLEAR, so a device can't join an IOMMU that is being
>+ * cleared, or one whose setup hasn't finished.
>+ */
>+static DEFINE_MUTEX(tdx_iommu_lock);
>+static LIST_HEAD(tdx_iommu_list);
>+
>+static void *tdx_iommu_alloc_pages(int node, size_t size)
>+{
>+	struct folio *folio;
>+
>+	if (node == NUMA_NO_NODE)
>+		node = numa_mem_id();
>+
>+	folio = __folio_alloc_node(GFP_KERNEL | __GFP_ZERO, get_order(size), node);

Would kzalloc_node() work here? kzalloc_node() handles NUMA_NO_NODE
itself, and there's no need to convert between folios and void *.


>+	if (!folio)
>+		return NULL;
>+
>+	return folio_address(folio);
>+}
>+
>+static void tdx_iommu_free_pages(void *virt)
>+{
>+	if (!virt)
>+		return;
>+
>+	folio_put(virt_to_folio(virt));
>+}
>+
>+/* Number of 4K pages backing IOMMU_MT entry @i. */
>+static unsigned int mt_entry_nr_pages(unsigned int i)
>+{
>+	return i < TDX_IOMMU_MT_IQ_NR ? IQ_BUFFER_SIZE >> PAGE_SHIFT : 1;
>+}
>+
>+/*
>+ * Drop the PAMT backing of the first @nr 4K pages handed to the TDX
>+ * module, in the order mt_pamt_get() took it. Pass UINT_MAX for all.
>+ */
>+static void mt_pamt_put(struct tdx_iommu_pages *mt, unsigned int nr)
>+{
>+	unsigned int i, j;
>+
>+	for (i = 0; i < mt->nr_entries && nr; i++) {
>+		unsigned long pfn = PHYS_PFN(__pa(mt->pages[i]));
>+
>+		for (j = 0; j < mt_entry_nr_pages(i) && nr; j++, nr--)
>+			tdx_pamt_put(pfn + j);
>+	}
>+}
>+
>+/*
>+ * With Dynamic PAMT, every 4K page handed to the TDX module needs PAMT
>+ * backing first. This covers the IQ buffers and the metadata pages, not
>+ * the root page, which only carries the list of pages into the
>+ * SEAMCALL. tdx_pamt_get() is a no-op without Dynamic PAMT.
>+ */
>+static int mt_pamt_get(struct tdx_iommu_pages *mt)
>+{
>+	unsigned int i, j, nr = 0;
>+	int ret;
>+
>+	for (i = 0; i < mt->nr_entries; i++) {
>+		unsigned long pfn = PHYS_PFN(__pa(mt->pages[i]));
>+
>+		for (j = 0; j < mt_entry_nr_pages(i); j++) {
>+			ret = tdx_pamt_get(pfn + j, NULL);
>+			if (ret) {
>+				mt_pamt_put(mt, nr);
>+				return ret;
>+			}
>+			nr++;
>+		}
>+	}
>+
>+	return 0;
>+}
>+
>+static void free_mt_pages(struct tdx_iommu_pages *array)
>+{
>+	unsigned int i;
>+
>+	if (IS_ERR_OR_NULL(array))
>+		return;
>+
>+	mt_pamt_put(array, UINT_MAX);
>+
>+	for (i = 0; i < array->nr_entries; i++)
>+		tdx_iommu_free_pages(array->pages[i]);
>+
>+	tdx_iommu_free_pages(array->root);
>+	kfree(array->pages);
>+	kfree(array);
>+}
>+
>+DEFINE_FREE(free_mt_pages, struct tdx_iommu_pages *, free_mt_pages(_T))
>+
>+static void **alloc_mt_pages(unsigned int nr_entries, int node)
>+{
>+	unsigned int i;
>+	void **pages;
>+	void *vaddr;
>+
>+	pages = kzalloc_objs(*pages, nr_entries);
>+	if (!pages)
>+		return NULL;
>+
>+	/* Allocate two contiguous buffers for the invalidation queue. */
>+	pages[0] = tdx_iommu_alloc_pages(node, IQ_BUFFER_SIZE);
>+	if (!pages[0])
>+		goto free_pages;
>+
>+	pages[1] = tdx_iommu_alloc_pages(node, IQ_BUFFER_SIZE);
>+	if (!pages[1])
>+		goto free_pages;

You can use mt_entry_nr_pages() to fold the two IQ buffer allocations
into the loop below.

>+
>+	/* Allocate the required number of pages for the IOMMU metadata. */
>+	for (i = TDX_IOMMU_MT_IQ_NR; i < nr_entries; i++) {
>+		vaddr = tdx_iommu_alloc_pages(node, SZ_4K);
>+		if (!vaddr)
>+			goto free_pages;
>+		pages[i] = vaddr;
>+	}
>+
>+	return pages;
>+
>+free_pages:
>+	for (i = 0; i < nr_entries; i++)
>+		tdx_iommu_free_pages(pages[i]);
>+	kfree(pages);
>+
>+	return NULL;
>+}

      reply	other threads:[~2026-11-18  8:11 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-12  4:30 [PATCH v3 0/9] coco/tdx-host: Introduce trusted DMA initialization support Lu Baolu
2026-11-12  4:30 ` [PATCH v3 1/9] x86/virt/tdx: Add SEAMCALL wrappers for IOMMU setup/clear Lu Baolu
2026-11-12  4:30 ` [PATCH v3 2/9] x86/virt/tdx: Read global metadata for trusted IOMMU Lu Baolu
2026-11-12  4:30 ` [PATCH v3 3/9] x86/virt/tdx: Export Dynamic PAMT helpers to tdx-host Lu Baolu
2026-11-12  4:30 ` [PATCH v3 4/9] iommu/vt-d: Add interface for tdx-host when enabling trusted DMA Lu Baolu
2026-11-12  4:30 ` [PATCH v3 5/9] iommu/vt-d: Block GCMD and ECMD while acquired by the TDX module Lu Baolu
2026-11-12  4:30 ` [PATCH v3 6/9] iommu/vt-d: Refuse system suspend " Lu Baolu
2026-11-12  4:30 ` [PATCH v3 7/9] coco/tdx-host: Add skeleton to support PCI link encryption Lu Baolu
2026-11-18  7:16   ` Chao Gao
2026-11-12  4:30 ` [PATCH v3 8/9] coco/tdx-host: Rename tdx-host.c to main.c Lu Baolu
2026-11-12  4:30 ` [PATCH v3 9/9] coco/tdx-host: Enable trusted DMA on IOMMUs on device connect Lu Baolu
2026-11-18  8:11   ` Chao Gao [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=av1dr3IGMqWA1-7r@intel.com \
    --to=chao.gao@intel.com \
    --cc=baolu.lu@linux.intel.com \
    --cc=binbin.wu@linux.intel.com \
    --cc=iommu@lists.linux.dev \
    --cc=jgg@ziepe.ca \
    --cc=kas@kernel.org \
    --cc=kevin.tian@intel.com \
    --cc=kvm@vger.kernel.org \
    --cc=linux-coco@lists.linux.dev \
    --cc=linux-kernel@vger.kernel.org \
    --cc=rick.p.edgecombe@intel.com \
    --cc=x86@kernel.org \
    --cc=xiaoyao.li@intel.com \
    --cc=yilun.xu@linux.intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.