[PATCH v13 0/5] vfio/dma-buf: add TPH support for peer-to-peer access

[PATCH v13 0/5] vfio/dma-buf: add TPH support for peer-to-peer access

[Zhiping Zhang]

This series adds TLP Processing Hints (TPH) support to the VFIO dma-buf
export path, allowing importing drivers (e.g. mlx5) to use the
exporter's steering tag when performing peer-to-peer DMA into a
VFIO-owned device.

There is no separate in-tree vendor kernel driver for the target device:
vfio-pci is the in-tree driver and the targeted device is managed
from userspace via VFIO passthrough. That is why the ST has to flow
through a uAPI: userspace owns the device and its ST table, so it is the
entity that can configure a meaningful value for a given dma-buf. The
kernel-visible participants are still in-tree: vfio-pci exports the
dma-buf and mlx5 imports it.

On the effect: the endpoint's PCIe ingress block uses the ST as
an in-band instruction for the incoming P2P TLP -- selecting a target
cache partition and, on writes, an in-flight operation on the data
before it lands. The dma-buf callback keeps this opaque to the
framework -- only the producer (userspace owner of the VFIO device)
and the consumer (endpoint block) need to interpret the value. The
dma-buf get_pci_tph callback itself is optional, but workloads that
depend on the endpoint's in-flight operation need it because fallback
does not produce the same result.

The dma-buf hook is intentionally generic and discoverable rather than
a private side channel. The exporter owns the completing address
space for the dma-buf and decides whether it can provide a meaningful
ST/PH tuple for that completer; the dma-buf core keeps the tuple opaque,
and importers merely request the namespace they support and place the
returned value on generated TLPs. Exporters that cannot derive a
meaningful tuple simply return -EOPNOTSUPP.

TPH is advisory: a steering tag that is not honored on the path (for
example an intermediate routing element that does not forward the TPH
prefix) is ignored and the request completes as an ordinary,
non-TPH transaction (PCIe Base 6.4 sec 2.2.7.1). This series therefore
targets the same-Root-Port / common-switch topology, where the ST
reaches the completer; cross-Root-Port P2P is best-effort and is not
gated in the uAPI, since supplying an unused ST is harmless and there
is no discoverable "TPH routing" capability to test against.

Patch 1 folds the reserved 0b10 "TPH Completer Supported" encoding into
"not supported" in get_rp_completer_type(), so only architected values
can reach the TPH Requester Enable field. It was previously posted
standalone to linux-pci; per Alex Williamson's v12 review it now travels
with the series, which removes the cross-tree ordering dependency and
lets review tooling apply the series as posted.
Patch 2 adds small PCI/TPH type helpers so drivers can query the enabled
TPH requester mode and the device's TPH Completer Supported field
without reaching into pci_dev internals (and so callers in
CONFIG_PCIE_TPH=n builds get a clean fallback). pcie_tph_completer_type()
applies the same reserved-encoding fold as get_rp_completer_type(),
inlined locally so the helper is self-contained.
Patch 3 adds the optional dma_buf_ops::get_pci_tph callback plus the
dma_buf_get_pci_tph() importer wrapper so importers can fetch TPH
metadata from an exporter under dmabuf->resv.
Patch 4 implements get_pci_tph in vfio-pci and adds the new uAPI
(VFIO_DEVICE_FEATURE_DMA_BUF_TPH) for userspace to attach the metadata.
Patch 5 wires up the mlx5 RDMA driver as a consumer. It also enforces the
dma_buf_get_pci_tph() steering-tag lifetime: the tag is only valid for the
mapping it was queried against, and the mkey's TPH fields cannot be
reprogrammed in place. mlx5 therefore records the registration-time tuple
and re-queries after each dma-buf mapping is established under
dmabuf->resv; unchanged tuples continue with the existing mkey, while
changed or missing tuples fail the remap rather than continue with a
stale hint. For vfio-pci BAR dma-bufs this is expected to be a no-op
because invalidation is revoke/quiesce, not movement to a new backing
placement, and the userspace-provided tuple is not changed by the
revoke/un-revoke path.

Build-tested with both CONFIG_PCIE_TPH=y and CONFIG_PCIE_TPH=n.
Functional validation on the target topology: PCIe analyzer captures
on the P2P TLPs confirm the ST emitted by mlx5 matches the value
configured through VFIO_DEVICE_FEATURE_DMA_BUF_TPH, and the end-to-end
P2P workload only produces results consistent with the endpoint's
ST-selected in-flight operation. For example, with userspace
configuring 8-bit ST=0xf0 and PH=2, an analyzer capture of a peer-to-
peer MWr64 shows "STP MWr64 TC=0 OHC=2 ..." followed by "OHC-B
ST=F0h PH=2 HV=1":
(TLP Captures)
08000260 -> STP MWr64 TC=0 OHC=2 TS=0 Attr=0 L=8
F0000004 -> RID=4h:0h.0h EP- Tag=F0h
E0200000 -> AddrH=000020E0h
00080006 -> AddrL=06000800h
90F00000 -> OHC-B ST=F0h PH=2 HV=1 AMA=0 AV-

The dma-buf get_pci_tph interface has also been exercised by a second,
independent importer: a different vendor's NIC whose driver is not yet
upstream, locally taught to call dma_buf_get_pci_tph(). A PCIe analyzer
confirmed the ST it placed on outbound P2P TLPs matches the value
configured through VFIO_DEVICE_FEATURE_DMA_BUF_TPH, the same result as
with mlx5. Two unrelated importer drivers exercising the callback
end-to-end shows the interface is not tied to a single consumer. That
importer change is out-of-tree and not part of this series. For that
second importer, with userspace configuring 8-bit ST=0xe0 and PH=0,
an analyzer capture shows:
(TLP Captures)
08200260 -> STP MWr64 TC=0 OHC=2 TS=1 Attr=0 L=8
4E00004C -> RID=4Ch:0h.0h EP- Tag=4Eh
00170000 -> AddrH=00001700h
00200006 -> AddrL=06002000h
10E00000 -> OHC-B ST=E0h PH=0 HV=1 AMA=0 AV-

Changes since v12:
  Patch 1 (PCI/TPH, new to the series): the reserved-encoding fold,
  previously posted standalone to linux-pci [1], is now the first patch
  here (Alex Williamson). Sashiko could not apply v12 because of that
  external dependency; with the fold in-series and the mlx5 leak fix in
  linux-next, v13 has none. The code is unchanged from the standalone
  v3; the Fixes: tag is dropped, since no code path can reach the
  reserved encoding today and the patch is hardening rather than a fix
  for observed silicon.

  Patch 2 (PCI/TPH): inline the reserved-encoding fold in
  pcie_tph_completer_type() rather than calling the helper that earlier
  folding revisions added; that helper was dropped in folding v3 per
  Bjorn Helgaas and Wei Huang.

  Patch 3 (dma-buf): no functional change.

  Patch 4 (vfio/pci): also gate the DMA_BUF_TPH feature on
  vdev->pci_ops->get_dmabuf_phys, matching
  vfio_pci_core_feature_dma_buf(). Without it PROBE reported the feature
  as supported on a device that advertises TPH Completer support but
  cannot export a vfio dma-buf at all, so nothing could ever carry the
  metadata (Alex Williamson, who raised this to uAPI-affecting
  severity).

  Patch 5 (mlx5): keep the !dev->st early-out in mlx5_st_alloc_index()
  ahead of the pcie_tph_get_cpu_st() call, so splitting out
  mlx5_st_alloc_index_by_tag() neither adds an ACPI _DSM invocation on
  devices without ST support nor changes the errno userspace sees when
  the _DSM lookup fails (Alex Williamson). The commit message now
  describes this rather than presenting the split as a pure extraction.

Previous link:
v12: https://lore.kernel.org/linux-pci/20260715204008.3911275-1-zhipingz@meta.com/
v11: https://lore.kernel.org/linux-pci/20260702181025.2694961-1-zhipingz@meta.com/
v10: https://lore.kernel.org/linux-pci/20260630224328.3218796-1-zhipingz@meta.com/
v9: https://lore.kernel.org/dri-devel/20260622184211.2229399-1-zhipingz@meta.com/
v8: https://lore.kernel.org/dri-devel/20260615065912.2177918-1-zhipingz@meta.com/
v7: https://lore.kernel.org/dri-devel/20260611161546.4075580-1-zhipingz@meta.com/
v6: https://lore.kernel.org/dri-devel/20260608185646.4085127-1-zhipingz@meta.com/
v5: https://lore.kernel.org/dri-devel/20260526144401.1485788-1-zhipingz@meta.com/
v4: https://lore.kernel.org/linux-pci/20260519201401.1558410-1-zhipingz@meta.com/
v3: https://lore.kernel.org/linux-pci/20260512184755.4137227-1-zhipingz@meta.com/
v2: https://lore.kernel.org/linux-pci/20260430200704.352228-1-zhipingz@meta.com/

Zhiping Zhang (5):
  PCI/TPH: treat reserved 0b10 completer encoding as unsupported
  PCI/TPH: Add requester/completer type helpers
  dma-buf: add optional get_pci_tph() callback
  vfio/pci: implement get_pci_tph and DMA_BUF_TPH feature
  RDMA/mlx5: get tph for p2p access when registering dma-buf mr

 drivers/dma-buf/dma-buf.c                     |  32 ++++
 drivers/infiniband/hw/mlx5/main.c             |   1 +
 drivers/infiniband/hw/mlx5/mlx5_ib.h          |  11 ++
 drivers/infiniband/hw/mlx5/mr.c               | 151 +++++++++++++++++-
 drivers/infiniband/hw/mlx5/odp.c              |   7 +
 .../net/ethernet/mellanox/mlx5/core/lib/st.c  |  52 +++++-
 drivers/pci/tph.c                             |  55 ++++++-
 drivers/vfio/pci/vfio_pci_core.c              |   3 +
 drivers/vfio/pci/vfio_pci_dmabuf.c            | 120 +++++++++++++-
 drivers/vfio/pci/vfio_pci_priv.h              |  13 ++
 include/linux/dma-buf.h                       |  25 +++
 include/linux/mlx5/driver.h                   |  15 ++
 include/linux/pci-tph.h                       |   8 +
 include/uapi/linux/vfio.h                     |  43 +++++
 14 files changed, 517 insertions(+), 19 deletions(-)

-- 
2.53.0-Meta

[PATCH v13 2/5] PCI/TPH: Add requester/completer type helpers

[Zhiping Zhang]

Add pcie_tph_enabled_req_type() so drivers can query the enabled TPH
requester mode without reaching into pci_dev internals.

Add pcie_tph_completer_type() so drivers that publish TPH metadata for a
device acting as a completer can gate on the "TPH Completer Supported"
field of Device Capabilities 2 (bits 13:12, PCI_EXP_DEVCAP2_TPH_COMP_MASK)
rather than reusing requester-side state. It applies the same reserved-
encoding fold as get_rp_completer_type(), inlined, so the reserved 0b10
value never reaches callers.

This keeps pci_dev::tph_req_type and the completer-capability decode
inside the PCI/TPH code and provides !CONFIG_PCIE_TPH stubs for callers.

Signed-off-by: Zhiping Zhang <zhipingz@meta.com>
---
 drivers/pci/tph.c       | 44 +++++++++++++++++++++++++++++++++++++++++
 include/linux/pci-tph.h |  8 ++++++++
 2 files changed, 52 insertions(+)

diff --git a/drivers/pci/tph.c b/drivers/pci/tph.c
index 5cbb381e2a02..d0202ff7a1b2 100644
--- a/drivers/pci/tph.c
+++ b/drivers/pci/tph.c
@@ -223,6 +223,50 @@ static u8 get_rp_completer_type(struct pci_dev *pdev)
 	return PCI_EXP_DEVCAP2_TPH_COMP_NONE;
 }
 
+/**
+ * pcie_tph_enabled_req_type - Return the device's enabled TPH requester type
+ * @pdev: PCI device to query
+ *
+ * Return: PCI_TPH_REQ_DISABLE, PCI_TPH_REQ_TPH_ONLY or PCI_TPH_REQ_EXT_TPH.
+ */
+u8 pcie_tph_enabled_req_type(struct pci_dev *pdev)
+{
+	return pdev->tph_req_type;
+}
+EXPORT_SYMBOL(pcie_tph_enabled_req_type);
+
+/**
+ * pcie_tph_completer_type - Return the device's TPH Completer support
+ * @pdev: PCI device to query
+ *
+ * Reads the "TPH Completer Supported" field (bits 13:12) of Device
+ * Capabilities 2. The reserved 0b10 encoding is folded into
+ * "not supported" so callers only need to compare against the three
+ * defined values.
+ *
+ * Return: one of %PCI_EXP_DEVCAP2_TPH_COMP_NONE,
+ *         %PCI_EXP_DEVCAP2_TPH_COMP_TPH_ONLY or
+ *         %PCI_EXP_DEVCAP2_TPH_COMP_EXT_TPH.
+ */
+u8 pcie_tph_completer_type(struct pci_dev *pdev)
+{
+	u8 tph_comp;
+	u32 reg;
+
+	if (pcie_capability_read_dword(pdev, PCI_EXP_DEVCAP2, &reg))
+		return PCI_EXP_DEVCAP2_TPH_COMP_NONE;
+	if (PCI_POSSIBLE_ERROR(reg))
+		return PCI_EXP_DEVCAP2_TPH_COMP_NONE;
+
+	/* 0b10 is reserved; fold it into "not supported" (see above). */
+	tph_comp = FIELD_GET(PCI_EXP_DEVCAP2_TPH_COMP_MASK, reg);
+	if (tph_comp == PCI_EXP_DEVCAP2_TPH_COMP_TPH_ONLY ||
+	    tph_comp == PCI_EXP_DEVCAP2_TPH_COMP_EXT_TPH)
+		return tph_comp;
+	return PCI_EXP_DEVCAP2_TPH_COMP_NONE;
+}
+EXPORT_SYMBOL(pcie_tph_completer_type);
+
 /* Write tag to ST table - Return 0 if OK, otherwise -errno */
 static int write_tag_to_st_table(struct pci_dev *pdev, int index, u16 tag)
 {
diff --git a/include/linux/pci-tph.h b/include/linux/pci-tph.h
index be68cd17f2f8..7743af6fe432 100644
--- a/include/linux/pci-tph.h
+++ b/include/linux/pci-tph.h
@@ -9,6 +9,8 @@
 #ifndef LINUX_PCI_TPH_H
 #define LINUX_PCI_TPH_H
 
+#include <linux/pci_regs.h>
+
 /*
  * According to the ECN for PCI Firmware Spec, Steering Tag can be different
  * depending on the memory type: Volatile Memory or Persistent Memory. When a
@@ -30,6 +32,8 @@ void pcie_disable_tph(struct pci_dev *pdev);
 int pcie_enable_tph(struct pci_dev *pdev, int mode);
 u16 pcie_tph_get_st_table_size(struct pci_dev *pdev);
 u32 pcie_tph_get_st_table_loc(struct pci_dev *pdev);
+u8 pcie_tph_enabled_req_type(struct pci_dev *pdev);
+u8 pcie_tph_completer_type(struct pci_dev *pdev);
 #else
 static inline int pcie_tph_set_st_entry(struct pci_dev *pdev,
 					unsigned int index, u16 tag)
@@ -41,6 +45,10 @@ static inline int pcie_tph_get_cpu_st(struct pci_dev *dev,
 static inline void pcie_disable_tph(struct pci_dev *pdev) { }
 static inline int pcie_enable_tph(struct pci_dev *pdev, int mode)
 { return -EINVAL; }
+static inline u8 pcie_tph_enabled_req_type(struct pci_dev *pdev)
+{ return PCI_TPH_REQ_DISABLE; }
+static inline u8 pcie_tph_completer_type(struct pci_dev *pdev)
+{ return PCI_EXP_DEVCAP2_TPH_COMP_NONE; }
 #endif
 
 #endif /* LINUX_PCI_TPH_H */
-- 
2.53.0-Meta

[PATCH v13 5/5] RDMA/mlx5: get tph for p2p access when registering dma-buf mr

[Zhiping Zhang]

Peer-to-peer DMA between a mlx5 NIC and a foreign PCIe endpoint
(typically a GPU or a vfio-pci passthrough device) traverses the host
PCIe fabric. The endpoint exporting the dma-buf knows which PCIe TLP
Processing Hint (TPH) Steering Tag yields the best placement for the
traffic it will sink: per-endpoint hint selection lets the root complex
or switch direct DMA to a specific cache slice / NUMA node, cutting
cross-socket snoop traffic and DRAM pressure under sustained p2p
workloads.

Until now the mlx5 importer had no way to learn the exporter's chosen
ST tag, so dma-buf MRs were registered without TPH and ran with the
default (no-hint) routing. With dma_buf_get_pci_tph() in place this
patch wires up mlx5_ib to query that metadata at MR registration time
for p2p access and use it to program requester-side TPH on the outbound
mkey. If the exporter has no metadata, fall back to the existing
no-TPH path so behavior for non-TPH-aware exporters is unchanged.

Use mlx5_st_alloc_index_by_tag() to translate exporter-provided
steering tags into local ST entries when table mode is active, and add
mlx5_st_get_index() for DMAH-backed flows that already carry an ST
index.

Splitting mlx5_st_alloc_index() moves the !dev->st check into the new
by-tag helper, so keep an equivalent early-out in mlx5_st_alloc_index()
itself, ahead of the pcie_tph_get_cpu_st() call. A device without ST
support therefore still returns -EOPNOTSUPP without an ACPI _DSM
invocation, exactly as it did before the split. The check is repeated in
mlx5_st_alloc_index_by_tag() because that is now an exported entry point
in its own right.

For TPH-backed FRMRs, keep the extra ST-table reference tied to MR
lifetime rather than pooled mkey lifetime. Acquire the ref before MR
creation and release it again when the MR is returned to the pool or
the backing mkey is destroyed, while leaving the generic FRMR pool
core unchanged.

The exporter's steering tag is only valid for the mapping it was queried
against (see dma_buf_get_pci_tph()). The mkey's TPH fields cannot be
reprogrammed in place: they are create-time fields with no UMR update
mask, and rebuilding the mkey would change the rkey/lkey, which must stay
stable across invalidate/refault.

Keep the registration-time TPH tuple in the MR and re-query the exporter
after each dma-buf mapping is established under dmabuf->resv. If the
tuple is unchanged, continue using the existing mkey. If the tuple changed
or disappeared, unmap the pages and return -EFAULT rather than continue
with a stale mkey hint; userspace must create a new MR to get a mkey
programmed with the new tuple. For vfio-pci BAR dma-bufs this validation
is expected to be a no-op because invalidation is revoke/quiesce of BAR
access, not movement to a different backing placement, and the
userspace-provided TPH tuple is not changed by the revoke/un-revoke path.

Import the DMA_BUF namespace for the new dma_buf_get_pci_tph() call so
modular mlx5_ib builds link cleanly.

Signed-off-by: Zhiping Zhang <zhipingz@meta.com>
---
 drivers/infiniband/hw/mlx5/main.c             |   1 +
 drivers/infiniband/hw/mlx5/mlx5_ib.h          |  11 ++
 drivers/infiniband/hw/mlx5/mr.c               | 151 +++++++++++++++++-
 drivers/infiniband/hw/mlx5/odp.c              |   7 +
 .../net/ethernet/mellanox/mlx5/core/lib/st.c  |  52 +++++-
 include/linux/mlx5/driver.h                   |  15 ++
 6 files changed, 223 insertions(+), 14 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index 02809114fc79..a2b497f6b16b 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -60,6 +60,7 @@
 MODULE_AUTHOR("Eli Cohen <eli@mellanox.com>");
 MODULE_DESCRIPTION("Mellanox 5th generation network adapters (ConnectX series) IB driver");
 MODULE_LICENSE("Dual BSD/GPL");
+MODULE_IMPORT_NS("DMA_BUF");
 
 struct mlx5_ib_event_work {
 	struct work_struct	work;
diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index 522984d958bb..c6a956b2ee87 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -646,6 +646,15 @@ enum mlx5_mkey_type {
 /* Used for non-existent ph value */
 #define MLX5_IB_NO_PH 0xff
 
+struct dma_buf;
+
+struct mlx5_ib_dmabuf_tph {
+	u16 steering_tag;
+	u8 ph;
+	bool extended;
+	bool valid;
+};
+
 struct mlx5_ib_mkey {
 	u32 key;
 	enum mlx5_mkey_type type;
@@ -726,6 +735,7 @@ struct mlx5_ib_mr {
 			u8 revoked :1;
 			/* Indicates previous dmabuf page fault occurred */
 			u8 dmabuf_faulted:1;
+			struct mlx5_ib_dmabuf_tph dmabuf_tph;
 			struct mlx5_ib_mkey null_mmkey;
 		};
 	};
@@ -1341,6 +1351,7 @@ struct ib_mr *mlx5_ib_rereg_user_mr(struct ib_mr *ib_mr, int flags, u64 start,
 				    u64 length, u64 virt_addr, int access_flags,
 				    struct ib_pd *pd, struct ib_udata *udata);
 int mlx5_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata);
+int mlx5_ib_validate_dmabuf_tph(struct mlx5_ib_mr *mr, struct dma_buf *dmabuf);
 struct ib_mr *mlx5_ib_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
 			       u32 max_num_sg);
 struct ib_mr *mlx5_ib_alloc_mr_integrity(struct ib_pd *pd,
diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
index e6b74955d95d..404c5d081b8a 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -39,6 +39,7 @@
 #include <linux/delay.h>
 #include <linux/dma-buf.h>
 #include <linux/dma-resv.h>
+#include <linux/pci-tph.h>
 #include <rdma/frmr_pools.h>
 #include <rdma/ib_umem_odp.h>
 #include "dm.h"
@@ -167,6 +168,61 @@ static int get_unchangeable_access_flags(struct mlx5_ib_dev *dev,
 #define MLX5_FRMR_POOLS_KERNEL_KEY_PH_MASK GENMASK_ULL(23, 16)
 #define MLX5_FRMR_POOLS_KERNEL_KEY_ST_INDEX_MASK GENMASK_ULL(15, 0)
 
+static u8 mlx5_ib_tph_key_ph(u8 ph)
+{
+	if (ph == MLX5_IB_NO_PH || ph == 0)
+		ph ^= MLX5_IB_NO_PH;
+
+	return ph;
+}
+
+static int mlx5_ib_get_st_handle_ref(struct mlx5_ib_dev *dev, u16 st_index)
+{
+	if (st_index == MLX5_MKC_PCIE_TPH_NO_STEERING_TAG_INDEX)
+		return 0;
+
+	return mlx5_st_get_index(dev->mdev, st_index);
+}
+
+static void mlx5_ib_put_st_index_ref(struct mlx5_ib_dev *dev, u16 st_index)
+{
+	if (st_index == MLX5_MKC_PCIE_TPH_NO_STEERING_TAG_INDEX)
+		return;
+
+	mlx5_st_dealloc_index(dev->mdev, st_index);
+}
+
+static void mlx5_ib_put_st_handle_ref(struct mlx5_ib_dev *dev,
+				      u64 kernel_vendor_key)
+{
+	u16 st_index = FIELD_GET(MLX5_FRMR_POOLS_KERNEL_KEY_ST_INDEX_MASK,
+				 kernel_vendor_key);
+
+	mlx5_ib_put_st_index_ref(dev, st_index);
+}
+
+int mlx5_ib_validate_dmabuf_tph(struct mlx5_ib_mr *mr, struct dma_buf *dmabuf)
+{
+	struct mlx5_ib_dmabuf_tph *tph = &mr->dmabuf_tph;
+	u16 steering_tag;
+	u8 ph;
+	int ret;
+
+	if (!tph->valid)
+		return 0;
+
+	dma_resv_assert_held(dmabuf->resv);
+
+	ret = dma_buf_get_pci_tph(dmabuf, tph->extended, &steering_tag, &ph);
+	if (ret)
+		return -EFAULT;
+
+	if (steering_tag != tph->steering_tag || ph != tph->ph)
+		return -EFAULT;
+
+	return 0;
+}
+
 static struct mlx5_ib_mr *
 _mlx5_frmr_pool_alloc(struct mlx5_ib_dev *dev, struct ib_umem *umem,
 		      int access_flags, int access_mode,
@@ -189,13 +245,10 @@ _mlx5_frmr_pool_alloc(struct mlx5_ib_dev *dev, struct ib_umem *umem,
 			MLX5_FRMR_POOLS_KEY_ACCESS_MODE_KSM_MASK :
 			0;
 
-	/* Normalize ph: swap 0 and MLX5_IB_NO_PH */
-	if (ph == MLX5_IB_NO_PH || ph == 0)
-		ph ^= MLX5_IB_NO_PH;
-
 	mr->ibmr.frmr.key.kernel_vendor_key =
 		FIELD_PREP(MLX5_FRMR_POOLS_KERNEL_KEY_ST_INDEX_MASK, st_index) |
-		FIELD_PREP(MLX5_FRMR_POOLS_KERNEL_KEY_PH_MASK, ph);
+		FIELD_PREP(MLX5_FRMR_POOLS_KERNEL_KEY_PH_MASK,
+			   mlx5_ib_tph_key_ph(ph));
 	err = ib_frmr_pool_pop(&dev->ib_dev, &mr->ibmr);
 	if (err) {
 		kfree(mr);
@@ -218,7 +271,9 @@ struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev,
 				      0 :
 				      MLX5_FRMR_POOLS_KEY_ACCESS_MODE_KSM_MASK,
 		.num_dma_blocks = ndescs,
-		.kernel_vendor_key = 0, /* no PH and no ST index */
+		.kernel_vendor_key =
+			FIELD_PREP(MLX5_FRMR_POOLS_KERNEL_KEY_ST_INDEX_MASK,
+				   MLX5_MKC_PCIE_TPH_NO_STEERING_TAG_INDEX),
 	};
 	struct mlx5_ib_mr *mr;
 	int ret;
@@ -557,6 +612,10 @@ static struct mlx5_ib_mr *reg_create(struct ib_pd *pd, struct ib_umem *umem,
 	mr->ibmr.pd = pd;
 	mr->access_flags = access_flags;
 	mr->page_shift = order_base_2(page_size);
+	mr->ibmr.frmr.key.kernel_vendor_key =
+		FIELD_PREP(MLX5_FRMR_POOLS_KERNEL_KEY_ST_INDEX_MASK, st_index) |
+		FIELD_PREP(MLX5_FRMR_POOLS_KERNEL_KEY_PH_MASK,
+			   mlx5_ib_tph_key_ph(ph));
 
 	inlen = MLX5_ST_SZ_BYTES(create_mkey_in);
 	if (populate)
@@ -753,6 +812,12 @@ static struct ib_mr *create_real_mr(struct ib_pd *pd, struct ib_umem *umem,
 			st_index = mdmah->st_index;
 	}
 
+	err = mlx5_ib_get_st_handle_ref(dev, st_index);
+	if (err) {
+		ib_umem_release(umem);
+		return ERR_PTR(err);
+	}
+
 	xlt_with_umr = mlx5r_umr_can_load_pas(dev, umem->length);
 	if (xlt_with_umr) {
 		mr = alloc_cacheable_mr(pd, umem, iova, access_flags,
@@ -769,6 +834,7 @@ static struct ib_mr *create_real_mr(struct ib_pd *pd, struct ib_umem *umem,
 		mutex_unlock(&dev->slow_path_mutex);
 	}
 	if (IS_ERR(mr)) {
+		mlx5_ib_put_st_index_ref(dev, st_index);
 		ib_umem_release(umem);
 		return ERR_CAST(mr);
 	}
@@ -903,6 +969,63 @@ static struct dma_buf_attach_ops mlx5_ib_dmabuf_attach_ops = {
 	.invalidate_mappings = mlx5_ib_dmabuf_invalidate_cb,
 };
 
+static void get_pci_tph_mr_dmabuf(struct mlx5_ib_dev *dev, struct dma_buf *dmabuf,
+				  struct mlx5_ib_dmabuf_tph *tph,
+				  u16 *st_index, u8 *ph)
+{
+	u16 local_st_index;
+	u16 steering_tag;
+	u8 local_ph;
+	bool extended;
+	int ret;
+
+	switch (pcie_tph_enabled_req_type(dev->mdev->pdev)) {
+	case PCI_TPH_REQ_TPH_ONLY:
+		extended = false;
+		break;
+	case PCI_TPH_REQ_EXT_TPH:
+		extended = true;
+		break;
+	default:
+		return;
+	}
+
+	/*
+	 * dma_buf_get_pci_tph() must run under dmabuf->resv, but drop the lock
+	 * before mlx5_st_alloc_index_by_tag(): that allocates with GFP_KERNEL,
+	 * which must not run under dmabuf->resv since reclaim can re-enter the
+	 * same reservation object through a dynamic exporter's move_notify path.
+	 */
+	dma_resv_lock(dmabuf->resv, NULL);
+	ret = dma_buf_get_pci_tph(dmabuf, extended, &steering_tag, &local_ph);
+	dma_resv_unlock(dmabuf->resv);
+	if (ret) {
+		if (ret != -EOPNOTSUPP)
+			mlx5_ib_dbg(dev, "get_pci_tph failed (%d)\n", ret);
+		return;
+	}
+
+	ret = mlx5_st_alloc_index_by_tag(dev->mdev, steering_tag,
+					 &local_st_index);
+	if (ret) {
+		mlx5_ib_dbg(dev, "st_alloc_index_by_tag failed (%d)\n", ret);
+		return;
+	}
+
+	*st_index = local_st_index;
+	*ph = local_ph;
+	tph->steering_tag = steering_tag;
+	tph->ph = local_ph;
+	tph->extended = extended;
+	tph->valid = true;
+}
+
+static void mlx5_ib_mr_put_st_handle_ref(struct mlx5_ib_mr *mr)
+{
+	mlx5_ib_put_st_handle_ref(mr_to_mdev(mr),
+				  mr->ibmr.frmr.key.kernel_vendor_key);
+}
+
 static struct ib_mr *
 reg_user_mr_dmabuf(struct ib_pd *pd, struct device *dma_device,
 		   u64 offset, u64 length, u64 virt_addr,
@@ -913,6 +1036,7 @@ reg_user_mr_dmabuf(struct ib_pd *pd, struct device *dma_device,
 	struct mlx5_ib_dev *dev = to_mdev(pd->device);
 	struct mlx5_ib_mr *mr = NULL;
 	struct ib_umem_dmabuf *umem_dmabuf;
+	struct mlx5_ib_dmabuf_tph tph = {};
 	u16 st_index = MLX5_MKC_PCIE_TPH_NO_STEERING_TAG_INDEX;
 	u8 ph = MLX5_IB_NO_PH;
 	int err;
@@ -945,18 +1069,29 @@ reg_user_mr_dmabuf(struct ib_pd *pd, struct device *dma_device,
 		ph = dmah->ph;
 		if (dmah->valid_fields & BIT(IB_DMAH_CPU_ID_EXISTS))
 			st_index = mdmah->st_index;
+
+		err = mlx5_ib_get_st_handle_ref(dev, st_index);
+		if (err) {
+			ib_umem_release(&umem_dmabuf->umem);
+			return ERR_PTR(err);
+		}
+	} else {
+		get_pci_tph_mr_dmabuf(dev, umem_dmabuf->attach->dmabuf,
+				      &tph, &st_index, &ph);
 	}
 
 	mr = alloc_cacheable_mr(pd, &umem_dmabuf->umem, virt_addr,
 				access_flags, access_mode,
 				st_index, ph);
 	if (IS_ERR(mr)) {
+		mlx5_ib_put_st_index_ref(dev, st_index);
 		ib_umem_release(&umem_dmabuf->umem);
 		return ERR_CAST(mr);
 	}
 
 	mlx5_ib_dbg(dev, "mkey 0x%x\n", mr->mmkey.key);
 
+	mr->dmabuf_tph = tph;
 	atomic_add(ib_umem_num_pages(mr->umem), &dev->mdev->priv.reg_pages);
 	umem_dmabuf->private = mr;
 	if (!pinned_mode) {
@@ -1405,6 +1540,7 @@ static int mlx5r_handle_mkey_cleanup(struct mlx5_ib_mr *mr)
 	if (mr->ibmr.frmr.pool) {
 		if (!mlx5_umr_revoke_mr_with_lock(mr)) {
 			ib_frmr_pool_push(mr->ibmr.device, &mr->ibmr);
+			mlx5_ib_mr_put_st_handle_ref(mr);
 			return 0;
 		}
 	}
@@ -1432,6 +1568,9 @@ static int mlx5r_handle_mkey_cleanup(struct mlx5_ib_mr *mr)
 	if (mr->ibmr.frmr.pool && !ret)
 		ib_frmr_pool_drop(&mr->ibmr);
 
+	if (!ret)
+		mlx5_ib_mr_put_st_handle_ref(mr);
+
 	return ret;
 }
 
diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c
index 1badec9bf527..7c1731aa3dd9 100644
--- a/drivers/infiniband/hw/mlx5/odp.c
+++ b/drivers/infiniband/hw/mlx5/odp.c
@@ -857,6 +857,13 @@ static int pagefault_dmabuf_mr(struct mlx5_ib_mr *mr, size_t bcnt,
 		return err;
 	}
 
+	err = mlx5_ib_validate_dmabuf_tph(mr, umem_dmabuf->attach->dmabuf);
+	if (err) {
+		ib_umem_dmabuf_unmap_pages(umem_dmabuf);
+		dma_resv_unlock(umem_dmabuf->attach->dmabuf->resv);
+		return err;
+	}
+
 	page_size = mlx5_umem_dmabuf_find_best_pgsz(umem_dmabuf, access_mode);
 	if (!page_size) {
 		ib_umem_dmabuf_unmap_pages(umem_dmabuf);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/st.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/st.c
index 7cedc348790d..c62bcae7c542 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lib/st.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/st.c
@@ -92,23 +92,18 @@ void mlx5_st_destroy(struct mlx5_core_dev *dev)
 	kfree(st);
 }
 
-int mlx5_st_alloc_index(struct mlx5_core_dev *dev, enum tph_mem_type mem_type,
-			unsigned int cpu_uid, u16 *st_index)
+int mlx5_st_alloc_index_by_tag(struct mlx5_core_dev *dev, u16 tag,
+			       u16 *st_index)
 {
 	struct mlx5_st_idx_data *idx_data;
 	struct mlx5_st *st = dev->st;
 	unsigned long index;
 	u32 xa_id;
-	u16 tag;
-	int ret;
+	int ret = 0;
 
 	if (!st)
 		return -EOPNOTSUPP;
 
-	ret = pcie_tph_get_cpu_st(dev->pdev, mem_type, cpu_uid, &tag);
-	if (ret)
-		return ret;
-
 	if (st->direct_mode) {
 		*st_index = tag;
 		return 0;
@@ -152,8 +147,49 @@ int mlx5_st_alloc_index(struct mlx5_core_dev *dev, enum tph_mem_type mem_type,
 	mutex_unlock(&st->lock);
 	return ret;
 }
+EXPORT_SYMBOL_GPL(mlx5_st_alloc_index_by_tag);
+
+int mlx5_st_alloc_index(struct mlx5_core_dev *dev, enum tph_mem_type mem_type,
+			unsigned int cpu_uid, u16 *st_index)
+{
+	u16 tag;
+	int ret;
+
+	if (!dev->st)
+		return -EOPNOTSUPP;
+
+	ret = pcie_tph_get_cpu_st(dev->pdev, mem_type, cpu_uid, &tag);
+	if (ret)
+		return ret;
+
+	return mlx5_st_alloc_index_by_tag(dev, tag, st_index);
+}
 EXPORT_SYMBOL_GPL(mlx5_st_alloc_index);
 
+int mlx5_st_get_index(struct mlx5_core_dev *dev, u16 st_index)
+{
+	struct mlx5_st_idx_data *idx_data;
+	struct mlx5_st *st = dev->st;
+	int ret = 0;
+
+	if (!st)
+		return -EOPNOTSUPP;
+
+	if (st->direct_mode)
+		return 0;
+
+	mutex_lock(&st->lock);
+	idx_data = xa_load(&st->idx_xa, st_index);
+	if (WARN_ON_ONCE(!idx_data))
+		ret = -EINVAL;
+	else
+		refcount_inc(&idx_data->usecount);
+	mutex_unlock(&st->lock);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(mlx5_st_get_index);
+
 int mlx5_st_dealloc_index(struct mlx5_core_dev *dev, u16 st_index)
 {
 	struct mlx5_st_idx_data *idx_data;
diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h
index b1871c0821d0..d26e5744c843 100644
--- a/include/linux/mlx5/driver.h
+++ b/include/linux/mlx5/driver.h
@@ -1182,16 +1182,31 @@ int mlx5_dm_sw_icm_dealloc(struct mlx5_core_dev *dev, enum mlx5_sw_icm_type type
 			   u64 length, u16 uid, phys_addr_t addr, u32 obj_id);
 
 #ifdef CONFIG_PCIE_TPH
+int mlx5_st_alloc_index_by_tag(struct mlx5_core_dev *dev, u16 tag,
+			       u16 *st_index);
+int mlx5_st_get_index(struct mlx5_core_dev *dev, u16 st_index);
 int mlx5_st_alloc_index(struct mlx5_core_dev *dev, enum tph_mem_type mem_type,
 			unsigned int cpu_uid, u16 *st_index);
 int mlx5_st_dealloc_index(struct mlx5_core_dev *dev, u16 st_index);
 #else
+static inline int mlx5_st_alloc_index_by_tag(struct mlx5_core_dev *dev,
+					     u16 tag, u16 *st_index)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline int mlx5_st_get_index(struct mlx5_core_dev *dev, u16 st_index)
+{
+	return -EOPNOTSUPP;
+}
+
 static inline int mlx5_st_alloc_index(struct mlx5_core_dev *dev,
 				      enum tph_mem_type mem_type,
 				      unsigned int cpu_uid, u16 *st_index)
 {
 	return -EOPNOTSUPP;
 }
+
 static inline int mlx5_st_dealloc_index(struct mlx5_core_dev *dev, u16 st_index)
 {
 	return -EOPNOTSUPP;
-- 
2.53.0-Meta

Re: [PATCH v13 3/5] dma-buf: add optional get_pci_tph() callback

[sashiko-bot]

> Add an optional dma_buf_ops.get_pci_tph callback and a
> DMA-buf importer wrapper, dma_buf_get_pci_tph().
> 
> TPH is PCIe TLP Processing Hint. 8-bit ST and 16-bit Extended ST are
> distinct PCIe TPH namespaces, so the importer requests the namespace it
> can emit and the exporter returns the matching ST/PH tuple or
> -EOPNOTSUPP.
> 
> dma_buf_get_pci_tph() is the importer entry point. It requires
> &dmabuf->resv to be held while the callback runs and returns
> -EOPNOTSUPP when the exporter does not provide PCI TPH metadata.
> 
> The first user is VFIO_DEVICE_FEATURE_DMA_BUF_TPH in vfio-pci, with
> mlx5 as the first importer.
> 
> Signed-off-by: Zhiping Zhang <zhipingz@meta.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260731211601.3033906-1-zhipingz@meta.com?part=3

Re: [PATCH v13 1/5] PCI/TPH: treat reserved 0b10 completer encoding as unsupported

[sashiko-bot]

> get_rp_completer_type() returns the Root Port's "TPH Completer
> Supported" field (bits 13:12 of Device Capabilities 2) verbatim. The
> 0b10 encoding is reserved, but pcie_enable_tph() feeds the raw value
> into the requester type:
> 
> 	pdev->tph_req_type = min(pdev->tph_req_type, rp_req_type);
> 
> and later writes tph_req_type to the TPH Requester Enable field, which
> only defines 0b00 (disable), 0b01 (TPH only) and 0b11 (extended TPH).
> 
> No known hardware presents the reserved 0b10 in this field, so this is
> defensive hardening rather than a fix for observed silicon: fold the
> reserved encoding into "not supported" so only the three architected
> values can ever reach the Requester Enable field.
> 
> Signed-off-by: Zhiping Zhang <zhipingz@meta.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260731211601.3033906-1-zhipingz@meta.com?part=1

Re: [PATCH v13 4/5] vfio/pci: implement get_pci_tph and DMA_BUF_TPH feature

[sashiko-bot]

> Implement dma-buf get_pci_tph for vfio-pci exported dma-bufs and add
> VFIO_DEVICE_FEATURE_DMA_BUF_TPH so userspace can publish TPH metadata
> for a VFIO-owned device.
> 
> 8-bit ST and 16-bit Extended ST are distinct PCIe TPH namespaces; the
> uAPI carries both with explicit validity flags, and get_pci_tph()
> returns the value matching the importer's requested namespace or
> -EOPNOTSUPP.
> 
> Publish and read the TPH descriptor under dmabuf->resv, matching the
> locking used for other importer-visible dma-buf state. The SET ioctl
> takes dma_resv_lock_interruptible(), while the callback runs under
> DMA-buf's asserted resv lock.
> 
> Hold memory_lock in the SET path while checking that the dma-buf still
> [ ... ]
> supported on a device that advertises TPH Completer support but cannot
> export a vfio dma-buf at all, so no dma-buf can ever exist to carry the
> metadata and every SET would fail later at the ownership check.
> 
> Signed-off-by: Zhiping Zhang <zhipingz@meta.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260731211601.3033906-1-zhipingz@meta.com?part=4

Re: [PATCH v13 2/5] PCI/TPH: Add requester/completer type helpers

[sashiko-bot]

> Add pcie_tph_enabled_req_type() so drivers can query the enabled TPH
> requester mode without reaching into pci_dev internals.
> 
> Add pcie_tph_completer_type() so drivers that publish TPH metadata for a
> device acting as a completer can gate on the "TPH Completer Supported"
> field of Device Capabilities 2 (bits 13:12, PCI_EXP_DEVCAP2_TPH_COMP_MASK)
> rather than reusing requester-side state. It applies the same reserved-
> encoding fold as get_rp_completer_type(), inlined, so the reserved 0b10
> value never reaches callers.
> 
> This keeps pci_dev::tph_req_type and the completer-capability decode
> inside the PCI/TPH code and provides !CONFIG_PCIE_TPH stubs for callers.
> 
> Signed-off-by: Zhiping Zhang <zhipingz@meta.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260731211601.3033906-1-zhipingz@meta.com?part=2

Re: [PATCH v13 5/5] RDMA/mlx5: get tph for p2p access when registering dma-buf mr

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] mlx5_st_dealloc_index leaks the idx_data struct when erasing it from the xarray.
--

--- Patch [5]: [PATCH v13 5/5] RDMA/mlx5: get tph for p2p access when registering dma-buf mr ---
commit cb1e1bfb29faeb4a7f3d550176e3590d73486a3e
Author: Zhiping Zhang <zhipingz@meta.com>

RDMA/mlx5: get tph for p2p access when registering dma-buf mr

This commit wires up mlx5_ib to query the exporter's chosen PCIe TLP
Processing Hint (TPH) Steering Tag at MR registration time for peer-to-peer
DMA access. It translates the exporter-provided steering tags into local ST
entries or uses default routing if no hint is available.

> diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
> index 00e13028762ae..fdd7d94bd4aa4 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -167,6 +168,61 @@ static int get_unchangeable_access_flags(struct mlx5_ib_dev *dev,
>  #define MLX5_FRMR_POOLS_KERNEL_KEY_PH_MASK GENMASK_ULL(23, 16)
>  #define MLX5_FRMR_POOLS_KERNEL_KEY_ST_INDEX_MASK GENMASK_ULL(15, 0)
>  
> +static u8 mlx5_ib_tph_key_ph(u8 ph)
> +{
> +	if (ph == MLX5_IB_NO_PH || ph == 0)
> +		ph ^= MLX5_IB_NO_PH;
> +
> +	return ph;
> +}
> +
> +static int mlx5_ib_get_st_handle_ref(struct mlx5_ib_dev *dev, u16 st_index)
> +{
> +	if (st_index == MLX5_MKC_PCIE_TPH_NO_STEERING_TAG_INDEX)
> +		return 0;
> +
> +	return mlx5_st_get_index(dev->mdev, st_index);
> +}
> +
> +static void mlx5_ib_put_st_index_ref(struct mlx5_ib_dev *dev, u16 st_index)
> +{
> +	if (st_index == MLX5_MKC_PCIE_TPH_NO_STEERING_TAG_INDEX)
> +		return;
> +
> +	mlx5_st_dealloc_index(dev->mdev, st_index);

[Severity: High]
This is a pre-existing issue, but this patch introduces a new, heavily-exercised
reachable path for it.

Does mlx5_st_dealloc_index() leak the mlx5_st_idx_data struct when erasing it
from the xarray?

When a steering tag's reference count drops to zero, the entry is removed via
xa_erase():

drivers/net/ethernet/mellanox/mlx5/core/lib/st.c:mlx5_st_dealloc_index() {
    ...
	if (refcount_dec_and_test(&idx_data->usecount)) {
		xa_erase(&st->idx_xa, st_index);
		/* We leave PCI config space as was before, no mkey will refer to it */
	}
    ...
}

However, the underlying idx_data pointer is not freed with kfree(idx_data).

Because the next time the same tag is requested, a new struct is allocated via
kzalloc_obj() in mlx5_st_alloc_index_by_tag(), won't repeated allocations and
deallocations lead to an unbounded kernel memory leak?

> +}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260731211601.3033906-1-zhipingz@meta.com?part=5

Re: [PATCH v13 5/5] RDMA/mlx5: get tph for p2p access when registering dma-buf mr

[Zhiping Zhang]

On Fri, Jul 31, 2026 at 2:48 PM <sashiko-bot@kernel.org> wrote:
...
> [Severity: High]
> This is a pre-existing issue, but this patch introduces a new, heavily-exercised
> reachable path for it.
>
> Does mlx5_st_dealloc_index() leak the mlx5_st_idx_data struct when erasing it
> from the xarray?
>
> When a steering tag's reference count drops to zero, the entry is removed via
> xa_erase():
>
> drivers/net/ethernet/mellanox/mlx5/core/lib/st.c:mlx5_st_dealloc_index() {
>     ...
>         if (refcount_dec_and_test(&idx_data->usecount)) {
>                 xa_erase(&st->idx_xa, st_index);
>                 /* We leave PCI config space as was before, no mkey will refer to it */
>         }
>     ...
> }
>
> However, the underlying idx_data pointer is not freed with kfree(idx_data).
>
> Because the next time the same tag is requested, a new struct is allocated via
> kzalloc_obj() in mlx5_st_alloc_index_by_tag(), won't repeated allocations and
> deallocations lead to an unbounded kernel memory leak?
>

This is the leak fixed by "net/mlx5: free mlx5_st_idx_data on final
dealloc" (df6134b527a8). Sashiko appears to have applied the series to
a tree predating that commit.