Re: [PATCH v13 2/5] PCI/TPH: Add requester/completer type helpers

Re: [PATCH v13 2/5] PCI/TPH: Add requester/completer type helpers

[fengchengwen]

Acked-by: Chengwen Feng <fengchengwen@huawei.com>

On 8/1/2026 5:15 AM, Zhiping Zhang wrote:
> Add pcie_tph_enabled_req_type() so drivers can query the enabled TPH
> requester mode without reaching into pci_dev internals.
> 
> Add pcie_tph_completer_type() so drivers that publish TPH metadata for a
> device acting as a completer can gate on the "TPH Completer Supported"
> field of Device Capabilities 2 (bits 13:12, PCI_EXP_DEVCAP2_TPH_COMP_MASK)
> rather than reusing requester-side state. It applies the same reserved-
> encoding fold as get_rp_completer_type(), inlined, so the reserved 0b10
> value never reaches callers.
> 
> This keeps pci_dev::tph_req_type and the completer-capability decode
> inside the PCI/TPH code and provides !CONFIG_PCIE_TPH stubs for callers.
> 
> Signed-off-by: Zhiping Zhang <zhipingz@meta.com>

Re: [PATCH v13 0/5] vfio/dma-buf: add TPH support for peer-to-peer access

[Zhiping Zhang]

 Hi Christian,

A gentle ping on this series, especially patch 3, which adds the
optional dma-buf get_pci_tph() callback. Could you please review this
when you have a chance?

https://lore.kernel.org/linux-pci/20260731211601.3033906-4-zhipingz@meta.com/

Thanks,
Zhiping


On Fri, Jul 31, 2026 at 2:22 PM Zhiping Zhang <zhipingz@meta.com> wrote:
>
> This series adds TLP Processing Hints (TPH) support to the VFIO dma-buf
> export path, allowing importing drivers (e.g. mlx5) to use the
> exporter's steering tag when performing peer-to-peer DMA into a
> VFIO-owned device.
>
> There is no separate in-tree vendor kernel driver for the target device:
> vfio-pci is the in-tree driver and the targeted device is managed
> from userspace via VFIO passthrough. That is why the ST has to flow
> through a uAPI: userspace owns the device and its ST table, so it is the
> entity that can configure a meaningful value for a given dma-buf. The
> kernel-visible participants are still in-tree: vfio-pci exports the
> dma-buf and mlx5 imports it.
>
> On the effect: the endpoint's PCIe ingress block uses the ST as
> an in-band instruction for the incoming P2P TLP -- selecting a target
> cache partition and, on writes, an in-flight operation on the data
> before it lands. The dma-buf callback keeps this opaque to the
> framework -- only the producer (userspace owner of the VFIO device)
> and the consumer (endpoint block) need to interpret the value. The
> dma-buf get_pci_tph callback itself is optional, but workloads that
> depend on the endpoint's in-flight operation need it because fallback
> does not produce the same result.
>
> The dma-buf hook is intentionally generic and discoverable rather than
> a private side channel. The exporter owns the completing address
> space for the dma-buf and decides whether it can provide a meaningful
> ST/PH tuple for that completer; the dma-buf core keeps the tuple opaque,
> and importers merely request the namespace they support and place the
> returned value on generated TLPs. Exporters that cannot derive a
> meaningful tuple simply return -EOPNOTSUPP.
>
> TPH is advisory: a steering tag that is not honored on the path (for
> example an intermediate routing element that does not forward the TPH
> prefix) is ignored and the request completes as an ordinary,
> non-TPH transaction (PCIe Base 6.4 sec 2.2.7.1). This series therefore
> targets the same-Root-Port / common-switch topology, where the ST
> reaches the completer; cross-Root-Port P2P is best-effort and is not
> gated in the uAPI, since supplying an unused ST is harmless and there
> is no discoverable "TPH routing" capability to test against.
>
> Patch 1 folds the reserved 0b10 "TPH Completer Supported" encoding into
> "not supported" in get_rp_completer_type(), so only architected values
> can reach the TPH Requester Enable field. It was previously posted
> standalone to linux-pci; per Alex Williamson's v12 review it now travels
> with the series, which removes the cross-tree ordering dependency and
> lets review tooling apply the series as posted.
> Patch 2 adds small PCI/TPH type helpers so drivers can query the enabled
> TPH requester mode and the device's TPH Completer Supported field
> without reaching into pci_dev internals (and so callers in
> CONFIG_PCIE_TPH=n builds get a clean fallback). pcie_tph_completer_type()
> applies the same reserved-encoding fold as get_rp_completer_type(),
> inlined locally so the helper is self-contained.
> Patch 3 adds the optional dma_buf_ops::get_pci_tph callback plus the
> dma_buf_get_pci_tph() importer wrapper so importers can fetch TPH
> metadata from an exporter under dmabuf->resv.
> Patch 4 implements get_pci_tph in vfio-pci and adds the new uAPI
> (VFIO_DEVICE_FEATURE_DMA_BUF_TPH) for userspace to attach the metadata.
> Patch 5 wires up the mlx5 RDMA driver as a consumer. It also enforces the
> dma_buf_get_pci_tph() steering-tag lifetime: the tag is only valid for the
> mapping it was queried against, and the mkey's TPH fields cannot be
> reprogrammed in place. mlx5 therefore records the registration-time tuple
> and re-queries after each dma-buf mapping is established under
> dmabuf->resv; unchanged tuples continue with the existing mkey, while
> changed or missing tuples fail the remap rather than continue with a
> stale hint. For vfio-pci BAR dma-bufs this is expected to be a no-op
> because invalidation is revoke/quiesce, not movement to a new backing
> placement, and the userspace-provided tuple is not changed by the
> revoke/un-revoke path.
>
> Build-tested with both CONFIG_PCIE_TPH=y and CONFIG_PCIE_TPH=n.
> Functional validation on the target topology: PCIe analyzer captures
> on the P2P TLPs confirm the ST emitted by mlx5 matches the value
> configured through VFIO_DEVICE_FEATURE_DMA_BUF_TPH, and the end-to-end
> P2P workload only produces results consistent with the endpoint's
> ST-selected in-flight operation. For example, with userspace
> configuring 8-bit ST=0xf0 and PH=2, an analyzer capture of a peer-to-
> peer MWr64 shows "STP MWr64 TC=0 OHC=2 ..." followed by "OHC-B
> ST=F0h PH=2 HV=1":
> (TLP Captures)
> 08000260 -> STP MWr64 TC=0 OHC=2 TS=0 Attr=0 L=8
> F0000004 -> RID=4h:0h.0h EP- Tag=F0h
> E0200000 -> AddrH=000020E0h
> 00080006 -> AddrL=06000800h
> 90F00000 -> OHC-B ST=F0h PH=2 HV=1 AMA=0 AV-
>
> The dma-buf get_pci_tph interface has also been exercised by a second,
> independent importer: a different vendor's NIC whose driver is not yet
> upstream, locally taught to call dma_buf_get_pci_tph(). A PCIe analyzer
> confirmed the ST it placed on outbound P2P TLPs matches the value
> configured through VFIO_DEVICE_FEATURE_DMA_BUF_TPH, the same result as
> with mlx5. Two unrelated importer drivers exercising the callback
> end-to-end shows the interface is not tied to a single consumer. That
> importer change is out-of-tree and not part of this series. For that
> second importer, with userspace configuring 8-bit ST=0xe0 and PH=0,
> an analyzer capture shows:
> (TLP Captures)
> 08200260 -> STP MWr64 TC=0 OHC=2 TS=1 Attr=0 L=8
> 4E00004C -> RID=4Ch:0h.0h EP- Tag=4Eh
> 00170000 -> AddrH=00001700h
> 00200006 -> AddrL=06002000h
> 10E00000 -> OHC-B ST=E0h PH=0 HV=1 AMA=0 AV-
>
> Changes since v12:
>   Patch 1 (PCI/TPH, new to the series): the reserved-encoding fold,
>   previously posted standalone to linux-pci [1], is now the first patch
>   here (Alex Williamson). Sashiko could not apply v12 because of that
>   external dependency; with the fold in-series and the mlx5 leak fix in
>   linux-next, v13 has none. The code is unchanged from the standalone
>   v3; the Fixes: tag is dropped, since no code path can reach the
>   reserved encoding today and the patch is hardening rather than a fix
>   for observed silicon.
>
>   Patch 2 (PCI/TPH): inline the reserved-encoding fold in
>   pcie_tph_completer_type() rather than calling the helper that earlier
>   folding revisions added; that helper was dropped in folding v3 per
>   Bjorn Helgaas and Wei Huang.
>
>   Patch 3 (dma-buf): no functional change.
>
>   Patch 4 (vfio/pci): also gate the DMA_BUF_TPH feature on
>   vdev->pci_ops->get_dmabuf_phys, matching
>   vfio_pci_core_feature_dma_buf(). Without it PROBE reported the feature
>   as supported on a device that advertises TPH Completer support but
>   cannot export a vfio dma-buf at all, so nothing could ever carry the
>   metadata (Alex Williamson, who raised this to uAPI-affecting
>   severity).
>
>   Patch 5 (mlx5): keep the !dev->st early-out in mlx5_st_alloc_index()
>   ahead of the pcie_tph_get_cpu_st() call, so splitting out
>   mlx5_st_alloc_index_by_tag() neither adds an ACPI _DSM invocation on
>   devices without ST support nor changes the errno userspace sees when
>   the _DSM lookup fails (Alex Williamson). The commit message now
>   describes this rather than presenting the split as a pure extraction.
>
> Previous link:
> v12: https://lore.kernel.org/linux-pci/20260715204008.3911275-1-zhipingz@meta.com/
> v11: https://lore.kernel.org/linux-pci/20260702181025.2694961-1-zhipingz@meta.com/
> v10: https://lore.kernel.org/linux-pci/20260630224328.3218796-1-zhipingz@meta.com/
> v9: https://lore.kernel.org/dri-devel/20260622184211.2229399-1-zhipingz@meta.com/
> v8: https://lore.kernel.org/dri-devel/20260615065912.2177918-1-zhipingz@meta.com/
> v7: https://lore.kernel.org/dri-devel/20260611161546.4075580-1-zhipingz@meta.com/
> v6: https://lore.kernel.org/dri-devel/20260608185646.4085127-1-zhipingz@meta.com/
> v5: https://lore.kernel.org/dri-devel/20260526144401.1485788-1-zhipingz@meta.com/
> v4: https://lore.kernel.org/linux-pci/20260519201401.1558410-1-zhipingz@meta.com/
> v3: https://lore.kernel.org/linux-pci/20260512184755.4137227-1-zhipingz@meta.com/
> v2: https://lore.kernel.org/linux-pci/20260430200704.352228-1-zhipingz@meta.com/
>
> Zhiping Zhang (5):
>   PCI/TPH: treat reserved 0b10 completer encoding as unsupported
>   PCI/TPH: Add requester/completer type helpers
>   dma-buf: add optional get_pci_tph() callback
>   vfio/pci: implement get_pci_tph and DMA_BUF_TPH feature
>   RDMA/mlx5: get tph for p2p access when registering dma-buf mr
>
>  drivers/dma-buf/dma-buf.c                     |  32 ++++
>  drivers/infiniband/hw/mlx5/main.c             |   1 +
>  drivers/infiniband/hw/mlx5/mlx5_ib.h          |  11 ++
>  drivers/infiniband/hw/mlx5/mr.c               | 151 +++++++++++++++++-
>  drivers/infiniband/hw/mlx5/odp.c              |   7 +
>  .../net/ethernet/mellanox/mlx5/core/lib/st.c  |  52 +++++-
>  drivers/pci/tph.c                             |  55 ++++++-
>  drivers/vfio/pci/vfio_pci_core.c              |   3 +
>  drivers/vfio/pci/vfio_pci_dmabuf.c            | 120 +++++++++++++-
>  drivers/vfio/pci/vfio_pci_priv.h              |  13 ++
>  include/linux/dma-buf.h                       |  25 +++
>  include/linux/mlx5/driver.h                   |  15 ++
>  include/linux/pci-tph.h                       |   8 +
>  include/uapi/linux/vfio.h                     |  43 +++++
>  14 files changed, 517 insertions(+), 19 deletions(-)
>
> --
> 2.53.0-Meta
>

Re: [PATCH v13 0/5] vfio/dma-buf: add TPH support for peer-to-peer access

[Zhiping Zhang]

Hi Christian,

Another ping on this series for your attention. Pls see below for more details.

Thanks,
Zhiping

On Fri, Aug 14, 2026 at 11:08 AM Zhiping Zhang <zhipingz@meta.com> wrote:
>
>  Hi Christian,
>
> A gentle ping on this series, especially patch 3, which adds the
> optional dma-buf get_pci_tph() callback. Could you please review this
> when you have a chance?
>
> https://lore.kernel.org/linux-pci/20260731211601.3033906-4-zhipingz@meta.com/
>
> Thanks,
> Zhiping
>
>
> On Fri, Jul 31, 2026 at 2:22 PM Zhiping Zhang <zhipingz@meta.com> wrote:
> >
> > This series adds TLP Processing Hints (TPH) support to the VFIO dma-buf
> > export path, allowing importing drivers (e.g. mlx5) to use the
> > exporter's steering tag when performing peer-to-peer DMA into a
> > VFIO-owned device.
> >
> > There is no separate in-tree vendor kernel driver for the target device:
> > vfio-pci is the in-tree driver and the targeted device is managed
> > from userspace via VFIO passthrough. That is why the ST has to flow
> > through a uAPI: userspace owns the device and its ST table, so it is the
> > entity that can configure a meaningful value for a given dma-buf. The
> > kernel-visible participants are still in-tree: vfio-pci exports the
> > dma-buf and mlx5 imports it.
> >
> > On the effect: the endpoint's PCIe ingress block uses the ST as
> > an in-band instruction for the incoming P2P TLP -- selecting a target
> > cache partition and, on writes, an in-flight operation on the data
> > before it lands. The dma-buf callback keeps this opaque to the
> > framework -- only the producer (userspace owner of the VFIO device)
> > and the consumer (endpoint block) need to interpret the value. The
> > dma-buf get_pci_tph callback itself is optional, but workloads that
> > depend on the endpoint's in-flight operation need it because fallback
> > does not produce the same result.
> >
> > The dma-buf hook is intentionally generic and discoverable rather than
> > a private side channel. The exporter owns the completing address
> > space for the dma-buf and decides whether it can provide a meaningful
> > ST/PH tuple for that completer; the dma-buf core keeps the tuple opaque,
> > and importers merely request the namespace they support and place the
> > returned value on generated TLPs. Exporters that cannot derive a
> > meaningful tuple simply return -EOPNOTSUPP.
> >
> > TPH is advisory: a steering tag that is not honored on the path (for
> > example an intermediate routing element that does not forward the TPH
> > prefix) is ignored and the request completes as an ordinary,
> > non-TPH transaction (PCIe Base 6.4 sec 2.2.7.1). This series therefore
> > targets the same-Root-Port / common-switch topology, where the ST
> > reaches the completer; cross-Root-Port P2P is best-effort and is not
> > gated in the uAPI, since supplying an unused ST is harmless and there
> > is no discoverable "TPH routing" capability to test against.
> >
> > Patch 1 folds the reserved 0b10 "TPH Completer Supported" encoding into
> > "not supported" in get_rp_completer_type(), so only architected values
> > can reach the TPH Requester Enable field. It was previously posted
> > standalone to linux-pci; per Alex Williamson's v12 review it now travels
> > with the series, which removes the cross-tree ordering dependency and
> > lets review tooling apply the series as posted.
> > Patch 2 adds small PCI/TPH type helpers so drivers can query the enabled
> > TPH requester mode and the device's TPH Completer Supported field
> > without reaching into pci_dev internals (and so callers in
> > CONFIG_PCIE_TPH=n builds get a clean fallback). pcie_tph_completer_type()
> > applies the same reserved-encoding fold as get_rp_completer_type(),
> > inlined locally so the helper is self-contained.
> > Patch 3 adds the optional dma_buf_ops::get_pci_tph callback plus the
> > dma_buf_get_pci_tph() importer wrapper so importers can fetch TPH
> > metadata from an exporter under dmabuf->resv.
> > Patch 4 implements get_pci_tph in vfio-pci and adds the new uAPI
> > (VFIO_DEVICE_FEATURE_DMA_BUF_TPH) for userspace to attach the metadata.
> > Patch 5 wires up the mlx5 RDMA driver as a consumer. It also enforces the
> > dma_buf_get_pci_tph() steering-tag lifetime: the tag is only valid for the
> > mapping it was queried against, and the mkey's TPH fields cannot be
> > reprogrammed in place. mlx5 therefore records the registration-time tuple
> > and re-queries after each dma-buf mapping is established under
> > dmabuf->resv; unchanged tuples continue with the existing mkey, while
> > changed or missing tuples fail the remap rather than continue with a
> > stale hint. For vfio-pci BAR dma-bufs this is expected to be a no-op
> > because invalidation is revoke/quiesce, not movement to a new backing
> > placement, and the userspace-provided tuple is not changed by the
> > revoke/un-revoke path.
> >
> > Build-tested with both CONFIG_PCIE_TPH=y and CONFIG_PCIE_TPH=n.
> > Functional validation on the target topology: PCIe analyzer captures
> > on the P2P TLPs confirm the ST emitted by mlx5 matches the value
> > configured through VFIO_DEVICE_FEATURE_DMA_BUF_TPH, and the end-to-end
> > P2P workload only produces results consistent with the endpoint's
> > ST-selected in-flight operation. For example, with userspace
> > configuring 8-bit ST=0xf0 and PH=2, an analyzer capture of a peer-to-
> > peer MWr64 shows "STP MWr64 TC=0 OHC=2 ..." followed by "OHC-B
> > ST=F0h PH=2 HV=1":
> > (TLP Captures)
> > 08000260 -> STP MWr64 TC=0 OHC=2 TS=0 Attr=0 L=8
> > F0000004 -> RID=4h:0h.0h EP- Tag=F0h
> > E0200000 -> AddrH=000020E0h
> > 00080006 -> AddrL=06000800h
> > 90F00000 -> OHC-B ST=F0h PH=2 HV=1 AMA=0 AV-
> >
> > The dma-buf get_pci_tph interface has also been exercised by a second,
> > independent importer: a different vendor's NIC whose driver is not yet
> > upstream, locally taught to call dma_buf_get_pci_tph(). A PCIe analyzer
> > confirmed the ST it placed on outbound P2P TLPs matches the value
> > configured through VFIO_DEVICE_FEATURE_DMA_BUF_TPH, the same result as
> > with mlx5. Two unrelated importer drivers exercising the callback
> > end-to-end shows the interface is not tied to a single consumer. That
> > importer change is out-of-tree and not part of this series. For that
> > second importer, with userspace configuring 8-bit ST=0xe0 and PH=0,
> > an analyzer capture shows:
> > (TLP Captures)
> > 08200260 -> STP MWr64 TC=0 OHC=2 TS=1 Attr=0 L=8
> > 4E00004C -> RID=4Ch:0h.0h EP- Tag=4Eh
> > 00170000 -> AddrH=00001700h
> > 00200006 -> AddrL=06002000h
> > 10E00000 -> OHC-B ST=E0h PH=0 HV=1 AMA=0 AV-
> >
> > Changes since v12:
> >   Patch 1 (PCI/TPH, new to the series): the reserved-encoding fold,
> >   previously posted standalone to linux-pci [1], is now the first patch
> >   here (Alex Williamson). Sashiko could not apply v12 because of that
> >   external dependency; with the fold in-series and the mlx5 leak fix in
> >   linux-next, v13 has none. The code is unchanged from the standalone
> >   v3; the Fixes: tag is dropped, since no code path can reach the
> >   reserved encoding today and the patch is hardening rather than a fix
> >   for observed silicon.
> >
> >   Patch 2 (PCI/TPH): inline the reserved-encoding fold in
> >   pcie_tph_completer_type() rather than calling the helper that earlier
> >   folding revisions added; that helper was dropped in folding v3 per
> >   Bjorn Helgaas and Wei Huang.
> >
> >   Patch 3 (dma-buf): no functional change.
> >
> >   Patch 4 (vfio/pci): also gate the DMA_BUF_TPH feature on
> >   vdev->pci_ops->get_dmabuf_phys, matching
> >   vfio_pci_core_feature_dma_buf(). Without it PROBE reported the feature
> >   as supported on a device that advertises TPH Completer support but
> >   cannot export a vfio dma-buf at all, so nothing could ever carry the
> >   metadata (Alex Williamson, who raised this to uAPI-affecting
> >   severity).
> >
> >   Patch 5 (mlx5): keep the !dev->st early-out in mlx5_st_alloc_index()
> >   ahead of the pcie_tph_get_cpu_st() call, so splitting out
> >   mlx5_st_alloc_index_by_tag() neither adds an ACPI _DSM invocation on
> >   devices without ST support nor changes the errno userspace sees when
> >   the _DSM lookup fails (Alex Williamson). The commit message now
> >   describes this rather than presenting the split as a pure extraction.
> >
> > Previous link:
> > v12: https://lore.kernel.org/linux-pci/20260715204008.3911275-1-zhipingz@meta.com/
> > v11: https://lore.kernel.org/linux-pci/20260702181025.2694961-1-zhipingz@meta.com/
> > v10: https://lore.kernel.org/linux-pci/20260630224328.3218796-1-zhipingz@meta.com/
> > v9: https://lore.kernel.org/dri-devel/20260622184211.2229399-1-zhipingz@meta.com/
> > v8: https://lore.kernel.org/dri-devel/20260615065912.2177918-1-zhipingz@meta.com/
> > v7: https://lore.kernel.org/dri-devel/20260611161546.4075580-1-zhipingz@meta.com/
> > v6: https://lore.kernel.org/dri-devel/20260608185646.4085127-1-zhipingz@meta.com/
> > v5: https://lore.kernel.org/dri-devel/20260526144401.1485788-1-zhipingz@meta.com/
> > v4: https://lore.kernel.org/linux-pci/20260519201401.1558410-1-zhipingz@meta.com/
> > v3: https://lore.kernel.org/linux-pci/20260512184755.4137227-1-zhipingz@meta.com/
> > v2: https://lore.kernel.org/linux-pci/20260430200704.352228-1-zhipingz@meta.com/
> >
> > Zhiping Zhang (5):
> >   PCI/TPH: treat reserved 0b10 completer encoding as unsupported
> >   PCI/TPH: Add requester/completer type helpers
> >   dma-buf: add optional get_pci_tph() callback
> >   vfio/pci: implement get_pci_tph and DMA_BUF_TPH feature
> >   RDMA/mlx5: get tph for p2p access when registering dma-buf mr
> >
> >  drivers/dma-buf/dma-buf.c                     |  32 ++++
> >  drivers/infiniband/hw/mlx5/main.c             |   1 +
> >  drivers/infiniband/hw/mlx5/mlx5_ib.h          |  11 ++
> >  drivers/infiniband/hw/mlx5/mr.c               | 151 +++++++++++++++++-
> >  drivers/infiniband/hw/mlx5/odp.c              |   7 +
> >  .../net/ethernet/mellanox/mlx5/core/lib/st.c  |  52 +++++-
> >  drivers/pci/tph.c                             |  55 ++++++-
> >  drivers/vfio/pci/vfio_pci_core.c              |   3 +
> >  drivers/vfio/pci/vfio_pci_dmabuf.c            | 120 +++++++++++++-
> >  drivers/vfio/pci/vfio_pci_priv.h              |  13 ++
> >  include/linux/dma-buf.h                       |  25 +++
> >  include/linux/mlx5/driver.h                   |  15 ++
> >  include/linux/pci-tph.h                       |   8 +
> >  include/uapi/linux/vfio.h                     |  43 +++++
> >  14 files changed, 517 insertions(+), 19 deletions(-)
> >
> > --
> > 2.53.0-Meta
> >

Re: [PATCH v13 0/5] vfio/dma-buf: add TPH support for peer-to-peer access - ping

[Zhiping Zhang]

Hi Christian,

Third ping on this series. The only patch still without review is 3/5,
which adds an optional get_pci_tph() callback to dma_buf_ops:

https://lore.kernel.org/kvm/20260731211601.3033906-4-zhipingz@meta.com/

It lets an exporter report the PCIe TPH steering tag for a
peer-to-peer mapping. The callback is optional -- exporters that don't
implement it behave exactly as today -- and nothing else in dma-buf
changes.

The rest of the series is settled: Chengwen acked patch 2, the
automated review is clean, and there are no open comments from v12.
Alex is holding his ack until the dma-buf side has been looked at, so
this one callback is gating the whole series.

If you have concerns about the API shape or the locking, I'll respin.
If someone else should review the dma-buf side, tell me who and I'll
take it there. Otherwise I'll check with Alex and Jason next week on
how to proceed, since 7.3 is in rc and the next window is about a
month out.

Thanks,
Zhiping

On Wed, Sep 2, 2026 at 1:53 PM Zhiping Zhang <zhipingz@meta.com> wrote:
>
> Hi Christian,
>
> Another ping on this series for your attention. Pls see below for more details.
>
> Thanks,
> Zhiping
>
> On Fri, Aug 14, 2026 at 11:08 AM Zhiping Zhang <zhipingz@meta.com> wrote:
> >
> >  Hi Christian,
> >
> > A gentle ping on this series, especially patch 3, which adds the
> > optional dma-buf get_pci_tph() callback. Could you please review this
> > when you have a chance?
> >
> > https://lore.kernel.org/linux-pci/20260731211601.3033906-4-zhipingz@meta.com/
> >
> > Thanks,
> > Zhiping
> >
> >
> > On Fri, Jul 31, 2026 at 2:22 PM Zhiping Zhang <zhipingz@meta.com> wrote:
> > >
> > > This series adds TLP Processing Hints (TPH) support to the VFIO dma-buf
> > > export path, allowing importing drivers (e.g. mlx5) to use the
> > > exporter's steering tag when performing peer-to-peer DMA into a
> > > VFIO-owned device.
> > >
> > > There is no separate in-tree vendor kernel driver for the target device:
> > > vfio-pci is the in-tree driver and the targeted device is managed
> > > from userspace via VFIO passthrough. That is why the ST has to flow
> > > through a uAPI: userspace owns the device and its ST table, so it is the
> > > entity that can configure a meaningful value for a given dma-buf. The
> > > kernel-visible participants are still in-tree: vfio-pci exports the
> > > dma-buf and mlx5 imports it.
> > >
> > > On the effect: the endpoint's PCIe ingress block uses the ST as
> > > an in-band instruction for the incoming P2P TLP -- selecting a target
> > > cache partition and, on writes, an in-flight operation on the data
> > > before it lands. The dma-buf callback keeps this opaque to the
> > > framework -- only the producer (userspace owner of the VFIO device)
> > > and the consumer (endpoint block) need to interpret the value. The
> > > dma-buf get_pci_tph callback itself is optional, but workloads that
> > > depend on the endpoint's in-flight operation need it because fallback
> > > does not produce the same result.
> > >
> > > The dma-buf hook is intentionally generic and discoverable rather than
> > > a private side channel. The exporter owns the completing address
> > > space for the dma-buf and decides whether it can provide a meaningful
> > > ST/PH tuple for that completer; the dma-buf core keeps the tuple opaque,
> > > and importers merely request the namespace they support and place the
> > > returned value on generated TLPs. Exporters that cannot derive a
> > > meaningful tuple simply return -EOPNOTSUPP.
> > >
> > > TPH is advisory: a steering tag that is not honored on the path (for
> > > example an intermediate routing element that does not forward the TPH
> > > prefix) is ignored and the request completes as an ordinary,
> > > non-TPH transaction (PCIe Base 6.4 sec 2.2.7.1). This series therefore
> > > targets the same-Root-Port / common-switch topology, where the ST
> > > reaches the completer; cross-Root-Port P2P is best-effort and is not
> > > gated in the uAPI, since supplying an unused ST is harmless and there
> > > is no discoverable "TPH routing" capability to test against.
> > >
> > > Patch 1 folds the reserved 0b10 "TPH Completer Supported" encoding into
> > > "not supported" in get_rp_completer_type(), so only architected values
> > > can reach the TPH Requester Enable field. It was previously posted
> > > standalone to linux-pci; per Alex Williamson's v12 review it now travels
> > > with the series, which removes the cross-tree ordering dependency and
> > > lets review tooling apply the series as posted.
> > > Patch 2 adds small PCI/TPH type helpers so drivers can query the enabled
> > > TPH requester mode and the device's TPH Completer Supported field
> > > without reaching into pci_dev internals (and so callers in
> > > CONFIG_PCIE_TPH=n builds get a clean fallback). pcie_tph_completer_type()
> > > applies the same reserved-encoding fold as get_rp_completer_type(),
> > > inlined locally so the helper is self-contained.
> > > Patch 3 adds the optional dma_buf_ops::get_pci_tph callback plus the
> > > dma_buf_get_pci_tph() importer wrapper so importers can fetch TPH
> > > metadata from an exporter under dmabuf->resv.
> > > Patch 4 implements get_pci_tph in vfio-pci and adds the new uAPI
> > > (VFIO_DEVICE_FEATURE_DMA_BUF_TPH) for userspace to attach the metadata.
> > > Patch 5 wires up the mlx5 RDMA driver as a consumer. It also enforces the
> > > dma_buf_get_pci_tph() steering-tag lifetime: the tag is only valid for the
> > > mapping it was queried against, and the mkey's TPH fields cannot be
> > > reprogrammed in place. mlx5 therefore records the registration-time tuple
> > > and re-queries after each dma-buf mapping is established under
> > > dmabuf->resv; unchanged tuples continue with the existing mkey, while
> > > changed or missing tuples fail the remap rather than continue with a
> > > stale hint. For vfio-pci BAR dma-bufs this is expected to be a no-op
> > > because invalidation is revoke/quiesce, not movement to a new backing
> > > placement, and the userspace-provided tuple is not changed by the
> > > revoke/un-revoke path.
> > >
> > > Build-tested with both CONFIG_PCIE_TPH=y and CONFIG_PCIE_TPH=n.
> > > Functional validation on the target topology: PCIe analyzer captures
> > > on the P2P TLPs confirm the ST emitted by mlx5 matches the value
> > > configured through VFIO_DEVICE_FEATURE_DMA_BUF_TPH, and the end-to-end
> > > P2P workload only produces results consistent with the endpoint's
> > > ST-selected in-flight operation. For example, with userspace
> > > configuring 8-bit ST=0xf0 and PH=2, an analyzer capture of a peer-to-
> > > peer MWr64 shows "STP MWr64 TC=0 OHC=2 ..." followed by "OHC-B
> > > ST=F0h PH=2 HV=1":
> > > (TLP Captures)
> > > 08000260 -> STP MWr64 TC=0 OHC=2 TS=0 Attr=0 L=8
> > > F0000004 -> RID=4h:0h.0h EP- Tag=F0h
> > > E0200000 -> AddrH=000020E0h
> > > 00080006 -> AddrL=06000800h
> > > 90F00000 -> OHC-B ST=F0h PH=2 HV=1 AMA=0 AV-
> > >
> > > The dma-buf get_pci_tph interface has also been exercised by a second,
> > > independent importer: a different vendor's NIC whose driver is not yet
> > > upstream, locally taught to call dma_buf_get_pci_tph(). A PCIe analyzer
> > > confirmed the ST it placed on outbound P2P TLPs matches the value
> > > configured through VFIO_DEVICE_FEATURE_DMA_BUF_TPH, the same result as
> > > with mlx5. Two unrelated importer drivers exercising the callback
> > > end-to-end shows the interface is not tied to a single consumer. That
> > > importer change is out-of-tree and not part of this series. For that
> > > second importer, with userspace configuring 8-bit ST=0xe0 and PH=0,
> > > an analyzer capture shows:
> > > (TLP Captures)
> > > 08200260 -> STP MWr64 TC=0 OHC=2 TS=1 Attr=0 L=8
> > > 4E00004C -> RID=4Ch:0h.0h EP- Tag=4Eh
> > > 00170000 -> AddrH=00001700h
> > > 00200006 -> AddrL=06002000h
> > > 10E00000 -> OHC-B ST=E0h PH=0 HV=1 AMA=0 AV-
> > >
> > > Changes since v12:
> > >   Patch 1 (PCI/TPH, new to the series): the reserved-encoding fold,
> > >   previously posted standalone to linux-pci [1], is now the first patch
> > >   here (Alex Williamson). Sashiko could not apply v12 because of that
> > >   external dependency; with the fold in-series and the mlx5 leak fix in
> > >   linux-next, v13 has none. The code is unchanged from the standalone
> > >   v3; the Fixes: tag is dropped, since no code path can reach the
> > >   reserved encoding today and the patch is hardening rather than a fix
> > >   for observed silicon.
> > >
> > >   Patch 2 (PCI/TPH): inline the reserved-encoding fold in
> > >   pcie_tph_completer_type() rather than calling the helper that earlier
> > >   folding revisions added; that helper was dropped in folding v3 per
> > >   Bjorn Helgaas and Wei Huang.
> > >
> > >   Patch 3 (dma-buf): no functional change.
> > >
> > >   Patch 4 (vfio/pci): also gate the DMA_BUF_TPH feature on
> > >   vdev->pci_ops->get_dmabuf_phys, matching
> > >   vfio_pci_core_feature_dma_buf(). Without it PROBE reported the feature
> > >   as supported on a device that advertises TPH Completer support but
> > >   cannot export a vfio dma-buf at all, so nothing could ever carry the
> > >   metadata (Alex Williamson, who raised this to uAPI-affecting
> > >   severity).
> > >
> > >   Patch 5 (mlx5): keep the !dev->st early-out in mlx5_st_alloc_index()
> > >   ahead of the pcie_tph_get_cpu_st() call, so splitting out
> > >   mlx5_st_alloc_index_by_tag() neither adds an ACPI _DSM invocation on
> > >   devices without ST support nor changes the errno userspace sees when
> > >   the _DSM lookup fails (Alex Williamson). The commit message now
> > >   describes this rather than presenting the split as a pure extraction.
> > >
> > > Previous link:
> > > v12: https://lore.kernel.org/linux-pci/20260715204008.3911275-1-zhipingz@meta.com/
> > > v11: https://lore.kernel.org/linux-pci/20260702181025.2694961-1-zhipingz@meta.com/
> > > v10: https://lore.kernel.org/linux-pci/20260630224328.3218796-1-zhipingz@meta.com/
> > > v9: https://lore.kernel.org/dri-devel/20260622184211.2229399-1-zhipingz@meta.com/
> > > v8: https://lore.kernel.org/dri-devel/20260615065912.2177918-1-zhipingz@meta.com/
> > > v7: https://lore.kernel.org/dri-devel/20260611161546.4075580-1-zhipingz@meta.com/
> > > v6: https://lore.kernel.org/dri-devel/20260608185646.4085127-1-zhipingz@meta.com/
> > > v5: https://lore.kernel.org/dri-devel/20260526144401.1485788-1-zhipingz@meta.com/
> > > v4: https://lore.kernel.org/linux-pci/20260519201401.1558410-1-zhipingz@meta.com/
> > > v3: https://lore.kernel.org/linux-pci/20260512184755.4137227-1-zhipingz@meta.com/
> > > v2: https://lore.kernel.org/linux-pci/20260430200704.352228-1-zhipingz@meta.com/
> > >
> > > Zhiping Zhang (5):
> > >   PCI/TPH: treat reserved 0b10 completer encoding as unsupported
> > >   PCI/TPH: Add requester/completer type helpers
> > >   dma-buf: add optional get_pci_tph() callback
> > >   vfio/pci: implement get_pci_tph and DMA_BUF_TPH feature
> > >   RDMA/mlx5: get tph for p2p access when registering dma-buf mr
> > >
> > >  drivers/dma-buf/dma-buf.c                     |  32 ++++
> > >  drivers/infiniband/hw/mlx5/main.c             |   1 +
> > >  drivers/infiniband/hw/mlx5/mlx5_ib.h          |  11 ++
> > >  drivers/infiniband/hw/mlx5/mr.c               | 151 +++++++++++++++++-
> > >  drivers/infiniband/hw/mlx5/odp.c              |   7 +
> > >  .../net/ethernet/mellanox/mlx5/core/lib/st.c  |  52 +++++-
> > >  drivers/pci/tph.c                             |  55 ++++++-
> > >  drivers/vfio/pci/vfio_pci_core.c              |   3 +
> > >  drivers/vfio/pci/vfio_pci_dmabuf.c            | 120 +++++++++++++-
> > >  drivers/vfio/pci/vfio_pci_priv.h              |  13 ++
> > >  include/linux/dma-buf.h                       |  25 +++
> > >  include/linux/mlx5/driver.h                   |  15 ++
> > >  include/linux/pci-tph.h                       |   8 +
> > >  include/uapi/linux/vfio.h                     |  43 +++++
> > >  14 files changed, 517 insertions(+), 19 deletions(-)
> > >
> > > --
> > > 2.53.0-Meta
> > >

Re: [PATCH v13 0/5] vfio/dma-buf: add TPH support for peer-to-peer access - ping

[Christian König]

Hi Zhiping,

sorry I'm completely underwater at the moment and don't have time to take another look at the full set.

But IIRC you already fixed my documentation requirements and skimming over the patch once more I can't see anything wrong of hand.

So feel free to add Acked-by: Christian König <christian.koenig@amd.com> to patch "dma-buf: add optional get_pci_tph() callback" and push it upstream through the VFIO channels.

Regards,
Christian.

On 9/10/26 23:46, Zhiping Zhang wrote:
> Hi Christian,
> 
> Third ping on this series. The only patch still without review is 3/5,
> which adds an optional get_pci_tph() callback to dma_buf_ops:
> 
> https://lore.kernel.org/kvm/20260731211601.3033906-4-zhipingz@meta.com/
> 
> It lets an exporter report the PCIe TPH steering tag for a
> peer-to-peer mapping. The callback is optional -- exporters that don't
> implement it behave exactly as today -- and nothing else in dma-buf
> changes.
> 
> The rest of the series is settled: Chengwen acked patch 2, the
> automated review is clean, and there are no open comments from v12.
> Alex is holding his ack until the dma-buf side has been looked at, so
> this one callback is gating the whole series.
> 
> If you have concerns about the API shape or the locking, I'll respin.
> If someone else should review the dma-buf side, tell me who and I'll
> take it there. Otherwise I'll check with Alex and Jason next week on
> how to proceed, since 7.3 is in rc and the next window is about a
> month out.
> 
> Thanks,
> Zhiping
> 
> On Wed, Sep 2, 2026 at 1:53 PM Zhiping Zhang <zhipingz@meta.com> wrote:
>>
>> Hi Christian,
>>
>> Another ping on this series for your attention. Pls see below for more details.
>>
>> Thanks,
>> Zhiping
>>
>> On Fri, Aug 14, 2026 at 11:08 AM Zhiping Zhang <zhipingz@meta.com> wrote:
>>>
>>>  Hi Christian,
>>>
>>> A gentle ping on this series, especially patch 3, which adds the
>>> optional dma-buf get_pci_tph() callback. Could you please review this
>>> when you have a chance?
>>>
>>> https://lore.kernel.org/linux-pci/20260731211601.3033906-4-zhipingz@meta.com/
>>>
>>> Thanks,
>>> Zhiping
>>>
>>>
>>> On Fri, Jul 31, 2026 at 2:22 PM Zhiping Zhang <zhipingz@meta.com> wrote:
>>>>
>>>> This series adds TLP Processing Hints (TPH) support to the VFIO dma-buf
>>>> export path, allowing importing drivers (e.g. mlx5) to use the
>>>> exporter's steering tag when performing peer-to-peer DMA into a
>>>> VFIO-owned device.
>>>>
>>>> There is no separate in-tree vendor kernel driver for the target device:
>>>> vfio-pci is the in-tree driver and the targeted device is managed
>>>> from userspace via VFIO passthrough. That is why the ST has to flow
>>>> through a uAPI: userspace owns the device and its ST table, so it is the
>>>> entity that can configure a meaningful value for a given dma-buf. The
>>>> kernel-visible participants are still in-tree: vfio-pci exports the
>>>> dma-buf and mlx5 imports it.
>>>>
>>>> On the effect: the endpoint's PCIe ingress block uses the ST as
>>>> an in-band instruction for the incoming P2P TLP -- selecting a target
>>>> cache partition and, on writes, an in-flight operation on the data
>>>> before it lands. The dma-buf callback keeps this opaque to the
>>>> framework -- only the producer (userspace owner of the VFIO device)
>>>> and the consumer (endpoint block) need to interpret the value. The
>>>> dma-buf get_pci_tph callback itself is optional, but workloads that
>>>> depend on the endpoint's in-flight operation need it because fallback
>>>> does not produce the same result.
>>>>
>>>> The dma-buf hook is intentionally generic and discoverable rather than
>>>> a private side channel. The exporter owns the completing address
>>>> space for the dma-buf and decides whether it can provide a meaningful
>>>> ST/PH tuple for that completer; the dma-buf core keeps the tuple opaque,
>>>> and importers merely request the namespace they support and place the
>>>> returned value on generated TLPs. Exporters that cannot derive a
>>>> meaningful tuple simply return -EOPNOTSUPP.
>>>>
>>>> TPH is advisory: a steering tag that is not honored on the path (for
>>>> example an intermediate routing element that does not forward the TPH
>>>> prefix) is ignored and the request completes as an ordinary,
>>>> non-TPH transaction (PCIe Base 6.4 sec 2.2.7.1). This series therefore
>>>> targets the same-Root-Port / common-switch topology, where the ST
>>>> reaches the completer; cross-Root-Port P2P is best-effort and is not
>>>> gated in the uAPI, since supplying an unused ST is harmless and there
>>>> is no discoverable "TPH routing" capability to test against.
>>>>
>>>> Patch 1 folds the reserved 0b10 "TPH Completer Supported" encoding into
>>>> "not supported" in get_rp_completer_type(), so only architected values
>>>> can reach the TPH Requester Enable field. It was previously posted
>>>> standalone to linux-pci; per Alex Williamson's v12 review it now travels
>>>> with the series, which removes the cross-tree ordering dependency and
>>>> lets review tooling apply the series as posted.
>>>> Patch 2 adds small PCI/TPH type helpers so drivers can query the enabled
>>>> TPH requester mode and the device's TPH Completer Supported field
>>>> without reaching into pci_dev internals (and so callers in
>>>> CONFIG_PCIE_TPH=n builds get a clean fallback). pcie_tph_completer_type()
>>>> applies the same reserved-encoding fold as get_rp_completer_type(),
>>>> inlined locally so the helper is self-contained.
>>>> Patch 3 adds the optional dma_buf_ops::get_pci_tph callback plus the
>>>> dma_buf_get_pci_tph() importer wrapper so importers can fetch TPH
>>>> metadata from an exporter under dmabuf->resv.
>>>> Patch 4 implements get_pci_tph in vfio-pci and adds the new uAPI
>>>> (VFIO_DEVICE_FEATURE_DMA_BUF_TPH) for userspace to attach the metadata.
>>>> Patch 5 wires up the mlx5 RDMA driver as a consumer. It also enforces the
>>>> dma_buf_get_pci_tph() steering-tag lifetime: the tag is only valid for the
>>>> mapping it was queried against, and the mkey's TPH fields cannot be
>>>> reprogrammed in place. mlx5 therefore records the registration-time tuple
>>>> and re-queries after each dma-buf mapping is established under
>>>> dmabuf->resv; unchanged tuples continue with the existing mkey, while
>>>> changed or missing tuples fail the remap rather than continue with a
>>>> stale hint. For vfio-pci BAR dma-bufs this is expected to be a no-op
>>>> because invalidation is revoke/quiesce, not movement to a new backing
>>>> placement, and the userspace-provided tuple is not changed by the
>>>> revoke/un-revoke path.
>>>>
>>>> Build-tested with both CONFIG_PCIE_TPH=y and CONFIG_PCIE_TPH=n.
>>>> Functional validation on the target topology: PCIe analyzer captures
>>>> on the P2P TLPs confirm the ST emitted by mlx5 matches the value
>>>> configured through VFIO_DEVICE_FEATURE_DMA_BUF_TPH, and the end-to-end
>>>> P2P workload only produces results consistent with the endpoint's
>>>> ST-selected in-flight operation. For example, with userspace
>>>> configuring 8-bit ST=0xf0 and PH=2, an analyzer capture of a peer-to-
>>>> peer MWr64 shows "STP MWr64 TC=0 OHC=2 ..." followed by "OHC-B
>>>> ST=F0h PH=2 HV=1":
>>>> (TLP Captures)
>>>> 08000260 -> STP MWr64 TC=0 OHC=2 TS=0 Attr=0 L=8
>>>> F0000004 -> RID=4h:0h.0h EP- Tag=F0h
>>>> E0200000 -> AddrH=000020E0h
>>>> 00080006 -> AddrL=06000800h
>>>> 90F00000 -> OHC-B ST=F0h PH=2 HV=1 AMA=0 AV-
>>>>
>>>> The dma-buf get_pci_tph interface has also been exercised by a second,
>>>> independent importer: a different vendor's NIC whose driver is not yet
>>>> upstream, locally taught to call dma_buf_get_pci_tph(). A PCIe analyzer
>>>> confirmed the ST it placed on outbound P2P TLPs matches the value
>>>> configured through VFIO_DEVICE_FEATURE_DMA_BUF_TPH, the same result as
>>>> with mlx5. Two unrelated importer drivers exercising the callback
>>>> end-to-end shows the interface is not tied to a single consumer. That
>>>> importer change is out-of-tree and not part of this series. For that
>>>> second importer, with userspace configuring 8-bit ST=0xe0 and PH=0,
>>>> an analyzer capture shows:
>>>> (TLP Captures)
>>>> 08200260 -> STP MWr64 TC=0 OHC=2 TS=1 Attr=0 L=8
>>>> 4E00004C -> RID=4Ch:0h.0h EP- Tag=4Eh
>>>> 00170000 -> AddrH=00001700h
>>>> 00200006 -> AddrL=06002000h
>>>> 10E00000 -> OHC-B ST=E0h PH=0 HV=1 AMA=0 AV-
>>>>
>>>> Changes since v12:
>>>>   Patch 1 (PCI/TPH, new to the series): the reserved-encoding fold,
>>>>   previously posted standalone to linux-pci [1], is now the first patch
>>>>   here (Alex Williamson). Sashiko could not apply v12 because of that
>>>>   external dependency; with the fold in-series and the mlx5 leak fix in
>>>>   linux-next, v13 has none. The code is unchanged from the standalone
>>>>   v3; the Fixes: tag is dropped, since no code path can reach the
>>>>   reserved encoding today and the patch is hardening rather than a fix
>>>>   for observed silicon.
>>>>
>>>>   Patch 2 (PCI/TPH): inline the reserved-encoding fold in
>>>>   pcie_tph_completer_type() rather than calling the helper that earlier
>>>>   folding revisions added; that helper was dropped in folding v3 per
>>>>   Bjorn Helgaas and Wei Huang.
>>>>
>>>>   Patch 3 (dma-buf): no functional change.
>>>>
>>>>   Patch 4 (vfio/pci): also gate the DMA_BUF_TPH feature on
>>>>   vdev->pci_ops->get_dmabuf_phys, matching
>>>>   vfio_pci_core_feature_dma_buf(). Without it PROBE reported the feature
>>>>   as supported on a device that advertises TPH Completer support but
>>>>   cannot export a vfio dma-buf at all, so nothing could ever carry the
>>>>   metadata (Alex Williamson, who raised this to uAPI-affecting
>>>>   severity).
>>>>
>>>>   Patch 5 (mlx5): keep the !dev->st early-out in mlx5_st_alloc_index()
>>>>   ahead of the pcie_tph_get_cpu_st() call, so splitting out
>>>>   mlx5_st_alloc_index_by_tag() neither adds an ACPI _DSM invocation on
>>>>   devices without ST support nor changes the errno userspace sees when
>>>>   the _DSM lookup fails (Alex Williamson). The commit message now
>>>>   describes this rather than presenting the split as a pure extraction.
>>>>
>>>> Previous link:
>>>> v12: https://lore.kernel.org/linux-pci/20260715204008.3911275-1-zhipingz@meta.com/
>>>> v11: https://lore.kernel.org/linux-pci/20260702181025.2694961-1-zhipingz@meta.com/
>>>> v10: https://lore.kernel.org/linux-pci/20260630224328.3218796-1-zhipingz@meta.com/
>>>> v9: https://lore.kernel.org/dri-devel/20260622184211.2229399-1-zhipingz@meta.com/
>>>> v8: https://lore.kernel.org/dri-devel/20260615065912.2177918-1-zhipingz@meta.com/
>>>> v7: https://lore.kernel.org/dri-devel/20260611161546.4075580-1-zhipingz@meta.com/
>>>> v6: https://lore.kernel.org/dri-devel/20260608185646.4085127-1-zhipingz@meta.com/
>>>> v5: https://lore.kernel.org/dri-devel/20260526144401.1485788-1-zhipingz@meta.com/
>>>> v4: https://lore.kernel.org/linux-pci/20260519201401.1558410-1-zhipingz@meta.com/
>>>> v3: https://lore.kernel.org/linux-pci/20260512184755.4137227-1-zhipingz@meta.com/
>>>> v2: https://lore.kernel.org/linux-pci/20260430200704.352228-1-zhipingz@meta.com/
>>>>
>>>> Zhiping Zhang (5):
>>>>   PCI/TPH: treat reserved 0b10 completer encoding as unsupported
>>>>   PCI/TPH: Add requester/completer type helpers
>>>>   dma-buf: add optional get_pci_tph() callback
>>>>   vfio/pci: implement get_pci_tph and DMA_BUF_TPH feature
>>>>   RDMA/mlx5: get tph for p2p access when registering dma-buf mr
>>>>
>>>>  drivers/dma-buf/dma-buf.c                     |  32 ++++
>>>>  drivers/infiniband/hw/mlx5/main.c             |   1 +
>>>>  drivers/infiniband/hw/mlx5/mlx5_ib.h          |  11 ++
>>>>  drivers/infiniband/hw/mlx5/mr.c               | 151 +++++++++++++++++-
>>>>  drivers/infiniband/hw/mlx5/odp.c              |   7 +
>>>>  .../net/ethernet/mellanox/mlx5/core/lib/st.c  |  52 +++++-
>>>>  drivers/pci/tph.c                             |  55 ++++++-
>>>>  drivers/vfio/pci/vfio_pci_core.c              |   3 +
>>>>  drivers/vfio/pci/vfio_pci_dmabuf.c            | 120 +++++++++++++-
>>>>  drivers/vfio/pci/vfio_pci_priv.h              |  13 ++
>>>>  include/linux/dma-buf.h                       |  25 +++
>>>>  include/linux/mlx5/driver.h                   |  15 ++
>>>>  include/linux/pci-tph.h                       |   8 +
>>>>  include/uapi/linux/vfio.h                     |  43 +++++
>>>>  14 files changed, 517 insertions(+), 19 deletions(-)
>>>>
>>>> --
>>>> 2.53.0-Meta
>>>>

Re: [PATCH v13 0/5] vfio/dma-buf: add TPH support for peer-to-peer access - ping

[Zhiping Zhang]

Thanks Christian -- no worries on the bandwidth, I'll add the Acked-by
to patch 3.

Hi Alex, Jason, Bjorn,

Could you help with reviews on patch 4 (vfio/pci) and patch 5
(RDMA/mlx5), plus acks on patches 1-2 (PCI TPH core) so the set can
travel whole if it goes through the VFIO tree, which is the route
Christian suggests?

I've also rebased the set onto 7.3 (base 5225b8eec4c9) and retested on
hardware -- PCIe analyzer captures confirming the requested ST and PH
on outbound TLPs, no splats.

One thing to decide: upstream took 13 for
VFIO_DEVICE_FEATURE_ZPCI_ERROR after v13 went out, so
VFIO_DEVICE_FEATURE_DMA_BUF_TPH becomes 14 in the respin -- let me
know if you'd rather it be something else.

I'll post v14 after I hear from you. Delta from v13 is the rebase,
that feature-number move, and nothing else functional.

Thanks,
Zhiping

On Fri, Sep 11, 2026 at 12:58 AM Christian König
<christian.koenig@amd.com> wrote:
>
> >
> Hi Zhiping,
>
> sorry I'm completely underwater at the moment and don't have time to take another look at the full set.
>
> But IIRC you already fixed my documentation requirements and skimming over the patch once more I can't see anything wrong of hand.
>
> So feel free to add Acked-by: Christian König <christian.koenig@amd.com> to patch "dma-buf: add optional get_pci_tph() callback" and push it upstream through the VFIO channels.
>
> Regards,
> Christian.
>
> On 9/10/26 23:46, Zhiping Zhang wrote:
> > Hi Christian,
> >
> > Third ping on this series. The only patch still without review is 3/5,
> > which adds an optional get_pci_tph() callback to dma_buf_ops:
> >
> > https://lore.kernel.org/kvm/20260731211601.3033906-4-zhipingz@meta.com/
> >
> > It lets an exporter report the PCIe TPH steering tag for a
> > peer-to-peer mapping. The callback is optional -- exporters that don't
> > implement it behave exactly as today -- and nothing else in dma-buf
> > changes.
> >
> > The rest of the series is settled: Chengwen acked patch 2, the
> > automated review is clean, and there are no open comments from v12.
> > Alex is holding his ack until the dma-buf side has been looked at, so
> > this one callback is gating the whole series.
> >
> > If you have concerns about the API shape or the locking, I'll respin.
> > If someone else should review the dma-buf side, tell me who and I'll
> > take it there. Otherwise I'll check with Alex and Jason next week on
> > how to proceed, since 7.3 is in rc and the next window is about a
> > month out.
> >
> > Thanks,
> > Zhiping
> >
> > On Wed, Sep 2, 2026 at 1:53 PM Zhiping Zhang <zhipingz@meta.com> wrote:
> >>
> >> Hi Christian,
> >>
> >> Another ping on this series for your attention. Pls see below for more details.
> >>
> >> Thanks,
> >> Zhiping
> >>
> >> On Fri, Aug 14, 2026 at 11:08 AM Zhiping Zhang <zhipingz@meta.com> wrote:
> >>>
> >>>  Hi Christian,
> >>>
> >>> A gentle ping on this series, especially patch 3, which adds the
> >>> optional dma-buf get_pci_tph() callback. Could you please review this
> >>> when you have a chance?
> >>>
> >>> https://lore.kernel.org/linux-pci/20260731211601.3033906-4-zhipingz@meta.com/
> >>>
> >>> Thanks,
> >>> Zhiping
> >>>
> >>>
> >>> On Fri, Jul 31, 2026 at 2:22 PM Zhiping Zhang <zhipingz@meta.com> wrote:
> >>>>
> >>>> This series adds TLP Processing Hints (TPH) support to the VFIO dma-buf
> >>>> export path, allowing importing drivers (e.g. mlx5) to use the
> >>>> exporter's steering tag when performing peer-to-peer DMA into a
> >>>> VFIO-owned device.
> >>>>
> >>>> There is no separate in-tree vendor kernel driver for the target device:
> >>>> vfio-pci is the in-tree driver and the targeted device is managed
> >>>> from userspace via VFIO passthrough. That is why the ST has to flow
> >>>> through a uAPI: userspace owns the device and its ST table, so it is the
> >>>> entity that can configure a meaningful value for a given dma-buf. The
> >>>> kernel-visible participants are still in-tree: vfio-pci exports the
> >>>> dma-buf and mlx5 imports it.
> >>>>
> >>>> On the effect: the endpoint's PCIe ingress block uses the ST as
> >>>> an in-band instruction for the incoming P2P TLP -- selecting a target
> >>>> cache partition and, on writes, an in-flight operation on the data
> >>>> before it lands. The dma-buf callback keeps this opaque to the
> >>>> framework -- only the producer (userspace owner of the VFIO device)
> >>>> and the consumer (endpoint block) need to interpret the value. The
> >>>> dma-buf get_pci_tph callback itself is optional, but workloads that
> >>>> depend on the endpoint's in-flight operation need it because fallback
> >>>> does not produce the same result.
> >>>>
> >>>> The dma-buf hook is intentionally generic and discoverable rather than
> >>>> a private side channel. The exporter owns the completing address
> >>>> space for the dma-buf and decides whether it can provide a meaningful
> >>>> ST/PH tuple for that completer; the dma-buf core keeps the tuple opaque,
> >>>> and importers merely request the namespace they support and place the
> >>>> returned value on generated TLPs. Exporters that cannot derive a
> >>>> meaningful tuple simply return -EOPNOTSUPP.
> >>>>
> >>>> TPH is advisory: a steering tag that is not honored on the path (for
> >>>> example an intermediate routing element that does not forward the TPH
> >>>> prefix) is ignored and the request completes as an ordinary,
> >>>> non-TPH transaction (PCIe Base 6.4 sec 2.2.7.1). This series therefore
> >>>> targets the same-Root-Port / common-switch topology, where the ST
> >>>> reaches the completer; cross-Root-Port P2P is best-effort and is not
> >>>> gated in the uAPI, since supplying an unused ST is harmless and there
> >>>> is no discoverable "TPH routing" capability to test against.
> >>>>
> >>>> Patch 1 folds the reserved 0b10 "TPH Completer Supported" encoding into
> >>>> "not supported" in get_rp_completer_type(), so only architected values
> >>>> can reach the TPH Requester Enable field. It was previously posted
> >>>> standalone to linux-pci; per Alex Williamson's v12 review it now travels
> >>>> with the series, which removes the cross-tree ordering dependency and
> >>>> lets review tooling apply the series as posted.
> >>>> Patch 2 adds small PCI/TPH type helpers so drivers can query the enabled
> >>>> TPH requester mode and the device's TPH Completer Supported field
> >>>> without reaching into pci_dev internals (and so callers in
> >>>> CONFIG_PCIE_TPH=n builds get a clean fallback). pcie_tph_completer_type()
> >>>> applies the same reserved-encoding fold as get_rp_completer_type(),
> >>>> inlined locally so the helper is self-contained.
> >>>> Patch 3 adds the optional dma_buf_ops::get_pci_tph callback plus the
> >>>> dma_buf_get_pci_tph() importer wrapper so importers can fetch TPH
> >>>> metadata from an exporter under dmabuf->resv.
> >>>> Patch 4 implements get_pci_tph in vfio-pci and adds the new uAPI
> >>>> (VFIO_DEVICE_FEATURE_DMA_BUF_TPH) for userspace to attach the metadata.
> >>>> Patch 5 wires up the mlx5 RDMA driver as a consumer. It also enforces the
> >>>> dma_buf_get_pci_tph() steering-tag lifetime: the tag is only valid for the
> >>>> mapping it was queried against, and the mkey's TPH fields cannot be
> >>>> reprogrammed in place. mlx5 therefore records the registration-time tuple
> >>>> and re-queries after each dma-buf mapping is established under
> >>>> dmabuf->resv; unchanged tuples continue with the existing mkey, while
> >>>> changed or missing tuples fail the remap rather than continue with a
> >>>> stale hint. For vfio-pci BAR dma-bufs this is expected to be a no-op
> >>>> because invalidation is revoke/quiesce, not movement to a new backing
> >>>> placement, and the userspace-provided tuple is not changed by the
> >>>> revoke/un-revoke path.
> >>>>
> >>>> Build-tested with both CONFIG_PCIE_TPH=y and CONFIG_PCIE_TPH=n.
> >>>> Functional validation on the target topology: PCIe analyzer captures
> >>>> on the P2P TLPs confirm the ST emitted by mlx5 matches the value
> >>>> configured through VFIO_DEVICE_FEATURE_DMA_BUF_TPH, and the end-to-end
> >>>> P2P workload only produces results consistent with the endpoint's
> >>>> ST-selected in-flight operation. For example, with userspace
> >>>> configuring 8-bit ST=0xf0 and PH=2, an analyzer capture of a peer-to-
> >>>> peer MWr64 shows "STP MWr64 TC=0 OHC=2 ..." followed by "OHC-B
> >>>> ST=F0h PH=2 HV=1":
> >>>> (TLP Captures)
> >>>> 08000260 -> STP MWr64 TC=0 OHC=2 TS=0 Attr=0 L=8
> >>>> F0000004 -> RID=4h:0h.0h EP- Tag=F0h
> >>>> E0200000 -> AddrH=000020E0h
> >>>> 00080006 -> AddrL=06000800h
> >>>> 90F00000 -> OHC-B ST=F0h PH=2 HV=1 AMA=0 AV-
> >>>>
> >>>> The dma-buf get_pci_tph interface has also been exercised by a second,
> >>>> independent importer: a different vendor's NIC whose driver is not yet
> >>>> upstream, locally taught to call dma_buf_get_pci_tph(). A PCIe analyzer
> >>>> confirmed the ST it placed on outbound P2P TLPs matches the value
> >>>> configured through VFIO_DEVICE_FEATURE_DMA_BUF_TPH, the same result as
> >>>> with mlx5. Two unrelated importer drivers exercising the callback
> >>>> end-to-end shows the interface is not tied to a single consumer. That
> >>>> importer change is out-of-tree and not part of this series. For that
> >>>> second importer, with userspace configuring 8-bit ST=0xe0 and PH=0,
> >>>> an analyzer capture shows:
> >>>> (TLP Captures)
> >>>> 08200260 -> STP MWr64 TC=0 OHC=2 TS=1 Attr=0 L=8
> >>>> 4E00004C -> RID=4Ch:0h.0h EP- Tag=4Eh
> >>>> 00170000 -> AddrH=00001700h
> >>>> 00200006 -> AddrL=06002000h
> >>>> 10E00000 -> OHC-B ST=E0h PH=0 HV=1 AMA=0 AV-
> >>>>
> >>>> Changes since v12:
> >>>>   Patch 1 (PCI/TPH, new to the series): the reserved-encoding fold,
> >>>>   previously posted standalone to linux-pci [1], is now the first patch
> >>>>   here (Alex Williamson). Sashiko could not apply v12 because of that
> >>>>   external dependency; with the fold in-series and the mlx5 leak fix in
> >>>>   linux-next, v13 has none. The code is unchanged from the standalone
> >>>>   v3; the Fixes: tag is dropped, since no code path can reach the
> >>>>   reserved encoding today and the patch is hardening rather than a fix
> >>>>   for observed silicon.
> >>>>
> >>>>   Patch 2 (PCI/TPH): inline the reserved-encoding fold in
> >>>>   pcie_tph_completer_type() rather than calling the helper that earlier
> >>>>   folding revisions added; that helper was dropped in folding v3 per
> >>>>   Bjorn Helgaas and Wei Huang.
> >>>>
> >>>>   Patch 3 (dma-buf): no functional change.
> >>>>
> >>>>   Patch 4 (vfio/pci): also gate the DMA_BUF_TPH feature on
> >>>>   vdev->pci_ops->get_dmabuf_phys, matching
> >>>>   vfio_pci_core_feature_dma_buf(). Without it PROBE reported the feature
> >>>>   as supported on a device that advertises TPH Completer support but
> >>>>   cannot export a vfio dma-buf at all, so nothing could ever carry the
> >>>>   metadata (Alex Williamson, who raised this to uAPI-affecting
> >>>>   severity).
> >>>>
> >>>>   Patch 5 (mlx5): keep the !dev->st early-out in mlx5_st_alloc_index()
> >>>>   ahead of the pcie_tph_get_cpu_st() call, so splitting out
> >>>>   mlx5_st_alloc_index_by_tag() neither adds an ACPI _DSM invocation on
> >>>>   devices without ST support nor changes the errno userspace sees when
> >>>>   the _DSM lookup fails (Alex Williamson). The commit message now
> >>>>   describes this rather than presenting the split as a pure extraction.
> >>>>
> >>>> Previous link:
> >>>> v12: https://lore.kernel.org/linux-pci/20260715204008.3911275-1-zhipingz@meta.com/
> >>>> v11: https://lore.kernel.org/linux-pci/20260702181025.2694961-1-zhipingz@meta.com/
> >>>> v10: https://lore.kernel.org/linux-pci/20260630224328.3218796-1-zhipingz@meta.com/
> >>>> v9: https://lore.kernel.org/dri-devel/20260622184211.2229399-1-zhipingz@meta.com/
> >>>> v8: https://lore.kernel.org/dri-devel/20260615065912.2177918-1-zhipingz@meta.com/
> >>>> v7: https://lore.kernel.org/dri-devel/20260611161546.4075580-1-zhipingz@meta.com/
> >>>> v6: https://lore.kernel.org/dri-devel/20260608185646.4085127-1-zhipingz@meta.com/
> >>>> v5: https://lore.kernel.org/dri-devel/20260526144401.1485788-1-zhipingz@meta.com/
> >>>> v4: https://lore.kernel.org/linux-pci/20260519201401.1558410-1-zhipingz@meta.com/
> >>>> v3: https://lore.kernel.org/linux-pci/20260512184755.4137227-1-zhipingz@meta.com/
> >>>> v2: https://lore.kernel.org/linux-pci/20260430200704.352228-1-zhipingz@meta.com/
> >>>>
> >>>> Zhiping Zhang (5):
> >>>>   PCI/TPH: treat reserved 0b10 completer encoding as unsupported
> >>>>   PCI/TPH: Add requester/completer type helpers
> >>>>   dma-buf: add optional get_pci_tph() callback
> >>>>   vfio/pci: implement get_pci_tph and DMA_BUF_TPH feature
> >>>>   RDMA/mlx5: get tph for p2p access when registering dma-buf mr
> >>>>
> >>>>  drivers/dma-buf/dma-buf.c                     |  32 ++++
> >>>>  drivers/infiniband/hw/mlx5/main.c             |   1 +
> >>>>  drivers/infiniband/hw/mlx5/mlx5_ib.h          |  11 ++
> >>>>  drivers/infiniband/hw/mlx5/mr.c               | 151 +++++++++++++++++-
> >>>>  drivers/infiniband/hw/mlx5/odp.c              |   7 +
> >>>>  .../net/ethernet/mellanox/mlx5/core/lib/st.c  |  52 +++++-
> >>>>  drivers/pci/tph.c                             |  55 ++++++-
> >>>>  drivers/vfio/pci/vfio_pci_core.c              |   3 +
> >>>>  drivers/vfio/pci/vfio_pci_dmabuf.c            | 120 +++++++++++++-
> >>>>  drivers/vfio/pci/vfio_pci_priv.h              |  13 ++
> >>>>  include/linux/dma-buf.h                       |  25 +++
> >>>>  include/linux/mlx5/driver.h                   |  15 ++
> >>>>  include/linux/pci-tph.h                       |   8 +
> >>>>  include/uapi/linux/vfio.h                     |  43 +++++
> >>>>  14 files changed, 517 insertions(+), 19 deletions(-)
> >>>>
> >>>> --
> >>>> 2.53.0-Meta
> >>>>
>

Re: [PATCH v13 0/5] vfio/dma-buf: add TPH support for peer-to-peer access - ping

[Leon Romanovsky]

On Fri, Sep 11, 2026 at 09:58:28AM +0200, Christian König wrote:
> Hi Zhiping,
> 
> sorry I'm completely underwater at the moment and don't have time to take another look at the full set.
> 
> But IIRC you already fixed my documentation requirements and skimming over the patch once more I can't see anything wrong of hand.
> 
> So feel free to add Acked-by: Christian König <christian.koenig@amd.com> to patch "dma-buf: add optional get_pci_tph() callback" and push it upstream through the VFIO channels.

Christian,

I'm slightly confused here. In the other thread, you expressed concerns
about having PCI details common to every device in the P2P world in the
dma-buf code, but here you seem fine with adding a niche P2P feature.

Thanks,

Re: [PATCH v13 0/5] vfio/dma-buf: add TPH support for peer-to-peer access - ping

[Christian König]

On 9/15/26 13:37, Leon Romanovsky wrote:
> On Fri, Sep 11, 2026 at 09:58:28AM +0200, Christian König wrote:
>> Hi Zhiping,
>>
>> sorry I'm completely underwater at the moment and don't have time to take another look at the full set.
>>
>> But IIRC you already fixed my documentation requirements and skimming over the patch once more I can't see anything wrong of hand.
>>
>> So feel free to add Acked-by: Christian König <christian.koenig@amd.com> to patch "dma-buf: add optional get_pci_tph() callback" and push it upstream through the VFIO channels.
> 
> Christian,
> 
> I'm slightly confused here. In the other thread, you expressed concerns
> about having PCI details common to every device in the P2P world in the
> dma-buf code, but here you seem fine with adding a niche P2P feature.

That one is only using u16 and u8 types and has no dependency on the PCI subsystem whatsoever.

Giving those adjacent information sounds perfectly fine to me, we just can't have any hard dependency on PCI subsystem types or functions.

DMA-buf itself is massively used with vendor specific P2P interconnects, PCI P2P is just a special case of that.

Regards,
Christian.

> 
> Thanks,

Re: [PATCH v13 0/5] vfio/dma-buf: add TPH support for peer-to-peer access - ping

[Leon Romanovsky]

On Tue, Sep 15, 2026 at 02:59:43PM +0200, Christian König wrote:
> On 9/15/26 13:37, Leon Romanovsky wrote:
> > On Fri, Sep 11, 2026 at 09:58:28AM +0200, Christian König wrote:
> >> Hi Zhiping,
> >>
> >> sorry I'm completely underwater at the moment and don't have time to take another look at the full set.
> >>
> >> But IIRC you already fixed my documentation requirements and skimming over the patch once more I can't see anything wrong of hand.
> >>
> >> So feel free to add Acked-by: Christian König <christian.koenig@amd.com> to patch "dma-buf: add optional get_pci_tph() callback" and push it upstream through the VFIO channels.
> > 
> > Christian,
> > 
> > I'm slightly confused here. In the other thread, you expressed concerns
> > about having PCI details common to every device in the P2P world in the
> > dma-buf code, but here you seem fine with adding a niche P2P feature.
> 
> That one is only using u16 and u8 types and has no dependency on the PCI subsystem whatsoever.
> 
> Giving those adjacent information sounds perfectly fine to me, we just can't have any hard dependency on PCI subsystem types or functions.
> 
> DMA-buf itself is massively used with vendor specific P2P interconnects, PCI P2P is just a special case of that.

You are looking at it from the DRM perspective. Outside the DRM world, PCI is
THE interconnect, everything else is a special case :).

Thanks

> 
> Regards,
> Christian.
> 
> > 
> > Thanks,

Re: [PATCH v13 0/5] vfio/dma-buf: add TPH support for peer-to-peer access - ping

[Alex Williamson]

On Mon, 14 Sep 2026 09:44:57 -0700
Zhiping Zhang <zhipingz@meta.com> wrote:

> Thanks Christian -- no worries on the bandwidth, I'll add the Acked-by
> to patch 3.
> 
> Hi Alex, Jason, Bjorn,
> 
> Could you help with reviews on patch 4 (vfio/pci) and patch 5
> (RDMA/mlx5), plus acks on patches 1-2 (PCI TPH core) so the set can
> travel whole if it goes through the VFIO tree, which is the route
> Christian suggests?
> 
> I've also rebased the set onto 7.3 (base 5225b8eec4c9) and retested on
> hardware -- PCIe analyzer captures confirming the requested ST and PH
> on outbound TLPs, no splats.
> 
> One thing to decide: upstream took 13 for
> VFIO_DEVICE_FEATURE_ZPCI_ERROR after v13 went out, so
> VFIO_DEVICE_FEATURE_DMA_BUF_TPH becomes 14 in the respin -- let me
> know if you'd rather it be something else.
> 
> I'll post v14 after I hear from you. Delta from v13 is the rebase,
> that feature-number move, and nothing else functional.

IIRC, we're pretty well settled on the vfio-pci front.  The feature
number does need to be iterated to the next available.  We still need
acks from PCI and mlx5, and given the extent of the mlx5 changes I
expect I need to provide a branch for Jason/Leon once I merge it.
Thanks,

Alex


> On Fri, Sep 11, 2026 at 12:58 AM Christian König
> <christian.koenig@amd.com> wrote:
> >  
> > >  
> > Hi Zhiping,
> >
> > sorry I'm completely underwater at the moment and don't have time to take another look at the full set.
> >
> > But IIRC you already fixed my documentation requirements and skimming over the patch once more I can't see anything wrong of hand.
> >
> > So feel free to add Acked-by: Christian König <christian.koenig@amd.com> to patch "dma-buf: add optional get_pci_tph() callback" and push it upstream through the VFIO channels.
> >
> > Regards,
> > Christian.
> >
> > On 9/10/26 23:46, Zhiping Zhang wrote:  
> > > Hi Christian,
> > >
> > > Third ping on this series. The only patch still without review is 3/5,
> > > which adds an optional get_pci_tph() callback to dma_buf_ops:
> > >
> > > https://lore.kernel.org/kvm/20260731211601.3033906-4-zhipingz@meta.com/
> > >
> > > It lets an exporter report the PCIe TPH steering tag for a
> > > peer-to-peer mapping. The callback is optional -- exporters that don't
> > > implement it behave exactly as today -- and nothing else in dma-buf
> > > changes.
> > >
> > > The rest of the series is settled: Chengwen acked patch 2, the
> > > automated review is clean, and there are no open comments from v12.
> > > Alex is holding his ack until the dma-buf side has been looked at, so
> > > this one callback is gating the whole series.
> > >
> > > If you have concerns about the API shape or the locking, I'll respin.
> > > If someone else should review the dma-buf side, tell me who and I'll
> > > take it there. Otherwise I'll check with Alex and Jason next week on
> > > how to proceed, since 7.3 is in rc and the next window is about a
> > > month out.
> > >
> > > Thanks,
> > > Zhiping
> > >
> > > On Wed, Sep 2, 2026 at 1:53 PM Zhiping Zhang <zhipingz@meta.com> wrote:  
> > >>
> > >> Hi Christian,
> > >>
> > >> Another ping on this series for your attention. Pls see below for more details.
> > >>
> > >> Thanks,
> > >> Zhiping
> > >>
> > >> On Fri, Aug 14, 2026 at 11:08 AM Zhiping Zhang <zhipingz@meta.com> wrote:  
> > >>>
> > >>>  Hi Christian,
> > >>>
> > >>> A gentle ping on this series, especially patch 3, which adds the
> > >>> optional dma-buf get_pci_tph() callback. Could you please review this
> > >>> when you have a chance?
> > >>>
> > >>> https://lore.kernel.org/linux-pci/20260731211601.3033906-4-zhipingz@meta.com/
> > >>>
> > >>> Thanks,
> > >>> Zhiping
> > >>>
> > >>>
> > >>> On Fri, Jul 31, 2026 at 2:22 PM Zhiping Zhang <zhipingz@meta.com> wrote:  
> > >>>>
> > >>>> This series adds TLP Processing Hints (TPH) support to the VFIO dma-buf
> > >>>> export path, allowing importing drivers (e.g. mlx5) to use the
> > >>>> exporter's steering tag when performing peer-to-peer DMA into a
> > >>>> VFIO-owned device.
> > >>>>
> > >>>> There is no separate in-tree vendor kernel driver for the target device:
> > >>>> vfio-pci is the in-tree driver and the targeted device is managed
> > >>>> from userspace via VFIO passthrough. That is why the ST has to flow
> > >>>> through a uAPI: userspace owns the device and its ST table, so it is the
> > >>>> entity that can configure a meaningful value for a given dma-buf. The
> > >>>> kernel-visible participants are still in-tree: vfio-pci exports the
> > >>>> dma-buf and mlx5 imports it.
> > >>>>
> > >>>> On the effect: the endpoint's PCIe ingress block uses the ST as
> > >>>> an in-band instruction for the incoming P2P TLP -- selecting a target
> > >>>> cache partition and, on writes, an in-flight operation on the data
> > >>>> before it lands. The dma-buf callback keeps this opaque to the
> > >>>> framework -- only the producer (userspace owner of the VFIO device)
> > >>>> and the consumer (endpoint block) need to interpret the value. The
> > >>>> dma-buf get_pci_tph callback itself is optional, but workloads that
> > >>>> depend on the endpoint's in-flight operation need it because fallback
> > >>>> does not produce the same result.
> > >>>>
> > >>>> The dma-buf hook is intentionally generic and discoverable rather than
> > >>>> a private side channel. The exporter owns the completing address
> > >>>> space for the dma-buf and decides whether it can provide a meaningful
> > >>>> ST/PH tuple for that completer; the dma-buf core keeps the tuple opaque,
> > >>>> and importers merely request the namespace they support and place the
> > >>>> returned value on generated TLPs. Exporters that cannot derive a
> > >>>> meaningful tuple simply return -EOPNOTSUPP.
> > >>>>
> > >>>> TPH is advisory: a steering tag that is not honored on the path (for
> > >>>> example an intermediate routing element that does not forward the TPH
> > >>>> prefix) is ignored and the request completes as an ordinary,
> > >>>> non-TPH transaction (PCIe Base 6.4 sec 2.2.7.1). This series therefore
> > >>>> targets the same-Root-Port / common-switch topology, where the ST
> > >>>> reaches the completer; cross-Root-Port P2P is best-effort and is not
> > >>>> gated in the uAPI, since supplying an unused ST is harmless and there
> > >>>> is no discoverable "TPH routing" capability to test against.
> > >>>>
> > >>>> Patch 1 folds the reserved 0b10 "TPH Completer Supported" encoding into
> > >>>> "not supported" in get_rp_completer_type(), so only architected values
> > >>>> can reach the TPH Requester Enable field. It was previously posted
> > >>>> standalone to linux-pci; per Alex Williamson's v12 review it now travels
> > >>>> with the series, which removes the cross-tree ordering dependency and
> > >>>> lets review tooling apply the series as posted.
> > >>>> Patch 2 adds small PCI/TPH type helpers so drivers can query the enabled
> > >>>> TPH requester mode and the device's TPH Completer Supported field
> > >>>> without reaching into pci_dev internals (and so callers in
> > >>>> CONFIG_PCIE_TPH=n builds get a clean fallback). pcie_tph_completer_type()
> > >>>> applies the same reserved-encoding fold as get_rp_completer_type(),
> > >>>> inlined locally so the helper is self-contained.
> > >>>> Patch 3 adds the optional dma_buf_ops::get_pci_tph callback plus the
> > >>>> dma_buf_get_pci_tph() importer wrapper so importers can fetch TPH
> > >>>> metadata from an exporter under dmabuf->resv.
> > >>>> Patch 4 implements get_pci_tph in vfio-pci and adds the new uAPI
> > >>>> (VFIO_DEVICE_FEATURE_DMA_BUF_TPH) for userspace to attach the metadata.
> > >>>> Patch 5 wires up the mlx5 RDMA driver as a consumer. It also enforces the
> > >>>> dma_buf_get_pci_tph() steering-tag lifetime: the tag is only valid for the
> > >>>> mapping it was queried against, and the mkey's TPH fields cannot be
> > >>>> reprogrammed in place. mlx5 therefore records the registration-time tuple
> > >>>> and re-queries after each dma-buf mapping is established under
> > >>>> dmabuf->resv; unchanged tuples continue with the existing mkey, while
> > >>>> changed or missing tuples fail the remap rather than continue with a
> > >>>> stale hint. For vfio-pci BAR dma-bufs this is expected to be a no-op
> > >>>> because invalidation is revoke/quiesce, not movement to a new backing
> > >>>> placement, and the userspace-provided tuple is not changed by the
> > >>>> revoke/un-revoke path.
> > >>>>
> > >>>> Build-tested with both CONFIG_PCIE_TPH=y and CONFIG_PCIE_TPH=n.
> > >>>> Functional validation on the target topology: PCIe analyzer captures
> > >>>> on the P2P TLPs confirm the ST emitted by mlx5 matches the value
> > >>>> configured through VFIO_DEVICE_FEATURE_DMA_BUF_TPH, and the end-to-end
> > >>>> P2P workload only produces results consistent with the endpoint's
> > >>>> ST-selected in-flight operation. For example, with userspace
> > >>>> configuring 8-bit ST=0xf0 and PH=2, an analyzer capture of a peer-to-
> > >>>> peer MWr64 shows "STP MWr64 TC=0 OHC=2 ..." followed by "OHC-B
> > >>>> ST=F0h PH=2 HV=1":
> > >>>> (TLP Captures)
> > >>>> 08000260 -> STP MWr64 TC=0 OHC=2 TS=0 Attr=0 L=8
> > >>>> F0000004 -> RID=4h:0h.0h EP- Tag=F0h
> > >>>> E0200000 -> AddrH=000020E0h
> > >>>> 00080006 -> AddrL=06000800h
> > >>>> 90F00000 -> OHC-B ST=F0h PH=2 HV=1 AMA=0 AV-
> > >>>>
> > >>>> The dma-buf get_pci_tph interface has also been exercised by a second,
> > >>>> independent importer: a different vendor's NIC whose driver is not yet
> > >>>> upstream, locally taught to call dma_buf_get_pci_tph(). A PCIe analyzer
> > >>>> confirmed the ST it placed on outbound P2P TLPs matches the value
> > >>>> configured through VFIO_DEVICE_FEATURE_DMA_BUF_TPH, the same result as
> > >>>> with mlx5. Two unrelated importer drivers exercising the callback
> > >>>> end-to-end shows the interface is not tied to a single consumer. That
> > >>>> importer change is out-of-tree and not part of this series. For that
> > >>>> second importer, with userspace configuring 8-bit ST=0xe0 and PH=0,
> > >>>> an analyzer capture shows:
> > >>>> (TLP Captures)
> > >>>> 08200260 -> STP MWr64 TC=0 OHC=2 TS=1 Attr=0 L=8
> > >>>> 4E00004C -> RID=4Ch:0h.0h EP- Tag=4Eh
> > >>>> 00170000 -> AddrH=00001700h
> > >>>> 00200006 -> AddrL=06002000h
> > >>>> 10E00000 -> OHC-B ST=E0h PH=0 HV=1 AMA=0 AV-
> > >>>>
> > >>>> Changes since v12:
> > >>>>   Patch 1 (PCI/TPH, new to the series): the reserved-encoding fold,
> > >>>>   previously posted standalone to linux-pci [1], is now the first patch
> > >>>>   here (Alex Williamson). Sashiko could not apply v12 because of that
> > >>>>   external dependency; with the fold in-series and the mlx5 leak fix in
> > >>>>   linux-next, v13 has none. The code is unchanged from the standalone
> > >>>>   v3; the Fixes: tag is dropped, since no code path can reach the
> > >>>>   reserved encoding today and the patch is hardening rather than a fix
> > >>>>   for observed silicon.
> > >>>>
> > >>>>   Patch 2 (PCI/TPH): inline the reserved-encoding fold in
> > >>>>   pcie_tph_completer_type() rather than calling the helper that earlier
> > >>>>   folding revisions added; that helper was dropped in folding v3 per
> > >>>>   Bjorn Helgaas and Wei Huang.
> > >>>>
> > >>>>   Patch 3 (dma-buf): no functional change.
> > >>>>
> > >>>>   Patch 4 (vfio/pci): also gate the DMA_BUF_TPH feature on
> > >>>>   vdev->pci_ops->get_dmabuf_phys, matching
> > >>>>   vfio_pci_core_feature_dma_buf(). Without it PROBE reported the feature
> > >>>>   as supported on a device that advertises TPH Completer support but
> > >>>>   cannot export a vfio dma-buf at all, so nothing could ever carry the
> > >>>>   metadata (Alex Williamson, who raised this to uAPI-affecting
> > >>>>   severity).
> > >>>>
> > >>>>   Patch 5 (mlx5): keep the !dev->st early-out in mlx5_st_alloc_index()
> > >>>>   ahead of the pcie_tph_get_cpu_st() call, so splitting out
> > >>>>   mlx5_st_alloc_index_by_tag() neither adds an ACPI _DSM invocation on
> > >>>>   devices without ST support nor changes the errno userspace sees when
> > >>>>   the _DSM lookup fails (Alex Williamson). The commit message now
> > >>>>   describes this rather than presenting the split as a pure extraction.
> > >>>>
> > >>>> Previous link:
> > >>>> v12: https://lore.kernel.org/linux-pci/20260715204008.3911275-1-zhipingz@meta.com/
> > >>>> v11: https://lore.kernel.org/linux-pci/20260702181025.2694961-1-zhipingz@meta.com/
> > >>>> v10: https://lore.kernel.org/linux-pci/20260630224328.3218796-1-zhipingz@meta.com/
> > >>>> v9: https://lore.kernel.org/dri-devel/20260622184211.2229399-1-zhipingz@meta.com/
> > >>>> v8: https://lore.kernel.org/dri-devel/20260615065912.2177918-1-zhipingz@meta.com/
> > >>>> v7: https://lore.kernel.org/dri-devel/20260611161546.4075580-1-zhipingz@meta.com/
> > >>>> v6: https://lore.kernel.org/dri-devel/20260608185646.4085127-1-zhipingz@meta.com/
> > >>>> v5: https://lore.kernel.org/dri-devel/20260526144401.1485788-1-zhipingz@meta.com/
> > >>>> v4: https://lore.kernel.org/linux-pci/20260519201401.1558410-1-zhipingz@meta.com/
> > >>>> v3: https://lore.kernel.org/linux-pci/20260512184755.4137227-1-zhipingz@meta.com/
> > >>>> v2: https://lore.kernel.org/linux-pci/20260430200704.352228-1-zhipingz@meta.com/
> > >>>>
> > >>>> Zhiping Zhang (5):
> > >>>>   PCI/TPH: treat reserved 0b10 completer encoding as unsupported
> > >>>>   PCI/TPH: Add requester/completer type helpers
> > >>>>   dma-buf: add optional get_pci_tph() callback
> > >>>>   vfio/pci: implement get_pci_tph and DMA_BUF_TPH feature
> > >>>>   RDMA/mlx5: get tph for p2p access when registering dma-buf mr
> > >>>>
> > >>>>  drivers/dma-buf/dma-buf.c                     |  32 ++++
> > >>>>  drivers/infiniband/hw/mlx5/main.c             |   1 +
> > >>>>  drivers/infiniband/hw/mlx5/mlx5_ib.h          |  11 ++
> > >>>>  drivers/infiniband/hw/mlx5/mr.c               | 151 +++++++++++++++++-
> > >>>>  drivers/infiniband/hw/mlx5/odp.c              |   7 +
> > >>>>  .../net/ethernet/mellanox/mlx5/core/lib/st.c  |  52 +++++-
> > >>>>  drivers/pci/tph.c                             |  55 ++++++-
> > >>>>  drivers/vfio/pci/vfio_pci_core.c              |   3 +
> > >>>>  drivers/vfio/pci/vfio_pci_dmabuf.c            | 120 +++++++++++++-
> > >>>>  drivers/vfio/pci/vfio_pci_priv.h              |  13 ++
> > >>>>  include/linux/dma-buf.h                       |  25 +++
> > >>>>  include/linux/mlx5/driver.h                   |  15 ++
> > >>>>  include/linux/pci-tph.h                       |   8 +
> > >>>>  include/uapi/linux/vfio.h                     |  43 +++++
> > >>>>  14 files changed, 517 insertions(+), 19 deletions(-)
> > >>>>
> > >>>> --
> > >>>> 2.53.0-Meta
> > >>>>  
> >  

Re: [PATCH v13 2/5] PCI/TPH: Add requester/completer type helpers

[Bjorn Helgaas]

On Fri, Jul 31, 2026 at 02:15:49PM -0700, Zhiping Zhang wrote:
> Add pcie_tph_enabled_req_type() so drivers can query the enabled TPH
> requester mode without reaching into pci_dev internals.
> 
> Add pcie_tph_completer_type() so drivers that publish TPH metadata for a
> device acting as a completer can gate on the "TPH Completer Supported"
> field of Device Capabilities 2 (bits 13:12, PCI_EXP_DEVCAP2_TPH_COMP_MASK)
> rather than reusing requester-side state. It applies the same reserved-
> encoding fold as get_rp_completer_type(), inlined, so the reserved 0b10
> value never reaches callers.
> 
> This keeps pci_dev::tph_req_type and the completer-capability decode
> inside the PCI/TPH code and provides !CONFIG_PCIE_TPH stubs for callers.
> 
> Signed-off-by: Zhiping Zhang <zhipingz@meta.com>

Acked-by: Bjorn Helgaas <bhelgaas@google.com>

> ---
>  drivers/pci/tph.c       | 44 +++++++++++++++++++++++++++++++++++++++++
>  include/linux/pci-tph.h |  8 ++++++++
>  2 files changed, 52 insertions(+)
> 
> diff --git a/drivers/pci/tph.c b/drivers/pci/tph.c
> index 5cbb381e2a02..d0202ff7a1b2 100644
> --- a/drivers/pci/tph.c
> +++ b/drivers/pci/tph.c
> @@ -223,6 +223,50 @@ static u8 get_rp_completer_type(struct pci_dev *pdev)
>  	return PCI_EXP_DEVCAP2_TPH_COMP_NONE;
>  }
>  
> +/**
> + * pcie_tph_enabled_req_type - Return the device's enabled TPH requester type
> + * @pdev: PCI device to query
> + *
> + * Return: PCI_TPH_REQ_DISABLE, PCI_TPH_REQ_TPH_ONLY or PCI_TPH_REQ_EXT_TPH.
> + */
> +u8 pcie_tph_enabled_req_type(struct pci_dev *pdev)
> +{
> +	return pdev->tph_req_type;
> +}
> +EXPORT_SYMBOL(pcie_tph_enabled_req_type);
> +
> +/**
> + * pcie_tph_completer_type - Return the device's TPH Completer support
> + * @pdev: PCI device to query
> + *
> + * Reads the "TPH Completer Supported" field (bits 13:12) of Device

s/Reads/Read/ to match other doc ("Return ... type" above, "Write tag" below)

> + * Capabilities 2. The reserved 0b10 encoding is folded into
> + * "not supported" so callers only need to compare against the three
> + * defined values.

s/The reserved 0b10 encoding is folded/Fold reserved 0b10 encoding/

> + *
> + * Return: one of %PCI_EXP_DEVCAP2_TPH_COMP_NONE,
> + *         %PCI_EXP_DEVCAP2_TPH_COMP_TPH_ONLY or
> + *         %PCI_EXP_DEVCAP2_TPH_COMP_EXT_TPH.
> + */
> +u8 pcie_tph_completer_type(struct pci_dev *pdev)
> +{
> +	u8 tph_comp;
> +	u32 reg;
> +
> +	if (pcie_capability_read_dword(pdev, PCI_EXP_DEVCAP2, &reg))
> +		return PCI_EXP_DEVCAP2_TPH_COMP_NONE;
> +	if (PCI_POSSIBLE_ERROR(reg))
> +		return PCI_EXP_DEVCAP2_TPH_COMP_NONE;
> +
> +	/* 0b10 is reserved; fold it into "not supported" (see above). */
> +	tph_comp = FIELD_GET(PCI_EXP_DEVCAP2_TPH_COMP_MASK, reg);
> +	if (tph_comp == PCI_EXP_DEVCAP2_TPH_COMP_TPH_ONLY ||
> +	    tph_comp == PCI_EXP_DEVCAP2_TPH_COMP_EXT_TPH)
> +		return tph_comp;
> +	return PCI_EXP_DEVCAP2_TPH_COMP_NONE;
> +}
> +EXPORT_SYMBOL(pcie_tph_completer_type);
> +
>  /* Write tag to ST table - Return 0 if OK, otherwise -errno */
>  static int write_tag_to_st_table(struct pci_dev *pdev, int index, u16 tag)
>  {
> diff --git a/include/linux/pci-tph.h b/include/linux/pci-tph.h
> index be68cd17f2f8..7743af6fe432 100644
> --- a/include/linux/pci-tph.h
> +++ b/include/linux/pci-tph.h
> @@ -9,6 +9,8 @@
>  #ifndef LINUX_PCI_TPH_H
>  #define LINUX_PCI_TPH_H
>  
> +#include <linux/pci_regs.h>
> +
>  /*
>   * According to the ECN for PCI Firmware Spec, Steering Tag can be different
>   * depending on the memory type: Volatile Memory or Persistent Memory. When a
> @@ -30,6 +32,8 @@ void pcie_disable_tph(struct pci_dev *pdev);
>  int pcie_enable_tph(struct pci_dev *pdev, int mode);
>  u16 pcie_tph_get_st_table_size(struct pci_dev *pdev);
>  u32 pcie_tph_get_st_table_loc(struct pci_dev *pdev);
> +u8 pcie_tph_enabled_req_type(struct pci_dev *pdev);
> +u8 pcie_tph_completer_type(struct pci_dev *pdev);
>  #else
>  static inline int pcie_tph_set_st_entry(struct pci_dev *pdev,
>  					unsigned int index, u16 tag)
> @@ -41,6 +45,10 @@ static inline int pcie_tph_get_cpu_st(struct pci_dev *dev,
>  static inline void pcie_disable_tph(struct pci_dev *pdev) { }
>  static inline int pcie_enable_tph(struct pci_dev *pdev, int mode)
>  { return -EINVAL; }
> +static inline u8 pcie_tph_enabled_req_type(struct pci_dev *pdev)
> +{ return PCI_TPH_REQ_DISABLE; }
> +static inline u8 pcie_tph_completer_type(struct pci_dev *pdev)
> +{ return PCI_EXP_DEVCAP2_TPH_COMP_NONE; }
>  #endif
>  
>  #endif /* LINUX_PCI_TPH_H */
> -- 
> 2.53.0-Meta
> 

Re: [PATCH v13 2/5] PCI/TPH: Add requester/completer type helpers

[Zhiping Zhang]

On Tue, Sep 15, 2026 at 3:23 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> >
> On Fri, Jul 31, 2026 at 02:15:49PM -0700, Zhiping Zhang wrote:
> > Add pcie_tph_enabled_req_type() so drivers can query the enabled TPH
> > requester mode without reaching into pci_dev internals.
> >
> > Add pcie_tph_completer_type() so drivers that publish TPH metadata for a
> > device acting as a completer can gate on the "TPH Completer Supported"
> > field of Device Capabilities 2 (bits 13:12, PCI_EXP_DEVCAP2_TPH_COMP_MASK)
> > rather than reusing requester-side state. It applies the same reserved-
> > encoding fold as get_rp_completer_type(), inlined, so the reserved 0b10
> > value never reaches callers.
> >
> > This keeps pci_dev::tph_req_type and the completer-capability decode
> > inside the PCI/TPH code and provides !CONFIG_PCIE_TPH stubs for callers.
> >
> > Signed-off-by: Zhiping Zhang <zhipingz@meta.com>
>
> Acked-by: Bjorn Helgaas <bhelgaas@google.com>
>
> > ---
> >  drivers/pci/tph.c       | 44 +++++++++++++++++++++++++++++++++++++++++
> >  include/linux/pci-tph.h |  8 ++++++++
> >  2 files changed, 52 insertions(+)
> >
> > diff --git a/drivers/pci/tph.c b/drivers/pci/tph.c
> > index 5cbb381e2a02..d0202ff7a1b2 100644
> > --- a/drivers/pci/tph.c
> > +++ b/drivers/pci/tph.c
> > @@ -223,6 +223,50 @@ static u8 get_rp_completer_type(struct pci_dev *pdev)
> >       return PCI_EXP_DEVCAP2_TPH_COMP_NONE;
> >  }
> >
> > +/**
> > + * pcie_tph_enabled_req_type - Return the device's enabled TPH requester type
> > + * @pdev: PCI device to query
> > + *
> > + * Return: PCI_TPH_REQ_DISABLE, PCI_TPH_REQ_TPH_ONLY or PCI_TPH_REQ_EXT_TPH.
> > + */
> > +u8 pcie_tph_enabled_req_type(struct pci_dev *pdev)
> > +{
> > +     return pdev->tph_req_type;
> > +}
> > +EXPORT_SYMBOL(pcie_tph_enabled_req_type);
> > +
> > +/**
> > + * pcie_tph_completer_type - Return the device's TPH Completer support
> > + * @pdev: PCI device to query
> > + *
> > + * Reads the "TPH Completer Supported" field (bits 13:12) of Device
>
> s/Reads/Read/ to match other doc ("Return ... type" above, "Write tag" below)
>
> > + * Capabilities 2. The reserved 0b10 encoding is folded into
> > + * "not supported" so callers only need to compare against the three
> > + * defined values.
>
> s/The reserved 0b10 encoding is folded/Fold reserved 0b10 encoding/
>

Thanks Bjorn for the ack. I'll incorporate both wording fixes in v14.
The kernel-doc becomes:

/**
 * pcie_tph_completer_type - Return the device's TPH Completer support
 * @pdev: PCI device to query
 *
 * Read the "TPH Completer Supported" field (bits 13:12) of Device
 * Capabilities 2. Fold reserved 0b10 encoding into "not supported" so
 * callers only need to compare against the three defined values.
 *
 * Return: one of %PCI_EXP_DEVCAP2_TPH_COMP_NONE,
 *         %PCI_EXP_DEVCAP2_TPH_COMP_TPH_ONLY or
 *         %PCI_EXP_DEVCAP2_TPH_COMP_EXT_TPH.
 */

  Thanks,
  Zhiping

Re: [PATCH v13 0/5] vfio/dma-buf: add TPH support for peer-to-peer access - ping

[Zhiping Zhang]

On Tue, Sep 15, 2026 at 1:49 PM Alex Williamson <alex@shazbot.org> wrote:
>
> >
> On Mon, 14 Sep 2026 09:44:57 -0700
> Zhiping Zhang <zhipingz@meta.com> wrote:
>
> > Thanks Christian -- no worries on the bandwidth, I'll add the Acked-by
> > to patch 3.
> >
> > Hi Alex, Jason, Bjorn,
> >
> > Could you help with reviews on patch 4 (vfio/pci) and patch 5
> > (RDMA/mlx5), plus acks on patches 1-2 (PCI TPH core) so the set can
> > travel whole if it goes through the VFIO tree, which is the route
> > Christian suggests?
> >
> > I've also rebased the set onto 7.3 (base 5225b8eec4c9) and retested on
> > hardware -- PCIe analyzer captures confirming the requested ST and PH
> > on outbound TLPs, no splats.
> >
> > One thing to decide: upstream took 13 for
> > VFIO_DEVICE_FEATURE_ZPCI_ERROR after v13 went out, so
> > VFIO_DEVICE_FEATURE_DMA_BUF_TPH becomes 14 in the respin -- let me
> > know if you'd rather it be something else.
> >
> > I'll post v14 after I hear from you. Delta from v13 is the rebase,
> > that feature-number move, and nothing else functional.
>
> IIRC, we're pretty well settled on the vfio-pci front.  The feature
> number does need to be iterated to the next available.  We still need
> acks from PCI and mlx5, and given the extent of the mlx5 changes I
> expect I need to provide a branch for Jason/Leon once I merge it.
> Thanks,
>
> Alex
>

Thanks Alex, it is great to know the vfio-pci side is settled! Bjorn
acked patches 1 and 2 today, so PCI is covered.

Hi Jason, Leon, could you take a look at patch 5, the RDMA/mlx5 consumer?

https://lore.kernel.org/linux-pci/20260731211601.3033906-6-zhipingz@meta.com/

Thanks,
Zhiping

Re: [PATCH v13 5/5] RDMA/mlx5: get tph for p2p access when registering dma-buf mr

[Zhiping Zhang]

Hi Jason, Leon,

Following up on patch 5, the RDMA/mlx5 consumer, this is the last
piece of the series still waiting on review:
https://lore.kernel.org/linux-pci/20260731211601.3033906-6-zhipingz@meta.com/

Where the rest stands: Bjorn acked patches 1-2 (PCI/TPH), Christian
acked patch 3 (dma-buf), and Alex considers the vfio-pci side settled.
On merge logistics, Alex wrote that given the extent of the mlx5
changes he expects to provide a branch for you two once he takes the
series. Michael reviewed this patch in v11:
https://lore.kernel.org/linux-rdma/a97ff986-d625-44e1-90cf-027b079cb1d0@nvidia.com/
I didn't carry his tag forward because patch 5 in v13 has a small
change: it keeps the !dev->st early-out in mlx5_st_alloc_index() ahead
of pcie_tph_get_cpu_st(), per Alex's v12 review.

I do owe a v14 -- a rebase onto 7.3 plus moving the feature number to
14, since upstream took 13 for VFIO_DEVICE_FEATURE_ZPCI_ERROR. Patch 5
itself is unchanged from v13, so a review now still applies. I'm happy
to post v14 first if you'd rather look at that. If the approach or the
placement is wrong, I'm glad to rework it.

Thanks,
Zhiping

On Fri, Jul 31, 2026 at 2:22 PM Zhiping Zhang <zhipingz@meta.com> wrote:
>
> Peer-to-peer DMA between a mlx5 NIC and a foreign PCIe endpoint
> (typically a GPU or a vfio-pci passthrough device) traverses the host
> PCIe fabric. The endpoint exporting the dma-buf knows which PCIe TLP
> Processing Hint (TPH) Steering Tag yields the best placement for the
> traffic it will sink: per-endpoint hint selection lets the root complex
> or switch direct DMA to a specific cache slice / NUMA node, cutting
> cross-socket snoop traffic and DRAM pressure under sustained p2p
> workloads.
>
> Until now the mlx5 importer had no way to learn the exporter's chosen
> ST tag, so dma-buf MRs were registered without TPH and ran with the
> default (no-hint) routing. With dma_buf_get_pci_tph() in place this
> patch wires up mlx5_ib to query that metadata at MR registration time
> for p2p access and use it to program requester-side TPH on the outbound
> mkey. If the exporter has no metadata, fall back to the existing
> no-TPH path so behavior for non-TPH-aware exporters is unchanged.
>
> Use mlx5_st_alloc_index_by_tag() to translate exporter-provided
> steering tags into local ST entries when table mode is active, and add
> mlx5_st_get_index() for DMAH-backed flows that already carry an ST
> index.
>
> Splitting mlx5_st_alloc_index() moves the !dev->st check into the new
> by-tag helper, so keep an equivalent early-out in mlx5_st_alloc_index()
> itself, ahead of the pcie_tph_get_cpu_st() call. A device without ST
> support therefore still returns -EOPNOTSUPP without an ACPI _DSM
> invocation, exactly as it did before the split. The check is repeated in
> mlx5_st_alloc_index_by_tag() because that is now an exported entry point
> in its own right.
>
> For TPH-backed FRMRs, keep the extra ST-table reference tied to MR
> lifetime rather than pooled mkey lifetime. Acquire the ref before MR
> creation and release it again when the MR is returned to the pool or
> the backing mkey is destroyed, while leaving the generic FRMR pool
> core unchanged.
>
> The exporter's steering tag is only valid for the mapping it was queried
> against (see dma_buf_get_pci_tph()). The mkey's TPH fields cannot be
> reprogrammed in place: they are create-time fields with no UMR update
> mask, and rebuilding the mkey would change the rkey/lkey, which must stay
> stable across invalidate/refault.
>
> Keep the registration-time TPH tuple in the MR and re-query the exporter
> after each dma-buf mapping is established under dmabuf->resv. If the
> tuple is unchanged, continue using the existing mkey. If the tuple changed
> or disappeared, unmap the pages and return -EFAULT rather than continue
> with a stale mkey hint; userspace must create a new MR to get a mkey
> programmed with the new tuple. For vfio-pci BAR dma-bufs this validation
> is expected to be a no-op because invalidation is revoke/quiesce of BAR
> access, not movement to a different backing placement, and the
> userspace-provided TPH tuple is not changed by the revoke/un-revoke path.
>
> Import the DMA_BUF namespace for the new dma_buf_get_pci_tph() call so
> modular mlx5_ib builds link cleanly.
>
> Signed-off-by: Zhiping Zhang <zhipingz@meta.com>
> ---
>  drivers/infiniband/hw/mlx5/main.c             |   1 +
>  drivers/infiniband/hw/mlx5/mlx5_ib.h          |  11 ++
>  drivers/infiniband/hw/mlx5/mr.c               | 151 +++++++++++++++++-
>  drivers/infiniband/hw/mlx5/odp.c              |   7 +
>  .../net/ethernet/mellanox/mlx5/core/lib/st.c  |  52 +++++-
>  include/linux/mlx5/driver.h                   |  15 ++
>  6 files changed, 223 insertions(+), 14 deletions(-)
>
> diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
> index 02809114fc79..a2b497f6b16b 100644
> --- a/drivers/infiniband/hw/mlx5/main.c
> +++ b/drivers/infiniband/hw/mlx5/main.c
> @@ -60,6 +60,7 @@
>  MODULE_AUTHOR("Eli Cohen <eli@mellanox.com>");
>  MODULE_DESCRIPTION("Mellanox 5th generation network adapters (ConnectX series) IB driver");
>  MODULE_LICENSE("Dual BSD/GPL");
> +MODULE_IMPORT_NS("DMA_BUF");
>
>  struct mlx5_ib_event_work {
>         struct work_struct      work;
> diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
> index 522984d958bb..c6a956b2ee87 100644
> --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
> +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
> @@ -646,6 +646,15 @@ enum mlx5_mkey_type {
>  /* Used for non-existent ph value */
>  #define MLX5_IB_NO_PH 0xff
>
> +struct dma_buf;
> +
> +struct mlx5_ib_dmabuf_tph {
> +       u16 steering_tag;
> +       u8 ph;
> +       bool extended;
> +       bool valid;
> +};
> +
>  struct mlx5_ib_mkey {
>         u32 key;
>         enum mlx5_mkey_type type;
> @@ -726,6 +735,7 @@ struct mlx5_ib_mr {
>                         u8 revoked :1;
>                         /* Indicates previous dmabuf page fault occurred */
>                         u8 dmabuf_faulted:1;
> +                       struct mlx5_ib_dmabuf_tph dmabuf_tph;
>                         struct mlx5_ib_mkey null_mmkey;
>                 };
>         };
> @@ -1341,6 +1351,7 @@ struct ib_mr *mlx5_ib_rereg_user_mr(struct ib_mr *ib_mr, int flags, u64 start,
>                                     u64 length, u64 virt_addr, int access_flags,
>                                     struct ib_pd *pd, struct ib_udata *udata);
>  int mlx5_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata);
> +int mlx5_ib_validate_dmabuf_tph(struct mlx5_ib_mr *mr, struct dma_buf *dmabuf);
>  struct ib_mr *mlx5_ib_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
>                                u32 max_num_sg);
>  struct ib_mr *mlx5_ib_alloc_mr_integrity(struct ib_pd *pd,
> diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
> index e6b74955d95d..404c5d081b8a 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -39,6 +39,7 @@
>  #include <linux/delay.h>
>  #include <linux/dma-buf.h>
>  #include <linux/dma-resv.h>
> +#include <linux/pci-tph.h>
>  #include <rdma/frmr_pools.h>
>  #include <rdma/ib_umem_odp.h>
>  #include "dm.h"
> @@ -167,6 +168,61 @@ static int get_unchangeable_access_flags(struct mlx5_ib_dev *dev,
>  #define MLX5_FRMR_POOLS_KERNEL_KEY_PH_MASK GENMASK_ULL(23, 16)
>  #define MLX5_FRMR_POOLS_KERNEL_KEY_ST_INDEX_MASK GENMASK_ULL(15, 0)
>
> +static u8 mlx5_ib_tph_key_ph(u8 ph)
> +{
> +       if (ph == MLX5_IB_NO_PH || ph == 0)
> +               ph ^= MLX5_IB_NO_PH;
> +
> +       return ph;
> +}
> +
> +static int mlx5_ib_get_st_handle_ref(struct mlx5_ib_dev *dev, u16 st_index)
> +{
> +       if (st_index == MLX5_MKC_PCIE_TPH_NO_STEERING_TAG_INDEX)
> +               return 0;
> +
> +       return mlx5_st_get_index(dev->mdev, st_index);
> +}
> +
> +static void mlx5_ib_put_st_index_ref(struct mlx5_ib_dev *dev, u16 st_index)
> +{
> +       if (st_index == MLX5_MKC_PCIE_TPH_NO_STEERING_TAG_INDEX)
> +               return;
> +
> +       mlx5_st_dealloc_index(dev->mdev, st_index);
> +}
> +
> +static void mlx5_ib_put_st_handle_ref(struct mlx5_ib_dev *dev,
> +                                     u64 kernel_vendor_key)
> +{
> +       u16 st_index = FIELD_GET(MLX5_FRMR_POOLS_KERNEL_KEY_ST_INDEX_MASK,
> +                                kernel_vendor_key);
> +
> +       mlx5_ib_put_st_index_ref(dev, st_index);
> +}
> +
> +int mlx5_ib_validate_dmabuf_tph(struct mlx5_ib_mr *mr, struct dma_buf *dmabuf)
> +{
> +       struct mlx5_ib_dmabuf_tph *tph = &mr->dmabuf_tph;
> +       u16 steering_tag;
> +       u8 ph;
> +       int ret;
> +
> +       if (!tph->valid)
> +               return 0;
> +
> +       dma_resv_assert_held(dmabuf->resv);
> +
> +       ret = dma_buf_get_pci_tph(dmabuf, tph->extended, &steering_tag, &ph);
> +       if (ret)
> +               return -EFAULT;
> +
> +       if (steering_tag != tph->steering_tag || ph != tph->ph)
> +               return -EFAULT;
> +
> +       return 0;
> +}
> +
>  static struct mlx5_ib_mr *
>  _mlx5_frmr_pool_alloc(struct mlx5_ib_dev *dev, struct ib_umem *umem,
>                       int access_flags, int access_mode,
> @@ -189,13 +245,10 @@ _mlx5_frmr_pool_alloc(struct mlx5_ib_dev *dev, struct ib_umem *umem,
>                         MLX5_FRMR_POOLS_KEY_ACCESS_MODE_KSM_MASK :
>                         0;
>
> -       /* Normalize ph: swap 0 and MLX5_IB_NO_PH */
> -       if (ph == MLX5_IB_NO_PH || ph == 0)
> -               ph ^= MLX5_IB_NO_PH;
> -
>         mr->ibmr.frmr.key.kernel_vendor_key =
>                 FIELD_PREP(MLX5_FRMR_POOLS_KERNEL_KEY_ST_INDEX_MASK, st_index) |
> -               FIELD_PREP(MLX5_FRMR_POOLS_KERNEL_KEY_PH_MASK, ph);
> +               FIELD_PREP(MLX5_FRMR_POOLS_KERNEL_KEY_PH_MASK,
> +                          mlx5_ib_tph_key_ph(ph));
>         err = ib_frmr_pool_pop(&dev->ib_dev, &mr->ibmr);
>         if (err) {
>                 kfree(mr);
> @@ -218,7 +271,9 @@ struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev,
>                                       0 :
>                                       MLX5_FRMR_POOLS_KEY_ACCESS_MODE_KSM_MASK,
>                 .num_dma_blocks = ndescs,
> -               .kernel_vendor_key = 0, /* no PH and no ST index */
> +               .kernel_vendor_key =
> +                       FIELD_PREP(MLX5_FRMR_POOLS_KERNEL_KEY_ST_INDEX_MASK,
> +                                  MLX5_MKC_PCIE_TPH_NO_STEERING_TAG_INDEX),
>         };
>         struct mlx5_ib_mr *mr;
>         int ret;
> @@ -557,6 +612,10 @@ static struct mlx5_ib_mr *reg_create(struct ib_pd *pd, struct ib_umem *umem,
>         mr->ibmr.pd = pd;
>         mr->access_flags = access_flags;
>         mr->page_shift = order_base_2(page_size);
> +       mr->ibmr.frmr.key.kernel_vendor_key =
> +               FIELD_PREP(MLX5_FRMR_POOLS_KERNEL_KEY_ST_INDEX_MASK, st_index) |
> +               FIELD_PREP(MLX5_FRMR_POOLS_KERNEL_KEY_PH_MASK,
> +                          mlx5_ib_tph_key_ph(ph));
>
>         inlen = MLX5_ST_SZ_BYTES(create_mkey_in);
>         if (populate)
> @@ -753,6 +812,12 @@ static struct ib_mr *create_real_mr(struct ib_pd *pd, struct ib_umem *umem,
>                         st_index = mdmah->st_index;
>         }
>
> +       err = mlx5_ib_get_st_handle_ref(dev, st_index);
> +       if (err) {
> +               ib_umem_release(umem);
> +               return ERR_PTR(err);
> +       }
> +
>         xlt_with_umr = mlx5r_umr_can_load_pas(dev, umem->length);
>         if (xlt_with_umr) {
>                 mr = alloc_cacheable_mr(pd, umem, iova, access_flags,
> @@ -769,6 +834,7 @@ static struct ib_mr *create_real_mr(struct ib_pd *pd, struct ib_umem *umem,
>                 mutex_unlock(&dev->slow_path_mutex);
>         }
>         if (IS_ERR(mr)) {
> +               mlx5_ib_put_st_index_ref(dev, st_index);
>                 ib_umem_release(umem);
>                 return ERR_CAST(mr);
>         }
> @@ -903,6 +969,63 @@ static struct dma_buf_attach_ops mlx5_ib_dmabuf_attach_ops = {
>         .invalidate_mappings = mlx5_ib_dmabuf_invalidate_cb,
>  };
>
> +static void get_pci_tph_mr_dmabuf(struct mlx5_ib_dev *dev, struct dma_buf *dmabuf,
> +                                 struct mlx5_ib_dmabuf_tph *tph,
> +                                 u16 *st_index, u8 *ph)
> +{
> +       u16 local_st_index;
> +       u16 steering_tag;
> +       u8 local_ph;
> +       bool extended;
> +       int ret;
> +
> +       switch (pcie_tph_enabled_req_type(dev->mdev->pdev)) {
> +       case PCI_TPH_REQ_TPH_ONLY:
> +               extended = false;
> +               break;
> +       case PCI_TPH_REQ_EXT_TPH:
> +               extended = true;
> +               break;
> +       default:
> +               return;
> +       }
> +
> +       /*
> +        * dma_buf_get_pci_tph() must run under dmabuf->resv, but drop the lock
> +        * before mlx5_st_alloc_index_by_tag(): that allocates with GFP_KERNEL,
> +        * which must not run under dmabuf->resv since reclaim can re-enter the
> +        * same reservation object through a dynamic exporter's move_notify path.
> +        */
> +       dma_resv_lock(dmabuf->resv, NULL);
> +       ret = dma_buf_get_pci_tph(dmabuf, extended, &steering_tag, &local_ph);
> +       dma_resv_unlock(dmabuf->resv);
> +       if (ret) {
> +               if (ret != -EOPNOTSUPP)
> +                       mlx5_ib_dbg(dev, "get_pci_tph failed (%d)\n", ret);
> +               return;
> +       }
> +
> +       ret = mlx5_st_alloc_index_by_tag(dev->mdev, steering_tag,
> +                                        &local_st_index);
> +       if (ret) {
> +               mlx5_ib_dbg(dev, "st_alloc_index_by_tag failed (%d)\n", ret);
> +               return;
> +       }
> +
> +       *st_index = local_st_index;
> +       *ph = local_ph;
> +       tph->steering_tag = steering_tag;
> +       tph->ph = local_ph;
> +       tph->extended = extended;
> +       tph->valid = true;
> +}
> +
> +static void mlx5_ib_mr_put_st_handle_ref(struct mlx5_ib_mr *mr)
> +{
> +       mlx5_ib_put_st_handle_ref(mr_to_mdev(mr),
> +                                 mr->ibmr.frmr.key.kernel_vendor_key);
> +}
> +
>  static struct ib_mr *
>  reg_user_mr_dmabuf(struct ib_pd *pd, struct device *dma_device,
>                    u64 offset, u64 length, u64 virt_addr,
> @@ -913,6 +1036,7 @@ reg_user_mr_dmabuf(struct ib_pd *pd, struct device *dma_device,
>         struct mlx5_ib_dev *dev = to_mdev(pd->device);
>         struct mlx5_ib_mr *mr = NULL;
>         struct ib_umem_dmabuf *umem_dmabuf;
> +       struct mlx5_ib_dmabuf_tph tph = {};
>         u16 st_index = MLX5_MKC_PCIE_TPH_NO_STEERING_TAG_INDEX;
>         u8 ph = MLX5_IB_NO_PH;
>         int err;
> @@ -945,18 +1069,29 @@ reg_user_mr_dmabuf(struct ib_pd *pd, struct device *dma_device,
>                 ph = dmah->ph;
>                 if (dmah->valid_fields & BIT(IB_DMAH_CPU_ID_EXISTS))
>                         st_index = mdmah->st_index;
> +
> +               err = mlx5_ib_get_st_handle_ref(dev, st_index);
> +               if (err) {
> +                       ib_umem_release(&umem_dmabuf->umem);
> +                       return ERR_PTR(err);
> +               }
> +       } else {
> +               get_pci_tph_mr_dmabuf(dev, umem_dmabuf->attach->dmabuf,
> +                                     &tph, &st_index, &ph);
>         }
>
>         mr = alloc_cacheable_mr(pd, &umem_dmabuf->umem, virt_addr,
>                                 access_flags, access_mode,
>                                 st_index, ph);
>         if (IS_ERR(mr)) {
> +               mlx5_ib_put_st_index_ref(dev, st_index);
>                 ib_umem_release(&umem_dmabuf->umem);
>                 return ERR_CAST(mr);
>         }
>
>         mlx5_ib_dbg(dev, "mkey 0x%x\n", mr->mmkey.key);
>
> +       mr->dmabuf_tph = tph;
>         atomic_add(ib_umem_num_pages(mr->umem), &dev->mdev->priv.reg_pages);
>         umem_dmabuf->private = mr;
>         if (!pinned_mode) {
> @@ -1405,6 +1540,7 @@ static int mlx5r_handle_mkey_cleanup(struct mlx5_ib_mr *mr)
>         if (mr->ibmr.frmr.pool) {
>                 if (!mlx5_umr_revoke_mr_with_lock(mr)) {
>                         ib_frmr_pool_push(mr->ibmr.device, &mr->ibmr);
> +                       mlx5_ib_mr_put_st_handle_ref(mr);
>                         return 0;
>                 }
>         }
> @@ -1432,6 +1568,9 @@ static int mlx5r_handle_mkey_cleanup(struct mlx5_ib_mr *mr)
>         if (mr->ibmr.frmr.pool && !ret)
>                 ib_frmr_pool_drop(&mr->ibmr);
>
> +       if (!ret)
> +               mlx5_ib_mr_put_st_handle_ref(mr);
> +
>         return ret;
>  }
>
> diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c
> index 1badec9bf527..7c1731aa3dd9 100644
> --- a/drivers/infiniband/hw/mlx5/odp.c
> +++ b/drivers/infiniband/hw/mlx5/odp.c
> @@ -857,6 +857,13 @@ static int pagefault_dmabuf_mr(struct mlx5_ib_mr *mr, size_t bcnt,
>                 return err;
>         }
>
> +       err = mlx5_ib_validate_dmabuf_tph(mr, umem_dmabuf->attach->dmabuf);
> +       if (err) {
> +               ib_umem_dmabuf_unmap_pages(umem_dmabuf);
> +               dma_resv_unlock(umem_dmabuf->attach->dmabuf->resv);
> +               return err;
> +       }
> +
>         page_size = mlx5_umem_dmabuf_find_best_pgsz(umem_dmabuf, access_mode);
>         if (!page_size) {
>                 ib_umem_dmabuf_unmap_pages(umem_dmabuf);
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/st.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/st.c
> index 7cedc348790d..c62bcae7c542 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/lib/st.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/st.c
> @@ -92,23 +92,18 @@ void mlx5_st_destroy(struct mlx5_core_dev *dev)
>         kfree(st);
>  }
>
> -int mlx5_st_alloc_index(struct mlx5_core_dev *dev, enum tph_mem_type mem_type,
> -                       unsigned int cpu_uid, u16 *st_index)
> +int mlx5_st_alloc_index_by_tag(struct mlx5_core_dev *dev, u16 tag,
> +                              u16 *st_index)
>  {
>         struct mlx5_st_idx_data *idx_data;
>         struct mlx5_st *st = dev->st;
>         unsigned long index;
>         u32 xa_id;
> -       u16 tag;
> -       int ret;
> +       int ret = 0;
>
>         if (!st)
>                 return -EOPNOTSUPP;
>
> -       ret = pcie_tph_get_cpu_st(dev->pdev, mem_type, cpu_uid, &tag);
> -       if (ret)
> -               return ret;
> -
>         if (st->direct_mode) {
>                 *st_index = tag;
>                 return 0;
> @@ -152,8 +147,49 @@ int mlx5_st_alloc_index(struct mlx5_core_dev *dev, enum tph_mem_type mem_type,
>         mutex_unlock(&st->lock);
>         return ret;
>  }
> +EXPORT_SYMBOL_GPL(mlx5_st_alloc_index_by_tag);
> +
> +int mlx5_st_alloc_index(struct mlx5_core_dev *dev, enum tph_mem_type mem_type,
> +                       unsigned int cpu_uid, u16 *st_index)
> +{
> +       u16 tag;
> +       int ret;
> +
> +       if (!dev->st)
> +               return -EOPNOTSUPP;
> +
> +       ret = pcie_tph_get_cpu_st(dev->pdev, mem_type, cpu_uid, &tag);
> +       if (ret)
> +               return ret;
> +
> +       return mlx5_st_alloc_index_by_tag(dev, tag, st_index);
> +}
>  EXPORT_SYMBOL_GPL(mlx5_st_alloc_index);
>
> +int mlx5_st_get_index(struct mlx5_core_dev *dev, u16 st_index)
> +{
> +       struct mlx5_st_idx_data *idx_data;
> +       struct mlx5_st *st = dev->st;
> +       int ret = 0;
> +
> +       if (!st)
> +               return -EOPNOTSUPP;
> +
> +       if (st->direct_mode)
> +               return 0;
> +
> +       mutex_lock(&st->lock);
> +       idx_data = xa_load(&st->idx_xa, st_index);
> +       if (WARN_ON_ONCE(!idx_data))
> +               ret = -EINVAL;
> +       else
> +               refcount_inc(&idx_data->usecount);
> +       mutex_unlock(&st->lock);
> +
> +       return ret;
> +}
> +EXPORT_SYMBOL_GPL(mlx5_st_get_index);
> +
>  int mlx5_st_dealloc_index(struct mlx5_core_dev *dev, u16 st_index)
>  {
>         struct mlx5_st_idx_data *idx_data;
> diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h
> index b1871c0821d0..d26e5744c843 100644
> --- a/include/linux/mlx5/driver.h
> +++ b/include/linux/mlx5/driver.h
> @@ -1182,16 +1182,31 @@ int mlx5_dm_sw_icm_dealloc(struct mlx5_core_dev *dev, enum mlx5_sw_icm_type type
>                            u64 length, u16 uid, phys_addr_t addr, u32 obj_id);
>
>  #ifdef CONFIG_PCIE_TPH
> +int mlx5_st_alloc_index_by_tag(struct mlx5_core_dev *dev, u16 tag,
> +                              u16 *st_index);
> +int mlx5_st_get_index(struct mlx5_core_dev *dev, u16 st_index);
>  int mlx5_st_alloc_index(struct mlx5_core_dev *dev, enum tph_mem_type mem_type,
>                         unsigned int cpu_uid, u16 *st_index);
>  int mlx5_st_dealloc_index(struct mlx5_core_dev *dev, u16 st_index);
>  #else
> +static inline int mlx5_st_alloc_index_by_tag(struct mlx5_core_dev *dev,
> +                                            u16 tag, u16 *st_index)
> +{
> +       return -EOPNOTSUPP;
> +}
> +
> +static inline int mlx5_st_get_index(struct mlx5_core_dev *dev, u16 st_index)
> +{
> +       return -EOPNOTSUPP;
> +}
> +
>  static inline int mlx5_st_alloc_index(struct mlx5_core_dev *dev,
>                                       enum tph_mem_type mem_type,
>                                       unsigned int cpu_uid, u16 *st_index)
>  {
>         return -EOPNOTSUPP;
>  }
> +
>  static inline int mlx5_st_dealloc_index(struct mlx5_core_dev *dev, u16 st_index)
>  {
>         return -EOPNOTSUPP;
> --
> 2.53.0-Meta
>

Re: [PATCH v13 5/5] RDMA/mlx5: get tph for p2p access when registering dma-buf mr

[fengchengwen]

Hi Zhiping,

On 8/1/2026 5:15 AM, Zhiping Zhang wrote:
> Peer-to-peer DMA between a mlx5 NIC and a foreign PCIe endpoint
> (typically a GPU or a vfio-pci passthrough device) traverses the host
> PCIe fabric. The endpoint exporting the dma-buf knows which PCIe TLP
> Processing Hint (TPH) Steering Tag yields the best placement for the
> traffic it will sink: per-endpoint hint selection lets the root complex
> or switch direct DMA to a specific cache slice / NUMA node, cutting
> cross-socket snoop traffic and DRAM pressure under sustained p2p
> workloads.
> 
> Until now the mlx5 importer had no way to learn the exporter's chosen
> ST tag, so dma-buf MRs were registered without TPH and ran with the
> default (no-hint) routing. With dma_buf_get_pci_tph() in place this
> patch wires up mlx5_ib to query that metadata at MR registration time
> for p2p access and use it to program requester-side TPH on the outbound
> mkey. If the exporter has no metadata, fall back to the existing
> no-TPH path so behavior for non-TPH-aware exporters is unchanged.


While working on v21 of the VFIO PCIe TPH series, we got review
feedback that importers must add pci_p2pdma_distance() validation
before retrieving ST values from a dma-buf.

Your mlx5 patch follows the same importer pattern – it consumes
dma-buf TPH metadata. I think this patch should also include the
same distance check.

Thanks,
Chengwen

Re: [PATCH v13 5/5] RDMA/mlx5: get tph for p2p access when registering dma-buf mr

[Zhiping Zhang]

On Wed, Sep 23, 2026 at 6:13 PM fengchengwen <fengchengwen@huawei.com> wrote:
>
> >
> Hi Zhiping,
>
> On 8/1/2026 5:15 AM, Zhiping Zhang wrote:
> > Peer-to-peer DMA between a mlx5 NIC and a foreign PCIe endpoint
> > (typically a GPU or a vfio-pci passthrough device) traverses the host
> > PCIe fabric. The endpoint exporting the dma-buf knows which PCIe TLP
> > Processing Hint (TPH) Steering Tag yields the best placement for the
> > traffic it will sink: per-endpoint hint selection lets the root complex
> > or switch direct DMA to a specific cache slice / NUMA node, cutting
> > cross-socket snoop traffic and DRAM pressure under sustained p2p
> > workloads.
> >
> > Until now the mlx5 importer had no way to learn the exporter's chosen
> > ST tag, so dma-buf MRs were registered without TPH and ran with the
> > default (no-hint) routing. With dma_buf_get_pci_tph() in place this
> > patch wires up mlx5_ib to query that metadata at MR registration time
> > for p2p access and use it to program requester-side TPH on the outbound
> > mkey. If the exporter has no metadata, fall back to the existing
> > no-TPH path so behavior for non-TPH-aware exporters is unchanged.
>
>
> While working on v21 of the VFIO PCIe TPH series, we got review
> feedback that importers must add pci_p2pdma_distance() validation
> before retrieving ST values from a dma-buf.
>
> Your mlx5 patch follows the same importer pattern – it consumes
> dma-buf TPH metadata. I think this patch should also include the
> same distance check.
>
> Thanks,
> Chengwen
>

Hi Chengwen,

I don't think this belongs in the importer. Every in-tree dma-buf
caller of pci_p2pdma_distance() is the exporter, in its .attach,
clearing attach->peer2peer -- amdgpu_dma_buf.c, xe_dma_buf.c,
habanalabs memory.c. There is no importer-side caller.

mlx5 also can't be one: pci_p2pdma_distance() takes the exporter's
struct pci_dev, and an importer only has a struct dma_buf. Reaching it
would mean introducing PCI types into the dma-buf core, which is what
Christian's ack on patch 3 rules out.

The routing check is already enforced by the core before any p2p DMA
can happen: vfio_pci_dma_buf_map() goes through
dma_buf_phys_vec_to_sgt(), which switches on
pci_p2pdma_map_type(provider, attach->dev) and fails -EINVAL on
PCI_P2PDMA_MAP_NOT_SUPPORTED -- the same calc_map_type_and_dist()
computation pci_p2pdma_distance() performs. An ST that mlx5 programmed
into the mkey cannot reach the wire unless that mapping has succeeded.

Thanks,
Zhiping

Re: [PATCH v13 5/5] RDMA/mlx5: get tph for p2p access when registering dma-buf mr

[Jason Gunthorpe]

On Wed, Sep 23, 2026 at 11:24:56PM -0700, Zhiping Zhang wrote:
> I don't think this belongs in the importer. Every in-tree dma-buf
> caller of pci_p2pdma_distance() is the exporter, in its .attach,
> clearing attach->peer2peer -- amdgpu_dma_buf.c, xe_dma_buf.c,
> habanalabs memory.c. There is no importer-side caller.

Right, and they shouldn't be doing that, but it still has to be
checked that the st is going directly to the peer device not the host
bridge. Not using the distance, but by the PCI_P2PDMA_MAP_BUS_ADDR
indication.

I fear you will need some of Leon's series to make that happen.

So probably the proposed change to dmabuf ops is far too simple.

Jason

Re: [PATCH v13 5/5] RDMA/mlx5: get tph for p2p access when registering dma-buf mr

[Zhiping Zhang]

On Thu, Sep 24, 2026 at 4:31 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> >
> On Wed, Sep 23, 2026 at 11:24:56PM -0700, Zhiping Zhang wrote:
> > I don't think this belongs in the importer. Every in-tree dma-buf
> > caller of pci_p2pdma_distance() is the exporter, in its .attach,
> > clearing attach->peer2peer -- amdgpu_dma_buf.c, xe_dma_buf.c,
> > habanalabs memory.c. There is no importer-side caller.
>
> Right, and they shouldn't be doing that, but it still has to be
> checked that the st is going directly to the peer device not the host
> bridge. Not using the distance, but by the PCI_P2PDMA_MAP_BUS_ADDR
> indication.
>
> I fear you will need some of Leon's series to make that happen.
>
> So probably the proposed change to dmabuf ops is far too simple.
>
> Jason

Hi Jason,

Thanks for the comments.

Agreed -- the tag should not be handed out unless the routing is
PCI_P2PDMA_MAP_BUS_ADDR. That can be enforced conservatively on 7.3
with two changes. I can fold both into patch 4.
```
In drivers/pci/p2pdma.c, make pci_p2pdma_map_type() callable from
tristate VFIO_PCI_CORE:

    EXPORT_SYMBOL_GPL(pci_p2pdma_map_type);

and in drivers/vfio/pci/vfio_pci_dmabuf.c,
vfio_pci_dma_buf_get_pci_tph() gains:

    struct dma_buf_attachment *attach;

    if (list_empty(&dmabuf->attachments))
        return -EOPNOTSUPP;

    list_for_each_entry(attach, &dmabuf->attachments, node)
        if (pci_p2pdma_map_type(priv->provider, attach->dev) !=
            PCI_P2PDMA_MAP_BUS_ADDR)
                return -EOPNOTSUPP;
```
For the query to succeed, every current attachment must therefore be
BUS_ADDR, and an empty list fails closed. The walk is protected by
dmabuf->resv, which vfio_pci_dma_buf_get_pci_tph() already asserts.
During initial registration, mlx5 treats -EOPNOTSUPP as no TPH and
registers a plain MR.

On the current behaviour, for the record:
PCI_P2PDMA_MAP_THRU_HOST_BRIDGE is a successful mapping path, so an ST
can currently be returned and programmed for host-bridge-routed
traffic. The mapping remains valid because TPH is advisory, but the ST
is interpreted in the root complex's own ST namespace rather than the
endpoint's; consequently, the endpoint's steering hint would be lost.
Returning -EOPNOTSUPP is therefore the honest answer when the route is
not
BUS_ADDR.

Two properties are worth stating explicitly:
 - The metadata and its callback are per-dmabuf, while routing is per
attachment, so this deliberately uses a conservative all-or-nothing
gate across the current attachments. It may withhold TPH from a direct
importer when another attachment is not BUS_ADDR, but it cannot return
a tag while any current attachment has a non-direct route. An importer
attaching later does not change an existing importer's route, and
future queries reevaluate the current attachment list.
 - I kept this on 7.3 rather than rebasing v14 onto Leon's unmerged
P2PDMA/ATS routing series. Any route that 7.3 does not report as
BUS_ADDR therefore falls back to no TPH. This keeps the set
self-contained on merged code and avoids queuing it behind another
series.

I plan to send v14 with these changes. Please let me know if you see
any issue with the implementation above.

Thanks,
Zhiping

Re: [PATCH v13 5/5] RDMA/mlx5: get tph for p2p access when registering dma-buf mr

[Jason Gunthorpe]

On Thu, Sep 24, 2026 at 11:17:57PM -0700, Zhiping Zhang wrote:
> On Thu, Sep 24, 2026 at 4:31 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> >
> > >
> > On Wed, Sep 23, 2026 at 11:24:56PM -0700, Zhiping Zhang wrote:
> > > I don't think this belongs in the importer. Every in-tree dma-buf
> > > caller of pci_p2pdma_distance() is the exporter, in its .attach,
> > > clearing attach->peer2peer -- amdgpu_dma_buf.c, xe_dma_buf.c,
> > > habanalabs memory.c. There is no importer-side caller.
> >
> > Right, and they shouldn't be doing that, but it still has to be
> > checked that the st is going directly to the peer device not the host
> > bridge. Not using the distance, but by the PCI_P2PDMA_MAP_BUS_ADDR
> > indication.
> >
> > I fear you will need some of Leon's series to make that happen.
> >
> > So probably the proposed change to dmabuf ops is far too simple.
> >
> > Jason
> 
> Hi Jason,
> 
> Thanks for the comments.
> 
> Agreed -- the tag should not be handed out unless the routing is
> PCI_P2PDMA_MAP_BUS_ADDR. That can be enforced conservatively on 7.3
> with two changes. I can fold both into patch 4.
> ```
> In drivers/pci/p2pdma.c, make pci_p2pdma_map_type() callable from
> tristate VFIO_PCI_CORE:
> 
>     EXPORT_SYMBOL_GPL(pci_p2pdma_map_type);

No, that's been rejected several times already.
 
> and in drivers/vfio/pci/vfio_pci_dmabuf.c,
> vfio_pci_dma_buf_get_pci_tph() gains:
> 
>     struct dma_buf_attachment *attach;
> 
>     if (list_empty(&dmabuf->attachments))
>         return -EOPNOTSUPP;
> 
>     list_for_each_entry(attach, &dmabuf->attachments, node)
>         if (pci_p2pdma_map_type(priv->provider, attach->dev) !=
>             PCI_P2PDMA_MAP_BUS_ADDR)
>                 return -EOPNOTSUPP;
> ```

Also no, this has to be done via PCI_P2PDMA_MAP_BUS_ADDR which also
enforces putting the determination in the right place in the code
flow..

> Two properties are worth stating explicitly:

Ah! AI!

Jason

Re: [PATCH v13 5/5] RDMA/mlx5: get tph for p2p access when registering dma-buf mr

[Zhiping Zhang]

On Mon, Sep 28, 2026 at 10:30 AM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> >
> On Thu, Sep 24, 2026 at 11:17:57PM -0700, Zhiping Zhang wrote:
> > On Thu, Sep 24, 2026 at 4:31 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> > >
> > > >
> > > On Wed, Sep 23, 2026 at 11:24:56PM -0700, Zhiping Zhang wrote:
> > > > I don't think this belongs in the importer. Every in-tree dma-buf
> > > > caller of pci_p2pdma_distance() is the exporter, in its .attach,
> > > > clearing attach->peer2peer -- amdgpu_dma_buf.c, xe_dma_buf.c,
> > > > habanalabs memory.c. There is no importer-side caller.
> > >
> > > Right, and they shouldn't be doing that, but it still has to be
> > > checked that the st is going directly to the peer device not the host
> > > bridge. Not using the distance, but by the PCI_P2PDMA_MAP_BUS_ADDR
> > > indication.
> > >
> > > I fear you will need some of Leon's series to make that happen.
> > >
> > > So probably the proposed change to dmabuf ops is far too simple.
> > >
> > > Jason
> >
> > Hi Jason,
> >
> > Thanks for the comments.
> >
> > Agreed -- the tag should not be handed out unless the routing is
> > PCI_P2PDMA_MAP_BUS_ADDR. That can be enforced conservatively on 7.3
> > with two changes. I can fold both into patch 4.
> > ```
> > In drivers/pci/p2pdma.c, make pci_p2pdma_map_type() callable from
> > tristate VFIO_PCI_CORE:
> >
> >     EXPORT_SYMBOL_GPL(pci_p2pdma_map_type);
>
> No, that's been rejected several times already.
>
> > and in drivers/vfio/pci/vfio_pci_dmabuf.c,
> > vfio_pci_dma_buf_get_pci_tph() gains:
> >
> >     struct dma_buf_attachment *attach;
> >
> >     if (list_empty(&dmabuf->attachments))
> >         return -EOPNOTSUPP;
> >
> >     list_for_each_entry(attach, &dmabuf->attachments, node)
> >         if (pci_p2pdma_map_type(priv->provider, attach->dev) !=
> >             PCI_P2PDMA_MAP_BUS_ADDR)
> >                 return -EOPNOTSUPP;
> > ```
>
> Also no, this has to be done via PCI_P2PDMA_MAP_BUS_ADDR which also
> enforces putting the determination in the right place in the code
> flow..
>
> > Two properties are worth stating explicitly:
>
> Ah! AI!
>
> Jason

Thanks Jason, I see what you meant by Leon's series now:
  https://lore.kernel.org/all/20260928-fix-p2p-acs-v4-0-v8-0-404453b9c435@nvidia.com/

Looking at v8, it seems patches 1-4 can stay functionally unchanged,
while patch 5 uses dma_buf_p2pdma_map_type() on its own attachment for
both the initial TPH query and revalidation. Is that what you expect?

If so, would you prefer that I wait for Leon's series and rebase the
whole stack on it, or split the series so patches 1-4 can land first
and patch 5 follows after Leon's series?

Thanks,
Zhiping

Re: [PATCH v13 5/5] RDMA/mlx5: get tph for p2p access when registering dma-buf mr

[Leon Romanovsky]

On Mon, Sep 28, 2026 at 02:47:46PM -0700, Zhiping Zhang wrote:
> On Mon, Sep 28, 2026 at 10:30 AM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> >
> > >
> > On Thu, Sep 24, 2026 at 11:17:57PM -0700, Zhiping Zhang wrote:
> > > On Thu, Sep 24, 2026 at 4:31 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> > > >
> > > > >
> > > > On Wed, Sep 23, 2026 at 11:24:56PM -0700, Zhiping Zhang wrote:
> > > > > I don't think this belongs in the importer. Every in-tree dma-buf
> > > > > caller of pci_p2pdma_distance() is the exporter, in its .attach,
> > > > > clearing attach->peer2peer -- amdgpu_dma_buf.c, xe_dma_buf.c,
> > > > > habanalabs memory.c. There is no importer-side caller.
> > > >
> > > > Right, and they shouldn't be doing that, but it still has to be
> > > > checked that the st is going directly to the peer device not the host
> > > > bridge. Not using the distance, but by the PCI_P2PDMA_MAP_BUS_ADDR
> > > > indication.
> > > >
> > > > I fear you will need some of Leon's series to make that happen.
> > > >
> > > > So probably the proposed change to dmabuf ops is far too simple.
> > > >
> > > > Jason
> > >
> > > Hi Jason,
> > >
> > > Thanks for the comments.
> > >
> > > Agreed -- the tag should not be handed out unless the routing is
> > > PCI_P2PDMA_MAP_BUS_ADDR. That can be enforced conservatively on 7.3
> > > with two changes. I can fold both into patch 4.
> > > ```
> > > In drivers/pci/p2pdma.c, make pci_p2pdma_map_type() callable from
> > > tristate VFIO_PCI_CORE:
> > >
> > >     EXPORT_SYMBOL_GPL(pci_p2pdma_map_type);
> >
> > No, that's been rejected several times already.
> >
> > > and in drivers/vfio/pci/vfio_pci_dmabuf.c,
> > > vfio_pci_dma_buf_get_pci_tph() gains:
> > >
> > >     struct dma_buf_attachment *attach;
> > >
> > >     if (list_empty(&dmabuf->attachments))
> > >         return -EOPNOTSUPP;
> > >
> > >     list_for_each_entry(attach, &dmabuf->attachments, node)
> > >         if (pci_p2pdma_map_type(priv->provider, attach->dev) !=
> > >             PCI_P2PDMA_MAP_BUS_ADDR)
> > >                 return -EOPNOTSUPP;
> > > ```
> >
> > Also no, this has to be done via PCI_P2PDMA_MAP_BUS_ADDR which also
> > enforces putting the determination in the right place in the code
> > flow..
> >
> > > Two properties are worth stating explicitly:
> >
> > Ah! AI!
> >
> > Jason
> 
> Thanks Jason, I see what you meant by Leon's series now:
>   https://lore.kernel.org/all/20260928-fix-p2p-acs-v4-0-v8-0-404453b9c435@nvidia.com/
> 
> Looking at v8, it seems patches 1-4 can stay functionally unchanged,
> while patch 5 uses dma_buf_p2pdma_map_type() on its own attachment for
> both the initial TPH query and revalidation. Is that what you expect?
> 
> If so, would you prefer that I wait for Leon's series and rebase the
> whole stack on it, or split the series so patches 1-4 can land first
> and patch 5 follows after Leon's series?

I would like you to join me there and explain to Christian why the
importer needs access to the exporter's internals.

You are already second user for the same functionally.

Thanks

> 
> Thanks,
> Zhiping

Re: [PATCH v13 5/5] RDMA/mlx5: get tph for p2p access when registering dma-buf mr

[Zhiping Zhang]

Hi Leon,

Sure, I'm happy to explain the TPH side. I see you've set the dma-buf
patches aside for now, so I'll just put the requirement on record
rather than ask for a particular API.

On the mlx5 ODP path the mkey must exist before the first dma-buf
mapping, and its ST index and PH are fixed at creation. It's the same
ordering problem you have with ATS. At that point mlx5 needs the ST/PH
tuple and an answer to Jason's question, whether its traffic to the
buffer is routed directly (PCI_P2PDMA_MAP_BUS_ADDR) rather than
through the host bridge. What mlx5 needs there is the answer, not the
exporter's p2pdma_provider. Wherever the route gets computed is fine,
as long as the result comes back for mlx5's own attachment.

In my earlier reply to Jason I said a later attach doesn't change an
existing importer's route. Christian pointed out in the patch 16
thread that it can, because the exporter only commits the access path
when the mapping is created. I was wrong there. The mlx5 patch
re-queries the ST/PH under dmabuf->resv after each
ib_umem_dmabuf_map_pages() and fails the fault if it changed; with
vfio-pci the tuple only changes if the device owner sets a new one.
The route check belongs in the same two places, which I think is close
to what Thomas suggested.

Alistair, adding you for visibility since you're picking up the
mapping-type work after LPC. mlx5 TPH would be the second user of
whatever lands there. I'm happy to help: I can implement the TPH side
on top of it and test on a CX-8 and a peer endpoint behind a PCIe
switch, including analyzer captures of the TLPs, or I can take on a
piece of the work if that's useful.

Thanks,
Zhiping

On Wed, Sep 30, 2026 at 5:13 AM Leon Romanovsky <leon@kernel.org> wrote:
>
> >
> On Mon, Sep 28, 2026 at 02:47:46PM -0700, Zhiping Zhang wrote:
> > On Mon, Sep 28, 2026 at 10:30 AM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> > >
> > > >
> > > On Thu, Sep 24, 2026 at 11:17:57PM -0700, Zhiping Zhang wrote:
> > > > On Thu, Sep 24, 2026 at 4:31 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> > > > >
> > > > > >
> > > > > On Wed, Sep 23, 2026 at 11:24:56PM -0700, Zhiping Zhang wrote:
> > > > > > I don't think this belongs in the importer. Every in-tree dma-buf
> > > > > > caller of pci_p2pdma_distance() is the exporter, in its .attach,
> > > > > > clearing attach->peer2peer -- amdgpu_dma_buf.c, xe_dma_buf.c,
> > > > > > habanalabs memory.c. There is no importer-side caller.
> > > > >
> > > > > Right, and they shouldn't be doing that, but it still has to be
> > > > > checked that the st is going directly to the peer device not the host
> > > > > bridge. Not using the distance, but by the PCI_P2PDMA_MAP_BUS_ADDR
> > > > > indication.
> > > > >
> > > > > I fear you will need some of Leon's series to make that happen.
> > > > >
> > > > > So probably the proposed change to dmabuf ops is far too simple.
> > > > >
> > > > > Jason
> > > >
> > > > Hi Jason,
> > > >
> > > > Thanks for the comments.
> > > >
> > > > Agreed -- the tag should not be handed out unless the routing is
> > > > PCI_P2PDMA_MAP_BUS_ADDR. That can be enforced conservatively on 7.3
> > > > with two changes. I can fold both into patch 4.
> > > > ```
> > > > In drivers/pci/p2pdma.c, make pci_p2pdma_map_type() callable from
> > > > tristate VFIO_PCI_CORE:
> > > >
> > > >     EXPORT_SYMBOL_GPL(pci_p2pdma_map_type);
> > >
> > > No, that's been rejected several times already.
> > >
> > > > and in drivers/vfio/pci/vfio_pci_dmabuf.c,
> > > > vfio_pci_dma_buf_get_pci_tph() gains:
> > > >
> > > >     struct dma_buf_attachment *attach;
> > > >
> > > >     if (list_empty(&dmabuf->attachments))
> > > >         return -EOPNOTSUPP;
> > > >
> > > >     list_for_each_entry(attach, &dmabuf->attachments, node)
> > > >         if (pci_p2pdma_map_type(priv->provider, attach->dev) !=
> > > >             PCI_P2PDMA_MAP_BUS_ADDR)
> > > >                 return -EOPNOTSUPP;
> > > > ```
> > >
> > > Also no, this has to be done via PCI_P2PDMA_MAP_BUS_ADDR which also
> > > enforces putting the determination in the right place in the code
> > > flow..
> > >
> > > > Two properties are worth stating explicitly:
> > >
> > > Ah! AI!
> > >
> > > Jason
> >
> > Thanks Jason, I see what you meant by Leon's series now:
> >   https://lore.kernel.org/all/20260928-fix-p2p-acs-v4-0-v8-0-404453b9c435@nvidia.com/
> >
> > Looking at v8, it seems patches 1-4 can stay functionally unchanged,
> > while patch 5 uses dma_buf_p2pdma_map_type() on its own attachment for
> > both the initial TPH query and revalidation. Is that what you expect?
> >
> > If so, would you prefer that I wait for Leon's series and rebase the
> > whole stack on it, or split the series so patches 1-4 can land first
> > and patch 5 follows after Leon's series?
>
> I would like you to join me there and explain to Christian why the
> importer needs access to the exporter's internals.
>
> You are already second user for the same functionally.
>
> Thanks
>
> >
> > Thanks,
> > Zhiping