Re: [PATCH V4 00/12] PCIe TPH and cache direct injection support

Wei Huang
2024-09-03T22:42:20Z

Hi Bjorn,

I have incorporated Jakub's feedback on patch 11 (i.e. bnxt.c) in my new
revision. Do you have additional comments or suggestions on V4 to
include in the next spin?

Thanks,
-Wei

On 8/22/24 3:41 PM, Wei Huang wrote:
> Hi All,
>
> TPH (TLP Processing Hints) is a PCIe feature that allows endpoint
> devices to provide optimization hints for requests that target memory
> space. These hints, in a format called steering tag (ST), are provided
> in the requester's TLP headers and allow the system hardware, including
> the Root Complex, to optimize the utilization of platform resources
> for the requests.
>
> Upcoming AMD hardware implement a new Cache Injection feature that
> leverages TPH. Cache Injection allows PCIe endpoints to inject I/O
> Coherent DMA writes directly into an L2 within the CCX (core complex)
> closest to the CPU core that will consume it. This technology is aimed
> at applications requiring high performance and low latency, such as
> networking and storage applications.
>
> This series introduces generic TPH support in Linux, allowing STs to be
> retrieved and used by PCIe endpoint drivers as needed. As a
> demonstration, it includes an example usage in the Broadcom BNXT driver.
> When running on Broadcom NICs with the appropriate firmware, it shows
> substantial memory bandwidth savings and better network bandwidth using
> real-world benchmarks. This solution is vendor-neutral and implemented
> based on industry standards (PCIe Spec and PCI FW Spec).
>
> V3->V4:
> * Rebase on top of the latest pci/next tree (tag: 6.11-rc1)
> * Add new API functioins to query/enable/disable TPH support
> * Make pcie_tph_set_st() completely independent from pcie_tph_get_cpu_st()
> * Rewrite bnxt.c based on new APIs
> * Remove documentation for now due to constantly changing API
> * Remove pci=notph, but keep pci=nostmode with better flow (Bjorn)
> * Lots of code rewrite in tph.c & pci-tph.h with cleaner interface (Bjorn)
> * Add TPH save/restore support (Paul Luse and Lukas Wunner)
>
> V2->V3:
> * Rebase on top of pci/next tree (tag: pci-v6.11-changes)
> * Redefine PCI TPH registers (pci_regs.h) without breaking uapi
> * Fix commit subjects/messages for kernel options (Jonathan and Bjorn)
> * Break API functions into three individual patches for easy review
> * Rewrite lots of code in tph.c/tph.h based (Jonathan and Bjorn)
>
> V1->V2:
> * Rebase on top of pci.git/for-linus (6.10-rc1)
> * Address mismatched data types reported by Sparse (Sparse check passed)
> * Add pcie_tph_intr_vec_supported() for checking IRQ mode support
> * Skip bnxt affinity notifier registration if
> pcie_tph_intr_vec_supported()=false
> * Minor fixes in bnxt driver (i.e. warning messages)
>
> Manoj Panicker (1):
> bnxt_en: Add TPH support in BNXT driver
>
> Michael Chan (1):
> bnxt_en: Pass NQ ID to the FW when allocating RX/RX AGG rings
>
> Paul Luse (1):
> PCI/TPH: Add save/restore support for TPH
>
> Wei Huang (9):
> PCI: Introduce PCIe TPH support framework
> PCI: Add TPH related register definition
> PCI/TPH: Add pcie_tph_modes() to query TPH modes
> PCI/TPH: Add pcie_enable_tph() to enable TPH
> PCI/TPH: Add pcie_disable_tph() to disable TPH
> PCI/TPH: Add pcie_tph_enabled() to check TPH state
> PCI/TPH: Add pcie_tph_set_st_entry() to set ST tag
> PCI/TPH: Add pcie_tph_get_cpu_st() to get ST tag
> PCI/TPH: Add pci=nostmode to force TPH No ST Mode
>
> .../admin-guide/kernel-parameters.txt | 3 +
> drivers/net/ethernet/broadcom/bnxt/bnxt.c | 86 ++-
> drivers/net/ethernet/broadcom/bnxt/bnxt.h | 4 +
> drivers/pci/pci.c | 4 +
> drivers/pci/pci.h | 12 +
> drivers/pci/pcie/Kconfig | 11 +
> drivers/pci/pcie/Makefile | 1 +
> drivers/pci/pcie/tph.c | 563 ++++++++++++++++++
> drivers/pci/probe.c | 1 +
> include/linux/pci-tph.h | 48 ++
> include/linux/pci.h | 7 +
> include/uapi/linux/pci_regs.h | 38 +-
> 12 files changed, 768 insertions(+), 10 deletions(-)
> create mode 100644 drivers/pci/pcie/tph.c
> create mode 100644 include/linux/pci-tph.h
>