Re: [PATCH v2 0/3] PCI: endpoint: Support hardware-owned MSI-X table and PBA

[Niklas Cassel <cassel@kernel.org>]

Hello Koichiro,

On Mon, Aug 31, 2026 at 12:19:45AM +0900, Koichiro Den wrote:
> Hi,
> 
> Some DWC endpoint controllers keep the MSI-X Table and PBA at fixed
> locations in a reserved BAR. On RK3588, the MSI-X doorbell uses this
> hardware-owned layout, as discussed in e.g. [1].
> 
> The reserved-region types for the Table and PBA were added with the
> Tegra194 description. The MSI-X setup API, however, still takes only the
> Table BAR and offset and assumes that the PBA immediately follows it in
> the same BAR.
> 
> This series passes the complete layout to pci_epc_set_msix() and adds
> pci_epc_get_hw_msix_layout() to expose a hardware-owned layout. The core
> does not select it automatically. The choice stays with the EPF.
> 
> A hardware-owned layout is not a direct replacement for every caller.
> pci-epf-ntb, for example, reads host-programmed Table entries from its
> own BAR to set up peer outbound mappings.
> 
> The series also fixes an interrupt-type mismatch in vNTB. ntb_hw_epf can
> select MSI-X, while pci-epf-vntb currently configures and raises only
> MSI. On RK3588, selecting the fixed BAR4 layout also selects the DWC
> MSI-X doorbell. EPF-owned layouts continue to use the regular path.
> 
> In short:
> 
> - pci-epf-vntb gains MSI-X support and uses the hardware-owned layout
>   when available.
> - pci-epf-test, pci-epf-ntb, and the NVMe PCI EPF keep their EPF-owned
>   layouts. This avoids unnecessary changes and reduces regression risk.
>   They can use a hardware-owned layout later if/when needed.
> 
> [1] https://lore.kernel.org/r/aY2q80zeRKSRO21H@fedora


Perhaps you could improve the cover letter to more clearly state why you
are doing this change.

Some guesses:
- Better performance. We avoid the need to map + unmap the MSI target
  address using an iATU each time we raise an MSI-X. We also avoid the
  need to flush posted write before unmap.
  Is there any performance difference? If so, it would be nice with some
  numbers.
- Allows more concurrent I/Os. By not using an iATU when raising an MSI-X,
  we have one more iATU available, so we can have one more outstanding I/O.
- Less waste of BAR space. (Since the MSI-X table and PBA already
  always takes up space in one of the BARs, it is wasteful to have
  the EPF drive duplicate it in another BAR.)



Personally, I don't see why we should only change pci-epf-vntb to use
the hardware-owned layout when available. I don't see why we would not
want to change pci-epf-test, pci-epf-ntb, and nvmet-pci-epf as well.
(If the EPC defines a HW defined MSI-X table + PBA, why not always
use that? If there is no HW defined MSI-X table + PBA, let the EPF
put the MSI-X table in any BAR it likes.)


I understand that you introduce dw_pcie_ep_msix_layout_is_hw_owned()
because you want an EPF driver optionally use the HW defined table.
But if all EPF drivers always use the HW defined table if available, I think
you can avoid introducing this helper, and let rockchip_pcie_raise_irq()
unconditionally call dw_pcie_ep_raise_msix_irq_doorbell() for case PCI_IRQ_MSIX.

See e.g. drivers/pci/controller/dwc/pci-layerscape-ep.c which already calls
dw_pcie_ep_raise_msix_irq_doorbell() unconditionally for case PCI_IRQ_MSIX.


Kind regards,
Niklas