Re: [PATCH v4 0/4] PCI: ep: dwc/imx6: Add bus address support for PCI endpoint devices

[Bjorn Helgaas <helgaas@kernel.org>]

On Thu, Oct 24, 2024 at 04:41:42PM -0400, Frank Li wrote:
> Endpoint          Root complex
>                              ┌───────┐        ┌─────────┐
>                ┌─────┐       │ EP    │        │         │      ┌─────┐
>                │     │       │ Ctrl  │        │         │      │ CPU │
>                │ DDR │       │       │        │ ┌────┐  │      └──┬──┘
>                │     │◄──────┼─ATU ◄─┼────────┼─┤BarN│◄─┼─────────┘
>                │     │       │       │        │ └────┘  │ Outbound Transfer
>                └─────┘       │       │        │         │
>                              │       │        │         │
>                              │       │        │         │
>                              │       │        │         │ Inbound Transfer
>                              │       │        │         │      ┌──▼──┐
>               ┌───────┐      │       │        │ ┌───────┼─────►│DDR  │
>               │       │ outbound Transfer*    │ │       │      └─────┘
>    ┌─────┐    │ Bus   ┼─────►│ ATU  ─┬────────┼─┘       │
>    │     │    │ Fabric│Bus   │       │ PCI Addr         │
>    │ CPU ├───►│       │Addr  │       │ 0xA000_0000      │
>    │     │CPU │       │0x8000_0000   │        │         │
>    └─────┘Addr└───────┘      │       │        │         │
>           0x7000_0000        └───────┘        └─────────┘
> 
> Add `bus_addr_base` to configure the outbound window address for CPU write.
> The BUS fabric generally passes the same address to the PCIe EP controller,
> but some BUS fabrics convert the address before sending it to the PCIe EP
> controller.
> 
> Above diagram, CPU write data to outbound windows address 0x7000_0000,
> Bus fabric convert it to 0x8000_0000. ATU should use BUS address
> 0x8000_0000 as input address and convert to PCI address 0xA000_0000.

The above doesn't match what's in patch 1/4, and I think the version
in 1/4 is better, so I'll comment there.

To avoid confusion, it might be better not to duplicate it in 0/4 and
1/4.

> Previously, `cpu_addr_fixup()` was used to handle address conversion. Now,
> the device tree provides this information, preferring a common method.
> 
> bus@5f000000 {
> 	compatible = "simple-bus";
> 	ranges = <0x80000000 0x0 0x70000000 0x10000000>;
> 
> 	pcie-ep@5f010000 {
> 		reg = <0x5f010000 0x00010000>,
> 		      <0x80000000 0x10000000>;
> 		reg-names = "dbi", "addr_space";
> 		...
> 	};
> 	...
> };
> 
> 'ranges' in bus@5f000000 descript how address convert from CPU address
> to BUS address.
> 
> Use `of_property_read_reg()` to obtain the BUS address and set it to the
> ATU correctly, eliminating the need for vendor-specific cpu_addr_fixup().
> 
> The 1st patch implement above method in dwc common driver.
> The 2nd patch update imx6's binding doc to add fsl,imx8q-pcie-ep.
> The 3rd patch fix a pci-mx6's a bug
> The 4th patch enable pci ep function.
> 
> The imx8q's dts is usptreaming, the pcie-ep part is below.
> 
> hsio_subsys: bus@5f000000 {
>         compatible = "simple-bus";
>         #address-cells = <1>;
>         #size-cells = <1>;
>         /* Only supports up to 32bits DMA, map all possible DDR as inbound ranges */
>         dma-ranges = <0x80000000 0 0x80000000 0x80000000>;
>         ranges = <0x5f000000 0x0 0x5f000000 0x01000000>,
>                  <0x80000000 0x0 0x70000000 0x10000000>;
> 
> 	pcieb_ep: pcie-ep@5f010000 {
>                 compatible = "fsl,imx8q-pcie-ep";
>                 reg = <0x5f010000 0x00010000>,
>                       <0x80000000 0x10000000>;
>                 reg-names = "dbi", "addr_space";
>                 num-lanes = <1>;
>                 interrupts = <GIC_SPI 104 IRQ_TYPE_LEVEL_HIGH>;
>                 interrupt-names = "dma";
>                 clocks = <&pcieb_lpcg IMX_LPCG_CLK_6>,
>                          <&pcieb_lpcg IMX_LPCG_CLK_4>,
>                          <&pcieb_lpcg IMX_LPCG_CLK_5>;
>                 clock-names = "dbi", "mstr", "slv";
>                 power-domains = <&pd IMX_SC_R_PCIE_B>;
>                 fsl,max-link-speed = <3>;
>                 num-ib-windows = <6>;
>                 num-ob-windows = <6>;
>         };
> };
> 
> Signed-off-by: Frank Li <Frank.Li@nxp.com>
> ---
> Changes in v4:
> - Fix 32bit build error
> | Reported-by: kernel test robot <lkp@intel.com>
> | Closes: https://lore.kernel.org/oe-kbuild-all/202410230328.BTHareG1-lkp@intel.com/
> - Link to v3: https://lore.kernel.org/r/20241021-pcie_ep_range-v3-0-b13526eb0089@nxp.com
> 
> Changes in v3:
> - Add mani' review tag for patch 3,4
> - Add varible using_dtbus_info to control use bus range information instead
> cpu_address_fixup().
> - Link to v2: https://lore.kernel.org/r/20240923-pcie_ep_range-v2-0-78d2ea434d9f@nxp.com
> 
> Changes in v2:
> - Totally rewrite with difference method. 'range' should in bus node
> instead pcie-ep node because address convert happen at bus fabric. Needn't
> add 'range' property at pci-ep node.
> - Link to v1: https://lore.kernel.org/r/20240919-pcie_ep_range-v1-0-b3e9d62780b7@nxp.com
> 
> ---
> Frank Li (4):
>       PCI: dwc: ep: Add bus_addr_base for outbound window
>       dt-bindings: PCI: fsl,imx6q-pcie-ep: Add compatible string fsl,imx8q-pcie-ep
>       PCI: imx6: Pass correct sub mode when calling phy_set_mode_ext()
>       PCI: imx6: Add i.MX8Q PCIe Endpoint (EP) support
> 
>  .../devicetree/bindings/pci/fsl,imx6q-pcie-ep.yaml | 38 +++++++++++++++++++++-
>  drivers/pci/controller/dwc/pci-imx6.c              | 26 ++++++++++++++-
>  drivers/pci/controller/dwc/pcie-designware-ep.c    | 14 +++++++-
>  drivers/pci/controller/dwc/pcie-designware.h       |  9 +++++
>  4 files changed, 84 insertions(+), 3 deletions(-)
> ---
> base-commit: afb15ca28055352101286046c1f9f01fdaa1ace1
> change-id: 20240918-pcie_ep_range-4c5c5e300e19
> 
> Best regards,
> ---
> Frank Li <Frank.Li@nxp.com>
>