Re: [PATCH 0/9] PCI-EP: Add 'ranges' support for PCI endpoint devices

[Frank Li <Frank.li@nxp.com>]

On Sat, Sep 21, 2024 at 09:43:17AM -0500, Rob Herring wrote:
> On Thu, Sep 19, 2024 at 5:03 PM Frank Li <Frank.Li@nxp.com> wrote:
> >
> > The PCI bus device tree supports 'ranges' properties that indicate
> > how to convert PCI addresses to CPU addresses. Many PCI controllers
> > are dual-role controllers, supporting both Root Complex (RC) and
> > Endpoint (EP) modes. The EP side also needs similar information for
> > proper address translation.
> >
> > This commit introduces several changes to add 'ranges' support for
> > PCI endpoint devices:
> >
> > 1. **Modify of_address.c**: Add support for the new `device_type`
> >    "pci-ep", enabling it to parse 'ranges' using the same functions
> >    as for PCI devices.
> >
> > 2. **Update DesignWare PCIe EP driver**: Enhance the driver to
> >    support 'ranges' when 'addr_space' is missing, maintaining
> >    compatibility with existing drivers.
> >
> > 3. **Update binding documentation**: Modify the device tree bindings
> >    to include 'ranges' support and make 'addr_space' an optional
> >    entry in 'reg-names'.
> >
> > 4. **Add i.MX8QXP EP support**: Incorporate support for the
> >    i.MX8QXP PCIe EP in the driver.
> >
> > i.MX8QXP PCIe dts is upstreaming.  Below is pcie-ep part.
> >
> > pcieb_ep: pcie-ep@5f010000 {
> >                 compatible = "fsl,imx8q-pcie-ep";
> >                 reg = <0x5f010000 0x00010000>;
> >                 reg-names = "dbi";
> >                 #address-cells = <3>;
> >                 #size-cells = <2>;
> >                 device_type = "pci-ep";
> >                 ranges = <0x82000000 0 0x80000000 0x70000000 0 0x10000000>;
>
> How does a PCI endpoint set PCI addresses? Those get assigned by the
> PCI host system. They can't be static in DT.

PCI address is set by other channel, such as RC write some place in bar0.

It indicates EP side outbound windows mapping. See below detail.


                                  Endpoint          Root complex
                                 ┌───────┐        ┌─────────┐
                   ┌─────┐       │ EP    │        │         │      ┌─────┐
                   │     │       │ Ctrl  │        │         │      │ CPU │
                   │ DDR │       │       │        │ ┌────┐  │      └──┬──┘
                   │     │◄──────┼─ATU ◄─┼────────┼─┤BarN│◄─┼─────────┘
                   │     │       │       │        │ └────┘  │ Outbound Transfer
                   └─────┘       │       │        │         │
                                 │       │        │         │
                                 │       │        │         │
                                 │       │        │         │ Inbound Transfer
                                 │       │        │         │      ┌──▼──┐
                  ┌───────┐      │       │        │ ┌───────┼─────►│DDR  │
                  │       │ outbound Transfer*    │ │       │      └─────┘
       ┌─────┐    │ Bus   ┼─────►│ ATU  ─┬────────┼─┘       │
       │     │    │ Fabric│Bus   │       │ PCI Addr         │
       │ CPU ├───►│       │Addr  │       │ 0xA000_0000      │
       │     │CPU │       │0x8000_0000   │        │         │
       └─────┘Addr└───────┘      │       │        │         │
              0x7000_0000        └───────┘        └─────────┘


This ranges descript above diagram Endpoint outbound Transfer*'s
information. There are address space (previous use addr_space in reg-name)
indicate such informaiton, such as [0x7000_00000, 0xB000_0000] as PCI EP
outbound windows. when cpu write 0x7000_0000, data will write to EP ctrl,
the ATU in EP ctrl convert to PCI address such 0xA000,0000, then write
to RC's DDR>

The PCI Addr 0xA000_0000 information was sent to EP driver by use other
channel, such as RC write it some place in Bar0.

The 'range' here indicated EP side's outbound windows information. Most
system CPU address is the same as bus address. but in imx8q, it is
difference. Bus fabric convert 0x7000_0000 to 0x8000_00000.

So need range indicate such address convertion.

>
> If you need the PCI address, just read your BAR registers.
>
> In general, why do you need this when none of the other PCI endpoint
> drivers have needed this?

Most system, the address is the same. Some system convert is simple, just
cut some high address bit, so their driver hardcode it. Maybe imx8QM have
first one, they have more than one controller and address map is not
such simple.

We use customer dt property in downstream kernel, but I think common
solution should be better, other drivers can remove their hardcode in
future. And it will be more symmetry with PCI host side's property.

Frank
>
> Rob