[PATCH V2] drivers: pci: dwc: configure multiple atu regions

[Shyam Saini <shyamsaini@linux.microsoft.com>]

Hi Serge,

Apologies for delayed response.

> > Hi Manivannan,
> > 
> > >> Before this change, the dwc PCIe driver configures only 1 ATU region,
> > >> which is sufficient for the devices with PCIe memory <= 4GB. However,
> > >> the driver probe fails when device uses more than 4GB of pcie memory.
> > >> 
> > > Something is not clear... This commit message implies that the driver used to
> > > work on your hardware (you haven't mentioned which one it is) and broken by the
> > > commit from Sergey.
> > 
> > sorry for any confusion, the driver use to work in v5.10 kernel, with v6.0 kernel it
> > fails to probe with following messages:
> > layerscape-pcie xx00000.pcie: Failed to set MEM range ...
> > layerscape-pcie: probe of xx00000.pcie failed with error -22
> > 
> > By tracing code, I found that the probe was failing on [1] this check,
> > which was added in [2] upstream commit from Serge and finally, the ATU upper bound limit was
> > set to 4G in [3] commit
> > 
> > pci driver probe succeeds either when
> >          1) I remove [1] check
> >          2) or by setting the ATU limit to the size of Mem64 range (my original patch [4])
> > 
> > > As per Sergey's commit, we auto detect the dw_pcie::region_limit. If the IP is <
> > > 4.60, then the limit is 4G, otherwise depends on CX_ATU_MAX_REGION_SIZE set in
> > > hw.
> > 
> 
> > I couldn't find any reference of CX_ATU_MAX_REGION_SIZE in my PCIe TRM, perhaps because it
> > is older than v4.60
> 
> Please find the line containing "iATU: unroll " in the boot log and

layerscape-pcie xx00000.pcie: iATU: unroll F, 256 ob, 24 ib, align 4K, limit 4G

> copy it here. Also please provide a content of the dw_pcie::version
> field _after_ the dw_pcie_version_detect() method is executed.
version is set as 0

i see that my driver always has [1] max = 0
While I am awaiting official info about CX_ATU_MAX_REGION_SIZE, but I think CX_ATU_MAX_REGION_SIZE may not exist

> > 
> > > So if your IP is < 4.60, you cannot map > 4GB of outbound memory anyway. But if
> > > it is > 4.60, you shouldn't see the failure that you reported for > 4G space
> > > (well you can see the failure if the limit is less than the region size). In the
> > > previous thread you mentioned that dw_pcie::region_limit is set to 4G. So how
> > > come your driver was working previously?
> > 
> 
> > The PCIe IP is from Synopsys and is older than 4.60,
> 
> If you know what the actual IP-core version is and it's older than
> 4.70a, then it's better to pre-define the version in the
> drivers/pci/controller/dwc/pci-layerscape.c probe procedure (like it's
> done in drivers/pci/controller/dwc/pci-keystone.c,
> drivers/pci/controller/dwc/pcie-bt1.c).

sure, I will look into that

> > the board is from Freescale LX2
> > family.
> 
> I failed to find the LX2 PCIe controller support in the
> drivers/pci/controller/dwc/pci-layerscape.c driver. AFAICS the
> drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c driver is responsible
> for working with the LX2 PCIe host. Am I missing something?

driver is drivers/pci/controller/dwc/pci-layerscape.c
device-tree compatible string is "fsl,ls2088a-pcie"

> Anyway I've checked all the DT-nodes defined for the Freescale Layerscape
> PCIe host controllers. None of them have the PCIe ranges defined with
> the window size greater than 4G (actually all of them of the 1G size).
> So the next question is: what DT source have you been using on your
> platform and what is the DT-node defined for the PCIe host controller
> in there?

It is internal and custom pcie range. It is not present in any upstream boards dts files.

> > The board uses drivers/pci/controller/dwc/pci-layerscape.c driver
> > Given PCIe IP is older than 4.60, I am not sure why it was working earlier for a memory range larger than 4G
> > Highly appreciate your guidance and help on this.
> 
> It has been working earlier because the kernel 5.10 didn't have
> PCI ranges check for not exceeding the iATU regions maximum limit.

true

> But once again, the DW PCIe Root-port driver has been designed to 
> allocate a _single_ iATU region for each PCIe memory ranges. Thus if the
> ranges exceeds the maximum limit, the mapping won't work for the
> addresses greater than the maximum limit. I've already explained it
> here:
> https://lore.kernel.org/linux-pci/hxluc6qth4temdyxloekbhoy4iielyvxmmhp3u47qwtcxb5t5v@v5hdzvqmrsyv
> 
> Moreover the IP-core databook not only explicitly prohibits to define the iATU
> ranges greater than 4GB on the DW PCIe device older than 4.60a (text
> from v4.21a databook), but has more strict constraint of the regions
> not crossing the 4GB boundary:
> "The upper 32 bits of the Target Address register always forms the
> upper 32 bits of the translated address because:
> ■ The maximum region size is 4 GB.
> ■ A region must not cross a 4 GB boundary."
> 
> (It's obvious though since the iATU Limit Address register is of the
> 32-bit size on the old controllers.)
> 
> So you either need to allocate several iATU regions to cover the
> requested PCIe ranges, or fix the PCIe controller DT-node to having the
> ranges not exceeding the maximum limit.

we have requirement for memory ranmge greater than 4G, so it seems range can't be changed.

I am looking into your other suggestions, highly appreciate your feedback

Thanks,
Shyam

[1] https://elixir.bootlin.com/linux/v6.10-rc5/source/drivers/pci/controller/dwc/pcie-designware.c#L832