Re: [PATCH 4/5] arm64: dts: imx95: add PCIe's msi-map and iommu-map property

[Frank Li <Frank.li@nxp.com>]

On Fri, Feb 28, 2025 at 12:11:47PM -0500, Frank Li wrote:
> On Fri, Feb 28, 2025 at 05:01:15PM +0100, Alexander Stein wrote:
> > Hi Frank,
> >
> > Am Freitag, 28. Februar 2025, 16:32:50 CET schrieb Frank Li:
> > > On Fri, Feb 28, 2025 at 10:08:58AM +0100, Alexander Stein wrote:
> > > > Hi Frank,
> > > >
> > > > Am Donnerstag, 27. Februar 2025, 17:39:47 CET schrieb Frank Li:
> > > > > On Thu, Feb 27, 2025 at 08:54:13AM +0100, Alexander Stein wrote:
> > > > > > Hi Frank,
> > > > > >
> > > > > > Am Mittwoch, 26. Februar 2025, 17:31:26 CET schrieb Frank Li:
> > > > > > > On Wed, Feb 26, 2025 at 01:11:37PM +0100, Alexander Stein wrote:
> > > > > > > > Hi Frank,
> > > > > > > >
> > > > > > > > Am Dienstag, 28. Januar 2025, 22:15:58 CET schrieb Frank Li:
> > > > > > > > > Add PCIe's msi-map and iommu-map property because i.MX95 support smmu and
> > > > > > > > > its.
> > > > > > > > >
> > > > > > > > > Signed-off-by: Frank Li <Frank.Li@nxp.com>
> > > > > > > > > ---
> > > > > > > > >  arch/arm64/boot/dts/freescale/imx95.dtsi | 14 ++++++++++++++
> > > > > > > > >  1 file changed, 14 insertions(+)
> > > > > > > > >
> > > > > > > > > diff --git a/arch/arm64/boot/dts/freescale/imx95.dtsi b/arch/arm64/boot/dts/freescale/imx95.dtsi
> > > > > > > > > index 6b8470cb3461a..2cebeda43a52d 100644
> > > > > > > > > --- a/arch/arm64/boot/dts/freescale/imx95.dtsi
> > > > > > > > > +++ b/arch/arm64/boot/dts/freescale/imx95.dtsi
> > > > > > > > > @@ -1573,6 +1573,12 @@ pcie0: pcie@4c300000 {
> > > > > > > > >  			assigned-clock-parents = <0>, <0>,
> > > > > > > > >  						 <&scmi_clk IMX95_CLK_SYSPLL1_PFD1_DIV2>;
> > > > > > > > >  			power-domains = <&scmi_devpd IMX95_PD_HSIO_TOP>;
> > > > > > > > > +			/* pcie0's Devid(BIT[7:6]) is 0x00, stream id(BIT[5:0]) is 0x10~0x17 */
> > > > > > > > > +			msi-map = <0x0 &its 0x10 0x1>,
> > > > > > > > > +				  <0x100 &its 0x11 0x7>;
> > > > > > > >
> > > > > > > > Aren't you missing msi-map-mask = <0x1ff>; here? Similar to pcie1.
> > > > > > > > Either way, with this change PCIe on pcie0 is not working anymore,
> > > > > > > > regardless of msi-map-mask.
> > > > > > >
> > > > > > > Yes, it should have msi-map-mask. During my test, I have not enable enetc
> > > > > > > so I have not found this problem.
> > > > > >
> > > > > > Just to be clear: This is not about enetc. This works fine here.
> > > > > >
> > > > > > > > Without msi-map-mask:
> > > > > > > > > OF: /soc/pcie@4c300000: iommu-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011
> > > > > > > > > OF: /soc/pcie@4c300000: no msi-map translation for id 0x300 on (null)
> > > > > > > > > r8169 0000:03:00.0: error -EINVAL: enable failure
> > > > > > > > > r8169 0000:03:00.0: probe with driver r8169 failed with error -22
> > > > > > > >
> > > > > > > > With msi-map-mask:
> > > > > > > > > OF: /soc/pcie@4c300000: iommu-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011
> > > > > > > > > OF: /soc/pcie@4c300000: msi-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011
> > > > > > > > > r8169 0000:03:00.0: enabling device (0000 -> 0003)
> > > > > > > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval
> > > > > > > > > r8169 0000:03:00.0: error -EIO: PCI read failed
> > > > > > > > > r8169 0000:03:00.0: probe with driver r8169 failed with error -5
> > > > > > >
> > > > > > > Can you try remove iommu-map and keep msi-map? then remove msi-map and
> > > > > > > keep iommu-map to check which one cause this problem.
> > > > > >
> > > > > > With only msi-map removed, but smmu enabled:
> > > > > > > arm-smmu-v3 490d0000.iommu: event 0x10 received:
> > > > > > > arm-smmu-v3 490d0000.iommu:      0x0000001100000010
> > > > > > > arm-smmu-v3 490d0000.iommu:      0x0000020a00000000
> > > > > > > arm-smmu-v3 490d0000.iommu:      0x000000009b0cc000
> > > > > > > arm-smmu-v3 490d0000.iommu:      0x0000000000000000
> > > > > > > arm-smmu-v3 490d0000.iommu: event: F_TRANSLATION client: 0000:01:00.0 sid: 0x11 ssid: 0x0 iova: 0x9b0cc000 ipa: 0x0
> > > > > > > arm-smmu-v3 490d0000.iommu: priv data read s1 "Input address caused fault" stag: 0x0 r8169 0000:03:00.0
> > > > > > > enp3s0: Link is Down
> > > > > >
> > > > > > With only iommu-map removed, both smmu enabled or disabled:
> > > > > > > OF: /soc/pcie@4c300000: msi-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011
> > > > > > > r8169 0000:03:00.0: enabling device (0000 -> 0003)
> > > > > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval
> > > > > > > r8169 0000:03:00.0: error -EIO: PCI read failed
> > > > > > > r8169 0000:03:00.0: probe with driver r8169 failed with error -5
> > > > > >
> > > > > > Only if smmu is disabled and msi-map is removed the driver probes
> > > > > > successfully:
> > > > > > > r8169 0000:03:00.0: enabling device (0000 -> 0003)
> > > > > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval
> > > > > > > r8169 0000:03:00.0 eth0: RTL8168g/8111g, d8:9d:b9:00:16:10, XID 4c0, IRQ 160
> > > > > > > r8169 0000:03:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko]
> > > > > > > r8169 0000:03:00.0 enp3s0: renamed from eth0
> > > > > > > r8169 0000:03:00.0: enabling bus mastering
> > > > > > > r8169 0000:03:00.0 enp3s0: Link is Down
> > > > > >
> > > > > > > >
> > > > > > > > Without msi-map/iommu-map:
> > > > > > > > > r8169 0000:03:00.0: enabling device (0000 -> 0003)
> > > > > > > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval
> > > > > > > > > r8169 0000:03:00.0 eth0: RTL8168g/8111g, d8:9d:b9:00:16:10, XID 4c0, IRQ 166
> > > > > > > > > r8169 0000:03:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko]
> > > > > > > > > r8169 0000:03:00.0 enp3s0: renamed from eth0
> > > > > > > > > r8169 0000:03:00.0: enabling bus mastering
> > > > > > > > > r8169 0000:03:00.0 enp3s0: Link is Down
> > > > > > > >
> > > > > > > > pcie1 works as expected. But this is only a single PCIe device, rather than
> > > > > > > > having a PCIe bridge.
> > > > > > > > Any idea what's wrong here?
> > > > > > >
> > > > > > > Can you help dump more information at for PCIe bridge case:
> > > > > > >
> > > > > > > imx_pcie_add_lut(), need rid and sid information.
> > > > > > > drivers/pci/controller/dwc/pci-imx6.c
> > > > > >
> > > > > > Just to be clear, without msi-map and iommu-map I get:
> > > > > > > imx6q-pcie 4c380000.pcie: rid: 0x0, sid: 0x18
> > > > > > > imx6q-pcie 4c380000.pcie: rid: 0x100, sid: 0x19
> > > > >
> > > > > Can you help dump register value PE0_LUT_CREQID offset 0x101 for your
> > > > > smmu-map or msi-map enable case
> > > >
> > > > I am assuming you meant offset 0x101c, as stated in the RM.
> > > > I added a dump directly before printing "PCI read failed" in r8169_main.c.
> > >
> > > Can you point me the code about where "error -EIO: PCI read failed"?
> >
> > The error messages comes from [1]. As far as I can tell, this is the first
> > read after pcim_iomap_table().
>
> Strange, it is CPU read MMIO, should not go through iommu at all.
>
> >
> > > I tested nvme devices worked at both PCIE0 and PCIE1.
> > >
> > > Which PCI switch do you use?
> >
> > This is a COTS MiniPCIe ethernet card [2]. If I'm reading correctly
> > the bridge seems to be a P17C9X20
>
> I order the same chipset one from amazon. Let me debug it after get it.
>

Finially we get realtek PCI card

it quite complex, there are one PCIe switch to split it to two pci bus.

 lspci -t
-[0000:00]---00.0-[01-ff]----00.0-[02-04]--+-03.0-[03]----00.0
                                           \-07.0-[04]----00.0


0000:00:00.0 PCI bridge: Philips Semiconductors Device 0000
0000:01:00.0 PCI bridge: ASMedia Technology Inc. ASM1182e 2-Port PCIe x1 Gen2 Packet Switch
0000:02:03.0 PCI bridge: ASMedia Technology Inc. ASM1182e 2-Port PCIe x1 Gen2 Packet Switch
0000:02:07.0 PCI bridge: ASMedia Technology Inc. ASM1182e 2-Port PCIe x1 Gen2 Packet Switch
0000:03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 09)
0000:04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 09)

It need below change

diff --git a/arch/arm64/boot/dts/freescale/imx95.dtsi b/arch/arm64/boot/dts/freescale/imx95.dtsi
index 9bb26b466a061..9dbf395b9a67b 100644
--- a/arch/arm64/boot/dts/freescale/imx95.dtsi
+++ b/arch/arm64/boot/dts/freescale/imx95.dtsi
@@ -1660,10 +1660,18 @@ pcie0: pcie@4c300000 {
                        power-domains = <&scmi_devpd IMX95_PD_HSIO_TOP>;
                        /* pcie0's Devid(BIT[7:6]) is 0x00, stream id(BIT[5:0]) is 0x10~0x17 */
                        msi-map = <0x0 &its 0x10 0x1>,
-                                 <0x100 &its 0x11 0x7>;
+                                 <0x100 &its 0x11 0x1>,
+                                 <0x218 &its 0x12 0x1>,
+                                 <0x238 &its 0x13 0x1>,
+                                 <0x300 &its 0x14 0x1>,
+                                 <0x400 &its 0x15 0x1>;
                        iommu-map = <0x000 &smmu 0x10 0x1>,
-                                   <0x100 &smmu 0x11 0x7>;
-                       iommu-map-mask = <0x1ff>;
+                                   <0x100 &smmu 0x11 0x1>,
+                                   <0x218 &smmu 0x12 0x1>,
+                                   <0x238 &smmu 0x13 0x1>,
+                                   <0x300 &smmu 0x14 0x1>,
+                                   <0x400 &smmu 0x15 0x1>;
+                       //iommu-map-mask = <0x1ff>;
                        fsl,max-link-speed = <3>;
                        status = "disabled";


Only 8 stream id assign to PCIe0 device, it is hard to dynamaic alloce one,
or need extra works

Frank

> Frank
>
> >
> > Thanks and best regards
> > Alexander
> >
> > [1] https://web.git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/net/ethernet/realtek/r8169_main.c?h=next-20250228#n5454
> > [2] https://www.delock.com/produkt/95237/merkmale.html?f=s
> > --
> > TQ-Systems GmbH | Mühlstraße 2, Gut Delling | 82229 Seefeld, Germany
> > Amtsgericht München, HRB 105018
> > Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider
> > http://www.tq-group.com/
> >
> >