Re: [PATCH] imx6: fix pcie enumeration

[Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>]

On Fri, Jan 05, 2018 at 02:39:57PM +0100, Koen Vandeputte wrote:
> 
> 
> On 2018-01-05 13:32, Lorenzo Pieralisi wrote:
> >
> >>	/* setup bus numbers */
> >>	val  =  dw_pcie_readl_dbi
> >>	val  &=  0xff000000;
> >>	val  |=  0x00010100; <---  hardcoded today
> >>	dw_pcie_writel_dbi
> 
> >>I *think* I understand what's going on - the kernel takes the primary,
> >>secondary and subordinate values in the host bridge as valid in:
> >>
> >>pci_scan_bridge_extend()
> >>
> >>and given that pcibios_assign_all_busses() returns false (guess) it sets-up
> >>the bus hierarchy with a bus resource with subordinate number as read from
> >>PCI host bridge config space - which, given that it is 1 according to your
> >>explanation - this triggers the issue you reported.
> >>
> >>After commit a20c7f36bd3d the root bus resource is propagated down the
> >>hierarchy, hence the problem.
> >>
> >>So, in order to fix the issue I think the best way is to programme the
> >>root bridge in:
> >>
> >>drivers/pci/dwc/pci-designware-host.c
> >>
> >>but with the value coming from the root bus IORESOURCE_BUS resource,
> >>not hardcoding 0xff.
> >>
> >>I would kindly ask you to send logs with debug turned on in:
> >>
> >>drivers/pci/probe.c
> >>
> >>since I would like to check my understanding is correct.
> >>
> >>Please CC all dwc host maintainers since this has potential widespread
> >>impact.
> >>
> >>Thanks,
> >>Lorenzo
> Hi Lorenzo,
> 
> This is exactly what I'm trying to explain:
> 
> The host starts of with a (hardcoded today) subord of 1. [bits 16:23]
> 
> Since commit a20c7f36bd3d, downstream devices cannot assign bus nr's
> higher than the subord of the upstream device.
> So in this case, scanning stops after the bridge as soon as bus 1 is
> assigned .. :)

There is one thing that I need to understand though. Before the commit
above, how would enumeration works given that the subordinate bus number
was set to 1 and that the kernel, AFAICS, does not overwrite it ?

Are you able to send me a log (enumeration with debugging enabled and
lspci) with the commit above reverted please ?

Thanks,
Lorenzo

> As other targets besides i.MX6 (layerscape, armada8k, ...) also use
> the same function to init PCIe, I believe those targets are also
> affected.
> 
> I've tested here setting the PCI_PRIMARY_BUS register to 0x 00 ff 01
> 00  (ignored-subord-secbus-primbus), and the whole scanning works
> again.
> I fully agree that hardcoding is not the final fix, as this param
> can be defined in a DT.
> 
> 
> Fixing this, combined with the upstream commit exposing the error,
> fixes all following pci boot errors:
> 
> ..
> [    0.466405] pci_bus 0000:05: [bus 05] partially hidden behind
> bridge 0000:01 [bus 01]
> ..
> [    0.466435] pci_bus 0000:02: busn_res: can not insert [bus 02-05]
> under [bus 01] (conflicts with (null) [bus 01])
> [    0.466454] pci_bus 0000:02: [bus 02-05] partially hidden behind
> bridge 0000:01 [bus 01]
> ..
> 
> 
> Watching the tree using lspci also shows that all primaries,
> secondaries and subords are perfectly logical as expected.
> 
> 
> Thanks,
> 
> Koen
> 
> 
> Log showing the initial issue without any fixup:
> 
> 
> [    0.116673] OF: PCI: host bridge /soc/pcie@0x01000000 ranges:
> [    0.116692] OF: PCI:   No bus range found for
> /soc/pcie@0x01000000, using [bus 00-ff]
> [    0.116719] OF: PCI:    IO 0x01f80000..0x01f8ffff -> 0x00000000
> [    0.116739] OF: PCI:   MEM 0x01000000..0x01efffff -> 0x01000000
> [    0.337752] imx6q-pcie 1ffc000.pcie: link up
> [    0.337771] imx6q-pcie 1ffc000.pcie: Link: Gen2 disabled
> [    0.337785] imx6q-pcie 1ffc000.pcie: link up
> [    0.337796] imx6q-pcie 1ffc000.pcie: Link up, Gen1
> [    0.338039] imx6q-pcie 1ffc000.pcie: PCI host bridge to bus 0000:00
> [    0.338055] pci_bus 0000:00: root bus resource [bus 00-ff]
> [    0.338069] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
> [    0.338082] pci_bus 0000:00: root bus resource [mem
> 0x01000000-0x01efffff]
> [    0.338094] pci_bus 0000:00: scanning bus
> [    0.338127] pci 0000:00:00.0: [16c3:abcd] type 01 class 0x060400
> [    0.338151] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
> [    0.338168] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
> [    0.338204] pci 0000:00:00.0: calling pci_fixup_ide_bases+0x0/0x4c
> [    0.338259] pci 0000:00:00.0: supports D1
> [    0.338267] pci 0000:00:00.0: PME# supported from D0 D1 D3hot D3cold
> [    0.338276] pci 0000:00:00.0: PME# disabled
> [    0.338512] pci_bus 0000:00: fixups for bus
> [    0.338525] PCI: bus0: Fast back to back transfers disabled
> [    0.338541] pci 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 0
> [    0.338673] pci_bus 0000:01: scanning bus
> [    0.338773] pci 0000:01:00.0: [10b5:8604] type 01 class 0x060400
> [    0.338816] pci 0000:01:00.0: calling ventana_pciesw_early_fixup+0x0/0xa4
> [    0.467817] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0001ffff]
> [    0.467999] pci 0000:01:00.0: calling pci_fixup_ide_bases+0x0/0x4c
> [    0.468467] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
> [    0.468491] pci 0000:01:00.0: PME# disabled
> [    0.468795] pci_bus 0000:01: fixups for bus
> [    0.468854] PCI: bus1: Fast back to back transfers disabled
> [    0.468877] pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 0
> [    0.468886] pci 0000:01:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
> [    0.468939] pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 1
> [    0.469265] pci_bus 0000:02: busn_res: can not insert [bus 02-01]
> under [bus 01] (conflicts with (null) [bus 01])
> [    0.469282] pci_bus 0000:02: scanning bus
> [    0.469554] pci_bus 0000:02: fixups for bus
> [    0.469559] PCI: bus2: Fast back to back transfers enabled
> [    0.469572] pci_bus 0000:02: bus scan returning with max=02
> [    0.469582] pci_bus 0000:02: busn_res: [bus 02-01] end is updated to 02
> [    0.469593] pci_bus 0000:02: busn_res: can not insert [bus 02]
> under [bus 01] (conflicts with (null) [bus 01])
> [    0.469615] pci_bus 0000:02: [bus 02] partially hidden behind
> bridge 0000:01 [bus 01]
> [    0.469636] pci_bus 0000:01: bus scan returning with max=02
> [    0.469643] pci 0000:00:00.0: bridge has subordinate 01 but max busn 02
> [    0.469661] pci 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 1
> [    0.469671] pci_bus 0000:00: bus scan returning with max=01
> [    0.469791] pci 0000:00:00.0: fixup irq: got 298
> [    0.469800] pci 0000:00:00.0: assigning IRQ 298
> [    0.469849] pci 0000:01:00.0: fixup irq: got 298
> [    0.469856] pci 0000:01:00.0: assigning IRQ 298
> [    0.469946] pci 0000:00:00.0: BAR 0: assigned [mem 0x01000000-0x010fffff]
> [    0.469965] pci 0000:00:00.0: BAR 8: assigned [mem 0x01100000-0x011fffff]
> [    0.469980] pci 0000:00:00.0: BAR 6: assigned [mem
> 0x01200000-0x0120ffff pref]
> [    0.469997] pci 0000:01:00.0: BAR 0: assigned [mem 0x01100000-0x0111ffff]
> [    0.470026] pci 0000:01:00.0: PCI bridge to [bus 02]
> [    0.470108] pci 0000:00:00.0: PCI bridge to [bus 01]
> [    0.470121] pci 0000:00:00.0:   bridge window [mem 0x01100000-0x011fffff]
> [    0.470381] pcieport 0000:00:00.0: Signaling PME through PCIe PME
> interrupt
> [    0.470397] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
> [    0.470412] pcie_pme 0000:00:00.0:pcie001: service driver pcie_pme loaded
> [    0.470660] pcieport 0000:01:00.0: enabling device (0140 -> 0142)
> [    0.470788] pcieport 0000:01:00.0: enabling bus mastering
> 
> 
> 
> [ Node 4 | node-4 ] lspci -tv
> -[0000:00]---00.0-[01]----00.0-[02]--
> [ Node 4 | node-4 ]
> 
> 
> 
> [ Node 4 | node-4 ] lspci -v
> 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> [Normal decode])
>     Flags: bus master, fast devsel, latency 0, IRQ 298
>     Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
>     Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
>     I/O behind bridge: None
>     Memory behind bridge: 01100000-011fffff [size=1M]
>     Prefetchable memory behind bridge: None
>     [virtual] Expansion ROM at 01200000 [disabled] [size=64K]
>     Capabilities: [40] Power Management version 3
>     Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
>     Capabilities: [70] Express Root Port (Slot-), MSI 00
>     Capabilities: [100] Advanced Error Reporting
>     Capabilities: [140] Virtual Channel
>     Kernel driver in use: pcieport
> lspci: Unable to load libkmod resources: error -12
> 
> 01:00.0 PCI bridge: PLX Technology, Inc. PEX 8604 4-lane, 4-Port PCI
> Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal
> decode])
>     Flags: bus master, fast devsel, latency 0, IRQ 298
>     Memory at 01100000 (32-bit, non-prefetchable) [size=128K]
>     Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
>     I/O behind bridge: None
>     Memory behind bridge: None
>     Prefetchable memory behind bridge: None
>     Capabilities: [40] Power Management version 3
>     Capabilities: [48] MSI: Enable- Count=1/4 Maskable+ 64bit+
>     Capabilities: [68] Express Upstream Port, MSI 00
>     Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8604
> 4-lane, 4-Port PCI Express Gen 2 (5.0 GT/s) Switch
>     Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00
>     Capabilities: [fb4] Advanced Error Reporting
>     Capabilities: [138] Power Budgeting <?>
>     Capabilities: [148] Virtual Channel
>     Capabilities: [448] Vendor Specific Information: ID=0000 Rev=0
> Len=0cc <?>
>     Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0
> Len=010 <?>
>     Kernel driver in use: pcieport
> [ Node 4 | node-4 ]