* Fwd: PCIe root port programming [not found] ` <CAH=XD58J+Yf-Rdz_bAJ-e9rU4ipu9opieFd0BuHeYAdX0nmrzg@mail.gmail.com> @ 2016-08-12 21:14 ` Kallol Biswas 2016-08-15 17:24 ` Bjorn Helgaas 0 siblings, 1 reply; 8+ messages in thread From: Kallol Biswas @ 2016-08-12 21:14 UTC (permalink / raw) To: linux-pci, helgaas I am trying to debug an issue that the PCIe memory transactions are not routed through a host bridge. The config transactions work fine. The primary, secondary and subordinate bus numbers look fine for the PCIe tree under the host bridge. The memory windows are programmed fine. Can someone point me to the code that programs the chipset/root port registers those direct processor transactions to a specific host bridge? Cat of /proc/mem is copied below. /root/physmem -p c8100000 -s 4 -r Operation:read to Addr: 0xc8100000 Value: 0 ffffffff ffffffff ffffffff ffffffff The PCIe analyzer sitting between host bridge (0:3.0 and 02:00.0) and external switches/devices does not capture any mem transactions. However, config transactions are visible in the trace. root@xlink-target:~# cat /proc/iomem 00000000-00000fff : reserved 00001000-0009abff : System RAM 0009ac00-0009ffff : reserved 000c0000-000c7fff : Video ROM 000c8000-000c8fff : Adapter ROM 000e0000-000fffff : reserved 000f0000-000fffff : System ROM 00100000-78f36fff : System RAM 01000000-01828f14 : Kernel code 01828f15-01f3fb3f : Kernel data 020bb000-021fdfff : Kernel bss 30000000-37ffffff : Crash kernel 78f37000-79839fff : reserved 79116018-79116018 : APEI ERST 7911601c-79116021 : APEI ERST 79116028-79116039 : APEI ERST 79116040-7911604c : APEI ERST 79116050-7911804f : APEI ERST 7983a000-79d4afff : ACPI Non-volatile Storage 79d4b000-8fffffff : reserved 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff] c6000000-c70fffff : PCI Bus 0000:2f c6000000-c70fffff : PCI Bus 0000:30 c6000000-c6ffffff : 0000:30:00.0 c7000000-c701ffff : 0000:30:00.0 c7100000-c71fffff : PCI Bus 0000:31 c7100000-c711ffff : 0000:31:00.1 c7100000-c711ffff : igb c7120000-c713ffff : 0000:31:00.0 c7120000-c713ffff : igb c7140000-c7143fff : 0000:31:00.1 c7140000-c7143fff : igb c7144000-c7147fff : 0000:31:00.0 c7144000-c7147fff : igb c7300000-c730ffff : 0000:00:14.0 c7300000-c730ffff : xhci-hcd c7310000-c73107ff : 0000:00:1f.2 c7310000-c73107ff : ahci c7311000-c73113ff : 0000:00:1d.0 c7311000-c73113ff : ehci_hcd c7312000-c73123ff : 0000:00:1a.0 c7312000-c73123ff : ehci_hcd c7314000-c731400f : 0000:00:16.1 c7315000-c731500f : 0000:00:16.0 c7316000-c73167ff : 0000:00:11.4 c7316000-c73167ff : ahci c7317000-c7317fff : 0000:00:05.4 c7ffc000-c7ffcfff : dmar1 c8000000-c8000000 : <BAD> c8000001-c8000001 : <BAD> c8000002-c8000002 : <BAD> c8000003-c8000003 : <BAD> c8100000-c91fffff : PCI Bus 0000:02 c8100000-c90fffff : PCI Bus 0000:03 c8100000-c90fffff : PCI Bus 0000:04 c8100000-c82fffff : PCI Bus 0000:05 c8100000-c81fffff : PCI Bus 0000:06 c8100000-c8100fff : 0000:06:00.0 c8200000-c82fffff : PCI Bus 0000:0a c8200000-c8203fff : 0000:0a:00.0 c9100000-c913ffff : 0000:02:00.0 fb000000-fb8fffff : PCI Bus 0000:81 fb000000-fb8fffff : PCI Bus 0000:82 fb000000-fb7fffff : PCI Bus 0000:8c fb000000-fb7fffff : 0000:8c:00.0 fb000000-fb7fffff : Plx_Mgr fb800000-fb8fffff : PCI Bus 0000:87 fb800000-fb800fff : 0000:87:00.0 fb900000-fb900fff : 0000:80:05.4 fbffc000-fbffcfff : dmar0 fec00000-fecfffff : PNP0003:00 fec00000-fec003ff : IOAPIC 0 fec01000-fec013ff : IOAPIC 1 fec40000-fec403ff : IOAPIC 2 fed00000-fed003ff : HPET 0 fed00000-fed003ff : PNP0103:00 fed12000-fed1200f : pnp 00:01 fed12010-fed1201f : pnp 00:01 fed1b000-fed1bfff : pnp 00:01 fed1c000-fed44fff : reserved fed1c000-fed3ffff : pnp 00:01 fed1f410-fed1f414 : iTCO_wdt.0.auto fed45000-fed8bfff : pnp 00:01 fee00000-feefffff : pnp 00:01 fee00000-fee00fff : Local APIC ff000000-ffffffff : reserved ff000000-ffffffff : pnp 00:01 100000000-47fffffff : System RAM 480000000-4807fffff : PCI Bus 0000:02 480000000-4807fffff : PCI Bus 0000:03 480000000-4807fffff : PCI Bus 0000:04 480000000-4800fffff : PCI Bus 0000:05 480000000-4800fffff : PCI Bus 0000:06 480000000-480003fff : 0000:06:00.0 383fffe00000-383fffefffff : PCI Bus 0000:31 383fffe00000-383fffe1ffff : 0000:31:00.1 383fffe20000-383fffe3ffff : 0000:31:00.1 383fffe40000-383fffe5ffff : 0000:31:00.0 383fffe60000-383fffe7ffff : 0000:31:00.0 383ffff00000-383ffff03fff : 0000:00:04.7 383ffff00000-383ffff03fff : ioatdma 383ffff04000-383ffff07fff : 0000:00:04.6 383ffff04000-383ffff07fff : ioatdma 383ffff08000-383ffff0bfff : 0000:00:04.5 383ffff08000-383ffff0bfff : ioatdma 383ffff0c000-383ffff0ffff : 0000:00:04.4 383ffff0c000-383ffff0ffff : ioatdma 383ffff10000-383ffff13fff : 0000:00:04.3 383ffff10000-383ffff13fff : ioatdma 383ffff14000-383ffff17fff : 0000:00:04.2 383ffff14000-383ffff17fff : ioatdma 383ffff18000-383ffff1bfff : 0000:00:04.1 383ffff18000-383ffff1bfff : ioatdma 383ffff1c000-383ffff1ffff : 0000:00:04.0 383ffff1c000-383ffff1ffff : ioatdma 383ffff21000-383ffff210ff : 0000:00:1f.3 387fffe00000-387fffefffff : PCI Bus 0000:81 387fffe00000-387fffefffff : PCI Bus 0000:82 387fffe00000-387fffefffff : PCI Bus 0000:87 387fffe00000-387fffe03fff : 0000:87:00.0 387ffff00000-387ffff03fff : 0000:80:04.7 387ffff00000-387ffff03fff : ioatdma 387ffff04000-387ffff07fff : 0000:80:04.6 387ffff04000-387ffff07fff : ioatdma 387ffff08000-387ffff0bfff : 0000:80:04.5 387ffff08000-387ffff0bfff : ioatdma 387ffff0c000-387ffff0ffff : 0000:80:04.4 387ffff0c000-387ffff0ffff : ioatdma 387ffff10000-387ffff13fff : 0000:80:04.3 387ffff10000-387ffff13fff : ioatdma 387ffff14000-387ffff17fff : 0000:80:04.2 387ffff14000-387ffff17fff : ioatdma 387ffff18000-387ffff1bfff : 0000:80:04.1 387ffff18000-387ffff1bfff : ioatdma 387ffff1c000-387ffff1ffff : 0000:80:04.0 387ffff1c000-387ffff1ffff : ioatdma Memory windows: | \-1f.2 \-[0000:00]-+-00.0 +-01.0-[01]-- +-03.0-[02-2c]----00.0-[03-2c]----08.0-[04-2c]----00.0-[05-0a]--+-00.0-[06]----00.0 | +-01.0-[07]-- | +-02.0-[08]-- | +-03.0-[09]-- | \-1e.0-[0a]----00.0 +-03.2-[2d]-- root@xlink-target:~# lspci -v -s 0:03.0 00:03.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 01) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0, IRQ 27 Bus: primary=00, secondary=02, subordinate=2c, sec-latency=0 Memory behind bridge: c8100000-c91fffff Prefetchable memory behind bridge: 0000000480000000-00000004807fffff root@xlink-target:~# lspci -v -s 02:00.0 02:00.0 PCI bridge: PLX Technology, Inc. PEX 8732 32-lane, 8-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca) (prog-if 00 [Normal decode]) Physical Slot: 4 Flags: bus master, fast devsel, latency 0, IRQ 32 Memory at c9100000 (32-bit, non-prefetchable) [size=256K] Bus: primary=02, secondary=03, subordinate=2c, sec-latency=0 Memory behind bridge: c8100000-c90fffff Prefetchable memory behind bridge: 0000000480000000-00000004807fffff root@xlink-target:~# lspci -v -s 03:08.0 03:08.0 PCI bridge: PLX Technology, Inc. PEX 8732 32-lane, 8-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0, IRQ 33 Bus: primary=03, secondary=04, subordinate=2c, sec-latency=0 Memory behind bridge: c8100000-c90fffff Prefetchable memory behind bridge: 0000000480000000-00000004807fffff root@xlink-target:~# lspci -v -s 04:00.0 04:00.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa) (prog-if 00 [Normal decode]) Physical Slot: 8-1 Flags: bus master, fast devsel, latency 0 Bus: primary=04, secondary=05, subordinate=0a, sec-latency=0 Memory behind bridge: c8100000-c82fffff Prefetchable memory behind bridge: 0000000480000000-00000004800fffff Capabilities: [40] Power Management version 3 root@xlink-target:~# lspci -v -s 05:00.0 05:00.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0, IRQ 96 Bus: primary=05, secondary=06, subordinate=06, sec-latency=0 Memory behind bridge: c8100000-c81fffff Prefetchable memory behind bridge: 0000000480000000-00000004800fffff root@xlink-target:~# lspci -v -s 06:00.0 06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06) Subsystem: TP-LINK Technologies Co., Ltd. TG-3468 Gigabit PCI Express Network Adapter Physical Slot: 1 Flags: bus master, fast devsel, latency 0 Memory at c8100000 (64-bit, non-prefetchable) [size=4K] Memory at 480000000 (64-bit, prefetchable) [size=16K] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Fwd: PCIe root port programming 2016-08-12 21:14 ` Fwd: PCIe root port programming Kallol Biswas @ 2016-08-15 17:24 ` Bjorn Helgaas 2016-08-15 18:39 ` Kallol Biswas 0 siblings, 1 reply; 8+ messages in thread From: Bjorn Helgaas @ 2016-08-15 17:24 UTC (permalink / raw) To: Kallol Biswas; +Cc: linux-pci Hi Kallol, Sorry I didn't respond to this earlier; I was on vacation for most of last week. On Fri, Aug 12, 2016 at 02:14:59PM -0700, Kallol Biswas wrote: > I am trying to debug an issue that the PCIe memory > transactions are not routed through a host bridge. > > The config transactions work fine. The primary, secondary and > subordinate bus numbers look fine for the PCIe tree under the host > bridge. The memory windows are programmed fine. > > Can someone point me to the code that programs the chipset/root port > registers those direct processor transactions to a specific host > bridge? It looks like you have an x86 system with ACPI. In that case, we use the ACPI pci_root.c host bridge driver. The chipset configuration for routing processor memory transactions to host bridges is done by the firmware, and the kernel doesn't touch it. There should be information in the chipset spec about how to do this, but there's no code in Linux itself that does it. > Cat of /proc/mem is copied below. > > /root/physmem -p c8100000 -s 4 -r > Operation:read to Addr: 0xc8100000 Value: 0 > ffffffff ffffffff ffffffff ffffffff > > The PCIe analyzer sitting between host bridge (0:3.0 and 02:00.0) and > external switches/devices does not capture any mem transactions. > However, config transactions are visible in the trace. 00:03.0 is not a host bridge; it's a Root Port, which logically connects two PCI buses. A host bridge logically connects a processor bus to a PCI bus. > root@xlink-target:~# cat /proc/iomem > 00000000-00000fff : reserved > 00001000-0009abff : System RAM > 0009ac00-0009ffff : reserved > 000c0000-000c7fff : Video ROM > 000c8000-000c8fff : Adapter ROM > 000e0000-000fffff : reserved > 000f0000-000fffff : System ROM > 00100000-78f36fff : System RAM > 01000000-01828f14 : Kernel code > 01828f15-01f3fb3f : Kernel data > 020bb000-021fdfff : Kernel bss > 30000000-37ffffff : Crash kernel > 78f37000-79839fff : reserved > 79116018-79116018 : APEI ERST > 7911601c-79116021 : APEI ERST > 79116028-79116039 : APEI ERST > 79116040-7911604c : APEI ERST > 79116050-7911804f : APEI ERST > 7983a000-79d4afff : ACPI Non-volatile Storage > 79d4b000-8fffffff : reserved > 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff] > c6000000-c70fffff : PCI Bus 0000:2f > c6000000-c70fffff : PCI Bus 0000:30 > c6000000-c6ffffff : 0000:30:00.0 > c7000000-c701ffff : 0000:30:00.0 > c7100000-c71fffff : PCI Bus 0000:31 > c7100000-c711ffff : 0000:31:00.1 > c7100000-c711ffff : igb > c7120000-c713ffff : 0000:31:00.0 > c7120000-c713ffff : igb > c7140000-c7143fff : 0000:31:00.1 > c7140000-c7143fff : igb > c7144000-c7147fff : 0000:31:00.0 > c7144000-c7147fff : igb > c7300000-c730ffff : 0000:00:14.0 > c7300000-c730ffff : xhci-hcd > c7310000-c73107ff : 0000:00:1f.2 > c7310000-c73107ff : ahci > c7311000-c73113ff : 0000:00:1d.0 > c7311000-c73113ff : ehci_hcd > c7312000-c73123ff : 0000:00:1a.0 > c7312000-c73123ff : ehci_hcd > c7314000-c731400f : 0000:00:16.1 > c7315000-c731500f : 0000:00:16.0 > c7316000-c73167ff : 0000:00:11.4 > c7316000-c73167ff : ahci > c7317000-c7317fff : 0000:00:05.4 > c7ffc000-c7ffcfff : dmar1 > c8000000-c8000000 : <BAD> > c8000001-c8000001 : <BAD> > c8000002-c8000002 : <BAD> > c8000003-c8000003 : <BAD> Interesting, what are these from? > c8100000-c91fffff : PCI Bus 0000:02 > c8100000-c90fffff : PCI Bus 0000:03 > c8100000-c90fffff : PCI Bus 0000:04 > c8100000-c82fffff : PCI Bus 0000:05 > c8100000-c81fffff : PCI Bus 0000:06 > c8100000-c8100fff : 0000:06:00.0 You're trying to read the 06:00.0 BAR at 0xc8100000. The windows above look fine, but the top-level host bridge window is missing. The c8100000-c91fffff window is a window through the 00:03.0 Root Port, which connects bus 00 to bus 02. The 00:03.0 bridge forwards memory transactions in the c8100000-c91fffff range from its primary bus (bus 00) to its secondary bus (bus 02). But we don't know how those transactions get to bus 00 in the first place. The host bridge is the connection between the processor and bus 00, and it should have a window that encloses c8100000-c91fffff. Your dmesg log should have information about this bridge, e.g., something like this: ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7f]) PCI host bridge to bus 0000:00 pci_bus 0000:00: root bus resource [mem 0xc8100000-0xc9ffffff window] and there should be a line in /proc/iomem something like this: c8100000-c9ffffff : PCI Bus 0000:00 Bjorn ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Fwd: PCIe root port programming 2016-08-15 17:24 ` Bjorn Helgaas @ 2016-08-15 18:39 ` Kallol Biswas 2016-08-15 19:06 ` Bjorn Helgaas 0 siblings, 1 reply; 8+ messages in thread From: Kallol Biswas @ 2016-08-15 18:39 UTC (permalink / raw) To: Bjorn Helgaas; +Cc: linux-pci [-- Attachment #1: Type: text/plain, Size: 7162 bytes --] Thank you for the explanation. Here is the relevant dmesg content: [ 0.790457] PCI: root bus ff: using default resources [ 0.790491] PCI host bridge to bus 0000:ff [ 0.790493] pci_bus 0000:ff: root bus resource [io 0x0000-0xffff] [ 0.790495] pci_bus 0000:ff: root bus resource [mem 0x00000000-0x3fffffffffff] [ 0.790497] pci_bus 0000:ff: root bus resource [bus ff] 0.795379] PCI host bridge to bus 0000:7f [ 0.795381] pci_bus 0000:7f: root bus resource [io 0x0000-0xffff] [ 0.795383] pci_bus 0000:7f: root bus resource [mem 0x00000000-0x3fffffffffff] [ 0.795384] pci_bus 0000:7f: root bus resource [bus 7f] 0.809149] PCI host bridge to bus 0000:00 [ 0.809151] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] [ 0.809153] pci_bus 0000:00: root bus resource [mem 0x00000000-0x3fffffffffff] [ 0.809155] pci_bus 0000:00: root bus resource [bus 00-7e] [ 0.852071] PCI host bridge to bus 0000:80 [ 0.852073] pci_bus 0000:80: root bus resource [io 0x0000-0xffff] [ 0.852075] pci_bus 0000:80: root bus resource [mem 0x00000000-0x3fffffffffff] [ 0.852077] pci_bus 0000:80: root bus resource [bus 80-fe] Following lines are from output of /proc/iomem c8000000-c8000000 : <BAD> c8000001-c8000001 : <BAD> c8000002-c8000002 : <BAD> c8000003-c8000003 : <BAD> c8100000-c91fffff : PCI Bus 0000:02 c8100000-c90fffff : PCI Bus 0000:03 c8100000-c90fffff : PCI Bus 0000:04 c8100000-c82fffff : PCI Bus 0000:05 c8100000-c81fffff : PCI Bus 0000:06 c8100000-c8100fff : 0000:06:00.0 c8200000-c82fffff : PCI Bus 0000:0a c8200000-c8203fff : 0000:0a:00.0 c9100000-c913ffff : 0000:02:00.0 fb000000-fb8fffff : PCI Bus 0000:81 fb000000-fb8fffff : PCI Bus 0000:82 fb000000-fb7fffff : PCI Bus 0000:8c fb000000-fb7fffff : 0000:8c:00.0 fb000000-fb7fffff : Plx_Mgr fb800000-fb8fffff : PCI Bus 0000:87 fb800000-fb800fff : 0000:87:00.0 fb900000-fb900fff : 0000:80:05.4 I can read the device BAR via fb800000-fb800fff : 0000:87:00.0. This device is available after reboot, and initial PCIe bus enumeration time. The device c8100000-c8100fff : 0000:06:00.0 shows up after a hotplug operation is done. Attached is the complete /proc/iomem and do not find the line c8100000-c9ffffff : PCI Bus 0000:00, but does it exist for PCI Bus 0000:80? BAR access works fine through this bus. On Mon, Aug 15, 2016 at 10:24 AM, Bjorn Helgaas <helgaas@kernel.org> wrote: > Hi Kallol, > > Sorry I didn't respond to this earlier; I was on vacation for most of > last week. > > On Fri, Aug 12, 2016 at 02:14:59PM -0700, Kallol Biswas wrote: >> I am trying to debug an issue that the PCIe memory >> transactions are not routed through a host bridge. >> >> The config transactions work fine. The primary, secondary and >> subordinate bus numbers look fine for the PCIe tree under the host >> bridge. The memory windows are programmed fine. >> >> Can someone point me to the code that programs the chipset/root port >> registers those direct processor transactions to a specific host >> bridge? > > It looks like you have an x86 system with ACPI. In that case, we use > the ACPI pci_root.c host bridge driver. The chipset configuration for > routing processor memory transactions to host bridges is done by the > firmware, and the kernel doesn't touch it. There should be > information in the chipset spec about how to do this, but there's no > code in Linux itself that does it. > >> Cat of /proc/mem is copied below. >> >> /root/physmem -p c8100000 -s 4 -r >> Operation:read to Addr: 0xc8100000 Value: 0 >> ffffffff ffffffff ffffffff ffffffff >> >> The PCIe analyzer sitting between host bridge (0:3.0 and 02:00.0) and >> external switches/devices does not capture any mem transactions. >> However, config transactions are visible in the trace. > > 00:03.0 is not a host bridge; it's a Root Port, which logically > connects two PCI buses. A host bridge logically connects a processor > bus to a PCI bus. > >> root@xlink-target:~# cat /proc/iomem >> 00000000-00000fff : reserved >> 00001000-0009abff : System RAM >> 0009ac00-0009ffff : reserved >> 000c0000-000c7fff : Video ROM >> 000c8000-000c8fff : Adapter ROM >> 000e0000-000fffff : reserved >> 000f0000-000fffff : System ROM >> 00100000-78f36fff : System RAM >> 01000000-01828f14 : Kernel code >> 01828f15-01f3fb3f : Kernel data >> 020bb000-021fdfff : Kernel bss >> 30000000-37ffffff : Crash kernel >> 78f37000-79839fff : reserved >> 79116018-79116018 : APEI ERST >> 7911601c-79116021 : APEI ERST >> 79116028-79116039 : APEI ERST >> 79116040-7911604c : APEI ERST >> 79116050-7911804f : APEI ERST >> 7983a000-79d4afff : ACPI Non-volatile Storage >> 79d4b000-8fffffff : reserved >> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff] >> c6000000-c70fffff : PCI Bus 0000:2f >> c6000000-c70fffff : PCI Bus 0000:30 >> c6000000-c6ffffff : 0000:30:00.0 >> c7000000-c701ffff : 0000:30:00.0 >> c7100000-c71fffff : PCI Bus 0000:31 >> c7100000-c711ffff : 0000:31:00.1 >> c7100000-c711ffff : igb >> c7120000-c713ffff : 0000:31:00.0 >> c7120000-c713ffff : igb >> c7140000-c7143fff : 0000:31:00.1 >> c7140000-c7143fff : igb >> c7144000-c7147fff : 0000:31:00.0 >> c7144000-c7147fff : igb >> c7300000-c730ffff : 0000:00:14.0 >> c7300000-c730ffff : xhci-hcd >> c7310000-c73107ff : 0000:00:1f.2 >> c7310000-c73107ff : ahci >> c7311000-c73113ff : 0000:00:1d.0 >> c7311000-c73113ff : ehci_hcd >> c7312000-c73123ff : 0000:00:1a.0 >> c7312000-c73123ff : ehci_hcd >> c7314000-c731400f : 0000:00:16.1 >> c7315000-c731500f : 0000:00:16.0 >> c7316000-c73167ff : 0000:00:11.4 >> c7316000-c73167ff : ahci >> c7317000-c7317fff : 0000:00:05.4 >> c7ffc000-c7ffcfff : dmar1 >> c8000000-c8000000 : <BAD> >> c8000001-c8000001 : <BAD> >> c8000002-c8000002 : <BAD> >> c8000003-c8000003 : <BAD> > > Interesting, what are these from? > >> c8100000-c91fffff : PCI Bus 0000:02 >> c8100000-c90fffff : PCI Bus 0000:03 >> c8100000-c90fffff : PCI Bus 0000:04 >> c8100000-c82fffff : PCI Bus 0000:05 >> c8100000-c81fffff : PCI Bus 0000:06 >> c8100000-c8100fff : 0000:06:00.0 > > You're trying to read the 06:00.0 BAR at 0xc8100000. The windows > above look fine, but the top-level host bridge window is missing. > > The c8100000-c91fffff window is a window through the 00:03.0 Root > Port, which connects bus 00 to bus 02. The 00:03.0 bridge forwards > memory transactions in the c8100000-c91fffff range from its primary > bus (bus 00) to its secondary bus (bus 02). > > But we don't know how those transactions get to bus 00 in the first > place. The host bridge is the connection between the processor and > bus 00, and it should have a window that encloses c8100000-c91fffff. > > Your dmesg log should have information about this bridge, e.g., > something like this: > > ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7f]) > PCI host bridge to bus 0000:00 > pci_bus 0000:00: root bus resource [mem 0xc8100000-0xc9ffffff window] > > and there should be a line in /proc/iomem something like this: > > c8100000-c9ffffff : PCI Bus 0000:00 > > Bjorn [-- Attachment #2: iomem.txt --] [-- Type: text/plain, Size: 4888 bytes --] 00000000-00000fff : reserved 00001000-0009abff : System RAM 0009ac00-0009ffff : reserved 000c0000-000c7fff : Video ROM 000c8000-000c8fff : Adapter ROM 000e0000-000fffff : reserved 000f0000-000fffff : System ROM 00100000-78f36fff : System RAM 01000000-01828f14 : Kernel code 01828f15-01f3fb3f : Kernel data 020bb000-021fdfff : Kernel bss 30000000-37ffffff : Crash kernel 78f37000-79839fff : reserved 79116018-79116018 : APEI ERST 7911601c-79116021 : APEI ERST 79116028-79116039 : APEI ERST 79116040-7911604c : APEI ERST 79116050-7911804f : APEI ERST 7983a000-79d4afff : ACPI Non-volatile Storage 79d4b000-8fffffff : reserved 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff] c6000000-c70fffff : PCI Bus 0000:2f c6000000-c70fffff : PCI Bus 0000:30 c6000000-c6ffffff : 0000:30:00.0 c7000000-c701ffff : 0000:30:00.0 c7100000-c71fffff : PCI Bus 0000:31 c7100000-c711ffff : 0000:31:00.1 c7100000-c711ffff : igb c7120000-c713ffff : 0000:31:00.0 c7120000-c713ffff : igb c7140000-c7143fff : 0000:31:00.1 c7140000-c7143fff : igb c7144000-c7147fff : 0000:31:00.0 c7144000-c7147fff : igb c7300000-c730ffff : 0000:00:14.0 c7300000-c730ffff : xhci-hcd c7310000-c73107ff : 0000:00:1f.2 c7310000-c73107ff : ahci c7311000-c73113ff : 0000:00:1d.0 c7311000-c73113ff : ehci_hcd c7312000-c73123ff : 0000:00:1a.0 c7312000-c73123ff : ehci_hcd c7314000-c731400f : 0000:00:16.1 c7315000-c731500f : 0000:00:16.0 c7316000-c73167ff : 0000:00:11.4 c7316000-c73167ff : ahci c7317000-c7317fff : 0000:00:05.4 c7ffc000-c7ffcfff : dmar1 c8000000-c8000000 : <BAD> c8000001-c8000001 : <BAD> c8000002-c8000002 : <BAD> c8000003-c8000003 : <BAD> c8100000-c91fffff : PCI Bus 0000:02 c8100000-c90fffff : PCI Bus 0000:03 c8100000-c90fffff : PCI Bus 0000:04 c8100000-c82fffff : PCI Bus 0000:05 c8100000-c81fffff : PCI Bus 0000:06 c8100000-c8100fff : 0000:06:00.0 c8200000-c82fffff : PCI Bus 0000:0a c8200000-c8203fff : 0000:0a:00.0 c9100000-c913ffff : 0000:02:00.0 fb000000-fb8fffff : PCI Bus 0000:81 fb000000-fb8fffff : PCI Bus 0000:82 fb000000-fb7fffff : PCI Bus 0000:8c fb000000-fb7fffff : 0000:8c:00.0 fb000000-fb7fffff : Plx_Mgr fb800000-fb8fffff : PCI Bus 0000:87 fb800000-fb800fff : 0000:87:00.0 fb900000-fb900fff : 0000:80:05.4 fbffc000-fbffcfff : dmar0 fec00000-fecfffff : PNP0003:00 fec00000-fec003ff : IOAPIC 0 fec01000-fec013ff : IOAPIC 1 fec40000-fec403ff : IOAPIC 2 fed00000-fed003ff : HPET 0 fed00000-fed003ff : PNP0103:00 fed12000-fed1200f : pnp 00:01 fed12010-fed1201f : pnp 00:01 fed1b000-fed1bfff : pnp 00:01 fed1c000-fed44fff : reserved fed1c000-fed3ffff : pnp 00:01 fed1f410-fed1f414 : iTCO_wdt.0.auto fed45000-fed8bfff : pnp 00:01 fee00000-feefffff : pnp 00:01 fee00000-fee00fff : Local APIC ff000000-ffffffff : reserved ff000000-ffffffff : pnp 00:01 100000000-47fffffff : System RAM 480000000-4807fffff : PCI Bus 0000:02 480000000-4807fffff : PCI Bus 0000:03 480000000-4807fffff : PCI Bus 0000:04 480000000-4800fffff : PCI Bus 0000:05 480000000-4800fffff : PCI Bus 0000:06 480000000-480003fff : 0000:06:00.0 383fffe00000-383fffefffff : PCI Bus 0000:31 383fffe00000-383fffe1ffff : 0000:31:00.1 383fffe20000-383fffe3ffff : 0000:31:00.1 383fffe40000-383fffe5ffff : 0000:31:00.0 383fffe60000-383fffe7ffff : 0000:31:00.0 383ffff00000-383ffff03fff : 0000:00:04.7 383ffff00000-383ffff03fff : ioatdma 383ffff04000-383ffff07fff : 0000:00:04.6 383ffff04000-383ffff07fff : ioatdma 383ffff08000-383ffff0bfff : 0000:00:04.5 383ffff08000-383ffff0bfff : ioatdma 383ffff0c000-383ffff0ffff : 0000:00:04.4 383ffff0c000-383ffff0ffff : ioatdma 383ffff10000-383ffff13fff : 0000:00:04.3 383ffff10000-383ffff13fff : ioatdma 383ffff14000-383ffff17fff : 0000:00:04.2 383ffff14000-383ffff17fff : ioatdma 383ffff18000-383ffff1bfff : 0000:00:04.1 383ffff18000-383ffff1bfff : ioatdma 383ffff1c000-383ffff1ffff : 0000:00:04.0 383ffff1c000-383ffff1ffff : ioatdma 383ffff21000-383ffff210ff : 0000:00:1f.3 387fffe00000-387fffefffff : PCI Bus 0000:81 387fffe00000-387fffefffff : PCI Bus 0000:82 387fffe00000-387fffefffff : PCI Bus 0000:87 387fffe00000-387fffe03fff : 0000:87:00.0 387ffff00000-387ffff03fff : 0000:80:04.7 387ffff00000-387ffff03fff : ioatdma 387ffff04000-387ffff07fff : 0000:80:04.6 387ffff04000-387ffff07fff : ioatdma 387ffff08000-387ffff0bfff : 0000:80:04.5 387ffff08000-387ffff0bfff : ioatdma 387ffff0c000-387ffff0ffff : 0000:80:04.4 387ffff0c000-387ffff0ffff : ioatdma 387ffff10000-387ffff13fff : 0000:80:04.3 387ffff10000-387ffff13fff : ioatdma 387ffff14000-387ffff17fff : 0000:80:04.2 387ffff14000-387ffff17fff : ioatdma 387ffff18000-387ffff1bfff : 0000:80:04.1 387ffff18000-387ffff1bfff : ioatdma 387ffff1c000-387ffff1ffff : 0000:80:04.0 387ffff1c000-387ffff1ffff : ioatdma ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Fwd: PCIe root port programming 2016-08-15 18:39 ` Kallol Biswas @ 2016-08-15 19:06 ` Bjorn Helgaas 2016-08-15 20:43 ` Kallol Biswas 0 siblings, 1 reply; 8+ messages in thread From: Bjorn Helgaas @ 2016-08-15 19:06 UTC (permalink / raw) To: Kallol Biswas; +Cc: linux-pci On Mon, Aug 15, 2016 at 11:39:56AM -0700, Kallol Biswas wrote: > Thank you for the explanation. > > > Here is the relevant dmesg content: > > [ 0.790457] PCI: root bus ff: using default resources > [ 0.790491] PCI host bridge to bus 0000:ff > [ 0.790493] pci_bus 0000:ff: root bus resource [io 0x0000-0xffff] > [ 0.790495] pci_bus 0000:ff: root bus resource [mem > 0x00000000-0x3fffffffffff] > [ 0.790497] pci_bus 0000:ff: root bus resource [bus ff] > > 0.795379] PCI host bridge to bus 0000:7f > [ 0.795381] pci_bus 0000:7f: root bus resource [io 0x0000-0xffff] > [ 0.795383] pci_bus 0000:7f: root bus resource [mem > 0x00000000-0x3fffffffffff] > [ 0.795384] pci_bus 0000:7f: root bus resource [bus 7f] > > 0.809149] PCI host bridge to bus 0000:00 > [ 0.809151] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] > [ 0.809153] pci_bus 0000:00: root bus resource [mem > 0x00000000-0x3fffffffffff] > [ 0.809155] pci_bus 0000:00: root bus resource [bus 00-7e] This is the host bridge that claims to lead to bus 0000:00. But the I/O and memory resources of all these bridges are wrong. The ranges they show are CPU addresses, not PCI bus addresses, so they cannot overlap. It could be that you're using a non-standard host bridge driver that did not apply the translation between CPU addresses and PCI bus addresses. In ACPI, this is the _TRA property. > [ 0.852071] PCI host bridge to bus 0000:80 > [ 0.852073] pci_bus 0000:80: root bus resource [io 0x0000-0xffff] > [ 0.852075] pci_bus 0000:80: root bus resource [mem > 0x00000000-0x3fffffffffff] > [ 0.852077] pci_bus 0000:80: root bus resource [bus 80-fe] > > Following lines are from output of /proc/iomem > > c8000000-c8000000 : <BAD> > c8000001-c8000001 : <BAD> > c8000002-c8000002 : <BAD> > c8000003-c8000003 : <BAD> > c8100000-c91fffff : PCI Bus 0000:02 > c8100000-c90fffff : PCI Bus 0000:03 > c8100000-c90fffff : PCI Bus 0000:04 > c8100000-c82fffff : PCI Bus 0000:05 > c8100000-c81fffff : PCI Bus 0000:06 > c8100000-c8100fff : 0000:06:00.0 > c8200000-c82fffff : PCI Bus 0000:0a > c8200000-c8203fff : 0000:0a:00.0 > c9100000-c913ffff : 0000:02:00.0 > fb000000-fb8fffff : PCI Bus 0000:81 > fb000000-fb8fffff : PCI Bus 0000:82 > fb000000-fb7fffff : PCI Bus 0000:8c > fb000000-fb7fffff : 0000:8c:00.0 > fb000000-fb7fffff : Plx_Mgr > fb800000-fb8fffff : PCI Bus 0000:87 > fb800000-fb800fff : 0000:87:00.0 > fb900000-fb900fff : 0000:80:05.4 > > > I can read the device BAR via fb800000-fb800fff : 0000:87:00.0. This > device is available after reboot, and initial PCIe bus enumeration > time. > > The device c8100000-c8100fff : 0000:06:00.0 shows up after a hotplug > operation is done. > > Attached is the complete /proc/iomem and do not find the line > c8100000-c9ffffff : PCI Bus 0000:00, but does it exist for > PCI Bus 0000:80? BAR access works fine through this bus. > > > > On Mon, Aug 15, 2016 at 10:24 AM, Bjorn Helgaas <helgaas@kernel.org> wrote: > > Hi Kallol, > > > > Sorry I didn't respond to this earlier; I was on vacation for most of > > last week. > > > > On Fri, Aug 12, 2016 at 02:14:59PM -0700, Kallol Biswas wrote: > >> I am trying to debug an issue that the PCIe memory > >> transactions are not routed through a host bridge. > >> > >> The config transactions work fine. The primary, secondary and > >> subordinate bus numbers look fine for the PCIe tree under the host > >> bridge. The memory windows are programmed fine. > >> > >> Can someone point me to the code that programs the chipset/root port > >> registers those direct processor transactions to a specific host > >> bridge? > > > > It looks like you have an x86 system with ACPI. In that case, we use > > the ACPI pci_root.c host bridge driver. The chipset configuration for > > routing processor memory transactions to host bridges is done by the > > firmware, and the kernel doesn't touch it. There should be > > information in the chipset spec about how to do this, but there's no > > code in Linux itself that does it. > > > >> Cat of /proc/mem is copied below. > >> > >> /root/physmem -p c8100000 -s 4 -r > >> Operation:read to Addr: 0xc8100000 Value: 0 > >> ffffffff ffffffff ffffffff ffffffff > >> > >> The PCIe analyzer sitting between host bridge (0:3.0 and 02:00.0) and > >> external switches/devices does not capture any mem transactions. > >> However, config transactions are visible in the trace. > > > > 00:03.0 is not a host bridge; it's a Root Port, which logically > > connects two PCI buses. A host bridge logically connects a processor > > bus to a PCI bus. > > > >> root@xlink-target:~# cat /proc/iomem > >> 00000000-00000fff : reserved > >> 00001000-0009abff : System RAM > >> 0009ac00-0009ffff : reserved > >> 000c0000-000c7fff : Video ROM > >> 000c8000-000c8fff : Adapter ROM > >> 000e0000-000fffff : reserved > >> 000f0000-000fffff : System ROM > >> 00100000-78f36fff : System RAM > >> 01000000-01828f14 : Kernel code > >> 01828f15-01f3fb3f : Kernel data > >> 020bb000-021fdfff : Kernel bss > >> 30000000-37ffffff : Crash kernel > >> 78f37000-79839fff : reserved > >> 79116018-79116018 : APEI ERST > >> 7911601c-79116021 : APEI ERST > >> 79116028-79116039 : APEI ERST > >> 79116040-7911604c : APEI ERST > >> 79116050-7911804f : APEI ERST > >> 7983a000-79d4afff : ACPI Non-volatile Storage > >> 79d4b000-8fffffff : reserved > >> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff] > >> c6000000-c70fffff : PCI Bus 0000:2f > >> c6000000-c70fffff : PCI Bus 0000:30 > >> c6000000-c6ffffff : 0000:30:00.0 > >> c7000000-c701ffff : 0000:30:00.0 > >> c7100000-c71fffff : PCI Bus 0000:31 > >> c7100000-c711ffff : 0000:31:00.1 > >> c7100000-c711ffff : igb > >> c7120000-c713ffff : 0000:31:00.0 > >> c7120000-c713ffff : igb > >> c7140000-c7143fff : 0000:31:00.1 > >> c7140000-c7143fff : igb > >> c7144000-c7147fff : 0000:31:00.0 > >> c7144000-c7147fff : igb > >> c7300000-c730ffff : 0000:00:14.0 > >> c7300000-c730ffff : xhci-hcd > >> c7310000-c73107ff : 0000:00:1f.2 > >> c7310000-c73107ff : ahci > >> c7311000-c73113ff : 0000:00:1d.0 > >> c7311000-c73113ff : ehci_hcd > >> c7312000-c73123ff : 0000:00:1a.0 > >> c7312000-c73123ff : ehci_hcd > >> c7314000-c731400f : 0000:00:16.1 > >> c7315000-c731500f : 0000:00:16.0 > >> c7316000-c73167ff : 0000:00:11.4 > >> c7316000-c73167ff : ahci > >> c7317000-c7317fff : 0000:00:05.4 > >> c7ffc000-c7ffcfff : dmar1 > >> c8000000-c8000000 : <BAD> > >> c8000001-c8000001 : <BAD> > >> c8000002-c8000002 : <BAD> > >> c8000003-c8000003 : <BAD> > > > > Interesting, what are these from? > > > >> c8100000-c91fffff : PCI Bus 0000:02 > >> c8100000-c90fffff : PCI Bus 0000:03 > >> c8100000-c90fffff : PCI Bus 0000:04 > >> c8100000-c82fffff : PCI Bus 0000:05 > >> c8100000-c81fffff : PCI Bus 0000:06 > >> c8100000-c8100fff : 0000:06:00.0 > > > > You're trying to read the 06:00.0 BAR at 0xc8100000. The windows > > above look fine, but the top-level host bridge window is missing. > > > > The c8100000-c91fffff window is a window through the 00:03.0 Root > > Port, which connects bus 00 to bus 02. The 00:03.0 bridge forwards > > memory transactions in the c8100000-c91fffff range from its primary > > bus (bus 00) to its secondary bus (bus 02). > > > > But we don't know how those transactions get to bus 00 in the first > > place. The host bridge is the connection between the processor and > > bus 00, and it should have a window that encloses c8100000-c91fffff. > > > > Your dmesg log should have information about this bridge, e.g., > > something like this: > > > > ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7f]) > > PCI host bridge to bus 0000:00 > > pci_bus 0000:00: root bus resource [mem 0xc8100000-0xc9ffffff window] > > > > and there should be a line in /proc/iomem something like this: > > > > c8100000-c9ffffff : PCI Bus 0000:00 > > > > Bjorn > 00000000-00000fff : reserved > 00001000-0009abff : System RAM > 0009ac00-0009ffff : reserved > 000c0000-000c7fff : Video ROM > 000c8000-000c8fff : Adapter ROM > 000e0000-000fffff : reserved > 000f0000-000fffff : System ROM > 00100000-78f36fff : System RAM > 01000000-01828f14 : Kernel code > 01828f15-01f3fb3f : Kernel data > 020bb000-021fdfff : Kernel bss > 30000000-37ffffff : Crash kernel > 78f37000-79839fff : reserved > 79116018-79116018 : APEI ERST > 7911601c-79116021 : APEI ERST > 79116028-79116039 : APEI ERST > 79116040-7911604c : APEI ERST > 79116050-7911804f : APEI ERST > 7983a000-79d4afff : ACPI Non-volatile Storage > 79d4b000-8fffffff : reserved > 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff] > c6000000-c70fffff : PCI Bus 0000:2f > c6000000-c70fffff : PCI Bus 0000:30 > c6000000-c6ffffff : 0000:30:00.0 > c7000000-c701ffff : 0000:30:00.0 > c7100000-c71fffff : PCI Bus 0000:31 > c7100000-c711ffff : 0000:31:00.1 > c7100000-c711ffff : igb > c7120000-c713ffff : 0000:31:00.0 > c7120000-c713ffff : igb > c7140000-c7143fff : 0000:31:00.1 > c7140000-c7143fff : igb > c7144000-c7147fff : 0000:31:00.0 > c7144000-c7147fff : igb > c7300000-c730ffff : 0000:00:14.0 > c7300000-c730ffff : xhci-hcd > c7310000-c73107ff : 0000:00:1f.2 > c7310000-c73107ff : ahci > c7311000-c73113ff : 0000:00:1d.0 > c7311000-c73113ff : ehci_hcd > c7312000-c73123ff : 0000:00:1a.0 > c7312000-c73123ff : ehci_hcd > c7314000-c731400f : 0000:00:16.1 > c7315000-c731500f : 0000:00:16.0 > c7316000-c73167ff : 0000:00:11.4 > c7316000-c73167ff : ahci > c7317000-c7317fff : 0000:00:05.4 > c7ffc000-c7ffcfff : dmar1 > c8000000-c8000000 : <BAD> > c8000001-c8000001 : <BAD> > c8000002-c8000002 : <BAD> > c8000003-c8000003 : <BAD> > c8100000-c91fffff : PCI Bus 0000:02 > c8100000-c90fffff : PCI Bus 0000:03 > c8100000-c90fffff : PCI Bus 0000:04 > c8100000-c82fffff : PCI Bus 0000:05 > c8100000-c81fffff : PCI Bus 0000:06 > c8100000-c8100fff : 0000:06:00.0 > c8200000-c82fffff : PCI Bus 0000:0a > c8200000-c8203fff : 0000:0a:00.0 > c9100000-c913ffff : 0000:02:00.0 > fb000000-fb8fffff : PCI Bus 0000:81 > fb000000-fb8fffff : PCI Bus 0000:82 > fb000000-fb7fffff : PCI Bus 0000:8c > fb000000-fb7fffff : 0000:8c:00.0 > fb000000-fb7fffff : Plx_Mgr > fb800000-fb8fffff : PCI Bus 0000:87 > fb800000-fb800fff : 0000:87:00.0 > fb900000-fb900fff : 0000:80:05.4 > fbffc000-fbffcfff : dmar0 > fec00000-fecfffff : PNP0003:00 > fec00000-fec003ff : IOAPIC 0 > fec01000-fec013ff : IOAPIC 1 > fec40000-fec403ff : IOAPIC 2 > fed00000-fed003ff : HPET 0 > fed00000-fed003ff : PNP0103:00 > fed12000-fed1200f : pnp 00:01 > fed12010-fed1201f : pnp 00:01 > fed1b000-fed1bfff : pnp 00:01 > fed1c000-fed44fff : reserved > fed1c000-fed3ffff : pnp 00:01 > fed1f410-fed1f414 : iTCO_wdt.0.auto > fed45000-fed8bfff : pnp 00:01 > fee00000-feefffff : pnp 00:01 > fee00000-fee00fff : Local APIC > ff000000-ffffffff : reserved > ff000000-ffffffff : pnp 00:01 > 100000000-47fffffff : System RAM > 480000000-4807fffff : PCI Bus 0000:02 > 480000000-4807fffff : PCI Bus 0000:03 > 480000000-4807fffff : PCI Bus 0000:04 > 480000000-4800fffff : PCI Bus 0000:05 > 480000000-4800fffff : PCI Bus 0000:06 > 480000000-480003fff : 0000:06:00.0 > 383fffe00000-383fffefffff : PCI Bus 0000:31 > 383fffe00000-383fffe1ffff : 0000:31:00.1 > 383fffe20000-383fffe3ffff : 0000:31:00.1 > 383fffe40000-383fffe5ffff : 0000:31:00.0 > 383fffe60000-383fffe7ffff : 0000:31:00.0 > 383ffff00000-383ffff03fff : 0000:00:04.7 > 383ffff00000-383ffff03fff : ioatdma > 383ffff04000-383ffff07fff : 0000:00:04.6 > 383ffff04000-383ffff07fff : ioatdma > 383ffff08000-383ffff0bfff : 0000:00:04.5 > 383ffff08000-383ffff0bfff : ioatdma > 383ffff0c000-383ffff0ffff : 0000:00:04.4 > 383ffff0c000-383ffff0ffff : ioatdma > 383ffff10000-383ffff13fff : 0000:00:04.3 > 383ffff10000-383ffff13fff : ioatdma > 383ffff14000-383ffff17fff : 0000:00:04.2 > 383ffff14000-383ffff17fff : ioatdma > 383ffff18000-383ffff1bfff : 0000:00:04.1 > 383ffff18000-383ffff1bfff : ioatdma > 383ffff1c000-383ffff1ffff : 0000:00:04.0 > 383ffff1c000-383ffff1ffff : ioatdma > 383ffff21000-383ffff210ff : 0000:00:1f.3 > 387fffe00000-387fffefffff : PCI Bus 0000:81 > 387fffe00000-387fffefffff : PCI Bus 0000:82 > 387fffe00000-387fffefffff : PCI Bus 0000:87 > 387fffe00000-387fffe03fff : 0000:87:00.0 > 387ffff00000-387ffff03fff : 0000:80:04.7 > 387ffff00000-387ffff03fff : ioatdma > 387ffff04000-387ffff07fff : 0000:80:04.6 > 387ffff04000-387ffff07fff : ioatdma > 387ffff08000-387ffff0bfff : 0000:80:04.5 > 387ffff08000-387ffff0bfff : ioatdma > 387ffff0c000-387ffff0ffff : 0000:80:04.4 > 387ffff0c000-387ffff0ffff : ioatdma > 387ffff10000-387ffff13fff : 0000:80:04.3 > 387ffff10000-387ffff13fff : ioatdma > 387ffff14000-387ffff17fff : 0000:80:04.2 > 387ffff14000-387ffff17fff : ioatdma > 387ffff18000-387ffff1bfff : 0000:80:04.1 > 387ffff18000-387ffff1bfff : ioatdma > 387ffff1c000-387ffff1ffff : 0000:80:04.0 > 387ffff1c000-387ffff1ffff : ioatdma ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Fwd: PCIe root port programming 2016-08-15 19:06 ` Bjorn Helgaas @ 2016-08-15 20:43 ` Kallol Biswas 2016-08-15 21:07 ` Bjorn Helgaas 0 siblings, 1 reply; 8+ messages in thread From: Kallol Biswas @ 2016-08-15 20:43 UTC (permalink / raw) To: Bjorn Helgaas; +Cc: linux-pci The kernel source code is 4.4. I have not done any modification except pre-allocated bus numbers and memory window for a DS port to allow successful hot plug operation of a PCIe port. 0.611919] acpi PNP0A08:00: host bridge window [io 0x0cf8-0x0cff] (ignored) [ 0.611921] acpi PNP0A08:00: host bridge window [io 0x0000-0x0cf7 window] (ignored) [ 0.611922] acpi PNP0A08:00: host bridge window [io 0x1000-0x7fff window] (ignored) [ 0.611924] acpi PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff window] (ignored) [ 0.611926] acpi PNP0A08:00: host bridge window [mem 0xfedb0000-0xfedb000f window] (ignored) [ 0.611928] acpi PNP0A08:00: host bridge window [mem 0xfedc0000-0xfedc000f window] (ignored) [ 0.611929] acpi PNP0A08:00: host bridge window [mem 0x90000000-0xc7ffbfff window] (ignored) [ 0.611931] acpi PNP0A08:00: host bridge window [mem 0x380000000000-0x383fffffffff window] (ignored) [ 0.611932] PCI: root bus 00: using default resources [ 0.612217] PCI host bridge to bus 0000:00 [ 0.612219] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] [ 0.612220] pci_bus 0000:00: root bus resource [mem 0x00000000-0x3fffffffffff] [ 0.612222] pci_bus 0000:00: root bus resource [bus 00-7e] kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.4.8-kb root=UUID=c6f326f6-7cc4-49f5-a3be-c511566165b4 ro pci=realloc,nocrs,assign-busses pcie_ports=native pciehp.pciehp_debug=1 pci=hpmemsize=0M console=ttyS1,115200n8 quiet splash crashkernel=384M-:128M vt.handoff=7 On Mon, Aug 15, 2016 at 12:06 PM, Bjorn Helgaas <helgaas@kernel.org> wrote: > On Mon, Aug 15, 2016 at 11:39:56AM -0700, Kallol Biswas wrote: >> Thank you for the explanation. >> >> >> Here is the relevant dmesg content: >> >> [ 0.790457] PCI: root bus ff: using default resources >> [ 0.790491] PCI host bridge to bus 0000:ff >> [ 0.790493] pci_bus 0000:ff: root bus resource [io 0x0000-0xffff] >> [ 0.790495] pci_bus 0000:ff: root bus resource [mem >> 0x00000000-0x3fffffffffff] >> [ 0.790497] pci_bus 0000:ff: root bus resource [bus ff] >> >> 0.795379] PCI host bridge to bus 0000:7f >> [ 0.795381] pci_bus 0000:7f: root bus resource [io 0x0000-0xffff] >> [ 0.795383] pci_bus 0000:7f: root bus resource [mem >> 0x00000000-0x3fffffffffff] >> [ 0.795384] pci_bus 0000:7f: root bus resource [bus 7f] >> >> 0.809149] PCI host bridge to bus 0000:00 >> [ 0.809151] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] >> [ 0.809153] pci_bus 0000:00: root bus resource [mem >> 0x00000000-0x3fffffffffff] >> [ 0.809155] pci_bus 0000:00: root bus resource [bus 00-7e] > > This is the host bridge that claims to lead to bus 0000:00. > > But the I/O and memory resources of all these bridges are wrong. The > ranges they show are CPU addresses, not PCI bus addresses, so they > cannot overlap. > > It could be that you're using a non-standard host bridge driver that > did not apply the translation between CPU addresses and PCI bus > addresses. In ACPI, this is the _TRA property. > >> [ 0.852071] PCI host bridge to bus 0000:80 >> [ 0.852073] pci_bus 0000:80: root bus resource [io 0x0000-0xffff] >> [ 0.852075] pci_bus 0000:80: root bus resource [mem >> 0x00000000-0x3fffffffffff] >> [ 0.852077] pci_bus 0000:80: root bus resource [bus 80-fe] >> >> Following lines are from output of /proc/iomem >> >> c8000000-c8000000 : <BAD> >> c8000001-c8000001 : <BAD> >> c8000002-c8000002 : <BAD> >> c8000003-c8000003 : <BAD> >> c8100000-c91fffff : PCI Bus 0000:02 >> c8100000-c90fffff : PCI Bus 0000:03 >> c8100000-c90fffff : PCI Bus 0000:04 >> c8100000-c82fffff : PCI Bus 0000:05 >> c8100000-c81fffff : PCI Bus 0000:06 >> c8100000-c8100fff : 0000:06:00.0 >> c8200000-c82fffff : PCI Bus 0000:0a >> c8200000-c8203fff : 0000:0a:00.0 >> c9100000-c913ffff : 0000:02:00.0 >> fb000000-fb8fffff : PCI Bus 0000:81 >> fb000000-fb8fffff : PCI Bus 0000:82 >> fb000000-fb7fffff : PCI Bus 0000:8c >> fb000000-fb7fffff : 0000:8c:00.0 >> fb000000-fb7fffff : Plx_Mgr >> fb800000-fb8fffff : PCI Bus 0000:87 >> fb800000-fb800fff : 0000:87:00.0 >> fb900000-fb900fff : 0000:80:05.4 >> >> >> I can read the device BAR via fb800000-fb800fff : 0000:87:00.0. This >> device is available after reboot, and initial PCIe bus enumeration >> time. >> >> The device c8100000-c8100fff : 0000:06:00.0 shows up after a hotplug >> operation is done. >> >> Attached is the complete /proc/iomem and do not find the line >> c8100000-c9ffffff : PCI Bus 0000:00, but does it exist for >> PCI Bus 0000:80? BAR access works fine through this bus. >> >> >> >> On Mon, Aug 15, 2016 at 10:24 AM, Bjorn Helgaas <helgaas@kernel.org> wrote: >> > Hi Kallol, >> > >> > Sorry I didn't respond to this earlier; I was on vacation for most of >> > last week. >> > >> > On Fri, Aug 12, 2016 at 02:14:59PM -0700, Kallol Biswas wrote: >> >> I am trying to debug an issue that the PCIe memory >> >> transactions are not routed through a host bridge. >> >> >> >> The config transactions work fine. The primary, secondary and >> >> subordinate bus numbers look fine for the PCIe tree under the host >> >> bridge. The memory windows are programmed fine. >> >> >> >> Can someone point me to the code that programs the chipset/root port >> >> registers those direct processor transactions to a specific host >> >> bridge? >> > >> > It looks like you have an x86 system with ACPI. In that case, we use >> > the ACPI pci_root.c host bridge driver. The chipset configuration for >> > routing processor memory transactions to host bridges is done by the >> > firmware, and the kernel doesn't touch it. There should be >> > information in the chipset spec about how to do this, but there's no >> > code in Linux itself that does it. >> > >> >> Cat of /proc/mem is copied below. >> >> >> >> /root/physmem -p c8100000 -s 4 -r >> >> Operation:read to Addr: 0xc8100000 Value: 0 >> >> ffffffff ffffffff ffffffff ffffffff >> >> >> >> The PCIe analyzer sitting between host bridge (0:3.0 and 02:00.0) and >> >> external switches/devices does not capture any mem transactions. >> >> However, config transactions are visible in the trace. >> > >> > 00:03.0 is not a host bridge; it's a Root Port, which logically >> > connects two PCI buses. A host bridge logically connects a processor >> > bus to a PCI bus. >> > >> >> root@xlink-target:~# cat /proc/iomem >> >> 00000000-00000fff : reserved >> >> 00001000-0009abff : System RAM >> >> 0009ac00-0009ffff : reserved >> >> 000c0000-000c7fff : Video ROM >> >> 000c8000-000c8fff : Adapter ROM >> >> 000e0000-000fffff : reserved >> >> 000f0000-000fffff : System ROM >> >> 00100000-78f36fff : System RAM >> >> 01000000-01828f14 : Kernel code >> >> 01828f15-01f3fb3f : Kernel data >> >> 020bb000-021fdfff : Kernel bss >> >> 30000000-37ffffff : Crash kernel >> >> 78f37000-79839fff : reserved >> >> 79116018-79116018 : APEI ERST >> >> 7911601c-79116021 : APEI ERST >> >> 79116028-79116039 : APEI ERST >> >> 79116040-7911604c : APEI ERST >> >> 79116050-7911804f : APEI ERST >> >> 7983a000-79d4afff : ACPI Non-volatile Storage >> >> 79d4b000-8fffffff : reserved >> >> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff] >> >> c6000000-c70fffff : PCI Bus 0000:2f >> >> c6000000-c70fffff : PCI Bus 0000:30 >> >> c6000000-c6ffffff : 0000:30:00.0 >> >> c7000000-c701ffff : 0000:30:00.0 >> >> c7100000-c71fffff : PCI Bus 0000:31 >> >> c7100000-c711ffff : 0000:31:00.1 >> >> c7100000-c711ffff : igb >> >> c7120000-c713ffff : 0000:31:00.0 >> >> c7120000-c713ffff : igb >> >> c7140000-c7143fff : 0000:31:00.1 >> >> c7140000-c7143fff : igb >> >> c7144000-c7147fff : 0000:31:00.0 >> >> c7144000-c7147fff : igb >> >> c7300000-c730ffff : 0000:00:14.0 >> >> c7300000-c730ffff : xhci-hcd >> >> c7310000-c73107ff : 0000:00:1f.2 >> >> c7310000-c73107ff : ahci >> >> c7311000-c73113ff : 0000:00:1d.0 >> >> c7311000-c73113ff : ehci_hcd >> >> c7312000-c73123ff : 0000:00:1a.0 >> >> c7312000-c73123ff : ehci_hcd >> >> c7314000-c731400f : 0000:00:16.1 >> >> c7315000-c731500f : 0000:00:16.0 >> >> c7316000-c73167ff : 0000:00:11.4 >> >> c7316000-c73167ff : ahci >> >> c7317000-c7317fff : 0000:00:05.4 >> >> c7ffc000-c7ffcfff : dmar1 >> >> c8000000-c8000000 : <BAD> >> >> c8000001-c8000001 : <BAD> >> >> c8000002-c8000002 : <BAD> >> >> c8000003-c8000003 : <BAD> >> > >> > Interesting, what are these from? >> > >> >> c8100000-c91fffff : PCI Bus 0000:02 >> >> c8100000-c90fffff : PCI Bus 0000:03 >> >> c8100000-c90fffff : PCI Bus 0000:04 >> >> c8100000-c82fffff : PCI Bus 0000:05 >> >> c8100000-c81fffff : PCI Bus 0000:06 >> >> c8100000-c8100fff : 0000:06:00.0 >> > >> > You're trying to read the 06:00.0 BAR at 0xc8100000. The windows >> > above look fine, but the top-level host bridge window is missing. >> > >> > The c8100000-c91fffff window is a window through the 00:03.0 Root >> > Port, which connects bus 00 to bus 02. The 00:03.0 bridge forwards >> > memory transactions in the c8100000-c91fffff range from its primary >> > bus (bus 00) to its secondary bus (bus 02). >> > >> > But we don't know how those transactions get to bus 00 in the first >> > place. The host bridge is the connection between the processor and >> > bus 00, and it should have a window that encloses c8100000-c91fffff. >> > >> > Your dmesg log should have information about this bridge, e.g., >> > something like this: >> > >> > ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7f]) >> > PCI host bridge to bus 0000:00 >> > pci_bus 0000:00: root bus resource [mem 0xc8100000-0xc9ffffff window] >> > >> > and there should be a line in /proc/iomem something like this: >> > >> > c8100000-c9ffffff : PCI Bus 0000:00 >> > >> > Bjorn > >> 00000000-00000fff : reserved >> 00001000-0009abff : System RAM >> 0009ac00-0009ffff : reserved >> 000c0000-000c7fff : Video ROM >> 000c8000-000c8fff : Adapter ROM >> 000e0000-000fffff : reserved >> 000f0000-000fffff : System ROM >> 00100000-78f36fff : System RAM >> 01000000-01828f14 : Kernel code >> 01828f15-01f3fb3f : Kernel data >> 020bb000-021fdfff : Kernel bss >> 30000000-37ffffff : Crash kernel >> 78f37000-79839fff : reserved >> 79116018-79116018 : APEI ERST >> 7911601c-79116021 : APEI ERST >> 79116028-79116039 : APEI ERST >> 79116040-7911604c : APEI ERST >> 79116050-7911804f : APEI ERST >> 7983a000-79d4afff : ACPI Non-volatile Storage >> 79d4b000-8fffffff : reserved >> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff] >> c6000000-c70fffff : PCI Bus 0000:2f >> c6000000-c70fffff : PCI Bus 0000:30 >> c6000000-c6ffffff : 0000:30:00.0 >> c7000000-c701ffff : 0000:30:00.0 >> c7100000-c71fffff : PCI Bus 0000:31 >> c7100000-c711ffff : 0000:31:00.1 >> c7100000-c711ffff : igb >> c7120000-c713ffff : 0000:31:00.0 >> c7120000-c713ffff : igb >> c7140000-c7143fff : 0000:31:00.1 >> c7140000-c7143fff : igb >> c7144000-c7147fff : 0000:31:00.0 >> c7144000-c7147fff : igb >> c7300000-c730ffff : 0000:00:14.0 >> c7300000-c730ffff : xhci-hcd >> c7310000-c73107ff : 0000:00:1f.2 >> c7310000-c73107ff : ahci >> c7311000-c73113ff : 0000:00:1d.0 >> c7311000-c73113ff : ehci_hcd >> c7312000-c73123ff : 0000:00:1a.0 >> c7312000-c73123ff : ehci_hcd >> c7314000-c731400f : 0000:00:16.1 >> c7315000-c731500f : 0000:00:16.0 >> c7316000-c73167ff : 0000:00:11.4 >> c7316000-c73167ff : ahci >> c7317000-c7317fff : 0000:00:05.4 >> c7ffc000-c7ffcfff : dmar1 >> c8000000-c8000000 : <BAD> >> c8000001-c8000001 : <BAD> >> c8000002-c8000002 : <BAD> >> c8000003-c8000003 : <BAD> >> c8100000-c91fffff : PCI Bus 0000:02 >> c8100000-c90fffff : PCI Bus 0000:03 >> c8100000-c90fffff : PCI Bus 0000:04 >> c8100000-c82fffff : PCI Bus 0000:05 >> c8100000-c81fffff : PCI Bus 0000:06 >> c8100000-c8100fff : 0000:06:00.0 >> c8200000-c82fffff : PCI Bus 0000:0a >> c8200000-c8203fff : 0000:0a:00.0 >> c9100000-c913ffff : 0000:02:00.0 >> fb000000-fb8fffff : PCI Bus 0000:81 >> fb000000-fb8fffff : PCI Bus 0000:82 >> fb000000-fb7fffff : PCI Bus 0000:8c >> fb000000-fb7fffff : 0000:8c:00.0 >> fb000000-fb7fffff : Plx_Mgr >> fb800000-fb8fffff : PCI Bus 0000:87 >> fb800000-fb800fff : 0000:87:00.0 >> fb900000-fb900fff : 0000:80:05.4 >> fbffc000-fbffcfff : dmar0 >> fec00000-fecfffff : PNP0003:00 >> fec00000-fec003ff : IOAPIC 0 >> fec01000-fec013ff : IOAPIC 1 >> fec40000-fec403ff : IOAPIC 2 >> fed00000-fed003ff : HPET 0 >> fed00000-fed003ff : PNP0103:00 >> fed12000-fed1200f : pnp 00:01 >> fed12010-fed1201f : pnp 00:01 >> fed1b000-fed1bfff : pnp 00:01 >> fed1c000-fed44fff : reserved >> fed1c000-fed3ffff : pnp 00:01 >> fed1f410-fed1f414 : iTCO_wdt.0.auto >> fed45000-fed8bfff : pnp 00:01 >> fee00000-feefffff : pnp 00:01 >> fee00000-fee00fff : Local APIC >> ff000000-ffffffff : reserved >> ff000000-ffffffff : pnp 00:01 >> 100000000-47fffffff : System RAM >> 480000000-4807fffff : PCI Bus 0000:02 >> 480000000-4807fffff : PCI Bus 0000:03 >> 480000000-4807fffff : PCI Bus 0000:04 >> 480000000-4800fffff : PCI Bus 0000:05 >> 480000000-4800fffff : PCI Bus 0000:06 >> 480000000-480003fff : 0000:06:00.0 >> 383fffe00000-383fffefffff : PCI Bus 0000:31 >> 383fffe00000-383fffe1ffff : 0000:31:00.1 >> 383fffe20000-383fffe3ffff : 0000:31:00.1 >> 383fffe40000-383fffe5ffff : 0000:31:00.0 >> 383fffe60000-383fffe7ffff : 0000:31:00.0 >> 383ffff00000-383ffff03fff : 0000:00:04.7 >> 383ffff00000-383ffff03fff : ioatdma >> 383ffff04000-383ffff07fff : 0000:00:04.6 >> 383ffff04000-383ffff07fff : ioatdma >> 383ffff08000-383ffff0bfff : 0000:00:04.5 >> 383ffff08000-383ffff0bfff : ioatdma >> 383ffff0c000-383ffff0ffff : 0000:00:04.4 >> 383ffff0c000-383ffff0ffff : ioatdma >> 383ffff10000-383ffff13fff : 0000:00:04.3 >> 383ffff10000-383ffff13fff : ioatdma >> 383ffff14000-383ffff17fff : 0000:00:04.2 >> 383ffff14000-383ffff17fff : ioatdma >> 383ffff18000-383ffff1bfff : 0000:00:04.1 >> 383ffff18000-383ffff1bfff : ioatdma >> 383ffff1c000-383ffff1ffff : 0000:00:04.0 >> 383ffff1c000-383ffff1ffff : ioatdma >> 383ffff21000-383ffff210ff : 0000:00:1f.3 >> 387fffe00000-387fffefffff : PCI Bus 0000:81 >> 387fffe00000-387fffefffff : PCI Bus 0000:82 >> 387fffe00000-387fffefffff : PCI Bus 0000:87 >> 387fffe00000-387fffe03fff : 0000:87:00.0 >> 387ffff00000-387ffff03fff : 0000:80:04.7 >> 387ffff00000-387ffff03fff : ioatdma >> 387ffff04000-387ffff07fff : 0000:80:04.6 >> 387ffff04000-387ffff07fff : ioatdma >> 387ffff08000-387ffff0bfff : 0000:80:04.5 >> 387ffff08000-387ffff0bfff : ioatdma >> 387ffff0c000-387ffff0ffff : 0000:80:04.4 >> 387ffff0c000-387ffff0ffff : ioatdma >> 387ffff10000-387ffff13fff : 0000:80:04.3 >> 387ffff10000-387ffff13fff : ioatdma >> 387ffff14000-387ffff17fff : 0000:80:04.2 >> 387ffff14000-387ffff17fff : ioatdma >> 387ffff18000-387ffff1bfff : 0000:80:04.1 >> 387ffff18000-387ffff1bfff : ioatdma >> 387ffff1c000-387ffff1ffff : 0000:80:04.0 >> 387ffff1c000-387ffff1ffff : ioatdma > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Fwd: PCIe root port programming 2016-08-15 20:43 ` Kallol Biswas @ 2016-08-15 21:07 ` Bjorn Helgaas 2016-08-15 21:51 ` Kallol Biswas 0 siblings, 1 reply; 8+ messages in thread From: Bjorn Helgaas @ 2016-08-15 21:07 UTC (permalink / raw) To: Kallol Biswas; +Cc: linux-pci On Mon, Aug 15, 2016 at 01:43:03PM -0700, Kallol Biswas wrote: > The kernel source code is 4.4. I have not done any modification except > pre-allocated bus numbers and memory window for a DS port to allow > successful hot plug operation of a PCIe port. > > 0.611919] acpi PNP0A08:00: host bridge window [io 0x0cf8-0x0cff] (ignored) > [ 0.611921] acpi PNP0A08:00: host bridge window [io 0x0000-0x0cf7 > window] (ignored) > [ 0.611922] acpi PNP0A08:00: host bridge window [io 0x1000-0x7fff > window] (ignored) > [ 0.611924] acpi PNP0A08:00: host bridge window [mem > 0x000a0000-0x000bffff window] (ignored) > [ 0.611926] acpi PNP0A08:00: host bridge window [mem > 0xfedb0000-0xfedb000f window] (ignored) > [ 0.611928] acpi PNP0A08:00: host bridge window [mem > 0xfedc0000-0xfedc000f window] (ignored) > [ 0.611929] acpi PNP0A08:00: host bridge window [mem > 0x90000000-0xc7ffbfff window] (ignored) > [ 0.611931] acpi PNP0A08:00: host bridge window [mem > 0x380000000000-0x383fffffffff window] (ignored) > [ 0.611932] PCI: root bus 00: using default resources > [ 0.612217] PCI host bridge to bus 0000:00 > [ 0.612219] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] > [ 0.612220] pci_bus 0000:00: root bus resource [mem > 0x00000000-0x3fffffffffff] > [ 0.612222] pci_bus 0000:00: root bus resource [bus 00-7e] > > kernel command line: > > BOOT_IMAGE=/boot/vmlinuz-4.4.8-kb > root=UUID=c6f326f6-7cc4-49f5-a3be-c511566165b4 ro > pci=realloc,nocrs,assign-busses pcie_ports=native > pciehp.pciehp_debug=1 pci=hpmemsize=0M console=ttyS1,115200n8 quiet > splash crashkernel=384M-:128M vt.handoff=7 You're booting with "pci=nocrs", which tells Linux to ignore the ACPI information about host bridge windows. You can't expect resource allocation and hotplug to work correctly when you're ignoring that. > On Mon, Aug 15, 2016 at 12:06 PM, Bjorn Helgaas <helgaas@kernel.org> wrote: > > On Mon, Aug 15, 2016 at 11:39:56AM -0700, Kallol Biswas wrote: > >> Thank you for the explanation. > >> > >> > >> Here is the relevant dmesg content: > >> > >> [ 0.790457] PCI: root bus ff: using default resources > >> [ 0.790491] PCI host bridge to bus 0000:ff > >> [ 0.790493] pci_bus 0000:ff: root bus resource [io 0x0000-0xffff] > >> [ 0.790495] pci_bus 0000:ff: root bus resource [mem > >> 0x00000000-0x3fffffffffff] > >> [ 0.790497] pci_bus 0000:ff: root bus resource [bus ff] > >> > >> 0.795379] PCI host bridge to bus 0000:7f > >> [ 0.795381] pci_bus 0000:7f: root bus resource [io 0x0000-0xffff] > >> [ 0.795383] pci_bus 0000:7f: root bus resource [mem > >> 0x00000000-0x3fffffffffff] > >> [ 0.795384] pci_bus 0000:7f: root bus resource [bus 7f] > >> > >> 0.809149] PCI host bridge to bus 0000:00 > >> [ 0.809151] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] > >> [ 0.809153] pci_bus 0000:00: root bus resource [mem > >> 0x00000000-0x3fffffffffff] > >> [ 0.809155] pci_bus 0000:00: root bus resource [bus 00-7e] > > > > This is the host bridge that claims to lead to bus 0000:00. > > > > But the I/O and memory resources of all these bridges are wrong. The > > ranges they show are CPU addresses, not PCI bus addresses, so they > > cannot overlap. > > > > It could be that you're using a non-standard host bridge driver that > > did not apply the translation between CPU addresses and PCI bus > > addresses. In ACPI, this is the _TRA property. > > > >> [ 0.852071] PCI host bridge to bus 0000:80 > >> [ 0.852073] pci_bus 0000:80: root bus resource [io 0x0000-0xffff] > >> [ 0.852075] pci_bus 0000:80: root bus resource [mem > >> 0x00000000-0x3fffffffffff] > >> [ 0.852077] pci_bus 0000:80: root bus resource [bus 80-fe] > >> > >> Following lines are from output of /proc/iomem > >> > >> c8000000-c8000000 : <BAD> > >> c8000001-c8000001 : <BAD> > >> c8000002-c8000002 : <BAD> > >> c8000003-c8000003 : <BAD> > >> c8100000-c91fffff : PCI Bus 0000:02 > >> c8100000-c90fffff : PCI Bus 0000:03 > >> c8100000-c90fffff : PCI Bus 0000:04 > >> c8100000-c82fffff : PCI Bus 0000:05 > >> c8100000-c81fffff : PCI Bus 0000:06 > >> c8100000-c8100fff : 0000:06:00.0 > >> c8200000-c82fffff : PCI Bus 0000:0a > >> c8200000-c8203fff : 0000:0a:00.0 > >> c9100000-c913ffff : 0000:02:00.0 > >> fb000000-fb8fffff : PCI Bus 0000:81 > >> fb000000-fb8fffff : PCI Bus 0000:82 > >> fb000000-fb7fffff : PCI Bus 0000:8c > >> fb000000-fb7fffff : 0000:8c:00.0 > >> fb000000-fb7fffff : Plx_Mgr > >> fb800000-fb8fffff : PCI Bus 0000:87 > >> fb800000-fb800fff : 0000:87:00.0 > >> fb900000-fb900fff : 0000:80:05.4 > >> > >> > >> I can read the device BAR via fb800000-fb800fff : 0000:87:00.0. This > >> device is available after reboot, and initial PCIe bus enumeration > >> time. > >> > >> The device c8100000-c8100fff : 0000:06:00.0 shows up after a hotplug > >> operation is done. > >> > >> Attached is the complete /proc/iomem and do not find the line > >> c8100000-c9ffffff : PCI Bus 0000:00, but does it exist for > >> PCI Bus 0000:80? BAR access works fine through this bus. > >> > >> > >> > >> On Mon, Aug 15, 2016 at 10:24 AM, Bjorn Helgaas <helgaas@kernel.org> wrote: > >> > Hi Kallol, > >> > > >> > Sorry I didn't respond to this earlier; I was on vacation for most of > >> > last week. > >> > > >> > On Fri, Aug 12, 2016 at 02:14:59PM -0700, Kallol Biswas wrote: > >> >> I am trying to debug an issue that the PCIe memory > >> >> transactions are not routed through a host bridge. > >> >> > >> >> The config transactions work fine. The primary, secondary and > >> >> subordinate bus numbers look fine for the PCIe tree under the host > >> >> bridge. The memory windows are programmed fine. > >> >> > >> >> Can someone point me to the code that programs the chipset/root port > >> >> registers those direct processor transactions to a specific host > >> >> bridge? > >> > > >> > It looks like you have an x86 system with ACPI. In that case, we use > >> > the ACPI pci_root.c host bridge driver. The chipset configuration for > >> > routing processor memory transactions to host bridges is done by the > >> > firmware, and the kernel doesn't touch it. There should be > >> > information in the chipset spec about how to do this, but there's no > >> > code in Linux itself that does it. > >> > > >> >> Cat of /proc/mem is copied below. > >> >> > >> >> /root/physmem -p c8100000 -s 4 -r > >> >> Operation:read to Addr: 0xc8100000 Value: 0 > >> >> ffffffff ffffffff ffffffff ffffffff > >> >> > >> >> The PCIe analyzer sitting between host bridge (0:3.0 and 02:00.0) and > >> >> external switches/devices does not capture any mem transactions. > >> >> However, config transactions are visible in the trace. > >> > > >> > 00:03.0 is not a host bridge; it's a Root Port, which logically > >> > connects two PCI buses. A host bridge logically connects a processor > >> > bus to a PCI bus. > >> > > >> >> root@xlink-target:~# cat /proc/iomem > >> >> 00000000-00000fff : reserved > >> >> 00001000-0009abff : System RAM > >> >> 0009ac00-0009ffff : reserved > >> >> 000c0000-000c7fff : Video ROM > >> >> 000c8000-000c8fff : Adapter ROM > >> >> 000e0000-000fffff : reserved > >> >> 000f0000-000fffff : System ROM > >> >> 00100000-78f36fff : System RAM > >> >> 01000000-01828f14 : Kernel code > >> >> 01828f15-01f3fb3f : Kernel data > >> >> 020bb000-021fdfff : Kernel bss > >> >> 30000000-37ffffff : Crash kernel > >> >> 78f37000-79839fff : reserved > >> >> 79116018-79116018 : APEI ERST > >> >> 7911601c-79116021 : APEI ERST > >> >> 79116028-79116039 : APEI ERST > >> >> 79116040-7911604c : APEI ERST > >> >> 79116050-7911804f : APEI ERST > >> >> 7983a000-79d4afff : ACPI Non-volatile Storage > >> >> 79d4b000-8fffffff : reserved > >> >> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff] > >> >> c6000000-c70fffff : PCI Bus 0000:2f > >> >> c6000000-c70fffff : PCI Bus 0000:30 > >> >> c6000000-c6ffffff : 0000:30:00.0 > >> >> c7000000-c701ffff : 0000:30:00.0 > >> >> c7100000-c71fffff : PCI Bus 0000:31 > >> >> c7100000-c711ffff : 0000:31:00.1 > >> >> c7100000-c711ffff : igb > >> >> c7120000-c713ffff : 0000:31:00.0 > >> >> c7120000-c713ffff : igb > >> >> c7140000-c7143fff : 0000:31:00.1 > >> >> c7140000-c7143fff : igb > >> >> c7144000-c7147fff : 0000:31:00.0 > >> >> c7144000-c7147fff : igb > >> >> c7300000-c730ffff : 0000:00:14.0 > >> >> c7300000-c730ffff : xhci-hcd > >> >> c7310000-c73107ff : 0000:00:1f.2 > >> >> c7310000-c73107ff : ahci > >> >> c7311000-c73113ff : 0000:00:1d.0 > >> >> c7311000-c73113ff : ehci_hcd > >> >> c7312000-c73123ff : 0000:00:1a.0 > >> >> c7312000-c73123ff : ehci_hcd > >> >> c7314000-c731400f : 0000:00:16.1 > >> >> c7315000-c731500f : 0000:00:16.0 > >> >> c7316000-c73167ff : 0000:00:11.4 > >> >> c7316000-c73167ff : ahci > >> >> c7317000-c7317fff : 0000:00:05.4 > >> >> c7ffc000-c7ffcfff : dmar1 > >> >> c8000000-c8000000 : <BAD> > >> >> c8000001-c8000001 : <BAD> > >> >> c8000002-c8000002 : <BAD> > >> >> c8000003-c8000003 : <BAD> > >> > > >> > Interesting, what are these from? > >> > > >> >> c8100000-c91fffff : PCI Bus 0000:02 > >> >> c8100000-c90fffff : PCI Bus 0000:03 > >> >> c8100000-c90fffff : PCI Bus 0000:04 > >> >> c8100000-c82fffff : PCI Bus 0000:05 > >> >> c8100000-c81fffff : PCI Bus 0000:06 > >> >> c8100000-c8100fff : 0000:06:00.0 > >> > > >> > You're trying to read the 06:00.0 BAR at 0xc8100000. The windows > >> > above look fine, but the top-level host bridge window is missing. > >> > > >> > The c8100000-c91fffff window is a window through the 00:03.0 Root > >> > Port, which connects bus 00 to bus 02. The 00:03.0 bridge forwards > >> > memory transactions in the c8100000-c91fffff range from its primary > >> > bus (bus 00) to its secondary bus (bus 02). > >> > > >> > But we don't know how those transactions get to bus 00 in the first > >> > place. The host bridge is the connection between the processor and > >> > bus 00, and it should have a window that encloses c8100000-c91fffff. > >> > > >> > Your dmesg log should have information about this bridge, e.g., > >> > something like this: > >> > > >> > ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7f]) > >> > PCI host bridge to bus 0000:00 > >> > pci_bus 0000:00: root bus resource [mem 0xc8100000-0xc9ffffff window] > >> > > >> > and there should be a line in /proc/iomem something like this: > >> > > >> > c8100000-c9ffffff : PCI Bus 0000:00 > >> > > >> > Bjorn > > > >> 00000000-00000fff : reserved > >> 00001000-0009abff : System RAM > >> 0009ac00-0009ffff : reserved > >> 000c0000-000c7fff : Video ROM > >> 000c8000-000c8fff : Adapter ROM > >> 000e0000-000fffff : reserved > >> 000f0000-000fffff : System ROM > >> 00100000-78f36fff : System RAM > >> 01000000-01828f14 : Kernel code > >> 01828f15-01f3fb3f : Kernel data > >> 020bb000-021fdfff : Kernel bss > >> 30000000-37ffffff : Crash kernel > >> 78f37000-79839fff : reserved > >> 79116018-79116018 : APEI ERST > >> 7911601c-79116021 : APEI ERST > >> 79116028-79116039 : APEI ERST > >> 79116040-7911604c : APEI ERST > >> 79116050-7911804f : APEI ERST > >> 7983a000-79d4afff : ACPI Non-volatile Storage > >> 79d4b000-8fffffff : reserved > >> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff] > >> c6000000-c70fffff : PCI Bus 0000:2f > >> c6000000-c70fffff : PCI Bus 0000:30 > >> c6000000-c6ffffff : 0000:30:00.0 > >> c7000000-c701ffff : 0000:30:00.0 > >> c7100000-c71fffff : PCI Bus 0000:31 > >> c7100000-c711ffff : 0000:31:00.1 > >> c7100000-c711ffff : igb > >> c7120000-c713ffff : 0000:31:00.0 > >> c7120000-c713ffff : igb > >> c7140000-c7143fff : 0000:31:00.1 > >> c7140000-c7143fff : igb > >> c7144000-c7147fff : 0000:31:00.0 > >> c7144000-c7147fff : igb > >> c7300000-c730ffff : 0000:00:14.0 > >> c7300000-c730ffff : xhci-hcd > >> c7310000-c73107ff : 0000:00:1f.2 > >> c7310000-c73107ff : ahci > >> c7311000-c73113ff : 0000:00:1d.0 > >> c7311000-c73113ff : ehci_hcd > >> c7312000-c73123ff : 0000:00:1a.0 > >> c7312000-c73123ff : ehci_hcd > >> c7314000-c731400f : 0000:00:16.1 > >> c7315000-c731500f : 0000:00:16.0 > >> c7316000-c73167ff : 0000:00:11.4 > >> c7316000-c73167ff : ahci > >> c7317000-c7317fff : 0000:00:05.4 > >> c7ffc000-c7ffcfff : dmar1 > >> c8000000-c8000000 : <BAD> > >> c8000001-c8000001 : <BAD> > >> c8000002-c8000002 : <BAD> > >> c8000003-c8000003 : <BAD> > >> c8100000-c91fffff : PCI Bus 0000:02 > >> c8100000-c90fffff : PCI Bus 0000:03 > >> c8100000-c90fffff : PCI Bus 0000:04 > >> c8100000-c82fffff : PCI Bus 0000:05 > >> c8100000-c81fffff : PCI Bus 0000:06 > >> c8100000-c8100fff : 0000:06:00.0 > >> c8200000-c82fffff : PCI Bus 0000:0a > >> c8200000-c8203fff : 0000:0a:00.0 > >> c9100000-c913ffff : 0000:02:00.0 > >> fb000000-fb8fffff : PCI Bus 0000:81 > >> fb000000-fb8fffff : PCI Bus 0000:82 > >> fb000000-fb7fffff : PCI Bus 0000:8c > >> fb000000-fb7fffff : 0000:8c:00.0 > >> fb000000-fb7fffff : Plx_Mgr > >> fb800000-fb8fffff : PCI Bus 0000:87 > >> fb800000-fb800fff : 0000:87:00.0 > >> fb900000-fb900fff : 0000:80:05.4 > >> fbffc000-fbffcfff : dmar0 > >> fec00000-fecfffff : PNP0003:00 > >> fec00000-fec003ff : IOAPIC 0 > >> fec01000-fec013ff : IOAPIC 1 > >> fec40000-fec403ff : IOAPIC 2 > >> fed00000-fed003ff : HPET 0 > >> fed00000-fed003ff : PNP0103:00 > >> fed12000-fed1200f : pnp 00:01 > >> fed12010-fed1201f : pnp 00:01 > >> fed1b000-fed1bfff : pnp 00:01 > >> fed1c000-fed44fff : reserved > >> fed1c000-fed3ffff : pnp 00:01 > >> fed1f410-fed1f414 : iTCO_wdt.0.auto > >> fed45000-fed8bfff : pnp 00:01 > >> fee00000-feefffff : pnp 00:01 > >> fee00000-fee00fff : Local APIC > >> ff000000-ffffffff : reserved > >> ff000000-ffffffff : pnp 00:01 > >> 100000000-47fffffff : System RAM > >> 480000000-4807fffff : PCI Bus 0000:02 > >> 480000000-4807fffff : PCI Bus 0000:03 > >> 480000000-4807fffff : PCI Bus 0000:04 > >> 480000000-4800fffff : PCI Bus 0000:05 > >> 480000000-4800fffff : PCI Bus 0000:06 > >> 480000000-480003fff : 0000:06:00.0 > >> 383fffe00000-383fffefffff : PCI Bus 0000:31 > >> 383fffe00000-383fffe1ffff : 0000:31:00.1 > >> 383fffe20000-383fffe3ffff : 0000:31:00.1 > >> 383fffe40000-383fffe5ffff : 0000:31:00.0 > >> 383fffe60000-383fffe7ffff : 0000:31:00.0 > >> 383ffff00000-383ffff03fff : 0000:00:04.7 > >> 383ffff00000-383ffff03fff : ioatdma > >> 383ffff04000-383ffff07fff : 0000:00:04.6 > >> 383ffff04000-383ffff07fff : ioatdma > >> 383ffff08000-383ffff0bfff : 0000:00:04.5 > >> 383ffff08000-383ffff0bfff : ioatdma > >> 383ffff0c000-383ffff0ffff : 0000:00:04.4 > >> 383ffff0c000-383ffff0ffff : ioatdma > >> 383ffff10000-383ffff13fff : 0000:00:04.3 > >> 383ffff10000-383ffff13fff : ioatdma > >> 383ffff14000-383ffff17fff : 0000:00:04.2 > >> 383ffff14000-383ffff17fff : ioatdma > >> 383ffff18000-383ffff1bfff : 0000:00:04.1 > >> 383ffff18000-383ffff1bfff : ioatdma > >> 383ffff1c000-383ffff1ffff : 0000:00:04.0 > >> 383ffff1c000-383ffff1ffff : ioatdma > >> 383ffff21000-383ffff210ff : 0000:00:1f.3 > >> 387fffe00000-387fffefffff : PCI Bus 0000:81 > >> 387fffe00000-387fffefffff : PCI Bus 0000:82 > >> 387fffe00000-387fffefffff : PCI Bus 0000:87 > >> 387fffe00000-387fffe03fff : 0000:87:00.0 > >> 387ffff00000-387ffff03fff : 0000:80:04.7 > >> 387ffff00000-387ffff03fff : ioatdma > >> 387ffff04000-387ffff07fff : 0000:80:04.6 > >> 387ffff04000-387ffff07fff : ioatdma > >> 387ffff08000-387ffff0bfff : 0000:80:04.5 > >> 387ffff08000-387ffff0bfff : ioatdma > >> 387ffff0c000-387ffff0ffff : 0000:80:04.4 > >> 387ffff0c000-387ffff0ffff : ioatdma > >> 387ffff10000-387ffff13fff : 0000:80:04.3 > >> 387ffff10000-387ffff13fff : ioatdma > >> 387ffff14000-387ffff17fff : 0000:80:04.2 > >> 387ffff14000-387ffff17fff : ioatdma > >> 387ffff18000-387ffff1bfff : 0000:80:04.1 > >> 387ffff18000-387ffff1bfff : ioatdma > >> 387ffff1c000-387ffff1ffff : 0000:80:04.0 > >> 387ffff1c000-387ffff1ffff : ioatdma > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Fwd: PCIe root port programming 2016-08-15 21:07 ` Bjorn Helgaas @ 2016-08-15 21:51 ` Kallol Biswas 2016-08-15 21:56 ` Kallol Biswas 0 siblings, 1 reply; 8+ messages in thread From: Kallol Biswas @ 2016-08-15 21:51 UTC (permalink / raw) To: Bjorn Helgaas; +Cc: linux-pci [-- Attachment #1: Type: text/plain, Size: 17899 bytes --] Hello Bjorn, Now memory access works: The following window has been setup properly: *90000000-c7ffbfff : PCI Bus 0000:00* 9d4b000-8fffffff : reserved 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff] 90000000-c7ffbfff : PCI Bus 0000:00 90000000-910fffff : PCI Bus 0000:02 90000000-90ffffff : PCI Bus 0000:03 90000000-90ffffff : PCI Bus 0000:04 90000000-901fffff : PCI Bus 0000:05 90000000-900fffff : PCI Bus 0000:06 90000000-90000fff : 0000:06:00.0 90000000-90000fff : r8169 90100000-901fffff : PCI Bus 0000:0a 90100000-90103fff : 0000:0a:00.0 91000000-9103ffff : 0000:02:00.0 c6000000-c70fffff : PCI Bus 0000:2f Thank you very much. nocrs was creating the problem. Just FYI, my setup is a regular supermicro PC, running Ubuntu 16.04 and my modified 4.4 kernel. I am trying to bring up devices via a PCIe fabric path. On Mon, Aug 15, 2016 at 2:07 PM, Bjorn Helgaas <helgaas@kernel.org> wrote: > On Mon, Aug 15, 2016 at 01:43:03PM -0700, Kallol Biswas wrote: > > The kernel source code is 4.4. I have not done any modification except > > pre-allocated bus numbers and memory window for a DS port to allow > > successful hot plug operation of a PCIe port. > > > > 0.611919] acpi PNP0A08:00: host bridge window [io 0x0cf8-0x0cff] > (ignored) > > [ 0.611921] acpi PNP0A08:00: host bridge window [io 0x0000-0x0cf7 > > window] (ignored) > > [ 0.611922] acpi PNP0A08:00: host bridge window [io 0x1000-0x7fff > > window] (ignored) > > [ 0.611924] acpi PNP0A08:00: host bridge window [mem > > 0x000a0000-0x000bffff window] (ignored) > > [ 0.611926] acpi PNP0A08:00: host bridge window [mem > > 0xfedb0000-0xfedb000f window] (ignored) > > [ 0.611928] acpi PNP0A08:00: host bridge window [mem > > 0xfedc0000-0xfedc000f window] (ignored) > > [ 0.611929] acpi PNP0A08:00: host bridge window [mem > > 0x90000000-0xc7ffbfff window] (ignored) > > [ 0.611931] acpi PNP0A08:00: host bridge window [mem > > 0x380000000000-0x383fffffffff window] (ignored) > > [ 0.611932] PCI: root bus 00: using default resources > > [ 0.612217] PCI host bridge to bus 0000:00 > > [ 0.612219] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] > > [ 0.612220] pci_bus 0000:00: root bus resource [mem > > 0x00000000-0x3fffffffffff] > > [ 0.612222] pci_bus 0000:00: root bus resource [bus 00-7e] > > > > kernel command line: > > > > BOOT_IMAGE=/boot/vmlinuz-4.4.8-kb > > root=UUID=c6f326f6-7cc4-49f5-a3be-c511566165b4 ro > > pci=realloc,nocrs,assign-busses pcie_ports=native > > pciehp.pciehp_debug=1 pci=hpmemsize=0M console=ttyS1,115200n8 quiet > > splash crashkernel=384M-:128M vt.handoff=7 > > You're booting with "pci=nocrs", which tells Linux to ignore the ACPI > information about host bridge windows. You can't expect resource > allocation and hotplug to work correctly when you're ignoring that. > > > On Mon, Aug 15, 2016 at 12:06 PM, Bjorn Helgaas <helgaas@kernel.org> > wrote: > > > On Mon, Aug 15, 2016 at 11:39:56AM -0700, Kallol Biswas wrote: > > >> Thank you for the explanation. > > >> > > >> > > >> Here is the relevant dmesg content: > > >> > > >> [ 0.790457] PCI: root bus ff: using default resources > > >> [ 0.790491] PCI host bridge to bus 0000:ff > > >> [ 0.790493] pci_bus 0000:ff: root bus resource [io 0x0000-0xffff] > > >> [ 0.790495] pci_bus 0000:ff: root bus resource [mem > > >> 0x00000000-0x3fffffffffff] > > >> [ 0.790497] pci_bus 0000:ff: root bus resource [bus ff] > > >> > > >> 0.795379] PCI host bridge to bus 0000:7f > > >> [ 0.795381] pci_bus 0000:7f: root bus resource [io 0x0000-0xffff] > > >> [ 0.795383] pci_bus 0000:7f: root bus resource [mem > > >> 0x00000000-0x3fffffffffff] > > >> [ 0.795384] pci_bus 0000:7f: root bus resource [bus 7f] > > >> > > >> 0.809149] PCI host bridge to bus 0000:00 > > >> [ 0.809151] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] > > >> [ 0.809153] pci_bus 0000:00: root bus resource [mem > > >> 0x00000000-0x3fffffffffff] > > >> [ 0.809155] pci_bus 0000:00: root bus resource [bus 00-7e] > > > > > > This is the host bridge that claims to lead to bus 0000:00. > > > > > > But the I/O and memory resources of all these bridges are wrong. The > > > ranges they show are CPU addresses, not PCI bus addresses, so they > > > cannot overlap. > > > > > > It could be that you're using a non-standard host bridge driver that > > > did not apply the translation between CPU addresses and PCI bus > > > addresses. In ACPI, this is the _TRA property. > > > > > >> [ 0.852071] PCI host bridge to bus 0000:80 > > >> [ 0.852073] pci_bus 0000:80: root bus resource [io 0x0000-0xffff] > > >> [ 0.852075] pci_bus 0000:80: root bus resource [mem > > >> 0x00000000-0x3fffffffffff] > > >> [ 0.852077] pci_bus 0000:80: root bus resource [bus 80-fe] > > >> > > >> Following lines are from output of /proc/iomem > > >> > > >> c8000000-c8000000 : <BAD> > > >> c8000001-c8000001 : <BAD> > > >> c8000002-c8000002 : <BAD> > > >> c8000003-c8000003 : <BAD> > > >> c8100000-c91fffff : PCI Bus 0000:02 > > >> c8100000-c90fffff : PCI Bus 0000:03 > > >> c8100000-c90fffff : PCI Bus 0000:04 > > >> c8100000-c82fffff : PCI Bus 0000:05 > > >> c8100000-c81fffff : PCI Bus 0000:06 > > >> c8100000-c8100fff : 0000:06:00.0 > > >> c8200000-c82fffff : PCI Bus 0000:0a > > >> c8200000-c8203fff : 0000:0a:00.0 > > >> c9100000-c913ffff : 0000:02:00.0 > > >> fb000000-fb8fffff : PCI Bus 0000:81 > > >> fb000000-fb8fffff : PCI Bus 0000:82 > > >> fb000000-fb7fffff : PCI Bus 0000:8c > > >> fb000000-fb7fffff : 0000:8c:00.0 > > >> fb000000-fb7fffff : Plx_Mgr > > >> fb800000-fb8fffff : PCI Bus 0000:87 > > >> fb800000-fb800fff : 0000:87:00.0 > > >> fb900000-fb900fff : 0000:80:05.4 > > >> > > >> > > >> I can read the device BAR via fb800000-fb800fff : 0000:87:00.0. This > > >> device is available after reboot, and initial PCIe bus enumeration > > >> time. > > >> > > >> The device c8100000-c8100fff : 0000:06:00.0 shows up after a hotplug > > >> operation is done. > > >> > > >> Attached is the complete /proc/iomem and do not find the line > > >> c8100000-c9ffffff : PCI Bus 0000:00, but does it exist for > > >> PCI Bus 0000:80? BAR access works fine through this bus. > > >> > > >> > > >> > > >> On Mon, Aug 15, 2016 at 10:24 AM, Bjorn Helgaas <helgaas@kernel.org> > wrote: > > >> > Hi Kallol, > > >> > > > >> > Sorry I didn't respond to this earlier; I was on vacation for most > of > > >> > last week. > > >> > > > >> > On Fri, Aug 12, 2016 at 02:14:59PM -0700, Kallol Biswas wrote: > > >> >> I am trying to debug an issue that the PCIe memory > > >> >> transactions are not routed through a host bridge. > > >> >> > > >> >> The config transactions work fine. The primary, secondary and > > >> >> subordinate bus numbers look fine for the PCIe tree under the host > > >> >> bridge. The memory windows are programmed fine. > > >> >> > > >> >> Can someone point me to the code that programs the chipset/root > port > > >> >> registers those direct processor transactions to a specific host > > >> >> bridge? > > >> > > > >> > It looks like you have an x86 system with ACPI. In that case, we > use > > >> > the ACPI pci_root.c host bridge driver. The chipset configuration > for > > >> > routing processor memory transactions to host bridges is done by the > > >> > firmware, and the kernel doesn't touch it. There should be > > >> > information in the chipset spec about how to do this, but there's no > > >> > code in Linux itself that does it. > > >> > > > >> >> Cat of /proc/mem is copied below. > > >> >> > > >> >> /root/physmem -p c8100000 -s 4 -r > > >> >> Operation:read to Addr: 0xc8100000 Value: 0 > > >> >> ffffffff ffffffff ffffffff ffffffff > > >> >> > > >> >> The PCIe analyzer sitting between host bridge (0:3.0 and 02:00.0) > and > > >> >> external switches/devices does not capture any mem transactions. > > >> >> However, config transactions are visible in the trace. > > >> > > > >> > 00:03.0 is not a host bridge; it's a Root Port, which logically > > >> > connects two PCI buses. A host bridge logically connects a > processor > > >> > bus to a PCI bus. > > >> > > > >> >> root@xlink-target:~# cat /proc/iomem > > >> >> 00000000-00000fff : reserved > > >> >> 00001000-0009abff : System RAM > > >> >> 0009ac00-0009ffff : reserved > > >> >> 000c0000-000c7fff : Video ROM > > >> >> 000c8000-000c8fff : Adapter ROM > > >> >> 000e0000-000fffff : reserved > > >> >> 000f0000-000fffff : System ROM > > >> >> 00100000-78f36fff : System RAM > > >> >> 01000000-01828f14 : Kernel code > > >> >> 01828f15-01f3fb3f : Kernel data > > >> >> 020bb000-021fdfff : Kernel bss > > >> >> 30000000-37ffffff : Crash kernel > > >> >> 78f37000-79839fff : reserved > > >> >> 79116018-79116018 : APEI ERST > > >> >> 7911601c-79116021 : APEI ERST > > >> >> 79116028-79116039 : APEI ERST > > >> >> 79116040-7911604c : APEI ERST > > >> >> 79116050-7911804f : APEI ERST > > >> >> 7983a000-79d4afff : ACPI Non-volatile Storage > > >> >> 79d4b000-8fffffff : reserved > > >> >> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff] > > >> >> c6000000-c70fffff : PCI Bus 0000:2f > > >> >> c6000000-c70fffff : PCI Bus 0000:30 > > >> >> c6000000-c6ffffff : 0000:30:00.0 > > >> >> c7000000-c701ffff : 0000:30:00.0 > > >> >> c7100000-c71fffff : PCI Bus 0000:31 > > >> >> c7100000-c711ffff : 0000:31:00.1 > > >> >> c7100000-c711ffff : igb > > >> >> c7120000-c713ffff : 0000:31:00.0 > > >> >> c7120000-c713ffff : igb > > >> >> c7140000-c7143fff : 0000:31:00.1 > > >> >> c7140000-c7143fff : igb > > >> >> c7144000-c7147fff : 0000:31:00.0 > > >> >> c7144000-c7147fff : igb > > >> >> c7300000-c730ffff : 0000:00:14.0 > > >> >> c7300000-c730ffff : xhci-hcd > > >> >> c7310000-c73107ff : 0000:00:1f.2 > > >> >> c7310000-c73107ff : ahci > > >> >> c7311000-c73113ff : 0000:00:1d.0 > > >> >> c7311000-c73113ff : ehci_hcd > > >> >> c7312000-c73123ff : 0000:00:1a.0 > > >> >> c7312000-c73123ff : ehci_hcd > > >> >> c7314000-c731400f : 0000:00:16.1 > > >> >> c7315000-c731500f : 0000:00:16.0 > > >> >> c7316000-c73167ff : 0000:00:11.4 > > >> >> c7316000-c73167ff : ahci > > >> >> c7317000-c7317fff : 0000:00:05.4 > > >> >> c7ffc000-c7ffcfff : dmar1 > > >> >> c8000000-c8000000 : <BAD> > > >> >> c8000001-c8000001 : <BAD> > > >> >> c8000002-c8000002 : <BAD> > > >> >> c8000003-c8000003 : <BAD> > > >> > > > >> > Interesting, what are these from? > > >> > > > >> >> c8100000-c91fffff : PCI Bus 0000:02 > > >> >> c8100000-c90fffff : PCI Bus 0000:03 > > >> >> c8100000-c90fffff : PCI Bus 0000:04 > > >> >> c8100000-c82fffff : PCI Bus 0000:05 > > >> >> c8100000-c81fffff : PCI Bus 0000:06 > > >> >> c8100000-c8100fff : 0000:06:00.0 > > >> > > > >> > You're trying to read the 06:00.0 BAR at 0xc8100000. The windows > > >> > above look fine, but the top-level host bridge window is missing. > > >> > > > >> > The c8100000-c91fffff window is a window through the 00:03.0 Root > > >> > Port, which connects bus 00 to bus 02. The 00:03.0 bridge forwards > > >> > memory transactions in the c8100000-c91fffff range from its primary > > >> > bus (bus 00) to its secondary bus (bus 02). > > >> > > > >> > But we don't know how those transactions get to bus 00 in the first > > >> > place. The host bridge is the connection between the processor and > > >> > bus 00, and it should have a window that encloses c8100000-c91fffff. > > >> > > > >> > Your dmesg log should have information about this bridge, e.g., > > >> > something like this: > > >> > > > >> > ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7f]) > > >> > PCI host bridge to bus 0000:00 > > >> > pci_bus 0000:00: root bus resource [mem 0xc8100000-0xc9ffffff > window] > > >> > > > >> > and there should be a line in /proc/iomem something like this: > > >> > > > >> > c8100000-c9ffffff : PCI Bus 0000:00 > > >> > > > >> > Bjorn > > > > > >> 00000000-00000fff : reserved > > >> 00001000-0009abff : System RAM > > >> 0009ac00-0009ffff : reserved > > >> 000c0000-000c7fff : Video ROM > > >> 000c8000-000c8fff : Adapter ROM > > >> 000e0000-000fffff : reserved > > >> 000f0000-000fffff : System ROM > > >> 00100000-78f36fff : System RAM > > >> 01000000-01828f14 : Kernel code > > >> 01828f15-01f3fb3f : Kernel data > > >> 020bb000-021fdfff : Kernel bss > > >> 30000000-37ffffff : Crash kernel > > >> 78f37000-79839fff : reserved > > >> 79116018-79116018 : APEI ERST > > >> 7911601c-79116021 : APEI ERST > > >> 79116028-79116039 : APEI ERST > > >> 79116040-7911604c : APEI ERST > > >> 79116050-7911804f : APEI ERST > > >> 7983a000-79d4afff : ACPI Non-volatile Storage > > >> 79d4b000-8fffffff : reserved > > >> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff] > > >> c6000000-c70fffff : PCI Bus 0000:2f > > >> c6000000-c70fffff : PCI Bus 0000:30 > > >> c6000000-c6ffffff : 0000:30:00.0 > > >> c7000000-c701ffff : 0000:30:00.0 > > >> c7100000-c71fffff : PCI Bus 0000:31 > > >> c7100000-c711ffff : 0000:31:00.1 > > >> c7100000-c711ffff : igb > > >> c7120000-c713ffff : 0000:31:00.0 > > >> c7120000-c713ffff : igb > > >> c7140000-c7143fff : 0000:31:00.1 > > >> c7140000-c7143fff : igb > > >> c7144000-c7147fff : 0000:31:00.0 > > >> c7144000-c7147fff : igb > > >> c7300000-c730ffff : 0000:00:14.0 > > >> c7300000-c730ffff : xhci-hcd > > >> c7310000-c73107ff : 0000:00:1f.2 > > >> c7310000-c73107ff : ahci > > >> c7311000-c73113ff : 0000:00:1d.0 > > >> c7311000-c73113ff : ehci_hcd > > >> c7312000-c73123ff : 0000:00:1a.0 > > >> c7312000-c73123ff : ehci_hcd > > >> c7314000-c731400f : 0000:00:16.1 > > >> c7315000-c731500f : 0000:00:16.0 > > >> c7316000-c73167ff : 0000:00:11.4 > > >> c7316000-c73167ff : ahci > > >> c7317000-c7317fff : 0000:00:05.4 > > >> c7ffc000-c7ffcfff : dmar1 > > >> c8000000-c8000000 : <BAD> > > >> c8000001-c8000001 : <BAD> > > >> c8000002-c8000002 : <BAD> > > >> c8000003-c8000003 : <BAD> > > >> c8100000-c91fffff : PCI Bus 0000:02 > > >> c8100000-c90fffff : PCI Bus 0000:03 > > >> c8100000-c90fffff : PCI Bus 0000:04 > > >> c8100000-c82fffff : PCI Bus 0000:05 > > >> c8100000-c81fffff : PCI Bus 0000:06 > > >> c8100000-c8100fff : 0000:06:00.0 > > >> c8200000-c82fffff : PCI Bus 0000:0a > > >> c8200000-c8203fff : 0000:0a:00.0 > > >> c9100000-c913ffff : 0000:02:00.0 > > >> fb000000-fb8fffff : PCI Bus 0000:81 > > >> fb000000-fb8fffff : PCI Bus 0000:82 > > >> fb000000-fb7fffff : PCI Bus 0000:8c > > >> fb000000-fb7fffff : 0000:8c:00.0 > > >> fb000000-fb7fffff : Plx_Mgr > > >> fb800000-fb8fffff : PCI Bus 0000:87 > > >> fb800000-fb800fff : 0000:87:00.0 > > >> fb900000-fb900fff : 0000:80:05.4 > > >> fbffc000-fbffcfff : dmar0 > > >> fec00000-fecfffff : PNP0003:00 > > >> fec00000-fec003ff : IOAPIC 0 > > >> fec01000-fec013ff : IOAPIC 1 > > >> fec40000-fec403ff : IOAPIC 2 > > >> fed00000-fed003ff : HPET 0 > > >> fed00000-fed003ff : PNP0103:00 > > >> fed12000-fed1200f : pnp 00:01 > > >> fed12010-fed1201f : pnp 00:01 > > >> fed1b000-fed1bfff : pnp 00:01 > > >> fed1c000-fed44fff : reserved > > >> fed1c000-fed3ffff : pnp 00:01 > > >> fed1f410-fed1f414 : iTCO_wdt.0.auto > > >> fed45000-fed8bfff : pnp 00:01 > > >> fee00000-feefffff : pnp 00:01 > > >> fee00000-fee00fff : Local APIC > > >> ff000000-ffffffff : reserved > > >> ff000000-ffffffff : pnp 00:01 > > >> 100000000-47fffffff : System RAM > > >> 480000000-4807fffff : PCI Bus 0000:02 > > >> 480000000-4807fffff : PCI Bus 0000:03 > > >> 480000000-4807fffff : PCI Bus 0000:04 > > >> 480000000-4800fffff : PCI Bus 0000:05 > > >> 480000000-4800fffff : PCI Bus 0000:06 > > >> 480000000-480003fff : 0000:06:00.0 > > >> 383fffe00000-383fffefffff : PCI Bus 0000:31 > > >> 383fffe00000-383fffe1ffff : 0000:31:00.1 > > >> 383fffe20000-383fffe3ffff : 0000:31:00.1 > > >> 383fffe40000-383fffe5ffff : 0000:31:00.0 > > >> 383fffe60000-383fffe7ffff : 0000:31:00.0 > > >> 383ffff00000-383ffff03fff : 0000:00:04.7 > > >> 383ffff00000-383ffff03fff : ioatdma > > >> 383ffff04000-383ffff07fff : 0000:00:04.6 > > >> 383ffff04000-383ffff07fff : ioatdma > > >> 383ffff08000-383ffff0bfff : 0000:00:04.5 > > >> 383ffff08000-383ffff0bfff : ioatdma > > >> 383ffff0c000-383ffff0ffff : 0000:00:04.4 > > >> 383ffff0c000-383ffff0ffff : ioatdma > > >> 383ffff10000-383ffff13fff : 0000:00:04.3 > > >> 383ffff10000-383ffff13fff : ioatdma > > >> 383ffff14000-383ffff17fff : 0000:00:04.2 > > >> 383ffff14000-383ffff17fff : ioatdma > > >> 383ffff18000-383ffff1bfff : 0000:00:04.1 > > >> 383ffff18000-383ffff1bfff : ioatdma > > >> 383ffff1c000-383ffff1ffff : 0000:00:04.0 > > >> 383ffff1c000-383ffff1ffff : ioatdma > > >> 383ffff21000-383ffff210ff : 0000:00:1f.3 > > >> 387fffe00000-387fffefffff : PCI Bus 0000:81 > > >> 387fffe00000-387fffefffff : PCI Bus 0000:82 > > >> 387fffe00000-387fffefffff : PCI Bus 0000:87 > > >> 387fffe00000-387fffe03fff : 0000:87:00.0 > > >> 387ffff00000-387ffff03fff : 0000:80:04.7 > > >> 387ffff00000-387ffff03fff : ioatdma > > >> 387ffff04000-387ffff07fff : 0000:80:04.6 > > >> 387ffff04000-387ffff07fff : ioatdma > > >> 387ffff08000-387ffff0bfff : 0000:80:04.5 > > >> 387ffff08000-387ffff0bfff : ioatdma > > >> 387ffff0c000-387ffff0ffff : 0000:80:04.4 > > >> 387ffff0c000-387ffff0ffff : ioatdma > > >> 387ffff10000-387ffff13fff : 0000:80:04.3 > > >> 387ffff10000-387ffff13fff : ioatdma > > >> 387ffff14000-387ffff17fff : 0000:80:04.2 > > >> 387ffff14000-387ffff17fff : ioatdma > > >> 387ffff18000-387ffff1bfff : 0000:80:04.1 > > >> 387ffff18000-387ffff1bfff : ioatdma > > >> 387ffff1c000-387ffff1ffff : 0000:80:04.0 > > >> 387ffff1c000-387ffff1ffff : ioatdma > > > > [-- Attachment #2: Type: text/html, Size: 23991 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Fwd: PCIe root port programming 2016-08-15 21:51 ` Kallol Biswas @ 2016-08-15 21:56 ` Kallol Biswas 0 siblings, 0 replies; 8+ messages in thread From: Kallol Biswas @ 2016-08-15 21:56 UTC (permalink / raw) To: Bjorn Helgaas; +Cc: linux-pci > Hello Bjorn, > Now memory access works: The following window has been setup properly: > 90000000-c7ffbfff : PCI Bus 0000:00 > > 9d4b000-8fffffff : reserved > 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff] > 90000000-c7ffbfff : PCI Bus 0000:00 > 90000000-910fffff : PCI Bus 0000:02 > 90000000-90ffffff : PCI Bus 0000:03 > 90000000-90ffffff : PCI Bus 0000:04 > 90000000-901fffff : PCI Bus 0000:05 > 90000000-900fffff : PCI Bus 0000:06 > 90000000-90000fff : 0000:06:00.0 > 90000000-90000fff : r8169 > 90100000-901fffff : PCI Bus 0000:0a > 90100000-90103fff : 0000:0a:00.0 > 91000000-9103ffff : 0000:02:00.0 > c6000000-c70fffff : PCI Bus 0000:2f > > Thank you very much. nocrs was creating the problem. > > Just FYI, my setup is a regular supermicro PC, running Ubuntu 16.04 and my modified 4.4 kernel. I am trying to bring up devices via a PCIe fabric path. > > > On Mon, Aug 15, 2016 at 2:07 PM, Bjorn Helgaas <helgaas@kernel.org> wrote: >> >> On Mon, Aug 15, 2016 at 01:43:03PM -0700, Kallol Biswas wrote: >> > The kernel source code is 4.4. I have not done any modification except >> > pre-allocated bus numbers and memory window for a DS port to allow >> > successful hot plug operation of a PCIe port. >> > >> > 0.611919] acpi PNP0A08:00: host bridge window [io 0x0cf8-0x0cff] (ignored) >> > [ 0.611921] acpi PNP0A08:00: host bridge window [io 0x0000-0x0cf7 >> > window] (ignored) >> > [ 0.611922] acpi PNP0A08:00: host bridge window [io 0x1000-0x7fff >> > window] (ignored) >> > [ 0.611924] acpi PNP0A08:00: host bridge window [mem >> > 0x000a0000-0x000bffff window] (ignored) >> > [ 0.611926] acpi PNP0A08:00: host bridge window [mem >> > 0xfedb0000-0xfedb000f window] (ignored) >> > [ 0.611928] acpi PNP0A08:00: host bridge window [mem >> > 0xfedc0000-0xfedc000f window] (ignored) >> > [ 0.611929] acpi PNP0A08:00: host bridge window [mem >> > 0x90000000-0xc7ffbfff window] (ignored) >> > [ 0.611931] acpi PNP0A08:00: host bridge window [mem >> > 0x380000000000-0x383fffffffff window] (ignored) >> > [ 0.611932] PCI: root bus 00: using default resources >> > [ 0.612217] PCI host bridge to bus 0000:00 >> > [ 0.612219] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] >> > [ 0.612220] pci_bus 0000:00: root bus resource [mem >> > 0x00000000-0x3fffffffffff] >> > [ 0.612222] pci_bus 0000:00: root bus resource [bus 00-7e] >> > >> > kernel command line: >> > >> > BOOT_IMAGE=/boot/vmlinuz-4.4.8-kb >> > root=UUID=c6f326f6-7cc4-49f5-a3be-c511566165b4 ro >> > pci=realloc,nocrs,assign-busses pcie_ports=native >> > pciehp.pciehp_debug=1 pci=hpmemsize=0M console=ttyS1,115200n8 quiet >> > splash crashkernel=384M-:128M vt.handoff=7 >> >> You're booting with "pci=nocrs", which tells Linux to ignore the ACPI >> information about host bridge windows. You can't expect resource >> allocation and hotplug to work correctly when you're ignoring that. >> >> > On Mon, Aug 15, 2016 at 12:06 PM, Bjorn Helgaas <helgaas@kernel.org> wrote: >> > > On Mon, Aug 15, 2016 at 11:39:56AM -0700, Kallol Biswas wrote: >> > >> Thank you for the explanation. >> > >> >> > >> >> > >> Here is the relevant dmesg content: >> > >> >> > >> [ 0.790457] PCI: root bus ff: using default resources >> > >> [ 0.790491] PCI host bridge to bus 0000:ff >> > >> [ 0.790493] pci_bus 0000:ff: root bus resource [io 0x0000-0xffff] >> > >> [ 0.790495] pci_bus 0000:ff: root bus resource [mem >> > >> 0x00000000-0x3fffffffffff] >> > >> [ 0.790497] pci_bus 0000:ff: root bus resource [bus ff] >> > >> >> > >> 0.795379] PCI host bridge to bus 0000:7f >> > >> [ 0.795381] pci_bus 0000:7f: root bus resource [io 0x0000-0xffff] >> > >> [ 0.795383] pci_bus 0000:7f: root bus resource [mem >> > >> 0x00000000-0x3fffffffffff] >> > >> [ 0.795384] pci_bus 0000:7f: root bus resource [bus 7f] >> > >> >> > >> 0.809149] PCI host bridge to bus 0000:00 >> > >> [ 0.809151] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] >> > >> [ 0.809153] pci_bus 0000:00: root bus resource [mem >> > >> 0x00000000-0x3fffffffffff] >> > >> [ 0.809155] pci_bus 0000:00: root bus resource [bus 00-7e] >> > > >> > > This is the host bridge that claims to lead to bus 0000:00. >> > > >> > > But the I/O and memory resources of all these bridges are wrong. The >> > > ranges they show are CPU addresses, not PCI bus addresses, so they >> > > cannot overlap. >> > > >> > > It could be that you're using a non-standard host bridge driver that >> > > did not apply the translation between CPU addresses and PCI bus >> > > addresses. In ACPI, this is the _TRA property. >> > > >> > >> [ 0.852071] PCI host bridge to bus 0000:80 >> > >> [ 0.852073] pci_bus 0000:80: root bus resource [io 0x0000-0xffff] >> > >> [ 0.852075] pci_bus 0000:80: root bus resource [mem >> > >> 0x00000000-0x3fffffffffff] >> > >> [ 0.852077] pci_bus 0000:80: root bus resource [bus 80-fe] >> > >> >> > >> Following lines are from output of /proc/iomem >> > >> >> > >> c8000000-c8000000 : <BAD> >> > >> c8000001-c8000001 : <BAD> >> > >> c8000002-c8000002 : <BAD> >> > >> c8000003-c8000003 : <BAD> >> > >> c8100000-c91fffff : PCI Bus 0000:02 >> > >> c8100000-c90fffff : PCI Bus 0000:03 >> > >> c8100000-c90fffff : PCI Bus 0000:04 >> > >> c8100000-c82fffff : PCI Bus 0000:05 >> > >> c8100000-c81fffff : PCI Bus 0000:06 >> > >> c8100000-c8100fff : 0000:06:00.0 >> > >> c8200000-c82fffff : PCI Bus 0000:0a >> > >> c8200000-c8203fff : 0000:0a:00.0 >> > >> c9100000-c913ffff : 0000:02:00.0 >> > >> fb000000-fb8fffff : PCI Bus 0000:81 >> > >> fb000000-fb8fffff : PCI Bus 0000:82 >> > >> fb000000-fb7fffff : PCI Bus 0000:8c >> > >> fb000000-fb7fffff : 0000:8c:00.0 >> > >> fb000000-fb7fffff : Plx_Mgr >> > >> fb800000-fb8fffff : PCI Bus 0000:87 >> > >> fb800000-fb800fff : 0000:87:00.0 >> > >> fb900000-fb900fff : 0000:80:05.4 >> > >> >> > >> >> > >> I can read the device BAR via fb800000-fb800fff : 0000:87:00.0. This >> > >> device is available after reboot, and initial PCIe bus enumeration >> > >> time. >> > >> >> > >> The device c8100000-c8100fff : 0000:06:00.0 shows up after a hotplug >> > >> operation is done. >> > >> >> > >> Attached is the complete /proc/iomem and do not find the line >> > >> c8100000-c9ffffff : PCI Bus 0000:00, but does it exist for >> > >> PCI Bus 0000:80? BAR access works fine through this bus. >> > >> >> > >> >> > >> >> > >> On Mon, Aug 15, 2016 at 10:24 AM, Bjorn Helgaas <helgaas@kernel.org> wrote: >> > >> > Hi Kallol, >> > >> > >> > >> > Sorry I didn't respond to this earlier; I was on vacation for most of >> > >> > last week. >> > >> > >> > >> > On Fri, Aug 12, 2016 at 02:14:59PM -0700, Kallol Biswas wrote: >> > >> >> I am trying to debug an issue that the PCIe memory >> > >> >> transactions are not routed through a host bridge. >> > >> >> >> > >> >> The config transactions work fine. The primary, secondary and >> > >> >> subordinate bus numbers look fine for the PCIe tree under the host >> > >> >> bridge. The memory windows are programmed fine. >> > >> >> >> > >> >> Can someone point me to the code that programs the chipset/root port >> > >> >> registers those direct processor transactions to a specific host >> > >> >> bridge? >> > >> > >> > >> > It looks like you have an x86 system with ACPI. In that case, we use >> > >> > the ACPI pci_root.c host bridge driver. The chipset configuration for >> > >> > routing processor memory transactions to host bridges is done by the >> > >> > firmware, and the kernel doesn't touch it. There should be >> > >> > information in the chipset spec about how to do this, but there's no >> > >> > code in Linux itself that does it. >> > >> > >> > >> >> Cat of /proc/mem is copied below. >> > >> >> >> > >> >> /root/physmem -p c8100000 -s 4 -r >> > >> >> Operation:read to Addr: 0xc8100000 Value: 0 >> > >> >> ffffffff ffffffff ffffffff ffffffff >> > >> >> >> > >> >> The PCIe analyzer sitting between host bridge (0:3.0 and 02:00.0) and >> > >> >> external switches/devices does not capture any mem transactions. >> > >> >> However, config transactions are visible in the trace. >> > >> > >> > >> > 00:03.0 is not a host bridge; it's a Root Port, which logically >> > >> > connects two PCI buses. A host bridge logically connects a processor >> > >> > bus to a PCI bus. >> > >> > >> > >> >> root@xlink-target:~# cat /proc/iomem >> > >> >> 00000000-00000fff : reserved >> > >> >> 00001000-0009abff : System RAM >> > >> >> 0009ac00-0009ffff : reserved >> > >> >> 000c0000-000c7fff : Video ROM >> > >> >> 000c8000-000c8fff : Adapter ROM >> > >> >> 000e0000-000fffff : reserved >> > >> >> 000f0000-000fffff : System ROM >> > >> >> 00100000-78f36fff : System RAM >> > >> >> 01000000-01828f14 : Kernel code >> > >> >> 01828f15-01f3fb3f : Kernel data >> > >> >> 020bb000-021fdfff : Kernel bss >> > >> >> 30000000-37ffffff : Crash kernel >> > >> >> 78f37000-79839fff : reserved >> > >> >> 79116018-79116018 : APEI ERST >> > >> >> 7911601c-79116021 : APEI ERST >> > >> >> 79116028-79116039 : APEI ERST >> > >> >> 79116040-7911604c : APEI ERST >> > >> >> 79116050-7911804f : APEI ERST >> > >> >> 7983a000-79d4afff : ACPI Non-volatile Storage >> > >> >> 79d4b000-8fffffff : reserved >> > >> >> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff] >> > >> >> c6000000-c70fffff : PCI Bus 0000:2f >> > >> >> c6000000-c70fffff : PCI Bus 0000:30 >> > >> >> c6000000-c6ffffff : 0000:30:00.0 >> > >> >> c7000000-c701ffff : 0000:30:00.0 >> > >> >> c7100000-c71fffff : PCI Bus 0000:31 >> > >> >> c7100000-c711ffff : 0000:31:00.1 >> > >> >> c7100000-c711ffff : igb >> > >> >> c7120000-c713ffff : 0000:31:00.0 >> > >> >> c7120000-c713ffff : igb >> > >> >> c7140000-c7143fff : 0000:31:00.1 >> > >> >> c7140000-c7143fff : igb >> > >> >> c7144000-c7147fff : 0000:31:00.0 >> > >> >> c7144000-c7147fff : igb >> > >> >> c7300000-c730ffff : 0000:00:14.0 >> > >> >> c7300000-c730ffff : xhci-hcd >> > >> >> c7310000-c73107ff : 0000:00:1f.2 >> > >> >> c7310000-c73107ff : ahci >> > >> >> c7311000-c73113ff : 0000:00:1d.0 >> > >> >> c7311000-c73113ff : ehci_hcd >> > >> >> c7312000-c73123ff : 0000:00:1a.0 >> > >> >> c7312000-c73123ff : ehci_hcd >> > >> >> c7314000-c731400f : 0000:00:16.1 >> > >> >> c7315000-c731500f : 0000:00:16.0 >> > >> >> c7316000-c73167ff : 0000:00:11.4 >> > >> >> c7316000-c73167ff : ahci >> > >> >> c7317000-c7317fff : 0000:00:05.4 >> > >> >> c7ffc000-c7ffcfff : dmar1 >> > >> >> c8000000-c8000000 : <BAD> >> > >> >> c8000001-c8000001 : <BAD> >> > >> >> c8000002-c8000002 : <BAD> >> > >> >> c8000003-c8000003 : <BAD> >> > >> > >> > >> > Interesting, what are these from? >> > >> > >> > >> >> c8100000-c91fffff : PCI Bus 0000:02 >> > >> >> c8100000-c90fffff : PCI Bus 0000:03 >> > >> >> c8100000-c90fffff : PCI Bus 0000:04 >> > >> >> c8100000-c82fffff : PCI Bus 0000:05 >> > >> >> c8100000-c81fffff : PCI Bus 0000:06 >> > >> >> c8100000-c8100fff : 0000:06:00.0 >> > >> > >> > >> > You're trying to read the 06:00.0 BAR at 0xc8100000. The windows >> > >> > above look fine, but the top-level host bridge window is missing. >> > >> > >> > >> > The c8100000-c91fffff window is a window through the 00:03.0 Root >> > >> > Port, which connects bus 00 to bus 02. The 00:03.0 bridge forwards >> > >> > memory transactions in the c8100000-c91fffff range from its primary >> > >> > bus (bus 00) to its secondary bus (bus 02). >> > >> > >> > >> > But we don't know how those transactions get to bus 00 in the first >> > >> > place. The host bridge is the connection between the processor and >> > >> > bus 00, and it should have a window that encloses c8100000-c91fffff. >> > >> > >> > >> > Your dmesg log should have information about this bridge, e.g., >> > >> > something like this: >> > >> > >> > >> > ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7f]) >> > >> > PCI host bridge to bus 0000:00 >> > >> > pci_bus 0000:00: root bus resource [mem 0xc8100000-0xc9ffffff window] >> > >> > >> > >> > and there should be a line in /proc/iomem something like this: >> > >> > >> > >> > c8100000-c9ffffff : PCI Bus 0000:00 >> > >> > >> > >> > Bjorn >> > > >> > >> 00000000-00000fff : reserved >> > >> 00001000-0009abff : System RAM >> > >> 0009ac00-0009ffff : reserved >> > >> 000c0000-000c7fff : Video ROM >> > >> 000c8000-000c8fff : Adapter ROM >> > >> 000e0000-000fffff : reserved >> > >> 000f0000-000fffff : System ROM >> > >> 00100000-78f36fff : System RAM >> > >> 01000000-01828f14 : Kernel code >> > >> 01828f15-01f3fb3f : Kernel data >> > >> 020bb000-021fdfff : Kernel bss >> > >> 30000000-37ffffff : Crash kernel >> > >> 78f37000-79839fff : reserved >> > >> 79116018-79116018 : APEI ERST >> > >> 7911601c-79116021 : APEI ERST >> > >> 79116028-79116039 : APEI ERST >> > >> 79116040-7911604c : APEI ERST >> > >> 79116050-7911804f : APEI ERST >> > >> 7983a000-79d4afff : ACPI Non-volatile Storage >> > >> 79d4b000-8fffffff : reserved >> > >> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff] >> > >> c6000000-c70fffff : PCI Bus 0000:2f >> > >> c6000000-c70fffff : PCI Bus 0000:30 >> > >> c6000000-c6ffffff : 0000:30:00.0 >> > >> c7000000-c701ffff : 0000:30:00.0 >> > >> c7100000-c71fffff : PCI Bus 0000:31 >> > >> c7100000-c711ffff : 0000:31:00.1 >> > >> c7100000-c711ffff : igb >> > >> c7120000-c713ffff : 0000:31:00.0 >> > >> c7120000-c713ffff : igb >> > >> c7140000-c7143fff : 0000:31:00.1 >> > >> c7140000-c7143fff : igb >> > >> c7144000-c7147fff : 0000:31:00.0 >> > >> c7144000-c7147fff : igb >> > >> c7300000-c730ffff : 0000:00:14.0 >> > >> c7300000-c730ffff : xhci-hcd >> > >> c7310000-c73107ff : 0000:00:1f.2 >> > >> c7310000-c73107ff : ahci >> > >> c7311000-c73113ff : 0000:00:1d.0 >> > >> c7311000-c73113ff : ehci_hcd >> > >> c7312000-c73123ff : 0000:00:1a.0 >> > >> c7312000-c73123ff : ehci_hcd >> > >> c7314000-c731400f : 0000:00:16.1 >> > >> c7315000-c731500f : 0000:00:16.0 >> > >> c7316000-c73167ff : 0000:00:11.4 >> > >> c7316000-c73167ff : ahci >> > >> c7317000-c7317fff : 0000:00:05.4 >> > >> c7ffc000-c7ffcfff : dmar1 >> > >> c8000000-c8000000 : <BAD> >> > >> c8000001-c8000001 : <BAD> >> > >> c8000002-c8000002 : <BAD> >> > >> c8000003-c8000003 : <BAD> >> > >> c8100000-c91fffff : PCI Bus 0000:02 >> > >> c8100000-c90fffff : PCI Bus 0000:03 >> > >> c8100000-c90fffff : PCI Bus 0000:04 >> > >> c8100000-c82fffff : PCI Bus 0000:05 >> > >> c8100000-c81fffff : PCI Bus 0000:06 >> > >> c8100000-c8100fff : 0000:06:00.0 >> > >> c8200000-c82fffff : PCI Bus 0000:0a >> > >> c8200000-c8203fff : 0000:0a:00.0 >> > >> c9100000-c913ffff : 0000:02:00.0 >> > >> fb000000-fb8fffff : PCI Bus 0000:81 >> > >> fb000000-fb8fffff : PCI Bus 0000:82 >> > >> fb000000-fb7fffff : PCI Bus 0000:8c >> > >> fb000000-fb7fffff : 0000:8c:00.0 >> > >> fb000000-fb7fffff : Plx_Mgr >> > >> fb800000-fb8fffff : PCI Bus 0000:87 >> > >> fb800000-fb800fff : 0000:87:00.0 >> > >> fb900000-fb900fff : 0000:80:05.4 >> > >> fbffc000-fbffcfff : dmar0 >> > >> fec00000-fecfffff : PNP0003:00 >> > >> fec00000-fec003ff : IOAPIC 0 >> > >> fec01000-fec013ff : IOAPIC 1 >> > >> fec40000-fec403ff : IOAPIC 2 >> > >> fed00000-fed003ff : HPET 0 >> > >> fed00000-fed003ff : PNP0103:00 >> > >> fed12000-fed1200f : pnp 00:01 >> > >> fed12010-fed1201f : pnp 00:01 >> > >> fed1b000-fed1bfff : pnp 00:01 >> > >> fed1c000-fed44fff : reserved >> > >> fed1c000-fed3ffff : pnp 00:01 >> > >> fed1f410-fed1f414 : iTCO_wdt.0.auto >> > >> fed45000-fed8bfff : pnp 00:01 >> > >> fee00000-feefffff : pnp 00:01 >> > >> fee00000-fee00fff : Local APIC >> > >> ff000000-ffffffff : reserved >> > >> ff000000-ffffffff : pnp 00:01 >> > >> 100000000-47fffffff : System RAM >> > >> 480000000-4807fffff : PCI Bus 0000:02 >> > >> 480000000-4807fffff : PCI Bus 0000:03 >> > >> 480000000-4807fffff : PCI Bus 0000:04 >> > >> 480000000-4800fffff : PCI Bus 0000:05 >> > >> 480000000-4800fffff : PCI Bus 0000:06 >> > >> 480000000-480003fff : 0000:06:00.0 >> > >> 383fffe00000-383fffefffff : PCI Bus 0000:31 >> > >> 383fffe00000-383fffe1ffff : 0000:31:00.1 >> > >> 383fffe20000-383fffe3ffff : 0000:31:00.1 >> > >> 383fffe40000-383fffe5ffff : 0000:31:00.0 >> > >> 383fffe60000-383fffe7ffff : 0000:31:00.0 >> > >> 383ffff00000-383ffff03fff : 0000:00:04.7 >> > >> 383ffff00000-383ffff03fff : ioatdma >> > >> 383ffff04000-383ffff07fff : 0000:00:04.6 >> > >> 383ffff04000-383ffff07fff : ioatdma >> > >> 383ffff08000-383ffff0bfff : 0000:00:04.5 >> > >> 383ffff08000-383ffff0bfff : ioatdma >> > >> 383ffff0c000-383ffff0ffff : 0000:00:04.4 >> > >> 383ffff0c000-383ffff0ffff : ioatdma >> > >> 383ffff10000-383ffff13fff : 0000:00:04.3 >> > >> 383ffff10000-383ffff13fff : ioatdma >> > >> 383ffff14000-383ffff17fff : 0000:00:04.2 >> > >> 383ffff14000-383ffff17fff : ioatdma >> > >> 383ffff18000-383ffff1bfff : 0000:00:04.1 >> > >> 383ffff18000-383ffff1bfff : ioatdma >> > >> 383ffff1c000-383ffff1ffff : 0000:00:04.0 >> > >> 383ffff1c000-383ffff1ffff : ioatdma >> > >> 383ffff21000-383ffff210ff : 0000:00:1f.3 >> > >> 387fffe00000-387fffefffff : PCI Bus 0000:81 >> > >> 387fffe00000-387fffefffff : PCI Bus 0000:82 >> > >> 387fffe00000-387fffefffff : PCI Bus 0000:87 >> > >> 387fffe00000-387fffe03fff : 0000:87:00.0 >> > >> 387ffff00000-387ffff03fff : 0000:80:04.7 >> > >> 387ffff00000-387ffff03fff : ioatdma >> > >> 387ffff04000-387ffff07fff : 0000:80:04.6 >> > >> 387ffff04000-387ffff07fff : ioatdma >> > >> 387ffff08000-387ffff0bfff : 0000:80:04.5 >> > >> 387ffff08000-387ffff0bfff : ioatdma >> > >> 387ffff0c000-387ffff0ffff : 0000:80:04.4 >> > >> 387ffff0c000-387ffff0ffff : ioatdma >> > >> 387ffff10000-387ffff13fff : 0000:80:04.3 >> > >> 387ffff10000-387ffff13fff : ioatdma >> > >> 387ffff14000-387ffff17fff : 0000:80:04.2 >> > >> 387ffff14000-387ffff17fff : ioatdma >> > >> 387ffff18000-387ffff1bfff : 0000:80:04.1 >> > >> 387ffff18000-387ffff1bfff : ioatdma >> > >> 387ffff1c000-387ffff1ffff : 0000:80:04.0 >> > >> 387ffff1c000-387ffff1ffff : ioatdma >> > > > > ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2016-08-15 21:56 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <CAH=XD5_kqkw=f1g_M69eUHsSDhry32s5VQJXC-_iVa+LZ+vUDA@mail.gmail.com>
[not found] ` <CAH=XD58J+Yf-Rdz_bAJ-e9rU4ipu9opieFd0BuHeYAdX0nmrzg@mail.gmail.com>
2016-08-12 21:14 ` Fwd: PCIe root port programming Kallol Biswas
2016-08-15 17:24 ` Bjorn Helgaas
2016-08-15 18:39 ` Kallol Biswas
2016-08-15 19:06 ` Bjorn Helgaas
2016-08-15 20:43 ` Kallol Biswas
2016-08-15 21:07 ` Bjorn Helgaas
2016-08-15 21:51 ` Kallol Biswas
2016-08-15 21:56 ` Kallol Biswas
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox