From: Bjorn Helgaas <helgaas@kernel.org>
To: Kallol Biswas <kallol@nucleodyne.com>
Cc: linux-pci@vger.kernel.org
Subject: Re: Fwd: PCIe root port programming
Date: Mon, 15 Aug 2016 14:06:25 -0500 [thread overview]
Message-ID: <20160815190625.GE9790@localhost> (raw)
In-Reply-To: <CAH=XD59GxwKnMp4uFfZpXB12rQtes0TFFshm1HgXLc3uMLQUVg@mail.gmail.com>
On Mon, Aug 15, 2016 at 11:39:56AM -0700, Kallol Biswas wrote:
> Thank you for the explanation.
>
>
> Here is the relevant dmesg content:
>
> [ 0.790457] PCI: root bus ff: using default resources
> [ 0.790491] PCI host bridge to bus 0000:ff
> [ 0.790493] pci_bus 0000:ff: root bus resource [io 0x0000-0xffff]
> [ 0.790495] pci_bus 0000:ff: root bus resource [mem
> 0x00000000-0x3fffffffffff]
> [ 0.790497] pci_bus 0000:ff: root bus resource [bus ff]
>
> 0.795379] PCI host bridge to bus 0000:7f
> [ 0.795381] pci_bus 0000:7f: root bus resource [io 0x0000-0xffff]
> [ 0.795383] pci_bus 0000:7f: root bus resource [mem
> 0x00000000-0x3fffffffffff]
> [ 0.795384] pci_bus 0000:7f: root bus resource [bus 7f]
>
> 0.809149] PCI host bridge to bus 0000:00
> [ 0.809151] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
> [ 0.809153] pci_bus 0000:00: root bus resource [mem
> 0x00000000-0x3fffffffffff]
> [ 0.809155] pci_bus 0000:00: root bus resource [bus 00-7e]
This is the host bridge that claims to lead to bus 0000:00.
But the I/O and memory resources of all these bridges are wrong. The
ranges they show are CPU addresses, not PCI bus addresses, so they
cannot overlap.
It could be that you're using a non-standard host bridge driver that
did not apply the translation between CPU addresses and PCI bus
addresses. In ACPI, this is the _TRA property.
> [ 0.852071] PCI host bridge to bus 0000:80
> [ 0.852073] pci_bus 0000:80: root bus resource [io 0x0000-0xffff]
> [ 0.852075] pci_bus 0000:80: root bus resource [mem
> 0x00000000-0x3fffffffffff]
> [ 0.852077] pci_bus 0000:80: root bus resource [bus 80-fe]
>
> Following lines are from output of /proc/iomem
>
> c8000000-c8000000 : <BAD>
> c8000001-c8000001 : <BAD>
> c8000002-c8000002 : <BAD>
> c8000003-c8000003 : <BAD>
> c8100000-c91fffff : PCI Bus 0000:02
> c8100000-c90fffff : PCI Bus 0000:03
> c8100000-c90fffff : PCI Bus 0000:04
> c8100000-c82fffff : PCI Bus 0000:05
> c8100000-c81fffff : PCI Bus 0000:06
> c8100000-c8100fff : 0000:06:00.0
> c8200000-c82fffff : PCI Bus 0000:0a
> c8200000-c8203fff : 0000:0a:00.0
> c9100000-c913ffff : 0000:02:00.0
> fb000000-fb8fffff : PCI Bus 0000:81
> fb000000-fb8fffff : PCI Bus 0000:82
> fb000000-fb7fffff : PCI Bus 0000:8c
> fb000000-fb7fffff : 0000:8c:00.0
> fb000000-fb7fffff : Plx_Mgr
> fb800000-fb8fffff : PCI Bus 0000:87
> fb800000-fb800fff : 0000:87:00.0
> fb900000-fb900fff : 0000:80:05.4
>
>
> I can read the device BAR via fb800000-fb800fff : 0000:87:00.0. This
> device is available after reboot, and initial PCIe bus enumeration
> time.
>
> The device c8100000-c8100fff : 0000:06:00.0 shows up after a hotplug
> operation is done.
>
> Attached is the complete /proc/iomem and do not find the line
> c8100000-c9ffffff : PCI Bus 0000:00, but does it exist for
> PCI Bus 0000:80? BAR access works fine through this bus.
>
>
>
> On Mon, Aug 15, 2016 at 10:24 AM, Bjorn Helgaas <helgaas@kernel.org> wrote:
> > Hi Kallol,
> >
> > Sorry I didn't respond to this earlier; I was on vacation for most of
> > last week.
> >
> > On Fri, Aug 12, 2016 at 02:14:59PM -0700, Kallol Biswas wrote:
> >> I am trying to debug an issue that the PCIe memory
> >> transactions are not routed through a host bridge.
> >>
> >> The config transactions work fine. The primary, secondary and
> >> subordinate bus numbers look fine for the PCIe tree under the host
> >> bridge. The memory windows are programmed fine.
> >>
> >> Can someone point me to the code that programs the chipset/root port
> >> registers those direct processor transactions to a specific host
> >> bridge?
> >
> > It looks like you have an x86 system with ACPI. In that case, we use
> > the ACPI pci_root.c host bridge driver. The chipset configuration for
> > routing processor memory transactions to host bridges is done by the
> > firmware, and the kernel doesn't touch it. There should be
> > information in the chipset spec about how to do this, but there's no
> > code in Linux itself that does it.
> >
> >> Cat of /proc/mem is copied below.
> >>
> >> /root/physmem -p c8100000 -s 4 -r
> >> Operation:read to Addr: 0xc8100000 Value: 0
> >> ffffffff ffffffff ffffffff ffffffff
> >>
> >> The PCIe analyzer sitting between host bridge (0:3.0 and 02:00.0) and
> >> external switches/devices does not capture any mem transactions.
> >> However, config transactions are visible in the trace.
> >
> > 00:03.0 is not a host bridge; it's a Root Port, which logically
> > connects two PCI buses. A host bridge logically connects a processor
> > bus to a PCI bus.
> >
> >> root@xlink-target:~# cat /proc/iomem
> >> 00000000-00000fff : reserved
> >> 00001000-0009abff : System RAM
> >> 0009ac00-0009ffff : reserved
> >> 000c0000-000c7fff : Video ROM
> >> 000c8000-000c8fff : Adapter ROM
> >> 000e0000-000fffff : reserved
> >> 000f0000-000fffff : System ROM
> >> 00100000-78f36fff : System RAM
> >> 01000000-01828f14 : Kernel code
> >> 01828f15-01f3fb3f : Kernel data
> >> 020bb000-021fdfff : Kernel bss
> >> 30000000-37ffffff : Crash kernel
> >> 78f37000-79839fff : reserved
> >> 79116018-79116018 : APEI ERST
> >> 7911601c-79116021 : APEI ERST
> >> 79116028-79116039 : APEI ERST
> >> 79116040-7911604c : APEI ERST
> >> 79116050-7911804f : APEI ERST
> >> 7983a000-79d4afff : ACPI Non-volatile Storage
> >> 79d4b000-8fffffff : reserved
> >> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
> >> c6000000-c70fffff : PCI Bus 0000:2f
> >> c6000000-c70fffff : PCI Bus 0000:30
> >> c6000000-c6ffffff : 0000:30:00.0
> >> c7000000-c701ffff : 0000:30:00.0
> >> c7100000-c71fffff : PCI Bus 0000:31
> >> c7100000-c711ffff : 0000:31:00.1
> >> c7100000-c711ffff : igb
> >> c7120000-c713ffff : 0000:31:00.0
> >> c7120000-c713ffff : igb
> >> c7140000-c7143fff : 0000:31:00.1
> >> c7140000-c7143fff : igb
> >> c7144000-c7147fff : 0000:31:00.0
> >> c7144000-c7147fff : igb
> >> c7300000-c730ffff : 0000:00:14.0
> >> c7300000-c730ffff : xhci-hcd
> >> c7310000-c73107ff : 0000:00:1f.2
> >> c7310000-c73107ff : ahci
> >> c7311000-c73113ff : 0000:00:1d.0
> >> c7311000-c73113ff : ehci_hcd
> >> c7312000-c73123ff : 0000:00:1a.0
> >> c7312000-c73123ff : ehci_hcd
> >> c7314000-c731400f : 0000:00:16.1
> >> c7315000-c731500f : 0000:00:16.0
> >> c7316000-c73167ff : 0000:00:11.4
> >> c7316000-c73167ff : ahci
> >> c7317000-c7317fff : 0000:00:05.4
> >> c7ffc000-c7ffcfff : dmar1
> >> c8000000-c8000000 : <BAD>
> >> c8000001-c8000001 : <BAD>
> >> c8000002-c8000002 : <BAD>
> >> c8000003-c8000003 : <BAD>
> >
> > Interesting, what are these from?
> >
> >> c8100000-c91fffff : PCI Bus 0000:02
> >> c8100000-c90fffff : PCI Bus 0000:03
> >> c8100000-c90fffff : PCI Bus 0000:04
> >> c8100000-c82fffff : PCI Bus 0000:05
> >> c8100000-c81fffff : PCI Bus 0000:06
> >> c8100000-c8100fff : 0000:06:00.0
> >
> > You're trying to read the 06:00.0 BAR at 0xc8100000. The windows
> > above look fine, but the top-level host bridge window is missing.
> >
> > The c8100000-c91fffff window is a window through the 00:03.0 Root
> > Port, which connects bus 00 to bus 02. The 00:03.0 bridge forwards
> > memory transactions in the c8100000-c91fffff range from its primary
> > bus (bus 00) to its secondary bus (bus 02).
> >
> > But we don't know how those transactions get to bus 00 in the first
> > place. The host bridge is the connection between the processor and
> > bus 00, and it should have a window that encloses c8100000-c91fffff.
> >
> > Your dmesg log should have information about this bridge, e.g.,
> > something like this:
> >
> > ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7f])
> > PCI host bridge to bus 0000:00
> > pci_bus 0000:00: root bus resource [mem 0xc8100000-0xc9ffffff window]
> >
> > and there should be a line in /proc/iomem something like this:
> >
> > c8100000-c9ffffff : PCI Bus 0000:00
> >
> > Bjorn
> 00000000-00000fff : reserved
> 00001000-0009abff : System RAM
> 0009ac00-0009ffff : reserved
> 000c0000-000c7fff : Video ROM
> 000c8000-000c8fff : Adapter ROM
> 000e0000-000fffff : reserved
> 000f0000-000fffff : System ROM
> 00100000-78f36fff : System RAM
> 01000000-01828f14 : Kernel code
> 01828f15-01f3fb3f : Kernel data
> 020bb000-021fdfff : Kernel bss
> 30000000-37ffffff : Crash kernel
> 78f37000-79839fff : reserved
> 79116018-79116018 : APEI ERST
> 7911601c-79116021 : APEI ERST
> 79116028-79116039 : APEI ERST
> 79116040-7911604c : APEI ERST
> 79116050-7911804f : APEI ERST
> 7983a000-79d4afff : ACPI Non-volatile Storage
> 79d4b000-8fffffff : reserved
> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
> c6000000-c70fffff : PCI Bus 0000:2f
> c6000000-c70fffff : PCI Bus 0000:30
> c6000000-c6ffffff : 0000:30:00.0
> c7000000-c701ffff : 0000:30:00.0
> c7100000-c71fffff : PCI Bus 0000:31
> c7100000-c711ffff : 0000:31:00.1
> c7100000-c711ffff : igb
> c7120000-c713ffff : 0000:31:00.0
> c7120000-c713ffff : igb
> c7140000-c7143fff : 0000:31:00.1
> c7140000-c7143fff : igb
> c7144000-c7147fff : 0000:31:00.0
> c7144000-c7147fff : igb
> c7300000-c730ffff : 0000:00:14.0
> c7300000-c730ffff : xhci-hcd
> c7310000-c73107ff : 0000:00:1f.2
> c7310000-c73107ff : ahci
> c7311000-c73113ff : 0000:00:1d.0
> c7311000-c73113ff : ehci_hcd
> c7312000-c73123ff : 0000:00:1a.0
> c7312000-c73123ff : ehci_hcd
> c7314000-c731400f : 0000:00:16.1
> c7315000-c731500f : 0000:00:16.0
> c7316000-c73167ff : 0000:00:11.4
> c7316000-c73167ff : ahci
> c7317000-c7317fff : 0000:00:05.4
> c7ffc000-c7ffcfff : dmar1
> c8000000-c8000000 : <BAD>
> c8000001-c8000001 : <BAD>
> c8000002-c8000002 : <BAD>
> c8000003-c8000003 : <BAD>
> c8100000-c91fffff : PCI Bus 0000:02
> c8100000-c90fffff : PCI Bus 0000:03
> c8100000-c90fffff : PCI Bus 0000:04
> c8100000-c82fffff : PCI Bus 0000:05
> c8100000-c81fffff : PCI Bus 0000:06
> c8100000-c8100fff : 0000:06:00.0
> c8200000-c82fffff : PCI Bus 0000:0a
> c8200000-c8203fff : 0000:0a:00.0
> c9100000-c913ffff : 0000:02:00.0
> fb000000-fb8fffff : PCI Bus 0000:81
> fb000000-fb8fffff : PCI Bus 0000:82
> fb000000-fb7fffff : PCI Bus 0000:8c
> fb000000-fb7fffff : 0000:8c:00.0
> fb000000-fb7fffff : Plx_Mgr
> fb800000-fb8fffff : PCI Bus 0000:87
> fb800000-fb800fff : 0000:87:00.0
> fb900000-fb900fff : 0000:80:05.4
> fbffc000-fbffcfff : dmar0
> fec00000-fecfffff : PNP0003:00
> fec00000-fec003ff : IOAPIC 0
> fec01000-fec013ff : IOAPIC 1
> fec40000-fec403ff : IOAPIC 2
> fed00000-fed003ff : HPET 0
> fed00000-fed003ff : PNP0103:00
> fed12000-fed1200f : pnp 00:01
> fed12010-fed1201f : pnp 00:01
> fed1b000-fed1bfff : pnp 00:01
> fed1c000-fed44fff : reserved
> fed1c000-fed3ffff : pnp 00:01
> fed1f410-fed1f414 : iTCO_wdt.0.auto
> fed45000-fed8bfff : pnp 00:01
> fee00000-feefffff : pnp 00:01
> fee00000-fee00fff : Local APIC
> ff000000-ffffffff : reserved
> ff000000-ffffffff : pnp 00:01
> 100000000-47fffffff : System RAM
> 480000000-4807fffff : PCI Bus 0000:02
> 480000000-4807fffff : PCI Bus 0000:03
> 480000000-4807fffff : PCI Bus 0000:04
> 480000000-4800fffff : PCI Bus 0000:05
> 480000000-4800fffff : PCI Bus 0000:06
> 480000000-480003fff : 0000:06:00.0
> 383fffe00000-383fffefffff : PCI Bus 0000:31
> 383fffe00000-383fffe1ffff : 0000:31:00.1
> 383fffe20000-383fffe3ffff : 0000:31:00.1
> 383fffe40000-383fffe5ffff : 0000:31:00.0
> 383fffe60000-383fffe7ffff : 0000:31:00.0
> 383ffff00000-383ffff03fff : 0000:00:04.7
> 383ffff00000-383ffff03fff : ioatdma
> 383ffff04000-383ffff07fff : 0000:00:04.6
> 383ffff04000-383ffff07fff : ioatdma
> 383ffff08000-383ffff0bfff : 0000:00:04.5
> 383ffff08000-383ffff0bfff : ioatdma
> 383ffff0c000-383ffff0ffff : 0000:00:04.4
> 383ffff0c000-383ffff0ffff : ioatdma
> 383ffff10000-383ffff13fff : 0000:00:04.3
> 383ffff10000-383ffff13fff : ioatdma
> 383ffff14000-383ffff17fff : 0000:00:04.2
> 383ffff14000-383ffff17fff : ioatdma
> 383ffff18000-383ffff1bfff : 0000:00:04.1
> 383ffff18000-383ffff1bfff : ioatdma
> 383ffff1c000-383ffff1ffff : 0000:00:04.0
> 383ffff1c000-383ffff1ffff : ioatdma
> 383ffff21000-383ffff210ff : 0000:00:1f.3
> 387fffe00000-387fffefffff : PCI Bus 0000:81
> 387fffe00000-387fffefffff : PCI Bus 0000:82
> 387fffe00000-387fffefffff : PCI Bus 0000:87
> 387fffe00000-387fffe03fff : 0000:87:00.0
> 387ffff00000-387ffff03fff : 0000:80:04.7
> 387ffff00000-387ffff03fff : ioatdma
> 387ffff04000-387ffff07fff : 0000:80:04.6
> 387ffff04000-387ffff07fff : ioatdma
> 387ffff08000-387ffff0bfff : 0000:80:04.5
> 387ffff08000-387ffff0bfff : ioatdma
> 387ffff0c000-387ffff0ffff : 0000:80:04.4
> 387ffff0c000-387ffff0ffff : ioatdma
> 387ffff10000-387ffff13fff : 0000:80:04.3
> 387ffff10000-387ffff13fff : ioatdma
> 387ffff14000-387ffff17fff : 0000:80:04.2
> 387ffff14000-387ffff17fff : ioatdma
> 387ffff18000-387ffff1bfff : 0000:80:04.1
> 387ffff18000-387ffff1bfff : ioatdma
> 387ffff1c000-387ffff1ffff : 0000:80:04.0
> 387ffff1c000-387ffff1ffff : ioatdma
next prev parent reply other threads:[~2016-08-15 19:16 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <CAH=XD5_kqkw=f1g_M69eUHsSDhry32s5VQJXC-_iVa+LZ+vUDA@mail.gmail.com>
[not found] ` <CAH=XD58J+Yf-Rdz_bAJ-e9rU4ipu9opieFd0BuHeYAdX0nmrzg@mail.gmail.com>
2016-08-12 21:14 ` Fwd: PCIe root port programming Kallol Biswas
2016-08-15 17:24 ` Bjorn Helgaas
2016-08-15 18:39 ` Kallol Biswas
2016-08-15 19:06 ` Bjorn Helgaas [this message]
2016-08-15 20:43 ` Kallol Biswas
2016-08-15 21:07 ` Bjorn Helgaas
2016-08-15 21:51 ` Kallol Biswas
2016-08-15 21:56 ` Kallol Biswas
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160815190625.GE9790@localhost \
--to=helgaas@kernel.org \
--cc=kallol@nucleodyne.com \
--cc=linux-pci@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox