Re: NULL Pointer in 3.x during PCI bus enumeration

[Bjorn Helgaas <bhelgaas@google.com>]

[+cc Yijing, Matthew]

On Thu, Mar 05, 2015 at 06:27:09PM -0800, Robert White wrote:
> It took me a while to get back into the lab and scrape together a
> working subsystem to collect the data you wanted.
> 
> Link to bug...
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=94361

Thanks!  Yijing added useful analysis to the bugzilla, but I'm going to
continue the email thread because there's interesting stuff here that
shouldn't be buried off in bugzilla.

Yijing noticed that you have an interesting system topology:

  pci 0000:00:1c.0: PCI bridge to [bus 02-0a]   Root Port (Slot+)
  pci 0000:02:00.0: PCI bridge to [bus 03-0a]   Downstream Port (Slot-)
  pci 0000:03:00.0: PCI bridge to [bus 04]      Upstream Port
  pci 0000:03:01.0: PCI bridge to [bus 05]      Downstream Port (Slot+)
  pci 0000:03:02.0: PCI bridge to [bus 06]      Downstream Port (Slot+)
  ...
  pci 0000:03:0a.0: PCI bridge to [bus 0a]      Downstream Port (Slot+)

It would be more typical for the Root Port to connect to an Upstream
Port of a switch, and Downstream Ports of the switch would connect to
endpoint devices.  But your system has the Root Port connected to a
Downstream Port.

This might be a legal topology, but I'm confused about how we should
interpret it.  ASPM configuration involves both ends of a link, and the
code allocates pcie_link_state structures for the device at the upstream
end of the link.  Normally the upstream end is a Root Port or a Downstream
Port.  Then it configures the other end by iterating over the list of
devices on the secondary bus ("pci_dev.subordinate" in the code).

In your system, there's a link from 00:1c.0 to 02:00.0, but if we're
looking at 02:00.0, the link is on the *upstream* side, not the downstream
side where we expect it.  So Linux allocates a pcie_link_state for 02:00.0,
but it thinks the other end of that link is at 03:00.0, but I think that's
wrong.

It might be possible for us to figure out where the other end of the link
is in a different way, without relying on the assumption that a Downstream
Port's link goes to its secondary bus.  But that would definitely require
some changes.

(I know your FADT told us not to touch ASPM anyway; that's another issue
that we also need to sort out.)

You said:

> Yes. It is part of the Advanced Telecommunication Computing
> Architecture (ATCA)
> http://en.wikipedia.org/wiki/Advanced_Telecommunications_Computing_Architecture

> The deal is that the "Field Replaceable Units" (FRUs) fit each fit
> into a carrier card that is little than a power control matrix and a
> PCIe bus. The CPU module itself is just another FRU just like the
> targets. So the computing module doesn't own the top level bus. The
> data trip to the final target devices, if they aren't co-resident on
> the CPU module, is up to the backplane and then back down to the
> target controller.

> So in this setup the CPU is a peer of all the other adapters.

Where do the devices listed above physically live?  Is the switch (devices
02:00.0, 03:00.0, 03:01.0, etc.) physically on the backplane, and the FRUs
(including the CPU) in slots connected to Downstream Ports of the switch?

Who assigns bus numbers to this fabric?  Would anything break if Linux
reassigned them?

Bjorn