Re: PCIe bus enumeration

[Bjorn Helgaas <bhelgaas@google.com>]

On Fri, Jul 04, 2014 at 09:55:20AM +0200, Federico Vaga wrote:
> > I assume these ports don't support hotplug.  If they *did* support
> > hotplug, those ports would have to exist because they handle the
> > hotplug events (presence detect, etc.)
> 
> I asked: yes, they do not support hotplug
> 
> > If you can collect the complete "lspci -vv" output from your machine
> > (with a device plugged in, so we can see the port leading to it),
> > that will help make this more concrete.  And maybe one with no
> > devices plugged in, so we can see exactly what changes.
> 
> I attached two files with the output. I putted a card in slot 10 and 
> took the output, then moved the card on slot 11 and took the output.
> 
> As you can see with diff the bridge behind the slot disappear when it 
> is empty.

Perfect, thanks!  For some reason, it really helps me to be able to stare
at the actual data.  Here's the situation with slot 10 occupied:

  00:01.0 82Q35 Root Port to [bus 05]          PCIe SltCap slot #21
  05:00.0 CERN/ECP/EDU Device                  slot 10
  00:1c.0 82801I Express Port 1 to [bus 04]    PCIe SltCap slot #22
  00:1c.3 (not present at all)
  00:1c.4 82801I Express Port 5 to [bus 03]    PCIe SltCap slot #0
  03:00.0 Realtek NIC

and here it is with slot 11 occupied:

  00:01.0 (not present at all)
  00:1c.0 82801I Express Port 1 to [bus 05]    PCIe SltCap slot #22
  00:1c.3 82801I Express Port 4 to [bus 04]    PCIe SltCap slot #25
  04:00.0 CERN/ECP/EDU Device                  slot 11
  00:1c.4 82801I Express Port 5 to [bus 03]    PCIe SltCap slot #0
  03:00.0 Realtek NIC

I'm pretty sure this is a function of your BIOS.  There are often
device-specific ways to enable or disable individual devices (like the root
ports here), and the BIOS is likely disabling these ports when there is
nothing below them.  I don't know why it would turn off 00:1c.3 when its
slot is empty, but it doesn't turn off 00:1c.0, which also leads to an
empty slot.  But I don't think Linux is involved in this, and if the BIOS
disables devices, there really isn't anything Linux can do about it.

If you can get to an EFI shell on this box, you might be able to confirm
this with the "pci" command.  Booting Linux with "pci=earlydump" is similar
in that it dumps PCI config space before we change anything.

To solve this problem, I think you need slot information even when there's
no hotplug.  This has been raised before [1, 2], and I think it's a good
idea, but nobody has implemented it yet.

Another curious thing is that you refer to "slot 10", but there's no
obvious connection between that and the "slot 21" in the PCIe capability of
the Root Port leading to that slot.  But I guess you said the slots are in
a backplane (they're not an integral part of the motherboard).  In that
case, there's no way for the motherboard to know what the labels on the
backplane are.

Bjorn

[1] http://lkml.kernel.org/r/CAErSpo45sDNPt6=Yw-qgqdojYL8+_JNOVNEnVxRLatga+bY+2A@mail.gmail.com
[2] https://bugzilla.kernel.org/show_bug.cgi?id=72681