PCIe hotplug

PCIe hotplug

[Ludwig Petrosyan]

Dear Linux PCI support team

I need help !

We are using MTCA crate system with the Ubuntu Linux for an accelerator 
control system.

The hotplug is very important for us, as most the crates are installed 
in the tunnel.

For a long time the PCIe hotplug works fine, we use the following boot 
parameters:

"noacpi noapic clock=tsc acpi=off apic=off pciehp.pciehp_force=1 
pciehp.pciehp_debug=1 pcie_ports=native"

After some point (seems after upgrading to kernel 3.x.x) we got problems 
(main problem is: the memories not remapped,

I mean: if I have some endpoint after removing and inserting back the 
memories are not remapped,

I know that I could not add new endpoint in running system, but it has 
to be possible to remove and insert back existing endpoint).

Now I am mixed out: what to use PCIe native hotplug or ACPI ..., I could 
not find any documentation about using PCIe hotplug

and how to configure.

Could somebody help me!

with best regards

Ludwig

Re: PCIe hotplug

[Bjorn Helgaas]

Hi Ludwig,

On Wed, Mar 29, 2017 at 03:34:47PM +0200, Ludwig Petrosyan wrote:
> Dear Linux PCI support team
> 
> I need help !
> 
> We are using MTCA crate system with the Ubuntu Linux for an
> accelerator control system.
> 
> The hotplug is very important for us, as most the crates are
> installed in the tunnel.
> 
> For a long time the PCIe hotplug works fine, we use the following
> boot parameters:
> 
> "noacpi noapic clock=tsc acpi=off apic=off pciehp.pciehp_force=1
> pciehp.pciehp_debug=1 pcie_ports=native"
> 
> After some point (seems after upgrading to kernel 3.x.x) we got
> problems (main problem is: the memories not remapped,
> 
> I mean: if I have some endpoint after removing and inserting back
> the memories are not remapped,
> 
> I know that I could not add new endpoint in running system, but it
> has to be possible to remove and insert back existing endpoint).
> 
> Now I am mixed out: what to use PCIe native hotplug or ACPI ..., I
> could not find any documentation about using PCIe hotplug
> 
> and how to configure.

Sounds like a possible regression, since it worked in an older kernel
but doesn't work any more.

Can you collect complete dmesg logs from the newest working kernel and
the oldest broken kernel, along with "lspci -vv" output (as root), and
open a report at https://bugzilla.kernel.org in the drivers/pci
component?  Please attach the logs instead of pasting them inline.

I'm slightly concerned about all the boot parameters because you're
turning off ACPI support in Linux, and there's some possibility of 
a conflict between ACPI hotplug support in the BIOS and the PCIe
native hotplug you're probably using now.

Bjorn

PCIe hotplug

[Sadanand Warrier]

[-- Attachment #1.1: Type: text/plain, Size: 846 bytes --]

Hi
   I had  question about PCIe hotplug. We have hardware that is connected
to the host by means of two PCIe switches. i.e. the host sees a PCIe switch
connected to one of its buses and on the far side of that switch another
PCIe switch which has a PCIe device.
   It is possible that this device does not train its host facing PCIe
links before the server enumerates down its PCI bus and reaches those
links. It is also possible the PCIe switch to which the device is attached
has not been able to train its own links before server enumeration.
  Is PCIe hotplug built to work on schemes like this? Let us assume that
the hardware has been designed to trasmit a presence signal once the links
are trained but this could happen after the server enumeration?
  Incidentally does the server take advantage of the BIOS/UEFI enumeration?

Thanks

S

[-- Attachment #1.2: Type: text/html, Size: 964 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: PCIe hotplug

[Greg KH]

On Thu, Feb 13, 2020 at 12:40:59PM -0500, Sadanand Warrier wrote:
> Hi
>    I had  question about PCIe hotplug. We have hardware that is connected
> to the host by means of two PCIe switches. i.e. the host sees a PCIe switch
> connected to one of its buses and on the far side of that switch another
> PCIe switch which has a PCIe device.
>    It is possible that this device does not train its host facing PCIe
> links before the server enumerates down its PCI bus and reaches those
> links. It is also possible the PCIe switch to which the device is attached
> has not been able to train its own links before server enumeration.
>   Is PCIe hotplug built to work on schemes like this? Let us assume that
> the hardware has been designed to trasmit a presence signal once the links
> are trained but this could happen after the server enumeration?

Look at the PCIe hotplug spec, it should answer all of your questions
about this.

>   Incidentally does the server take advantage of the BIOS/UEFI enumeration?

Yes, of course, how else would the kernel be able to enumerate PCI
devices?  :)

thanks,

greg k-h

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: PCIe hotplug

[Sadanand Warrier]

[-- Attachment #1.1: Type: text/plain, Size: 1440 bytes --]

Thank you Greg.
About the last one , it's been a while but I wasn't sure whether Linux was
going to do its own enumeration.
Of course it's best to take advantage of all the stuff done by UEFI ,
padding etc.

S

On Thu, 13 Feb 2020 at 12:48, Greg KH <greg@kroah.com> wrote:

> On Thu, Feb 13, 2020 at 12:40:59PM -0500, Sadanand Warrier wrote:
> > Hi
> >    I had  question about PCIe hotplug. We have hardware that is connected
> > to the host by means of two PCIe switches. i.e. the host sees a PCIe
> switch
> > connected to one of its buses and on the far side of that switch another
> > PCIe switch which has a PCIe device.
> >    It is possible that this device does not train its host facing PCIe
> > links before the server enumerates down its PCI bus and reaches those
> > links. It is also possible the PCIe switch to which the device is
> attached
> > has not been able to train its own links before server enumeration.
> >   Is PCIe hotplug built to work on schemes like this? Let us assume that
> > the hardware has been designed to trasmit a presence signal once the
> links
> > are trained but this could happen after the server enumeration?
>
> Look at the PCIe hotplug spec, it should answer all of your questions
> about this.
>
> >   Incidentally does the server take advantage of the BIOS/UEFI
> enumeration?
>
> Yes, of course, how else would the kernel be able to enumerate PCI
> devices?  :)
>
> thanks,
>
> greg k-h
>

[-- Attachment #1.2: Type: text/html, Size: 1900 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies