Re: PCIe: Unexpected .remove() and .probe() Callback Invocation Without Device Removal

[Bjorn Helgaas <helgaas@kernel.org>]

On Wed, May 07, 2025 at 08:01:09PM +0530, Subhashini Rao Beerisetty wrote:
> Hi All,
> 
> I’m reaching out for some guidance on a behavior we're observing with
> a Xilinx FPGA based PCIe device on our test systems. This device uses
> an out-of-tree driver.
> 
> We are seeing that, without any actual physical or hotplug
> removal/reinsertion of the PCIe device, the kernel invokes the
> pci_driver's .remove() callback followed shortly by the .probe()
> callback. This appears to be an unexpected re-enumeration or reset of
> the PCIe device.
> 
> Below is a snippet of the relevant kernel log captured during one such event.
> 
> Apr 30 20:01:05 xilinxtest1 kernel: [6612195.847385] XILINX_FPGA PCI
> 0000:01:00.0: PME# disabled
> Apr 30 20:01:05 xilinxtest1 kernel: [6612195.847410] XILINX_FPGA PCI:
> XILINX_FPGA_pci_remove
> Apr 30 20:01:05 xilinxtest1 kernel: [6612195.848110] pci 0000:01:00.0:
> EDR: Notify handler removed
> Apr 30 20:01:05 xilinxtest1 kernel: [6612195.848240] pci 0000:01:00.0:
> device released
> Apr 30 20:01:05 xilinxtest1 kernel: [6612195.876157] pci_bus 0000:00:
> scanning bus
> Apr 30 20:01:05 xilinxtest1 kernel: [6612195.876419] pcieport
> 0000:00:1c.0: scanning [bus 01-01] behind bridge, pass 0
> Apr 30 20:01:05 xilinxtest1 kernel: [6612195.876445] pci_bus 0000:01:
> scanning bus
> Apr 30 20:01:05 xilinxtest1 kernel: [6612195.876594] pci 0000:01:00.0:
> [1556:5555] type 00 class 0x050000
> Apr 30 20:01:05 xilinxtest1 kernel: [6612195.876678] pci 0000:01:00.0:
> reg 0x10: [mem 0xd0400000-0xd07fffff]
> Apr 30 20:01:05 xilinxtest1 kernel: [6612195.877426] pci 0000:01:00.0:
> EDR: Notify handler installed
> Apr 30 20:01:05 xilinxtest1 kernel: [6612195.877850] pci_bus 0000:01:
> bus scan returning with max=01
> Apr 30 20:01:05 xilinxtest1 kernel: [6612195.877872] pcieport
> 0000:00:1c.2: scanning [bus 02-02] behind bridge, pass 0
> Apr 30 20:01:05 xilinxtest1 kernel: [6612195.877898] pci_bus 0000:02:
> scanning bus
> Apr 30 20:01:05 xilinxtest1 kernel: [6612195.877915] pci_bus 0000:02:
> bus scan returning with max=02
> Apr 30 20:01:05 xilinxtest1 kernel: [6612195.877932] pcieport
> 0000:00:1c.3: scanning [bus 03-03] behind bridge, pass 0
> Apr 30 20:01:05 xilinxtest1 kernel: [6612195.877956] pci_bus 0000:03:
> scanning bus
> Apr 30 20:01:05 xilinxtest1 kernel: [6612195.877965] pci_bus 0000:03:
> bus scan returning with max=03
> Apr 30 20:01:05 xilinxtest1 kernel: [6612195.877982] pcieport
> 0000:00:1c.0: scanning [bus 01-01] behind bridge, pass 1
> Apr 30 20:01:05 xilinxtest1 kernel: [6612195.878013] pcieport
> 0000:00:1c.2: scanning [bus 02-02] behind bridge, pass 1
> Apr 30 20:01:05 xilinxtest1 kernel: [6612195.878043] pcieport
> 0000:00:1c.3: scanning [bus 03-03] behind bridge, pass 1
> Apr 30 20:01:05 xilinxtest1 kernel: [6612195.878066] pci_bus 0000:00:
> bus scan returning with max=03
> Apr 30 20:01:05 xilinxtest1 kernel: [6612195.878094] pci 0000:01:00.0:
> BAR 0: assigned [mem 0xd0400000-0xd07fffff]
> Apr 30 20:01:05 xilinxtest1 kernel: [6612195.878204] XILINX_FPGA PCI
> 0000:01:00.0: runtime IRQ mapping not provided by arch
> 
> 
> Our Questions:
> 
> What could trigger such a PCIe device re-enumeration without a physical event?
> 
> Are there any known kernel or platform-level triggers that might cause this?
> 
> Any debug hooks or sysfs entries we should monitor or enable to catch
> the root cause?

If you can point us to the source for the driver you're using, we
might be able to help.

Otherwise, adding dump_stack() in your .probe() and .remove()
functions might give clues.

Bjorn