PCI driver - hotplug of multiple devices

PCI driver - hotplug of multiple devices

[Jozef Matejcik (Nokia)]

Hi,

We are facing certain issues with the driver of PCI device.

The driver is Broadcom's linux-kernel-bde.

The device is a forwarding ASIC, acting as 2 PCI devices on one die. I'd like to note it's acting as 2 separate PCI devices, not a device with 2 virtual functions:

[*****@***** ~]$ lspci |grep -i broad
07:00.0 Ethernet controller: Broadcom Inc. and subsidiaries *****
08:00.0 Ethernet controller: Broadcom Inc. and subsidiaries *****

Both are managed by the driver mentioned above, linux-kernel-bde.

The issue:
 - kernel module is inserted
 - we de-assert reset of these PCI devices (previously in reset, i.e. effectively powered down)
 - sometimes, very infrequently, the box freezes for 1-3 minutes and the HW triggers reboot

When we removed the module before de-asserting the resets and then inserted it later, the issue did not occur. So we suspect that the kernel driver is accessing the device during probe too early and this is causing HW freeze.

The question:
Are PCI drivers expected to handle hotplug events of any number of their devices simultaneously? Can the issue I described above be considered a bug in vendor's implementation?

Please let me know if you need more information.

Thanks,
Jozef

Re: PCI driver - hotplug of multiple devices

[Lukas Wunner]

On Wed, Jul 22, 2026 at 07:12:24AM +0000, Jozef Matejcik (Nokia) wrote:
> The driver is Broadcom's linux-kernel-bde.

This seems to be an out-of-tree driver, i.e. not something that is
part of the mainline kernel.  The upstream community can only provide
very limited support for such drivers.  The vendor needs to upstream
the driver into the mainline kernel, otherwise there's only so much
we can do.

> The issue:
>  - kernel module is inserted
>  - we de-assert reset of these PCI devices (previously in reset, i.e.
>    effectively powered down)
>  - sometimes, very infrequently, the box freezes for 1-3 minutes and
>    the HW triggers reboot

We'd need logs to say anything about it.

> When we removed the module before de-asserting the resets and then
> inserted it later, the issue did not occur. So we suspect that the
> kernel driver is accessing the device during probe too early and
> this is causing HW freeze.

Or maybe the reset puts the device in a state which is unexpected by
the driver.

> Are PCI drivers expected to handle hotplug events of any number of
> their devices simultaneously?

Yes.  E.g. a global data structure in the driver needs to be properly
locked if accessed concurrently by the probe function for separate
devices.

> Can the issue I described above be considered a bug in vendor's
> implementation?

Sounds like it.  Usually out-of-tree drivers haven't gone through
the rigorous review process by the community and quality may thus
be subpar.  The recommendation is to ask vendors whether driver
support is in mainline already when selecting components for your
products.

Thanks,

Lukas

RE: PCI driver - hotplug of multiple devices

[Jozef Matejcik (Nokia)]

Hi Lukas,

Thanks for the answer.

> We'd need logs to say anything about it.

The code path taken in faulty case is the same as the one in successful case. However, we do see that some PCI reads failed:

 Good one:
[  294.400338] linux-kernel-bde (187): iproc version = 0x14 dma_hi_bits = 0x0 [1]
[  294.400341] linux-kernel-bde (187): iProc version: 0x00000014 (20)         [2]
[  294.400348] linux-kernel-bde (187): subwin: 0x02840000 (42205184)
[  294.400353] linux-kernel-bde (187): subwin: 0x00000000 (0)
[  294.400358] linux-kernel-bde (187): subwin: 0x0292c000 (43171840)
[  294.400362] linux-kernel-bde (187): subwin: 0x00000000 (0)
[  294.400367] linux-kernel-bde (187): subwin: 0x00000000 (0)
[  294.400371] linux-kernel-bde (187): subwin: 0x00000000 (0)
[  294.400375] linux-kernel-bde (187): subwin: 0x0292d000 (43175936)
[  294.400379] linux-kernel-bde (187): subwin: 0x02017000 (33648640)
[  294.400385] linux-kernel-bde (187): PCI resource len 8MB  

Bad one:
[   56.335260] linux-kernel-bde (187): iproc version = 0x14 dma_hi_bits = 0x0 [1]
[   56.335263] linux-kernel-bde (187): iProc version: 0x00000014 (20)         [2]
[   56.335271] linux-kernel-bde (5538): subwin: 0xfffff000 (-4096)
[   56.543324] linux-kernel-bde (5538): subwin: 0xfffff000 (-4096)
[   56.751375] linux-kernel-bde (5538): subwin: 0xfffff000 (-4096)
[   56.959441] linux-kernel-bde (5538): subwin: 0xfffff000 (-4096)
[   57.167681] linux-kernel-bde (5538): subwin: 0xfffff000 (-4096)
Freeze here.

> Or maybe the reset puts the device in a state which is unexpected by the driver.
Maybe, but this happens once between 15 and 90 chip resets, according to our observation. So most of the time driver/HW handle it as expected.

> E.g. a global data structure in the driver needs to be properly locked if accessed concurrently by the probe function for separate devices.
We are aware that there is existing race condition in this driver, which was causing different issues in the past. Vendor played dead bug here.

All this summarized, we need to go back to vendor's support.

Thanks again,
Jozef

-----Original Message-----
From: Lukas Wunner <lukas@wunner.de> 
Sent: Wednesday, July 22, 2026 10:36 AM
To: Jozef Matejcik (Nokia) <jozef.matejcik@nokia.com>
Cc: linux-pci@vger.kernel.org; Peter Sys (Nokia) <peter.sys@nokia.com>
Subject: Re: PCI driver - hotplug of multiple devices


CAUTION: This is an external email. Please be very careful when clicking links or opening attachments. See the URL nok.it/ext for additional information.



On Wed, Jul 22, 2026 at 07:12:24AM +0000, Jozef Matejcik (Nokia) wrote:
> The driver is Broadcom's linux-kernel-bde.

This seems to be an out-of-tree driver, i.e. not something that is part of the mainline kernel.  The upstream community can only provide very limited support for such drivers.  The vendor needs to upstream the driver into the mainline kernel, otherwise there's only so much we can do.

> The issue:
>  - kernel module is inserted
>  - we de-assert reset of these PCI devices (previously in reset, i.e.
>    effectively powered down)
>  - sometimes, very infrequently, the box freezes for 1-3 minutes and
>    the HW triggers reboot

We'd need logs to say anything about it.

> When we removed the module before de-asserting the resets and then 
> inserted it later, the issue did not occur. So we suspect that the 
> kernel driver is accessing the device during probe too early and this 
> is causing HW freeze.

Or maybe the reset puts the device in a state which is unexpected by the driver.

> Are PCI drivers expected to handle hotplug events of any number of 
> their devices simultaneously?

Yes.  E.g. a global data structure in the driver needs to be properly locked if accessed concurrently by the probe function for separate devices.

> Can the issue I described above be considered a bug in vendor's 
> implementation?

Sounds like it.  Usually out-of-tree drivers haven't gone through the rigorous review process by the community and quality may thus be subpar.  The recommendation is to ask vendors whether driver support is in mainline already when selecting components for your products.

Thanks,

Lukas