[PATCH v3 0/2] PCI/IOV: Fix deadlock when removing PF with enabled SR-IOV

[PATCH v3 0/2] PCI/IOV: Fix deadlock when removing PF with enabled SR-IOV

[Niklas Schnelle]

Hi Bjorn,

Doing additional testing for a distribution backport of commit
05703271c3cd ("PCI/IOV: Add PCI rescan-remove locking when
enabling/disabling SR-IOV") Benjamin found a hang with s390's
recover attribute. Further investigation showed this to be a deadlock by
recursively trying to take pci_rescan_remove lock when removing a PF
with enabled SR-IOV.

The issue can be reproduced on both s390 and x86_64 with:

    $ echo <NUM> > /sys/bus/pci/devices/<pf>/sriov_numvfs
    $ echo 1 > /sys/bus/pci/devices/<pf>/remove

As this seems worse than the original, hard to hit, race fixed by the
cited commit I think we first want to revert the broken fix.

Following that patch 2 attempts to fix the original issue by taking the
PCI rescan/remove lock directly before calling into the driver's
sriov_configure() callback enforcing the rule that this should only
be called with the pci_rescan_remove_lock held.

Thanks,
Niklas

Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
---
Changes in v3:
- Rebased on v6.19-rc1, also verified issue is still there and the fix
  still works
- Added more of the lockdep splat for better context
- Link to v2: https://lore.kernel.org/r/20251119-revert_sriov_lock-v2-0-ea50eb1e8f96@linux.ibm.com

Changes in v2:
- Collected R-b from Benjamin
- Link to v1: https://lore.kernel.org/r/20251030-revert_sriov_lock-v1-0-70f82ade426f@linux.ibm.com

---
Niklas Schnelle (2):
      Revert "PCI/IOV: Add PCI rescan-remove locking when enabling/disabling SR-IOV"
      PCI/IOV: Fix race between SR-IOV enable/disable and hotplug

 drivers/pci/iov.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)
---
base-commit: 8f0b4cce4481fb22653697cced8d0d04027cb1e8
change-id: 20251029-revert_sriov_lock-aef4557f360f

Best regards,
-- 
Niklas Schnelle

[PATCH v3 1/2] Revert "PCI/IOV: Add PCI rescan-remove locking when enabling/disabling SR-IOV"

[Niklas Schnelle]

This reverts commit 05703271c3cd ("PCI/IOV: Add PCI rescan-remove
locking when enabling/disabling SR-IOV") which causes a deadlock by
recursively taking pci_rescan_remove_lock when sriov_del_vfs() is called
as part of pci_stop_and_remove_bus_device(). For example with the
following sequence of commands:

$ echo <NUM> > /sys/bus/pci/devices/<pf>/sriov_numvfs
$ echo 1 > /sys/bus/pci/devices/<pf>/remove

A trimmed trace of the deadlock on a mlx5 device is as below:

    zsh/5715 is trying to acquire lock:
    000002597926ef50 (pci_rescan_remove_lock){+.+.}-{3:3}, at: sriov_disable+0x34/0x140

    but task is already holding lock:
    000002597926ef50 (pci_rescan_remove_lock){+.+.}-{3:3}, at: pci_stop_and_remove_bus_device_locked+0x24/0x80
    ...
    Call Trace:
     [<00000259778c4f90>] dump_stack_lvl+0xc0/0x110
     [<00000259779c844e>] print_deadlock_bug+0x31e/0x330
     [<00000259779c1908>] __lock_acquire+0x16c8/0x32f0
     [<00000259779bffac>] lock_acquire+0x14c/0x350
     [<00000259789643a6>] __mutex_lock_common+0xe6/0x1520
     [<000002597896413c>] mutex_lock_nested+0x3c/0x50
     [<00000259784a07e4>] sriov_disable+0x34/0x140
     [<00000258f7d6dd80>] mlx5_sriov_disable+0x50/0x80 [mlx5_core]
     [<00000258f7d5745e>] remove_one+0x5e/0xf0 [mlx5_core]
     [<00000259784857fc>] pci_device_remove+0x3c/0xa0
     [<000002597851012e>] device_release_driver_internal+0x18e/0x280
     [<000002597847ae22>] pci_stop_bus_device+0x82/0xa0
     [<000002597847afce>] pci_stop_and_remove_bus_device_locked+0x5e/0x80
     [<00000259784972c2>] remove_store+0x72/0x90
     [<0000025977e6661a>] kernfs_fop_write_iter+0x15a/0x200
     [<0000025977d7241c>] vfs_write+0x24c/0x300
     [<0000025977d72696>] ksys_write+0x86/0x110
     [<000002597895b61c>] __do_syscall+0x14c/0x400
     [<000002597896e0ee>] system_call+0x6e/0x90

This alone is not a complete fix as it restores the issue the cited
commit tried to solve. A new fix will be provided as a follow on.

Cc: stable@vger.kernel.org
Fixes: 05703271c3cd ("PCI/IOV: Add PCI rescan-remove locking when enabling/disabling SR-IOV")
Reported-by: Benjamin Block <bblock@linux.ibm.com>
Reviewed-by: Benjamin Block <bblock@linux.ibm.com>
Acked-by: Gerd Bayer <gbayer@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
---
 drivers/pci/iov.c | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
index 00784a60ba80bb55ff2790d8f87e15a90c652a24..7de5b18647beb69127ba11234fb9f1dec9b50540 100644
--- a/drivers/pci/iov.c
+++ b/drivers/pci/iov.c
@@ -629,18 +629,15 @@ static int sriov_add_vfs(struct pci_dev *dev, u16 num_vfs)
 	if (dev->no_vf_scan)
 		return 0;
 
-	pci_lock_rescan_remove();
 	for (i = 0; i < num_vfs; i++) {
 		rc = pci_iov_add_virtfn(dev, i);
 		if (rc)
 			goto failed;
 	}
-	pci_unlock_rescan_remove();
 	return 0;
 failed:
 	while (i--)
 		pci_iov_remove_virtfn(dev, i);
-	pci_unlock_rescan_remove();
 
 	return rc;
 }
@@ -765,10 +762,8 @@ static void sriov_del_vfs(struct pci_dev *dev)
 	struct pci_sriov *iov = dev->sriov;
 	int i;
 
-	pci_lock_rescan_remove();
 	for (i = 0; i < iov->num_VFs; i++)
 		pci_iov_remove_virtfn(dev, i);
-	pci_unlock_rescan_remove();
 }
 
 static void sriov_disable(struct pci_dev *dev)

-- 
2.51.0

[PATCH v3 2/2] PCI/IOV: Fix race between SR-IOV enable/disable and hotplug

[Niklas Schnelle]

Commit 05703271c3cd ("PCI/IOV: Add PCI rescan-remove locking when
enabling/disabling SR-IOV") tried to fix a race between the VF removal
inside sriov_del_vfs() and concurrent hot unplug by taking the PCI
rescan/remove lock in sriov_del_vfs(). Similarly the PCI rescan/remove
lock was also taken in sriov_add_vfs() to protect addition of VFs.

This approach however causes deadlock on trying to remove PFs with
SR-IOV enabled because PFs disable SR-IOV during removal and this
removal happens under the PCI rescan/remove lock. So the original fix
had to be reverted.

Instead of taking the PCI rescan/remove lock in sriov_add_vfs() and
sriov_del_vfs(), fix the race that occurs with SR-IOV enable and disable
vs hotplug higher up in the callchain by taking the lock in
sriov_numvfs_store() before calling into the driver's sriov_configure()
callback.

Cc: stable@vger.kernel.org
Fixes: 05703271c3cd ("PCI/IOV: Add PCI rescan-remove locking when enabling/disabling SR-IOV")
Reported-by: Benjamin Block <bblock@linux.ibm.com>
Reviewed-by: Benjamin Block <bblock@linux.ibm.com>
Reviewed-by: Gerd Bayer <gbayer@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
---
 drivers/pci/iov.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
index 7de5b18647beb69127ba11234fb9f1dec9b50540..4a659c34935e116dd6d0b4ce42ed12a1ba9418d1 100644
--- a/drivers/pci/iov.c
+++ b/drivers/pci/iov.c
@@ -495,7 +495,9 @@ static ssize_t sriov_numvfs_store(struct device *dev,
 
 	if (num_vfs == 0) {
 		/* disable VFs */
+		pci_lock_rescan_remove();
 		ret = pdev->driver->sriov_configure(pdev, 0);
+		pci_unlock_rescan_remove();
 		goto exit;
 	}
 
@@ -507,7 +509,9 @@ static ssize_t sriov_numvfs_store(struct device *dev,
 		goto exit;
 	}
 
+	pci_lock_rescan_remove();
 	ret = pdev->driver->sriov_configure(pdev, num_vfs);
+	pci_unlock_rescan_remove();
 	if (ret < 0)
 		goto exit;
 

-- 
2.51.0

Re: [PATCH v3 0/2] PCI/IOV: Fix deadlock when removing PF with enabled SR-IOV

[Thorsten Leemhuis]

Lo! Top-posting to facilitate processing.

The issue fixed by these patches is on my list on tracked regressions.
Makes me wonder: Did it fall through the cracks due to the festive
season, was this fixes in a different way, or is this obsolete for some
reason?

Ciao, Thorsten

On 12/16/25 23:14, Niklas Schnelle wrote:
> Hi Bjorn,
> 
> Doing additional testing for a distribution backport of commit
> 05703271c3cd ("PCI/IOV: Add PCI rescan-remove locking when
> enabling/disabling SR-IOV") Benjamin found a hang with s390's
> recover attribute. Further investigation showed this to be a deadlock by
> recursively trying to take pci_rescan_remove lock when removing a PF
> with enabled SR-IOV.
> 
> The issue can be reproduced on both s390 and x86_64 with:
> 
>     $ echo <NUM> > /sys/bus/pci/devices/<pf>/sriov_numvfs
>     $ echo 1 > /sys/bus/pci/devices/<pf>/remove
> 
> As this seems worse than the original, hard to hit, race fixed by the
> cited commit I think we first want to revert the broken fix.
> 
> Following that patch 2 attempts to fix the original issue by taking the
> PCI rescan/remove lock directly before calling into the driver's
> sriov_configure() callback enforcing the rule that this should only
> be called with the pci_rescan_remove_lock held.
> 
> Thanks,
> Niklas
> 
> Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
> ---
> Changes in v3:
> - Rebased on v6.19-rc1, also verified issue is still there and the fix
>   still works
> - Added more of the lockdep splat for better context
> - Link to v2: https://lore.kernel.org/r/20251119-revert_sriov_lock-v2-0-ea50eb1e8f96@linux.ibm.com
> 
> Changes in v2:
> - Collected R-b from Benjamin
> - Link to v1: https://lore.kernel.org/r/20251030-revert_sriov_lock-v1-0-70f82ade426f@linux.ibm.com
> 
> ---
> Niklas Schnelle (2):
>       Revert "PCI/IOV: Add PCI rescan-remove locking when enabling/disabling SR-IOV"
>       PCI/IOV: Fix race between SR-IOV enable/disable and hotplug
> 
>  drivers/pci/iov.c | 9 ++++-----
>  1 file changed, 4 insertions(+), 5 deletions(-)
> ---
> base-commit: 8f0b4cce4481fb22653697cced8d0d04027cb1e8
> change-id: 20251029-revert_sriov_lock-aef4557f360f
> 
> Best regards,

Re: [PATCH v3 0/2] PCI/IOV: Fix deadlock when removing PF with enabled SR-IOV

[Niklas Schnelle]

On Sun, 2026-02-01 at 16:56 +0100, Thorsten Leemhuis wrote:
> Lo! Top-posting to facilitate processing.
> 
> The issue fixed by these patches is on my list on tracked regressions.
> Makes me wonder: Did it fall through the cracks due to the festive
> season, was this fixes in a different way, or is this obsolete for some
> reason?
> 
> Ciao, Thorsten
> 
> On 12/16/25 23:14, Niklas Schnelle wrote:
> > 
--- snip ---
> > The issue can be reproduced on both s390 and x86_64 with:
> > 
> >     $ echo <NUM> > /sys/bus/pci/devices/<pf>/sriov_numvfs
> >     $ echo 1 > /sys/bus/pci/devices/<pf>/remove
> > 
> > As this seems worse than the original, hard to hit, race fixed by the
> > cited commit I think we first want to revert the broken fix.

Hi Thorsten,

The issue is definitely still current, I just reproduced with the above
instructions on v6.19-rc8.

I fear in general there may still be some further issues with SR-IOV PF
disable path but this is still the best fix I know for my broken commit
(see Fixes tag) and the issue it tried to fix.

Thanks,
Niklas

Re: [PATCH v3 0/2] PCI/IOV: Fix deadlock when removing PF with enabled SR-IOV

[Bjorn Helgaas]

On Tue, Dec 16, 2025 at 11:14:01PM +0100, Niklas Schnelle wrote:
> Hi Bjorn,
> 
> Doing additional testing for a distribution backport of commit
> 05703271c3cd ("PCI/IOV: Add PCI rescan-remove locking when
> enabling/disabling SR-IOV") Benjamin found a hang with s390's
> recover attribute. Further investigation showed this to be a deadlock by
> recursively trying to take pci_rescan_remove lock when removing a PF
> with enabled SR-IOV.
> 
> The issue can be reproduced on both s390 and x86_64 with:
> 
>     $ echo <NUM> > /sys/bus/pci/devices/<pf>/sriov_numvfs
>     $ echo 1 > /sys/bus/pci/devices/<pf>/remove
> 
> As this seems worse than the original, hard to hit, race fixed by the
> cited commit I think we first want to revert the broken fix.
> 
> Following that patch 2 attempts to fix the original issue by taking the
> PCI rescan/remove lock directly before calling into the driver's
> sriov_configure() callback enforcing the rule that this should only
> be called with the pci_rescan_remove_lock held.
> 
> Thanks,
> Niklas
> 
> Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
> ---
> Changes in v3:
> - Rebased on v6.19-rc1, also verified issue is still there and the fix
>   still works
> - Added more of the lockdep splat for better context
> - Link to v2: https://lore.kernel.org/r/20251119-revert_sriov_lock-v2-0-ea50eb1e8f96@linux.ibm.com
> 
> Changes in v2:
> - Collected R-b from Benjamin
> - Link to v1: https://lore.kernel.org/r/20251030-revert_sriov_lock-v1-0-70f82ade426f@linux.ibm.com
> 
> ---
> Niklas Schnelle (2):
>       Revert "PCI/IOV: Add PCI rescan-remove locking when enabling/disabling SR-IOV"
>       PCI/IOV: Fix race between SR-IOV enable/disable and hotplug
> 
>  drivers/pci/iov.c | 9 ++++-----
>  1 file changed, 4 insertions(+), 5 deletions(-)
> ---
> base-commit: 8f0b4cce4481fb22653697cced8d0d04027cb1e8
> change-id: 20251029-revert_sriov_lock-aef4557f360f

Applied to pci/iov for v6.20, thanks!

Re: [PATCH v3 0/2] PCI/IOV: Fix deadlock when removing PF with enabled SR-IOV

[Dragos Tatulea]

Hi Niklas,

On 03.02.26 01:48, Bjorn Helgaas wrote:
> On Tue, Dec 16, 2025 at 11:14:01PM +0100, Niklas Schnelle wrote:
>> Hi Bjorn,
>>
>> Doing additional testing for a distribution backport of commit
>> 05703271c3cd ("PCI/IOV: Add PCI rescan-remove locking when
>> enabling/disabling SR-IOV") Benjamin found a hang with s390's
>> recover attribute. Further investigation showed this to be a deadlock by
>> recursively trying to take pci_rescan_remove lock when removing a PF
>> with enabled SR-IOV.
>>
>> The issue can be reproduced on both s390 and x86_64 with:
>>
>>     $ echo <NUM> > /sys/bus/pci/devices/<pf>/sriov_numvfs
>>     $ echo 1 > /sys/bus/pci/devices/<pf>/remove
>>
>> As this seems worse than the original, hard to hit, race fixed by the
>> cited commit I think we first want to revert the broken fix.
>>
>> Following that patch 2 attempts to fix the original issue by taking the
>> PCI rescan/remove lock directly before calling into the driver's
>> sriov_configure() callback enforcing the rule that this should only
>> be called with the pci_rescan_remove_lock held.
>>
>> Thanks,
>> Niklas
>>
>> Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
>> ---
>> Changes in v3:
>> - Rebased on v6.19-rc1, also verified issue is still there and the fix
>>   still works
>> - Added more of the lockdep splat for better context
>> - Link to v2: https://lore.kernel.org/r/20251119-revert_sriov_lock-v2-0-ea50eb1e8f96@linux.ibm.com
>>
>> Changes in v2:
>> - Collected R-b from Benjamin
>> - Link to v1: https://lore.kernel.org/r/20251030-revert_sriov_lock-v1-0-70f82ade426f@linux.ibm.com
>>
>> ---
>> Niklas Schnelle (2):
>>       Revert "PCI/IOV: Add PCI rescan-remove locking when enabling/disabling SR-IOV"
>>       PCI/IOV: Fix race between SR-IOV enable/disable and hotplug
>>
>>  drivers/pci/iov.c | 9 ++++-----
>>  1 file changed, 4 insertions(+), 5 deletions(-)
>> ---
>> base-commit: 8f0b4cce4481fb22653697cced8d0d04027cb1e8
>> change-id: 20251029-revert_sriov_lock-aef4557f360f
> 
> Applied to pci/iov for v6.20, thanks!

After pulling in these commits in our internal tree we can see the
lockdep splat from below in many internal tests. We are still trying to
find an easy repro for this. We had to internally revert both of them.

I noticed some similar discussion in another thread [1] but there it
seems that these changes are actually fixing the issue which is not
the case for us.

------------[ cut here ]------------
WARNING: drivers/pci/remove.c:130 at pci_stop_and_remove_bus_device+0x39/0x40, CPU#2: modprobe/12956
Modules linked in: mlx5_core(-) act_tunnel_key vxlan dummy act_mirred act_gact cls_flower act_police act_ct nf_flow_table [...]
CPU: 2 UID: 0 PID: 12956 Comm: modprobe Not tainted 6.19.0net_next_e834b5e #1 PREEMPT 
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
RIP: 0010:pci_stop_and_remove_bus_device+0x39/0x40
Code: [...]
RSP: 0018:ffff888164c9fd10 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff888188ff2000 RCX: 0000000000000001
RDX: 0000000000000046 RSI: ffffffff8307e068 RDI: ffff88816bf4c9c0
RBP: ffff888188ff2000 R08: 00000000000000f4 R09: ffff88816bf4c080
R10: 0000000000000001 R11: 0000000000000003 R12: 0000000000000000
R13: ffff888164c9fd27 R14: 0000000000000002 R15: 0000000000000000
FS:  00007f52364bd740(0000) GS:ffff8885a9019000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005622dbf749d8 CR3: 0000000169132004 CR4: 0000000000372eb0
Call Trace:
 <TASK>
 pci_iov_remove_virtfn+0xbd/0x120
 sriov_disable+0x30/0xe0
 mlx5_sriov_disable+0x50/0xa0 [mlx5_core]
 remove_one+0x68/0xe0 [mlx5_core]
 pci_device_remove+0x39/0xa0
 device_release_driver_internal+0x1e4/0x240
 driver_detach+0x47/0x90
 bus_remove_driver+0x84/0x110
 pci_unregister_driver+0x3b/0x90
 mlx5_cleanup+0x13/0x40 [mlx5_core]
 __x64_sys_delete_module+0x16f/0x290
 ? kmem_cache_free+0x221/0x520
 do_syscall_64+0xa8/0x13f0
 entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x7f5235f2c3fb
Code: [...]
RSP: 002b:00007ffc6ba11518 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0
RAX: ffffffffffffffda RBX: 00005558c4278f30 RCX: 00007f5235f2c3fb
RDX: 0000000000000000 RSI: 0000000000000800 RDI: 00005558c4278f98
RBP: 00007ffc6ba11540 R08: 1999999999999999 R09: 0000000000000000
R10: 00007f5235fa5fe0 R11: 0000000000000206 R12: 0000000000000000
R13: 00007ffc6ba11570 R14: 0000000000000000 R15: 0000000000000000
 </TASK>
irq event stamp: 44859
hardirqs last  enabled at (44869): [<ffffffff814af7ca>] __up_console_sem+0x5a/0x70
hardirqs last disabled at (44878): [<ffffffff814af7af>] __up_console_sem+0x3f/0x70
softirqs last  enabled at (44844): [<ffffffff81430312>] irq_exit_rcu+0x82/0xe0
softirqs last disabled at (44821): [<ffffffff81430312>] irq_exit_rcu+0x82/0xe0
---[ end trace 0000000000000000 ]---

[1] https://lore.kernel.org/all/20260222112904.171858-1-ionut.nechita@windriver.com/

Thanks,
Dragos

Re: [PATCH v3 0/2] PCI/IOV: Fix deadlock when removing PF with enabled SR-IOV

[Benjamin Block]

On Mon, Feb 23, 2026 at 03:10:35PM +0100, Dragos Tatulea wrote:
> After pulling in these commits in our internal tree we can see the
> lockdep splat from below in many internal tests. We are still trying to
> find an easy repro for this. We had to internally revert both of them.
> 
> I noticed some similar discussion in another thread [1] but there it
> seems that these changes are actually fixing the issue which is not
> the case for us.
> 
> ------------[ cut here ]------------
> WARNING: drivers/pci/remove.c:130 at pci_stop_and_remove_bus_device+0x39/0x40, CPU#2: modprobe/12956
> Modules linked in: mlx5_core(-) act_tunnel_key vxlan dummy act_mirred act_gact cls_flower act_police act_ct nf_flow_table [...]
> CPU: 2 UID: 0 PID: 12956 Comm: modprobe Not tainted 6.19.0net_next_e834b5e #1 PREEMPT 
> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
> RIP: 0010:pci_stop_and_remove_bus_device+0x39/0x40
> Code: [...]
> RSP: 0018:ffff888164c9fd10 EFLAGS: 00010246
> RAX: 0000000000000000 RBX: ffff888188ff2000 RCX: 0000000000000001
> RDX: 0000000000000046 RSI: ffffffff8307e068 RDI: ffff88816bf4c9c0
> RBP: ffff888188ff2000 R08: 00000000000000f4 R09: ffff88816bf4c080
> R10: 0000000000000001 R11: 0000000000000003 R12: 0000000000000000
> R13: ffff888164c9fd27 R14: 0000000000000002 R15: 0000000000000000
> FS:  00007f52364bd740(0000) GS:ffff8885a9019000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00005622dbf749d8 CR3: 0000000169132004 CR4: 0000000000372eb0
> Call Trace:
>  <TASK>
>  pci_iov_remove_virtfn+0xbd/0x120
>  sriov_disable+0x30/0xe0
>  mlx5_sriov_disable+0x50/0xa0 [mlx5_core]
>  remove_one+0x68/0xe0 [mlx5_core]
>  pci_device_remove+0x39/0xa0
>  device_release_driver_internal+0x1e4/0x240
>  driver_detach+0x47/0x90
>  bus_remove_driver+0x84/0x110
>  pci_unregister_driver+0x3b/0x90

This looks pretty much like what Ionut is trying to fix in
v1: https://lore.kernel.org/linux-pci/20260214193235.262219-3-ionut.nechita@windriver.com/T/
v2: https://lore.kernel.org/linux-pci/20260219212648.82606-1-ionut.nechita@windriver.com/T/

Maybe try giving those patches a spin. I think one easy way to hit this sort
of thing is to try unbinding a PF that has 1 or more VFs attached to it from
some device driver. The "trick" is that SR-IOV has to be active.

>  mlx5_cleanup+0x13/0x40 [mlx5_core]
>  __x64_sys_delete_module+0x16f/0x290
>  ? kmem_cache_free+0x221/0x520
>  do_syscall_64+0xa8/0x13f0
>  entry_SYSCALL_64_after_hwframe+0x4b/0x53
> RIP: 0033:0x7f5235f2c3fb
> Code: [...]
> RSP: 002b:00007ffc6ba11518 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0
> RAX: ffffffffffffffda RBX: 00005558c4278f30 RCX: 00007f5235f2c3fb
> RDX: 0000000000000000 RSI: 0000000000000800 RDI: 00005558c4278f98
> RBP: 00007ffc6ba11540 R08: 1999999999999999 R09: 0000000000000000
> R10: 00007f5235fa5fe0 R11: 0000000000000206 R12: 0000000000000000
> R13: 00007ffc6ba11570 R14: 0000000000000000 R15: 0000000000000000
>  </TASK>
> irq event stamp: 44859
> hardirqs last  enabled at (44869): [<ffffffff814af7ca>] __up_console_sem+0x5a/0x70
> hardirqs last disabled at (44878): [<ffffffff814af7af>] __up_console_sem+0x3f/0x70
> softirqs last  enabled at (44844): [<ffffffff81430312>] irq_exit_rcu+0x82/0xe0
> softirqs last disabled at (44821): [<ffffffff81430312>] irq_exit_rcu+0x82/0xe0
> ---[ end trace 0000000000000000 ]---
> 
> [1] https://lore.kernel.org/all/20260222112904.171858-1-ionut.nechita@windriver.com/


-- 
Best Regards, Benjamin Block        /        Linux on IBM Z Kernel Development
IBM Deutschland Research & Development GmbH    /   https://www.ibm.com/privacy
Vors. Aufs.-R.: Wolfgang Wendt         /        Geschäftsführung: David Faller
Sitz der Ges.: Ehningen     /     Registergericht: AmtsG Stuttgart, HRB 243294

Re: [PATCH v3 0/2] PCI/IOV: Fix deadlock when removing PF with enabled SR-IOV

[Dragos Tatulea]

On 23.02.26 18:33, Benjamin Block wrote:
> On Mon, Feb 23, 2026 at 03:10:35PM +0100, Dragos Tatulea wrote:
>> After pulling in these commits in our internal tree we can see the
>> lockdep splat from below in many internal tests. We are still trying to
>> find an easy repro for this. We had to internally revert both of them.
>>
>> I noticed some similar discussion in another thread [1] but there it
>> seems that these changes are actually fixing the issue which is not
>> the case for us.
>>
>> ------------[ cut here ]------------
>> WARNING: drivers/pci/remove.c:130 at pci_stop_and_remove_bus_device+0x39/0x40, CPU#2: modprobe/12956
>> Modules linked in: mlx5_core(-) act_tunnel_key vxlan dummy act_mirred act_gact cls_flower act_police act_ct nf_flow_table [...]
>> CPU: 2 UID: 0 PID: 12956 Comm: modprobe Not tainted 6.19.0net_next_e834b5e #1 PREEMPT 
>> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
>> RIP: 0010:pci_stop_and_remove_bus_device+0x39/0x40
>> Code: [...]
>> RSP: 0018:ffff888164c9fd10 EFLAGS: 00010246
>> RAX: 0000000000000000 RBX: ffff888188ff2000 RCX: 0000000000000001
>> RDX: 0000000000000046 RSI: ffffffff8307e068 RDI: ffff88816bf4c9c0
>> RBP: ffff888188ff2000 R08: 00000000000000f4 R09: ffff88816bf4c080
>> R10: 0000000000000001 R11: 0000000000000003 R12: 0000000000000000
>> R13: ffff888164c9fd27 R14: 0000000000000002 R15: 0000000000000000
>> FS:  00007f52364bd740(0000) GS:ffff8885a9019000(0000) knlGS:0000000000000000
>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> CR2: 00005622dbf749d8 CR3: 0000000169132004 CR4: 0000000000372eb0
>> Call Trace:
>>  <TASK>
>>  pci_iov_remove_virtfn+0xbd/0x120
>>  sriov_disable+0x30/0xe0
>>  mlx5_sriov_disable+0x50/0xa0 [mlx5_core]
>>  remove_one+0x68/0xe0 [mlx5_core]
>>  pci_device_remove+0x39/0xa0
>>  device_release_driver_internal+0x1e4/0x240
>>  driver_detach+0x47/0x90
>>  bus_remove_driver+0x84/0x110
>>  pci_unregister_driver+0x3b/0x90
> 
> This looks pretty much like what Ionut is trying to fix in
> v1: https://lore.kernel.org/linux-pci/20260214193235.262219-3-ionut.nechita@windriver.com/T/
> v2: https://lore.kernel.org/linux-pci/20260219212648.82606-1-ionut.nechita@windriver.com/T/
> 
> Maybe try giving those patches a spin. I think one easy way to hit this sort
> of thing is to try unbinding a PF that has 1 or more VFs attached to it from
> some device driver. The "trick" is that SR-IOV has to be active.
Thanks or the pointer. Will try it.

Thanks,
Dragos

Re: [PATCH v3 0/2] PCI/IOV: Fix deadlock when removing PF with enabled SR-IOV

[Dragos Tatulea]

On 23.02.26 19:34, Dragos Tatulea wrote:
> 
> 
> On 23.02.26 18:33, Benjamin Block wrote:
>> On Mon, Feb 23, 2026 at 03:10:35PM +0100, Dragos Tatulea wrote:
>>> After pulling in these commits in our internal tree we can see the
>>> lockdep splat from below in many internal tests. We are still trying to
>>> find an easy repro for this. We had to internally revert both of them.
>>>
>>> I noticed some similar discussion in another thread [1] but there it
>>> seems that these changes are actually fixing the issue which is not
>>> the case for us.
>>>
>>> ------------[ cut here ]------------
>>> WARNING: drivers/pci/remove.c:130 at pci_stop_and_remove_bus_device+0x39/0x40, CPU#2: modprobe/12956
>>> Modules linked in: mlx5_core(-) act_tunnel_key vxlan dummy act_mirred act_gact cls_flower act_police act_ct nf_flow_table [...]
>>> CPU: 2 UID: 0 PID: 12956 Comm: modprobe Not tainted 6.19.0net_next_e834b5e #1 PREEMPT 
>>> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
>>> RIP: 0010:pci_stop_and_remove_bus_device+0x39/0x40
>>> Code: [...]
>>> RSP: 0018:ffff888164c9fd10 EFLAGS: 00010246
>>> RAX: 0000000000000000 RBX: ffff888188ff2000 RCX: 0000000000000001
>>> RDX: 0000000000000046 RSI: ffffffff8307e068 RDI: ffff88816bf4c9c0
>>> RBP: ffff888188ff2000 R08: 00000000000000f4 R09: ffff88816bf4c080
>>> R10: 0000000000000001 R11: 0000000000000003 R12: 0000000000000000
>>> R13: ffff888164c9fd27 R14: 0000000000000002 R15: 0000000000000000
>>> FS:  00007f52364bd740(0000) GS:ffff8885a9019000(0000) knlGS:0000000000000000
>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> CR2: 00005622dbf749d8 CR3: 0000000169132004 CR4: 0000000000372eb0
>>> Call Trace:
>>>  <TASK>
>>>  pci_iov_remove_virtfn+0xbd/0x120
>>>  sriov_disable+0x30/0xe0
>>>  mlx5_sriov_disable+0x50/0xa0 [mlx5_core]
>>>  remove_one+0x68/0xe0 [mlx5_core]
>>>  pci_device_remove+0x39/0xa0
>>>  device_release_driver_internal+0x1e4/0x240
>>>  driver_detach+0x47/0x90
>>>  bus_remove_driver+0x84/0x110
>>>  pci_unregister_driver+0x3b/0x90
>>
>> This looks pretty much like what Ionut is trying to fix in
>> v1: https://lore.kernel.org/linux-pci/20260214193235.262219-3-ionut.nechita@windriver.com/T/
>> v2: https://lore.kernel.org/linux-pci/20260219212648.82606-1-ionut.nechita@windriver.com/T/
>>
>> Maybe try giving those patches a spin. I think one easy way to hit this sort
>> of thing is to try unbinding a PF that has 1 or more VFs attached to it from
>> some device driver. The "trick" is that SR-IOV has to be active.
> Thanks or the pointer. Will try it.
> 
Took the v2 and it did the trick. Thanks!
Is it worth a Tested-by tag from me?

Thanks,
Dragos

Re: [PATCH v3 0/2] PCI/IOV: Fix deadlock when removing PF with enabled SR-IOV

[Bjorn Helgaas]

On Wed, Feb 25, 2026 at 03:59:49PM +0100, Dragos Tatulea wrote:
> On 23.02.26 19:34, Dragos Tatulea wrote:
> > On 23.02.26 18:33, Benjamin Block wrote:
> >> On Mon, Feb 23, 2026 at 03:10:35PM +0100, Dragos Tatulea wrote:
> >>> After pulling in these commits in our internal tree we can see the
> >>> lockdep splat from below in many internal tests. We are still trying to
> >>> find an easy repro for this. We had to internally revert both of them.
> >>>
> >>> I noticed some similar discussion in another thread [1] but there it
> >>> seems that these changes are actually fixing the issue which is not
> >>> the case for us.
> >>>
> >>> ------------[ cut here ]------------
> >>> WARNING: drivers/pci/remove.c:130 at pci_stop_and_remove_bus_device+0x39/0x40, CPU#2: modprobe/12956
> >>> Modules linked in: mlx5_core(-) act_tunnel_key vxlan dummy act_mirred act_gact cls_flower act_police act_ct nf_flow_table [...]
> >>> CPU: 2 UID: 0 PID: 12956 Comm: modprobe Not tainted 6.19.0net_next_e834b5e #1 PREEMPT 
> >>> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
> >>> RIP: 0010:pci_stop_and_remove_bus_device+0x39/0x40
> >>> Code: [...]
> >>> RSP: 0018:ffff888164c9fd10 EFLAGS: 00010246
> >>> RAX: 0000000000000000 RBX: ffff888188ff2000 RCX: 0000000000000001
> >>> RDX: 0000000000000046 RSI: ffffffff8307e068 RDI: ffff88816bf4c9c0
> >>> RBP: ffff888188ff2000 R08: 00000000000000f4 R09: ffff88816bf4c080
> >>> R10: 0000000000000001 R11: 0000000000000003 R12: 0000000000000000
> >>> R13: ffff888164c9fd27 R14: 0000000000000002 R15: 0000000000000000
> >>> FS:  00007f52364bd740(0000) GS:ffff8885a9019000(0000) knlGS:0000000000000000
> >>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> >>> CR2: 00005622dbf749d8 CR3: 0000000169132004 CR4: 0000000000372eb0
> >>> Call Trace:
> >>>  <TASK>
> >>>  pci_iov_remove_virtfn+0xbd/0x120
> >>>  sriov_disable+0x30/0xe0
> >>>  mlx5_sriov_disable+0x50/0xa0 [mlx5_core]
> >>>  remove_one+0x68/0xe0 [mlx5_core]
> >>>  pci_device_remove+0x39/0xa0
> >>>  device_release_driver_internal+0x1e4/0x240
> >>>  driver_detach+0x47/0x90
> >>>  bus_remove_driver+0x84/0x110
> >>>  pci_unregister_driver+0x3b/0x90
> >>
> >> This looks pretty much like what Ionut is trying to fix in
> >> v1: https://lore.kernel.org/linux-pci/20260214193235.262219-3-ionut.nechita@windriver.com/T/
> >> v2: https://lore.kernel.org/linux-pci/20260219212648.82606-1-ionut.nechita@windriver.com/T/
> >>
> >> Maybe try giving those patches a spin. I think one easy way to hit this sort
> >> of thing is to try unbinding a PF that has 1 or more VFs attached to it from
> >> some device driver. The "trick" is that SR-IOV has to be active.
> > Thanks or the pointer. Will try it.
> > 
> Took the v2 and it did the trick. Thanks!
> Is it worth a Tested-by tag from me?

Definitely, I always like to include a Tested-by, both to acknowledge
your effort in testing and to able to include you if we trip over
similar issues in the future.

If you respond to the patch you tested and include "Tested-by", the
tools will pick it up automatically.

Re: [PATCH v3 2/2] PCI/IOV: Fix race between SR-IOV enable/disable and hotplug

[Guenter Roeck]

Hi,

On Tue, Dec 16, 2025 at 11:14:03PM +0100, Niklas Schnelle wrote:
> Commit 05703271c3cd ("PCI/IOV: Add PCI rescan-remove locking when
> enabling/disabling SR-IOV") tried to fix a race between the VF removal
> inside sriov_del_vfs() and concurrent hot unplug by taking the PCI
> rescan/remove lock in sriov_del_vfs(). Similarly the PCI rescan/remove
> lock was also taken in sriov_add_vfs() to protect addition of VFs.
> 
> This approach however causes deadlock on trying to remove PFs with
> SR-IOV enabled because PFs disable SR-IOV during removal and this
> removal happens under the PCI rescan/remove lock. So the original fix
> had to be reverted.
> 
> Instead of taking the PCI rescan/remove lock in sriov_add_vfs() and
> sriov_del_vfs(), fix the race that occurs with SR-IOV enable and disable
> vs hotplug higher up in the callchain by taking the lock in
> sriov_numvfs_store() before calling into the driver's sriov_configure()
> callback.
> 
> Cc: stable@vger.kernel.org
> Fixes: 05703271c3cd ("PCI/IOV: Add PCI rescan-remove locking when enabling/disabling SR-IOV")
> Reported-by: Benjamin Block <bblock@linux.ibm.com>
> Reviewed-by: Benjamin Block <bblock@linux.ibm.com>
> Reviewed-by: Gerd Bayer <gbayer@linux.ibm.com>
> Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
> ---
>  drivers/pci/iov.c | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> index 7de5b18647beb69127ba11234fb9f1dec9b50540..4a659c34935e116dd6d0b4ce42ed12a1ba9418d1 100644
> --- a/drivers/pci/iov.c
> +++ b/drivers/pci/iov.c
> @@ -495,7 +495,9 @@ static ssize_t sriov_numvfs_store(struct device *dev,
>  
>  	if (num_vfs == 0) {
>  		/* disable VFs */
> +		pci_lock_rescan_remove();
>  		ret = pdev->driver->sriov_configure(pdev, 0);
> +		pci_unlock_rescan_remove();
>  		goto exit;
>  	}
>  
> @@ -507,7 +509,9 @@ static ssize_t sriov_numvfs_store(struct device *dev,
>  		goto exit;
>  	}
>  
> +	pci_lock_rescan_remove();
>  	ret = pdev->driver->sriov_configure(pdev, num_vfs);
> +	pci_unlock_rescan_remove();
>  	if (ret < 0)
>  		goto exit;
>  
Google's experimental AI review agent provided the following feedback
on this patch.

  Could this introduce an AB-BA deadlock between the device lock and the
  rescan/remove lock?

  Earlier in sriov_numvfs_store(), device_lock(&pdev->dev) is acquired. The
  patch then attempts to acquire pci_lock_rescan_remove() while holding the
  device lock.

  However, during a hotplug removal of the PF (for example, via sysfs),
  remove_store() first acquires pci_lock_rescan_remove() and subsequently
  calls pci_stop_and_remove_bus_device_locked(). That path eventually calls
  device_release_driver(), which attempts to acquire device_lock(&pdev->dev).

  If sriov_numvfs_store() and a concurrent removal of the PF race, it appears
  they could deadlock waiting on each other's locks.

The actual call sequence (at least in v6.12.y, where this patch was
backported to) is as follows.
  remove_store()
  -> pci_stop_and_remove_bus_device_locked()
     -> pci_lock_rescan_remove()
        -> pci_stop_and_remove_bus_device()
	   -> pci_stop_bus_device()
	   -> pci_remove_bus_device()
	      -> pci_remove_bus()
	         -> device_unregister()
		    -> device_del()
		       -> device_lock()

I don't claim to fully understand the code, but the AI does seem to have a
point. Please let me know if the AI analysis is correct or if it misses
something.

Thanks,
Guenter

Re: [PATCH v3 2/2] PCI/IOV: Fix race between SR-IOV enable/disable and hotplug

[Benjamin Block]

On Mon, Mar 16, 2026 at 06:57:53PM -0700, Guenter Roeck wrote:
> > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> > index 7de5b18647beb69127ba11234fb9f1dec9b50540..4a659c34935e116dd6d0b4ce42ed12a1ba9418d1 100644
> > --- a/drivers/pci/iov.c
> > +++ b/drivers/pci/iov.c
> > @@ -495,7 +495,9 @@ static ssize_t sriov_numvfs_store(struct device *dev,
> >  
> >  	if (num_vfs == 0) {
> >  		/* disable VFs */
> > +		pci_lock_rescan_remove();
> >  		ret = pdev->driver->sriov_configure(pdev, 0);
> > +		pci_unlock_rescan_remove();
> >  		goto exit;
> >  	}
> >  
> > @@ -507,7 +509,9 @@ static ssize_t sriov_numvfs_store(struct device *dev,
> >  		goto exit;
> >  	}
> >  
> > +	pci_lock_rescan_remove();
> >  	ret = pdev->driver->sriov_configure(pdev, num_vfs);
> > +	pci_unlock_rescan_remove();
> >  	if (ret < 0)
> >  		goto exit;
> >  
>
> Google's experimental AI review agent provided the following feedback
> on this patch.
> 
>   Could this introduce an AB-BA deadlock between the device lock and the
>   rescan/remove lock?
> 
>   Earlier in sriov_numvfs_store(), device_lock(&pdev->dev) is acquired. The
>   patch then attempts to acquire pci_lock_rescan_remove() while holding the
>   device lock.
> 
>   However, during a hotplug removal of the PF (for example, via sysfs),
>   remove_store() first acquires pci_lock_rescan_remove() and subsequently
>   calls pci_stop_and_remove_bus_device_locked(). That path eventually calls
>   device_release_driver(), which attempts to acquire device_lock(&pdev->dev).
> 
>   If sriov_numvfs_store() and a concurrent removal of the PF race, it appears
>   they could deadlock waiting on each other's locks.
> 
> The actual call sequence (at least in v6.12.y, where this patch was
> backported to) is as follows.
>   remove_store()
>   -> pci_stop_and_remove_bus_device_locked()
>      -> pci_lock_rescan_remove()
>         -> pci_stop_and_remove_bus_device()
> 	   -> pci_stop_bus_device()
> 	   -> pci_remove_bus_device()
> 	      -> pci_remove_bus()
> 	         -> device_unregister()
> 		    -> device_del()
> 		       -> device_lock()
> 
> I don't claim to fully understand the code, but the AI does seem to have a
> point. Please let me know if the AI analysis is correct or if it misses
> something.

Ugh. Well. That sucks. This lock is a sheer endless well of joy.
No, well, I think the AI is correct.

We've since discussed to move away from that patch again, or rather,
improve it further by applying this in top:
    https://lore.kernel.org/linux-pci/20260310074303.17480-2-ionut.nechita@windriver.com/

Because it improves some scenarios, such as driver core unbinds.
But looking at it from this angle, it suffers from the same AB-BA cyclic
deadlock.

  remove_store()
    |
    +- pci_stop_and_remove_bus_device_locked()
        |
        +- takes: pci_rescan_remove_lock                    # XXX
        |
        +- pci_stop_and_remove_bus_device()
            |
            +- pci_stop_bus_device()
                |
                +- pci_stop_dev()
                    |
                    +- device_release_driver()
                        |
                        +- device_release_driver_internal()
                            |
                            +- __device_driver_lock()
                                |
                                +- __device_driver_lock() - takes: pdev->dev

  unbind_store()
    |
    +- device_driver_detach()
        |
        +- device_release_driver_internal()
            |
            +- __device_driver_lock() - takes: pdev->dev    # XXX
            |
            +- __device_release_driver()
                |
                +- device_remove()
                    |
                    +- pci_device_remove()
                        |
                        +- vfio_pci_remove()
                            |
                            +- vfio_pci_core_sriov_configure()
                                |
                                +- pci_disable_sriov()
                                    |
                                    +- sriov_disable()
                                        |
                                        +- sriov_del_vfs()
                                            |
                                            +- takes: pci_rescan_remove_lock

And there is no way I can see how we can reverse the lock order in the
unbind_store() case, since everything above pci_device_remove() is owned
by the driver core itself. I don't see a way for us to put a hook in
there to take `pci_rescan_remove_lock`.

It's similar to what I'm trying to fix in:
    https://lore.kernel.org/linux-pci/354b9e4a54ced67f3c89df198041df19434fe4c8.1773235561.git.bblock@linux.ibm.com/
Taking `pci_rescan_remove_lock` inside the release functions is fraught
with traps, especially with SR-IOV in the mix.

One quick idea: can we somehow unbind the device from any device driver
in remove_store() before calling
pci_stop_and_remove_bus_device_locked()?  That way we would not have any
SR-IOV functions attached anymore at the point where we remove the PF,
since the DD are expected to clean them up.

-- 
Best Regards und Beste Grüße, Benjamin Block
               PGP KeyID: 9610 2BB8 2E17 6F65 2362  6DF2 46E0 4E05 67A3 2E9E

Re: [PATCH v3 2/2] PCI/IOV: Fix race between SR-IOV enable/disable and hotplug

[Benjamin Block]

On Tue, Mar 17, 2026 at 10:01:49AM +0100, Benjamin Block wrote:
> One quick idea: can we somehow unbind the device from any device driver
> in remove_store() before calling
> pci_stop_and_remove_bus_device_locked()?  That way we would not have any
> SR-IOV functions attached anymore at the point where we remove the PF,
> since the DD are expected to clean them up.

Very quick-and-dirty:

diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c
index 16eaaf749ba9..490b6c1f07d3 100644
--- a/drivers/pci/pci-sysfs.c
+++ b/drivers/pci/pci-sysfs.c
@@ -33,6 +33,8 @@
 #include <linux/unaligned.h>
 #include "pci.h"
 
+extern void device_driver_detach(struct device *dev);
+
 #ifndef ARCH_PCI_DEV_GROUPS
 #define ARCH_PCI_DEV_GROUPS
 #endif
@@ -518,8 +520,10 @@ static ssize_t remove_store(struct device *dev, struct device_attribute *attr,
 	if (kstrtoul(buf, 0, &val) < 0)
 		return -EINVAL;
 
-	if (val && device_remove_file_self(dev, attr))
+	if (val && device_remove_file_self(dev, attr)) {
+		device_driver_detach(dev);
 		pci_stop_and_remove_bus_device_locked(to_pci_dev(dev));
+	}
 	return count;
 }
 static DEVICE_ATTR_IGNORE_LOCKDEP(remove, 0220, NULL,

I run this with vfio-pci as device-driver and some sniff tests:

  root@hostname:~# lspci
  0103:00:00.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
  1100:00:00.0 Processing accelerators: IBM Spyre Accelerator (rev 02)
  1100:00:00.1 Processing accelerators: IBM Spyre Accelerator Virtual Function (rev 02)
  1180:00:00.0 Processing accelerators: IBM Spyre Accelerator (rev 02)
  1180:00:00.1 Processing accelerators: IBM Spyre Accelerator Virtual Function (rev 02)
  root@hostname:~# cd /sys/bus/pci/
  root@hostname:/sys/bus/pci# echo 0 > devices/1100\:00\:00.0/sriov_numvfs
  root@hostname:/sys/bus/pci# echo 1 > devices/1100\:00\:00.0/sriov_numvfs
  root@hostname:/sys/bus/pci# echo 1100:00:00.0 > drivers/vfio-pci/unbind
  root@hostname:/sys/bus/pci# echo 1100:00:00.0 > drivers/vfio-pci/bind
  root@hostname:/sys/bus/pci# echo 0 > devices/1100\:00\:00.0/sriov_numvfs
  root@hostname:/sys/bus/pci# echo 1 > devices/1100\:00\:00.0/sriov_numvfs
  root@hostname:/sys/bus/pci# echo 1 > devices/1100\:00\:00.0/remove
  root@hostname:/sys/bus/pci# lspci
  0103:00:00.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
  1180:00:00.0 Processing accelerators: IBM Spyre Accelerator (rev 02)
  1180:00:00.1 Processing accelerators: IBM Spyre Accelerator Virtual Function (rev 02)
  root@hostname:/sys/bus/pci# rmmod vfio_ccw vfio_pci vfio_pci_core vfio_iommu_type1 vfio
  root@hostname:/sys/bus/pci# modprobe -a vfio_ccw vfio_pci vfio_pci_core vfio_iommu_type1 vfio
  root@hostname:/sys/bus/pci# echo 1 > rescan
  root@hostname:/sys/bus/pci# echo 1 > devices/1100\:00\:00.0/sriov_numvfs
  root@hostname:/sys/bus/pci# echo 1 > devices/1180\:00\:00.0/sriov_numvfs
  root@hostname:/sys/bus/pci# lspci
  0103:00:00.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
  1100:00:00.0 Processing accelerators: IBM Spyre Accelerator (rev 02)
  1100:00:00.1 Processing accelerators: IBM Spyre Accelerator Virtual Function (rev 02)
  1180:00:00.0 Processing accelerators: IBM Spyre Accelerator (rev 02)
  1180:00:00.1 Processing accelerators: IBM Spyre Accelerator Virtual Function (rev 02)

All without any Warnings/Lockdep-Splats/Kmemleak-Splats/etc.

But the patch itself is very hacky I'm afraid.

-- 
Best Regards, Benjamin Block        /        Linux on IBM Z Kernel Development
IBM Deutschland Research & Development GmbH    /   https://www.ibm.com/privacy
Vors. Aufs.-R.: Wolfgang Wendt         /        Geschäftsführung: David Faller
Sitz der Ges.: Ehningen     /     Registergericht: AmtsG Stuttgart, HRB 243294

Re: [PATCH v3 2/2] PCI/IOV: Fix race between SR-IOV enable/disable and hotplug

[Benjamin Block]

On Tue, Mar 17, 2026 at 10:46:56AM +0100, Benjamin Block wrote:
> On Tue, Mar 17, 2026 at 10:01:49AM +0100, Benjamin Block wrote:
> > One quick idea: can we somehow unbind the device from any device driver
> > in remove_store() before calling
> > pci_stop_and_remove_bus_device_locked()?  That way we would not have any
> > SR-IOV functions attached anymore at the point where we remove the PF,
> > since the DD are expected to clean them up.
> 
> Very quick-and-dirty:
> 
> diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c
> index 16eaaf749ba9..490b6c1f07d3 100644
> --- a/drivers/pci/pci-sysfs.c
> +++ b/drivers/pci/pci-sysfs.c
> @@ -33,6 +33,8 @@
>  #include <linux/unaligned.h>
>  #include "pci.h"
>  
> +extern void device_driver_detach(struct device *dev);
> +
>  #ifndef ARCH_PCI_DEV_GROUPS
>  #define ARCH_PCI_DEV_GROUPS
>  #endif
> @@ -518,8 +520,10 @@ static ssize_t remove_store(struct device *dev, struct device_attribute *attr,
>  	if (kstrtoul(buf, 0, &val) < 0)
>  		return -EINVAL;
>  
> -	if (val && device_remove_file_self(dev, attr))
> +	if (val && device_remove_file_self(dev, attr)) {
> +		device_driver_detach(dev);

Hmm, thinking on it, this leaves open the possibility, that something re-binds
the DD in between these two calls. Although unlikely, it might happen.

>  		pci_stop_and_remove_bus_device_locked(to_pci_dev(dev));
> +	}
>  	return count;
>  }
>  static DEVICE_ATTR_IGNORE_LOCKDEP(remove, 0220, NULL,

I guess, I dig the hole a bit deeper.

diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c
index 16eaaf749ba9..d2877b805d6e 100644
--- a/drivers/pci/pci-sysfs.c
+++ b/drivers/pci/pci-sysfs.c
@@ -33,6 +33,8 @@
 #include <linux/unaligned.h>
 #include "pci.h"
 
+extern void device_driver_detach(struct device *dev);
+
 #ifndef ARCH_PCI_DEV_GROUPS
 #define ARCH_PCI_DEV_GROUPS
 #endif
@@ -518,8 +520,13 @@ static ssize_t remove_store(struct device *dev, struct device_attribute *attr,
 	if (kstrtoul(buf, 0, &val) < 0)
 		return -EINVAL;
 
-	if (val && device_remove_file_self(dev, attr))
+	if (val && device_remove_file_self(dev, attr)) {
+		device_lock(dev);
+		kill_device(dev);
+		device_unlock(dev);
+		device_driver_detach(dev);
 		pci_stop_and_remove_bus_device_locked(to_pci_dev(dev));
+	}
 	return count;
 }
 static DEVICE_ATTR_IGNORE_LOCKDEP(remove, 0220, NULL,

Marking the device dead prevents re-binding after the unbind.

Same sniff tests as before:

  root@b315lp04:~# cd /sys/bus/pci/
  root@b315lp04:/sys/bus/pci# lspci
  0103:00:00.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
  1100:00:00.0 Processing accelerators: IBM Spyre Accelerator (rev 02)
  1100:00:00.1 Processing accelerators: IBM Spyre Accelerator Virtual Function (rev 02)
  1180:00:00.0 Processing accelerators: IBM Spyre Accelerator (rev 02)
  1180:00:00.1 Processing accelerators: IBM Spyre Accelerator Virtual Function (rev 02)
  root@b315lp04:/sys/bus/pci# echo 0 > devices/1100\:00\:00.0/sriov_numvfs
  root@b315lp04:/sys/bus/pci# echo 1 > devices/1100\:00\:00.0/sriov_numvfs
  root@b315lp04:/sys/bus/pci# echo 1100:00:00.0 > drivers/vfio-pci/unbind
  root@b315lp04:/sys/bus/pci# echo 1100:00:00.0 > drivers/vfio-pci/bind
  root@b315lp04:/sys/bus/pci# echo 1 > devices/1100\:00\:00.0/sriov_numvfs
  root@b315lp04:/sys/bus/pci# echo 1 > devices/1100\:00\:00.0/remove
  root@b315lp04:/sys/bus/pci# lspci
  0103:00:00.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
  1180:00:00.0 Processing accelerators: IBM Spyre Accelerator (rev 02)
  1180:00:00.1 Processing accelerators: IBM Spyre Accelerator Virtual Function (rev 02)
  root@b315lp04:/sys/bus/pci# rmmod vfio_ccw vfio_pci vfio_pci_core vfio_iommu_type1 vfio
  root@b315lp04:/sys/bus/pci# modprobe -a vfio_ccw vfio_pci vfio_pci_core vfio_iommu_type1 vfio
  root@b315lp04:/sys/bus/pci# echo 1 > rescan
  root@b315lp04:/sys/bus/pci# echo 1 > devices/1100\:00\:00.0/sriov_numvfs
  root@b315lp04:/sys/bus/pci# echo 1 > devices/1180\:00\:00.0/sriov_numvfs
  root@b315lp04:/sys/bus/pci# lspci
  0103:00:00.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
  1100:00:00.0 Processing accelerators: IBM Spyre Accelerator (rev 02)
  1100:00:00.1 Processing accelerators: IBM Spyre Accelerator Virtual Function (rev 02)
  1180:00:00.0 Processing accelerators: IBM Spyre Accelerator (rev 02)
  1180:00:00.1 Processing accelerators: IBM Spyre Accelerator Virtual Function (rev 02)
  root@b315lp04:/sys/bus/pci# echo 0 > devices/1180\:00\:00.0/sriov_numvfs
  root@b315lp04:/sys/bus/pci# echo 1 > devices/1180\:00\:00.0/sriov_numvfs
  root@b315lp04:/sys/bus/pci# echo 1180:00:00.0 > drivers/vfio-pci/unbind
  root@b315lp04:/sys/bus/pci# echo 1180:00:00.0 > drivers/vfio-pci/bind
  root@b315lp04:/sys/bus/pci# echo 1 > devices/1180\:00\:00.0/sriov_numvfs
  root@b315lp04:/sys/bus/pci# echo 1 > devices/1180\:00\:00.0/remove
  root@b315lp04:/sys/bus/pci# lspci
  0103:00:00.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
  1100:00:00.0 Processing accelerators: IBM Spyre Accelerator (rev 02)
  1100:00:00.1 Processing accelerators: IBM Spyre Accelerator Virtual Function (rev 02)

Same result: all without any Warnings/Lockdep-Splats/Kmemleak-Splats/etc.

-- 
Best Regards, Benjamin Block        /        Linux on IBM Z Kernel Development
IBM Deutschland Research & Development GmbH    /   https://www.ibm.com/privacy
Vors. Aufs.-R.: Wolfgang Wendt         /        Geschäftsführung: David Faller
Sitz der Ges.: Ehningen     /     Registergericht: AmtsG Stuttgart, HRB 243294

Re: [PATCH v3 2/2] PCI/IOV: Fix race between SR-IOV enable/disable and hotplug

[Lukas Wunner]

On Tue, Mar 17, 2026 at 12:33:34PM +0100, Benjamin Block wrote:
> @@ -518,8 +520,13 @@ static ssize_t remove_store(struct device *dev, struct device_attribute *attr,
>  	if (kstrtoul(buf, 0, &val) < 0)
>  		return -EINVAL;
>  
> -	if (val && device_remove_file_self(dev, attr))
> +	if (val && device_remove_file_self(dev, attr)) {
> +		device_lock(dev);
> +		kill_device(dev);
> +		device_unlock(dev);
> +		device_driver_detach(dev);
>  		pci_stop_and_remove_bus_device_locked(to_pci_dev(dev));
> +	}
>  	return count;
>  }
>  static DEVICE_ATTR_IGNORE_LOCKDEP(remove, 0220, NULL,
> 
> Marking the device dead prevents re-binding after the unbind.

That can also be achieved through the PCI_DEV_ALLOW_BINDING flag,
though you'd have to add a pci_dev_disallow_binding() helper in
drivers/pci/pci.h to go alongside the existing
pci_dev_allow_binding() and pci_dev_binding_disallowed() helpers.

However pci_stop_and_remove_bus_device() implicitly unbinds the
driver before removing the device.  Remind me, what's the need
to unbind before calling that function?

Thanks,

Lukas

Re: [PATCH v3 2/2] PCI/IOV: Fix race between SR-IOV enable/disable and hotplug

[Lukas Wunner]

On Tue, Mar 17, 2026 at 02:08:11PM +0100, Lukas Wunner wrote:
> However pci_stop_and_remove_bus_device() implicitly unbinds the
> driver before removing the device.  Remind me, what's the need
> to unbind before calling that function?

Never mind, read Günter Röck's e-mail only now.  So this is just
a bandaid to work around the too coarse-grained pci_lock_rescan_remove().

I've been arguing for a while that we need to move to more
fine-grained locking, but it's difficult to get there without
breaking things, it's difficult to make sense of a lot of old code
and it's difficult to allocate time to tech debt problems like this
because employers always want developers to focus on enabling shiny
new features first. :(

Thanks,

Lukas

Re: [PATCH v3 2/2] PCI/IOV: Fix race between SR-IOV enable/disable and hotplug

[Benjamin Block]

On Tue, Mar 17, 2026 at 02:18:00PM +0100, Lukas Wunner wrote:
> On Tue, Mar 17, 2026 at 02:08:11PM +0100, Lukas Wunner wrote:
> > However pci_stop_and_remove_bus_device() implicitly unbinds the
> > driver before removing the device.  Remind me, what's the need
> > to unbind before calling that function?
> 
> Never mind, read Günter Röck's e-mail only now.  So this is just
> a bandaid to work around the too coarse-grained pci_lock_rescan_remove().
> 
> I've been arguing for a while that we need to move to more
> fine-grained locking, but it's difficult to get there without
> breaking things, it's difficult to make sense of a lot of old code
> and it's difficult to allocate time to tech debt problems like this
> because employers always want developers to focus on enabling shiny
> new features first. :(

I agree, it's a mess :/

The quick idea here was to prevent the found AB-BA circular lock by
moving the removal of the SR-IOV VFs in front of the call to
pci_stop_and_remove_bus_device_locked(), and that is typically done when
the device driver is unbound (it's the expectation in
pci_device_remove()).

But it's just a quick idea, maybe someone else has a better one. I'm
also not sure this is the only place where we'd need to deal with this
particular AB-BA circular lock scenario (`pdev->dev` and
`pci_rescan_remove_lock`).

I'll run a few more tests, and see if I can scare up some other/new
lockdep warning with this change.

-- 
Best Regards und Beste Grüße, Benjamin Block
               PGP KeyID: 9610 2BB8 2E17 6F65 2362  6DF2 46E0 4E05 67A3 2E9E