public inbox for linux-pci@vger.kernel.org
 help / color / mirror / Atom feed
From: Bjorn Helgaas <helgaas@kernel.org>
To: "Ionut Nechita (Wind River)" <ionut.nechita@windriver.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>,
	linux-pci@vger.kernel.org,
	Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
	Clark Williams <clrkwllms@kernel.org>,
	Steven Rostedt <rostedt@goodmis.org>,
	linux-rt-devel@lists.linux.dev, linux-kernel@vger.kernel.org,
	Ionut Nechita <ionut_n2001@yahoo.com>
Subject: Re: [PATCH] PCI/IOV: Fix recursive locking deadlock on pci_rescan_remove_lock
Date: Mon, 9 Feb 2026 10:14:43 -0600	[thread overview]
Message-ID: <20260209161443.GA190606@bhelgaas> (raw)
In-Reply-To: <20260209075706.16367-2-ionut.nechita@windriver.com>

On Mon, Feb 09, 2026 at 09:57:07AM +0200, Ionut Nechita (Wind River) wrote:
> From: Ionut Nechita <ionut.nechita@windriver.com>
> 
> When a PCI device is hot-removed via sysfs (e.g., echo 1 > /sys/.../remove),
> pci_stop_and_remove_bus_device_locked() acquires pci_rescan_remove_lock and
> then recursively walks the bus hierarchy calling driver .remove() callbacks.
> 
> If the removed device is a PF with SR-IOV enabled (e.g., i40e, ice), the
> driver's .remove() calls pci_disable_sriov() -> sriov_disable() ->
> sriov_del_vfs() which also tries to acquire pci_rescan_remove_lock.
> Since this is a non-recursive mutex and the same thread already holds it,
> this results in a deadlock.
> 
> On PREEMPT_RT kernels, where mutexes are backed by rtmutex with deadlock
> detection, this immediately triggers:
> 
>   WARNING: CPU: 15 PID: 11730 at kernel/locking/rtmutex.c:1663
>   Call Trace:
>    mutex_lock+0x47/0x60
>    sriov_disable+0x2a/0x100
>    i40e_free_vfs+0x415/0x470 [i40e]
>    i40e_remove+0x38d/0x3e0 [i40e]
>    pci_device_remove+0x3b/0xb0
>    device_release_driver_internal+0x193/0x200
>    pci_stop_bus_device+0x81/0xb0
>    pci_stop_and_remove_bus_device_locked+0x16/0x30
>    remove_store+0x79/0x90
> 
> On non-RT kernels the same recursive acquisition silently hangs the calling
> process, eventually causing netdev watchdog TX timeout splats.
> 
> This affects all drivers that call pci_disable_sriov() from their .remove()
> callback (i40e, ice, and others).
> 
> Fix this by tracking the owner of pci_rescan_remove_lock and skipping the
> redundant acquisition in sriov_del_vfs() when the current thread already
> holds it.  The VF removal is still serialized correctly because the caller
> already holds the lock.

Ionut, can you confirm whether Niklas's patches resolve this deadlock?
The following patches are queued for v7.0:

  2fa119c0e5e5 ("Revert "PCI/IOV: Add PCI rescan-remove locking when enabling/disabling SR-IOV"")
  a5338e365c45 ("PCI/IOV: Fix race between SR-IOV enable/disable and hotplug")

They are included in next-20260205.  They're probably in earlier
linux-next kernels, too, but I guess linux-next doesn't keep older
tags anymore, so I don't know how to figure out exactly when they were
included.  I put them in my tree on Feb 1.

Bjorn

  parent reply	other threads:[~2026-02-09 16:14 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-02-09  7:57 [PATCH] PCI/IOV: Fix recursive locking deadlock on pci_rescan_remove_lock Ionut Nechita (Wind River)
2026-02-09  8:25 ` Sebastian Andrzej Siewior
2026-02-09 10:12   ` Niklas Schnelle
2026-02-11  7:37     ` Sebastian Andrzej Siewior
2026-02-09 16:14 ` Bjorn Helgaas [this message]
2026-02-22 11:29   ` Ionut Nechita (Wind River)

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20260209161443.GA190606@bhelgaas \
    --to=helgaas@kernel.org \
    --cc=bhelgaas@google.com \
    --cc=bigeasy@linutronix.de \
    --cc=clrkwllms@kernel.org \
    --cc=ionut.nechita@windriver.com \
    --cc=ionut_n2001@yahoo.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=linux-rt-devel@lists.linux.dev \
    --cc=rostedt@goodmis.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox