The Linux Kernel Mailing List
 help / color / mirror / Atom feed
From: Alex Williamson <alex.williamson@nvidia.com>
To: "Ionut Nechita (Wind River)" <ionut.nechita@windriver.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>,
	helgaas@kernel.org, linux-pci@vger.kernel.org,
	linux-kernel@vger.kernel.org, stable@vger.kernel.org,
	Dave Jiang <dave.jiang@intel.com>,
	Alex Williamson <alex.williamson@redhat.com>,
	lukas@wunner.de, sebott@linux.ibm.com, schnelle@linux.ibm.com,
	bblock@linux.ibm.com, linux@roeck-us.net,
	matthew.brost@intel.com, michal.wajdeczko@intel.com,
	piotr.piorkowski@intel.com, dtatulea@nvidia.com, mani@kernel.org,
	kbusch@kernel.org, lkml@mageta.org, alifm@linux.ibm.com,
	julianr@linux.ibm.com, ionut_n2001@yahoo.com,
	sunlightlinux@gmail.com
Subject: Re: [PATCH 1/1] PCI: Don't block bridge config access while waiting for the device lock
Date: Thu, 19 Nov 2026 14:32:27 -0800	[thread overview]
Message-ID: <20261119143227.7c00a09e@nvidia.com> (raw)
In-Reply-To: <20261119123504.25906-2-ionut.nechita@windriver.com>

On Thu, 19 Nov 2026 14:35:04 +0200
"Ionut Nechita (Wind River)" <ionut.nechita@windriver.com> wrote:

> From: Ionut Nechita <ionut.nechita@windriver.com>
> 
> pci_reset_function() locks the upstream bridge with pci_dev_lock(),
> which takes the bridge's device lock and then blocks config access to
> the bridge, and only then takes the device lock of the function being
> reset. If another task holds that device lock, the reset sleeps with
> config access to the bridge blocked, and every user config read of the
> bridge sleeps in pci_wait_cfg() for as long.
> 
> This happens with vfio-pci when a userspace driver keeps a VF open:
> 
>   1. A driver unbind of the VF waits in vfio_unregister_group_dev()
>      for the user to release the device, holding the VF device lock.
>   2. Writing 0 to sriov_numvfs on the PF takes the PF device lock and
>      waits in sriov_disable() for the VF device lock.
>   3. Writing 1 to the PF "reset" attribute calls pci_reset_function(),
>      which blocks config access to the PF's upstream bridge and waits
>      for the PF device lock.
> 
> From then on lspci, which reads the config space of every device,
> hangs in D state, and so does any tool or agent that runs it, until
> the userspace driver releases the VF:
> 
>   lspci           D
>     pci_wait_cfg+0x60/0xc0
>     pci_user_read_config_dword+0xcc/0xf0
>     pci_read_config+0xf5/0x290
>     kernfs_fop_read_iter+0xb6/0x1b0
>     vfs_read+0x21f/0x340
> 
> This was hit with an Intel FPGA PAC N3000 FEC device bound to vfio-pci
> (PF and VFs), where the SR-IOV FEC operator reconfigures the PF while a
> DPDK application holds one of its VFs.
> 
> Take the device locks first, bridge before device as before, and block
> config access to the bridge and the device only once both are held.
> The order between the two device locks is unchanged. Config access is
> now blocked only while the reset runs, not while waiting for a lock.
> 
> The reset itself still waits for the device lock, so the tasks in
> steps 1-3 stay blocked until the user releases the VF. Only config
> reads of unrelated callers stop being blocked.
> 
> Fixes: 7e89efc6e9e4 ("PCI: Lock upstream bridge for pci_reset_function()")
> Cc: stable@vger.kernel.org
> Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
> ---
>  drivers/pci/pci.c | 24 ++++++++++++++++++++----
>  1 file changed, 20 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index d79f73b28..9c8d5dda9 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -5345,19 +5345,35 @@ int pci_reset_function(struct pci_dev *dev)
>  	 * no upstream bridge configuration to hold consistent.
>  	 */
>  	bridge = pci_upstream_bridge(dev);
> +
> +	/*
> +	 * Take the device locks, bridge before device, before blocking
> +	 * config access to either of them.  device_lock() can sleep for as
> +	 * long as another task holds it, e.g. a driver unbind waiting for
> +	 * a vfio user to release the device.  If config access to the
> +	 * bridge were already blocked at that point, every config read of
> +	 * the bridge (lspci, sysfs "config") would sleep in pci_wait_cfg()
> +	 * for the same time.
> +	 */
>  	if (bridge)
> -		pci_dev_lock(bridge);
> +		device_lock(&bridge->dev);
> +	device_lock(&dev->dev);
> +	if (bridge)
> +		pci_cfg_access_lock(bridge);
> +	pci_cfg_access_lock(dev);
>  
> -	pci_dev_lock(dev);
>  	pci_dev_save_and_disable(dev);
>  
>  	rc = __pci_reset_function_locked(dev);
>  
>  	pci_dev_restore(dev);
> -	pci_dev_unlock(dev);
>  
> +	pci_cfg_access_unlock(dev);
>  	if (bridge)
> -		pci_dev_unlock(bridge);
> +		pci_cfg_access_unlock(bridge);
> +	device_unlock(&dev->dev);
> +	if (bridge)
> +		device_unlock(&bridge->dev);
>  
>  	return rc;
>  }

Reviewed-by: Alex Williamson <alex.williamson@nvidia.com>

      reply	other threads:[~2026-11-19 22:32 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-19 12:35 [PATCH 0/1] PCI: Fix lspci hang when a reset waits for a vfio-held device Ionut Nechita (Wind River)
2026-11-19 12:35 ` [PATCH 1/1] PCI: Don't block bridge config access while waiting for the device lock Ionut Nechita (Wind River)
2026-11-19 22:32   ` Alex Williamson [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261119143227.7c00a09e@nvidia.com \
    --to=alex.williamson@nvidia.com \
    --cc=alex.williamson@redhat.com \
    --cc=alifm@linux.ibm.com \
    --cc=bblock@linux.ibm.com \
    --cc=bhelgaas@google.com \
    --cc=dave.jiang@intel.com \
    --cc=dtatulea@nvidia.com \
    --cc=helgaas@kernel.org \
    --cc=ionut.nechita@windriver.com \
    --cc=ionut_n2001@yahoo.com \
    --cc=julianr@linux.ibm.com \
    --cc=kbusch@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=linux@roeck-us.net \
    --cc=lkml@mageta.org \
    --cc=lukas@wunner.de \
    --cc=mani@kernel.org \
    --cc=matthew.brost@intel.com \
    --cc=michal.wajdeczko@intel.com \
    --cc=piotr.piorkowski@intel.com \
    --cc=schnelle@linux.ibm.com \
    --cc=sebott@linux.ibm.com \
    --cc=stable@vger.kernel.org \
    --cc=sunlightlinux@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox