From: <mhonap@nvidia.com>
To: <alex@shazbot.org>, <jgg@ziepe.ca>, <ankita@nvidia.com>,
<jic23@kernel.org>, <dave.jiang@intel.com>,
<alejandro.lucero-palau@amd.com>, <smadhavan@nvidia.com>,
<corbet@lwn.net>, <skhan@linuxfoundation.org>,
<dave@stgolabs.net>, <alison.schofield@intel.com>,
<vishal.l.verma@intel.com>, <iweiny@kernel.org>,
<ming.li@zohomail.com>, <yishaih@nvidia.com>,
<skolothumtho@nvidia.com>, <kevin.tian@intel.com>,
<bhelgaas@google.com>, <dmatlack@google.com>, <kees@kernel.org>,
<gustavoars@kernel.org>
Cc: <cjia@nvidia.com>, <kjaju@nvidia.com>, <vsethi@nvidia.com>,
<zhiw@nvidia.com>, <mhonap@nvidia.com>,
<linux-doc@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
<kvm@vger.kernel.org>, <linux-cxl@vger.kernel.org>,
<linux-pci@vger.kernel.org>, <linux-kselftest@vger.kernel.org>,
<linux-hardening@vger.kernel.org>
Subject: [PATCH v6 15/31] vfio/pci: Route every reset point through a try-reset helper
Date: Fri, 16 Oct 2026 21:48:22 +0530 [thread overview]
Message-ID: <20261016161839.2765592-16-mhonap@nvidia.com> (raw)
In-Reply-To: <20261016161839.2765592-1-mhonap@nvidia.com>
From: Manish Honap <mhonap@nvidia.com>
A provider such as vfio-cxl must reset its device itself: a CXL Type-2
function must not take an FLR, and its reset changes what the provider
lets the fault path map. vfio-pci-core resets a function at five points:
enable, close, VFIO_DEVICE_RESET, a virtualized PCIe or AF FLR, and the
guest's CXL Reset doorbell in the virtualized CXL DVSEC.
Add a mandatory reset op to the provider ops, and refuse to register ops
without it. Add vfio_pci_try_reset_function(), which keeps the trylock
of pci_try_reset_function(): it returns -EAGAIN when the device lock is
contended, and otherwise runs the provider's reset with the device lock
and memory_lock held. The close path keeps its bridge and device trylock
and calls a locked variant. vfio_pci_try_reset_function() saves the PCI
state first, as pci_try_reset_function() does, for the provider's reset
to restore. The locked variant does not save: on close the reset must
restore the open-time state that vfio_pci_core_disable() has just
loaded. A failed save is returned and the reset is not attempted.
The enable path still fails only on -EAGAIN. A provider checks at bind
that it can reset the device, so reset_works is set for a provider
device whatever the result of the first reset.
The CXL Reset doorbell now runs the reset and reports its result in
Status2: Reset Error on failure, Reset Done on success.
Suggested-by: Alex Williamson <alex@shazbot.org>
Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
drivers/vfio/pci/vfio_pci_config.c | 19 ++++++--
drivers/vfio/pci/vfio_pci_core.c | 71 ++++++++++++++++++++++++++++--
drivers/vfio/pci/vfio_pci_priv.h | 1 +
include/linux/vfio_pci_core.h | 7 +++
4 files changed, 90 insertions(+), 8 deletions(-)
diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c
index 1fc463540211..6fbece95cb9f 100644
--- a/drivers/vfio/pci/vfio_pci_config.c
+++ b/drivers/vfio/pci/vfio_pci_config.c
@@ -910,7 +910,7 @@ static int vfio_exp_config_write(struct vfio_pci_core_device *vdev, int pos,
if (!ret && (cap & PCI_EXP_DEVCAP_FLR)) {
vfio_pci_zap_and_down_write_memory_lock(vdev);
vfio_pci_dma_buf_move(vdev, true);
- pci_try_reset_function(vdev->pdev);
+ vfio_pci_try_reset_function(vdev);
if (__vfio_pci_memory_enabled(vdev))
vfio_pci_dma_buf_move(vdev, false);
up_write(&vdev->memory_lock);
@@ -995,7 +995,7 @@ static int vfio_af_config_write(struct vfio_pci_core_device *vdev, int pos,
if (!ret && (cap & PCI_AF_CAP_FLR) && (cap & PCI_AF_CAP_TP)) {
vfio_pci_zap_and_down_write_memory_lock(vdev);
vfio_pci_dma_buf_move(vdev, true);
- pci_try_reset_function(vdev->pdev);
+ vfio_pci_try_reset_function(vdev);
if (__vfio_pci_memory_enabled(vdev))
vfio_pci_dma_buf_move(vdev, false);
up_write(&vdev->memory_lock);
@@ -1786,9 +1786,20 @@ static int vfio_cxl_dvsec_write(struct vfio_pci_core_device *vdev, int pos,
if (!(ctrl2 & PCI_DVSEC_CXL_DISABLE_CACHING))
status2 &= ~PCI_DVSEC_CXL_CACHE_INV;
if (ctrl2 & PCI_DVSEC_CXL_INIT_CXL_RST) {
+ int ret;
+
ctrl2 &= ~PCI_DVSEC_CXL_INIT_CXL_RST;
- status2 &= ~PCI_DVSEC_CXL_RST_ERR;
- status2 |= PCI_DVSEC_CXL_RST_DONE;
+
+ /* Run the real reset here and report its result in Status2. */
+ vfio_pci_zap_and_down_write_memory_lock(vdev);
+ vfio_pci_dma_buf_move(vdev, true);
+ ret = vfio_pci_try_reset_function(vdev);
+ if (__vfio_pci_memory_enabled(vdev))
+ vfio_pci_dma_buf_move(vdev, false);
+ up_write(&vdev->memory_lock);
+
+ status2 &= ~(PCI_DVSEC_CXL_RST_DONE | PCI_DVSEC_CXL_RST_ERR);
+ status2 |= ret ? PCI_DVSEC_CXL_RST_ERR : PCI_DVSEC_CXL_RST_DONE;
}
*pctrl2 = cpu_to_le16(ctrl2);
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 1e0ac28b7564..6676c561465b 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -605,6 +605,62 @@ static void vfio_pci_core_unmap_bars(struct vfio_pci_core_device *vdev)
}
}
+/**
+ * vfio_pci_try_reset_function() - reset the function without blocking
+ * @vdev: vfio PCI device, with memory_lock held for write
+ *
+ * Reset through the PCI core, or let the provider reset a device it serves.
+ * The provider resets the device with its PCI device lock held, so take that
+ * lock here, without waiting, as pci_try_reset_function() does.
+ *
+ * Return: 0 on success, -EAGAIN when the device lock is contended, or the
+ * reset error. For a provider device a failed pci_save_state() is also
+ * returned, and the reset is not attempted: unlike pci_try_reset_function(),
+ * the provider's reset needs a valid saved state to restore.
+ */
+int vfio_pci_try_reset_function(struct vfio_pci_core_device *vdev)
+{
+ struct pci_dev *pdev = vdev->pdev;
+ int ret;
+
+ lockdep_assert_held_write(&vdev->memory_lock);
+
+ if (!vdev->cxl_ops)
+ return pci_try_reset_function(pdev);
+
+ if (!pci_dev_trylock(pdev))
+ return -EAGAIN;
+
+ /*
+ * The provider restores PCI state from the saved copy, so save the
+ * current state first, as pci_try_reset_function() does. The locked
+ * variant on the close path must not: it resets with the state that
+ * vfio_pci_core_disable() has just loaded.
+ */
+ ret = pci_save_state(pdev);
+ if (!ret)
+ ret = vdev->cxl_ops->reset(vdev);
+ pci_dev_unlock(pdev);
+
+ return ret;
+}
+
+/*
+ * As vfio_pci_try_reset_function(), for a caller that holds the device lock.
+ * Only the close path calls this. It takes memory_lock with the device lock
+ * held, so user config access is blocked meanwhile: the order a sysfs reset
+ * uses. That cannot deadlock against a PCI_COMMAND write holding memory_lock
+ * because no user remains at close.
+ */
+static int vfio_pci_reset_function_locked(struct vfio_pci_core_device *vdev)
+{
+ if (!vdev->cxl_ops)
+ return __pci_reset_function_locked(vdev->pdev);
+
+ guard(rwsem_write)(&vdev->memory_lock);
+ return vdev->cxl_ops->reset(vdev);
+}
+
static void vfio_pci_core_free_dev_regions(struct vfio_pci_core_device *vdev)
{
for (int i = 0; i < vdev->num_regions; i++)
@@ -636,11 +692,14 @@ int vfio_pci_core_enable(struct vfio_pci_core_device *vdev)
goto out_power;
/* If reset fails because of the device lock, fail this path entirely */
- ret = pci_try_reset_function(pdev);
+ down_write(&vdev->memory_lock);
+ ret = vfio_pci_try_reset_function(vdev);
+ up_write(&vdev->memory_lock);
if (ret == -EAGAIN)
goto out_disable_device;
- vdev->reset_works = !ret;
+ /* A provider checks at bind that it can reset the device. */
+ vdev->reset_works = vdev->cxl_ops || !ret;
pci_save_state(pdev);
vdev->pci_saved_state = pci_store_saved_state(pdev);
if (!vdev->pci_saved_state)
@@ -846,7 +905,7 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev)
if (bridge && !pci_dev_trylock(bridge))
goto out_restore_state;
if (pci_dev_trylock(pdev)) {
- if (!__pci_reset_function_locked(pdev))
+ if (!vfio_pci_reset_function_locked(vdev))
vdev->needs_reset = false;
pci_dev_unlock(pdev);
}
@@ -1541,7 +1600,7 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
vfio_pci_set_power_state(vdev, PCI_D0);
vfio_pci_dma_buf_move(vdev, true);
- ret = pci_try_reset_function(vdev->pdev);
+ ret = vfio_pci_try_reset_function(vdev);
if (__vfio_pci_memory_enabled(vdev))
vfio_pci_dma_buf_move(vdev, false);
up_write(&vdev->memory_lock);
@@ -2397,6 +2456,10 @@ int vfio_pci_core_register_cxl_ops(const struct vfio_cxl_ops *ops)
{
guard(rwsem_write)(&vfio_pci_cxl_ops_rwsem);
+ /* vfio-pci-core never resets a provider's device by itself. */
+ if (!ops->reset)
+ return -EINVAL;
+
if (vfio_pci_cxl_ops)
return -EBUSY;
diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h
index 09a68c99b43c..dbede6e24fa9 100644
--- a/drivers/vfio/pci/vfio_pci_priv.h
+++ b/drivers/vfio/pci/vfio_pci_priv.h
@@ -80,6 +80,7 @@ int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev,
pci_power_t state);
void vfio_pci_zap_and_down_write_memory_lock(struct vfio_pci_core_device *vdev);
+int vfio_pci_try_reset_function(struct vfio_pci_core_device *vdev);
u16 vfio_pci_memory_lock_and_enable(struct vfio_pci_core_device *vdev);
void vfio_pci_memory_unlock_and_restore(struct vfio_pci_core_device *vdev,
u16 cmd);
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 991a3fcd84eb..cb9ba92b696a 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -87,6 +87,13 @@ struct vfio_cxl_ops {
*/
void (*reset_prepare)(struct vfio_pci_core_device *vdev);
int (*reset_done)(struct vfio_pci_core_device *vdev);
+ /*
+ * Reset the function in place of the PCI core; mandatory. Called with
+ * the PCI device lock and memory_lock (for write) held. The caller
+ * provides the saved PCI state the reset restores: the try-reset path
+ * saves it first, the close path keeps the open-time copy.
+ */
+ int (*reset)(struct vfio_pci_core_device *vdev);
/* Pinned per bound CXL device so vfio-cxl cannot unload under usage */
struct module *owner;
};
--
2.53.0
next prev parent reply other threads:[~2026-10-16 16:21 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-16 16:18 [PATCH v6 00/31] vfio/pci: Add CXL Type-2 device passthrough support mhonap
2026-10-16 16:18 ` [PATCH v6 01/31] cxl/regs: Split the BAR block request and ioremap helpers mhonap
2026-10-16 16:18 ` [PATCH v6 02/31] cxl/regs: Let a BAR-owning driver own the component register block mhonap
2026-10-16 16:18 ` [PATCH v6 03/31] cxl: Move component register defines to uapi/cxl/cxl_regs.h mhonap
2026-10-16 16:18 ` [PATCH v6 04/31] PCI/CXL: Let the driver that owns the HDM range run CXL Reset mhonap
2026-10-16 16:18 ` [PATCH v6 05/31] PCI: Refuse bus and slot reset below a CXL port with SBR masked mhonap
2026-10-16 16:18 ` [PATCH v6 06/31] vfio/pci: Add the CXL provider ops registration interface mhonap
2026-10-16 16:18 ` [PATCH v6 07/31] vfio/pci: Detect CXL devices and load the CXL provider on demand mhonap
2026-10-16 16:18 ` [PATCH v6 08/31] vfio/pci: Add a generic excluded-range list mhonap
2026-10-16 16:18 ` [PATCH v6 09/31] vfio/pci: Advertise the mmap areas around excluded ranges mhonap
2026-10-16 16:18 ` [PATCH v6 10/31] vfio/pci: Refuse ioeventfds and dma-buf exports over " mhonap
2026-10-16 16:18 ` [PATCH v6 11/31] vfio/pci: Virtualize the CXL DVSEC in vfio_pci_config.c mhonap
2026-10-16 16:18 ` [PATCH v6 12/31] vfio/pci: Call the CXL open and close hooks around device use mhonap
2026-10-16 16:18 ` [PATCH v6 13/31] vfio/pci: Bracket PCI resets with the CXL reset hooks mhonap
2026-10-16 16:18 ` [PATCH v6 14/31] vfio/pci: Provide an opt-out for the CXL Type-2 extensions mhonap
2026-10-16 16:18 ` mhonap [this message]
2026-10-16 16:18 ` [PATCH v6 16/31] vfio/pci: Revoke and re-arm one device-specific region mhonap
2026-10-16 16:18 ` [PATCH v6 17/31] vfio/pci: Call the device provider across low-power transitions mhonap
2026-10-16 16:18 ` [PATCH v6 18/31] vfio/cxl: Add the vfio-cxl provider module skeleton mhonap
2026-10-16 16:18 ` [PATCH v6 19/31] vfio/cxl: Create the CXL memdev and set media ready at bind mhonap
2026-10-16 16:18 ` [PATCH v6 20/31] vfio/cxl: Own the whole component register BAR mhonap
2026-10-16 16:18 ` [PATCH v6 21/31] vfio/cxl: Expose the HDM memory region to the guest mhonap
2026-10-16 16:18 ` [PATCH v6 22/31] vfio/cxl: Allow mmap of the HDM memory region mhonap
2026-10-16 16:18 ` [PATCH v6 23/31] vfio/cxl: Contain HDM memory errors with memory_failure() mhonap
2026-10-16 16:18 ` [PATCH v6 24/31] vfio/cxl: Expose the HDM decoder registers read-only to the guest mhonap
2026-10-16 16:18 ` [PATCH v6 25/31] vfio/cxl: Exclude the HDM decoder registers from direct BAR access mhonap
2026-10-16 16:18 ` [PATCH v6 26/31] vfio/cxl: Restore the HDM decoder after a hot reset mhonap
2026-10-16 16:18 ` [PATCH v6 27/31] vfio/cxl: Describe the CXL device and decoder geometry to userspace mhonap
2026-10-16 16:18 ` [PATCH v6 28/31] vfio/cxl: Export the HDM memory region as a dma-buf mhonap
2026-10-16 16:18 ` [PATCH v6 29/31] vfio/cxl: Run the CXL reset at the vfio reset points mhonap
2026-10-16 16:18 ` [PATCH v6 30/31] Documentation: vfio-pci: Document CXL Type-2 device passthrough mhonap
2026-10-16 16:18 ` [PATCH v6 31/31] selftests/vfio: Add CXL Type-2 passthrough tests mhonap
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261016161839.2765592-16-mhonap@nvidia.com \
--to=mhonap@nvidia.com \
--cc=alejandro.lucero-palau@amd.com \
--cc=alex@shazbot.org \
--cc=alison.schofield@intel.com \
--cc=ankita@nvidia.com \
--cc=bhelgaas@google.com \
--cc=cjia@nvidia.com \
--cc=corbet@lwn.net \
--cc=dave.jiang@intel.com \
--cc=dave@stgolabs.net \
--cc=dmatlack@google.com \
--cc=gustavoars@kernel.org \
--cc=iweiny@kernel.org \
--cc=jgg@ziepe.ca \
--cc=jic23@kernel.org \
--cc=kees@kernel.org \
--cc=kevin.tian@intel.com \
--cc=kjaju@nvidia.com \
--cc=kvm@vger.kernel.org \
--cc=linux-cxl@vger.kernel.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-hardening@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=ming.li@zohomail.com \
--cc=skhan@linuxfoundation.org \
--cc=skolothumtho@nvidia.com \
--cc=smadhavan@nvidia.com \
--cc=vishal.l.verma@intel.com \
--cc=vsethi@nvidia.com \
--cc=yishaih@nvidia.com \
--cc=zhiw@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox