[RFC PATCH 0/1] vfio: circular locking dependency in pci_dev_reset_iommu_prepare()

[RFC PATCH 0/1] vfio: circular locking dependency in pci_dev_reset_iommu_prepare()

[Vipin Sharma]

Hello,

I have encountered a reproducible circular locking dependency (lockdep)
warning when performing a device reset via VFIO (e.g. VFIO_DEVICE_RESET ioctl
or kexec live update) on a device whose BARs have been mmapped and faulted.

This issue was introduced in commit f5b16b802174 ("PCI: Suspend iommu function
prior to resetting a device"), which added calls to
pci_dev_reset_iommu_prepare() / pci_dev_reset_iommu_done() inside core PCI
reset routines (pcie_flr(), pci_pm_reset(), etc.).

Patch 1 adds a new selftest in tools/testing/selftests/vfio/ that triggers
this warning reliably on systems with ATS and reset support.

I used Intel DSA device 8086:0b25 with the test in Patch 1.

This issue was identified in the VFIO Live Update series
  https://lore.kernel.org/kvm/20260723181436.GB260824.vipinsh@google.com/

Since issue is independent of Live Update, I am sending this out
separately to not mix two things.

I have uploaded the patch to github also:
  https://github.com/shvipin/linux vfio/lockdep-mmap-fault-device-reset

================================================================================
Lock Hierarchy Analysis (Helped by AI)
================================================================================

1. In VFIO, handling VFIO_DEVICE_RESET (vfio_pci_ioctl_reset()) or bus/hot reset
   acquires down_write(&vdev->memory_lock) to zap active MMIO BAR mappings
   and prevent concurrent MMIO faults during reset.
   Inside memory_lock, it calls pci_try_reset_function() / pcie_flr().

2. Commit f5b16b802174 added pci_dev_reset_iommu_prepare() into pcie_flr().
   pci_dev_reset_iommu_prepare() acquires guard(mutex)(&group->mutex).
   This establishes:
       &vdev->memory_lock --> &group->mutex

3. Meanwhile, during kernel boot or device discovery (iommu_probe_device() /
   bus_iommu_probe()), iommu.c holds &group->mutex while calling
   iommu_setup_dma_ops(), which calls iova_domain_init_rcaches().
   iova_domain_init_rcaches() registers a multi-instance cpuhp handler via
   cpuhp_state_add_instance_nocalls(), acquiring cpus_read_lock().
   This establishes:
       &group->mutex --> cpu_hotplug_lock

4. cpu_hotplug_lock is an outer lock for various kernel subsystems that
   execute callbacks or sysfs/firmware/perf interactions, which eventually
   perform copy_to_user(), copy_from_user(), or filldir() (which can trigger
   a page fault in user memory under mmap_lock).
   This establishes:
       cpu_hotplug_lock --> (kernfs_rwsem / inode->i_rwsem / cpuctx_mutex)
                        --> &mm->mmap_lock

5. When userspace touches an mmapped VFIO BAR, the page fault handler
   (vfio_pci_mmap_huge_fault() / vfio_pci_mmap_page_fault()) executes under
   &mm->mmap_lock and acquires down_read(&vdev->memory_lock).
   This establishes:
       &mm->mmap_lock --> &vdev->memory_lock

Combining (1) through (5) yields the circular dependency cycle:

    &group->mutex
      --> cpu_hotplug_lock
        --> [kernfs_rwsem | i_mutex_dir_key | cpuctx_mutex]
          --> &mm->mmap_lock
            --> &vdev->memory_lock
              --> &group->mutex  <== DEADLOCK DETECTED


================================================================================
Reproductions / Lockdep Traces
================================================================================

I observed this lockdep warning closing via multiple distinct paths:

--- Trace 1: via sysfs / kernfs readdir ---

[  438.457974] WARNING: possible circular locking dependency detected
[  438.477952] vfio_cdev_lockd/17678 is trying to acquire lock:
[  438.484320] (&group->mutex){+.+.}-{4:4}, at: pci_dev_reset_iommu_prepare+0x6e/0x200
[  438.494169] but task is already holding lock:
[  438.501401] (&vdev->memory_lock){++++}-{4:4}, at: vfio_pci_core_ioctl+0x460/0xb20
[  438.511064] which lock already depends on the new lock.
...
[  438.919866] Chain exists of:
[  438.919866]   &group->mutex --> &mm->mmap_lock --> &vdev->memory_lock
...
[  438.999115]  __mutex_lock+0x8c/0xd80
[  438.999122]  pci_dev_reset_iommu_prepare+0x6e/0x200
[  438.999123]  pcie_flr+0x32/0xc0
[  438.999125]  __pci_reset_function_locked+0x84/0x120
[  438.999126]  pci_try_reset_function+0x51/0xe0
[  438.999128]  vfio_pci_core_ioctl+0x72b/0xb20

--- Trace 2: via VFS lookup / microcode loading during kexec ---

[  584.552264] WARNING: possible circular locking dependency detected
[  584.572247] kexec/14239 is trying to acquire lock:
[  584.577754] (&group->mutex){+.+.}-{4:4}, at: pci_dev_reset_iommu_prepare+0x6e/0x1a0
[  584.587602] but task is already holding lock:
[  584.594838] (&vdev->memory_lock){++++}-{4:4}, at: vfio_pci_liveupdate_freeze+0x51/0x100
...
[  584.751130] -> #2 (&type->i_mutex_dir_key#4):
[  584.760319]        lookup_slow+0x26/0x50
[  584.786970]        _request_firmware+0x4ab/0x8c0
[  584.797979]        request_microcode_fw+0xf2/0x510
[  584.834601] -> #1 (cpu_hotplug_lock):
[  584.842183]        cpus_read_lock+0x3b/0xd0
[  584.858957]        iommu_setup_dma_ops+0x175/0x540

--- Trace 3: via perf read / watchdog cpuhp callback ---

[  233.859740] WARNING: possible circular locking dependency detected
[  233.879725] vfio_cdev_lockd/14999 is trying to acquire lock:
[  233.886094] (&group->mutex){+.+.}-{4:4}, at: pci_dev_reset_iommu_prepare+0x6e/0x200
[  233.895948] but task is already holding lock:
[  233.903183] (&vdev->memory_lock){++++}-{4:4}, at: vfio_pci_core_ioctl+0x460/0xb20
...
[  234.030037] -> #3 (&cpuctx_mutex):
[  234.040960]        perf_event_enable+0x18/0xa0
[  234.046126]        lockup_detector_online_cpu+0x22/0x30
[  234.057658]        cpuhp_thread_fun+0x164/0x1e0
[  234.115402] -> #1 (cpu_hotplug_lock):
[  234.122985]        cpus_read_lock+0x3b/0xd0
[  234.139757]        iommu_setup_dma_ops+0x18a/0x560


================================================================================
Potential Solutions Suggested by AI
================================================================================

1. Decouple iommu_setup_dma_ops() from group->mutex in drivers/iommu/iommu.c:
   iommu_setup_dma_ops() only requires struct device * and the domain pointer
   (group->default_domain); it does not mutate any fields in struct iommu_group.
   Moving the iommu_setup_dma_ops() calls after mutex_unlock(&group->mutex) in
   iommu_probe_device(), bus_iommu_probe(), and iommu_group_store_type() breaks
   the initial &group->mutex -> cpu_hotplug_lock dependency.

2. Avoid holding down_write(&vdev->memory_lock) across pci_try_reset_function()
   in VFIO:
   vfio-pci could zap active BAR mappings under memory_lock and set a state
   flag / disable memory decoding, drop memory_lock before calling
   pci_try_reset_function(), and then re-acquire memory_lock to re-enable
   memory. While resetting, any concurrent user fault will see the memory
   disabled condition and return VM_FAULT_SIGBUS safely.

3. Refine synchronization in pci_dev_reset_iommu_prepare():
   Evaluate if attaching to the blocking domain and pausing ATS during device
   reset can be protected using more fine-grained locking or atomic state
   flags without holding the coarse &group->mutex.

Feedback and suggestions on the preferred direction to resolve this would be
greatly appreciated.

Thanks,
Vipin Sharma

Vipin Sharma (1):
  selftests: vfio: Add mmap fault and device reset test

 tools/testing/selftests/vfio/Makefile         |  1 +
 .../selftests/vfio/vfio_pci_mmap_reset_test.c | 60 +++++++++++++++++++
 2 files changed, 61 insertions(+)
 create mode 100644 tools/testing/selftests/vfio/vfio_pci_mmap_reset_test.c

-- 
2.55.0.766.g2966f0265a-goog

[RFC PATCH 1/1] selftests: vfio: Add mmap fault and device reset test

[Vipin Sharma]

Add a selftest to verify VFIO PCI device reset on an mmapped and faulted
device. The test maps all available BARs on the device, faults them,
and triggers a device reset via the VFIO_DEVICE_RESET ioctl.

This exercise uncovers a circular locking dependency introduced in
commit f5b16b802174 ("PCI: Suspend iommu function prior to resetting a
device"), where pci_dev_reset_iommu_prepare() acquires group->mutex
under vfio's vdev->memory_lock.

Assisted-by: Jetski:gemini-3.1-pro
Signed-off-by: Vipin Sharma <vipinsh@google.com>
---
 tools/testing/selftests/vfio/Makefile         |  1 +
 .../selftests/vfio/vfio_pci_mmap_reset_test.c | 60 +++++++++++++++++++
 2 files changed, 61 insertions(+)
 create mode 100644 tools/testing/selftests/vfio/vfio_pci_mmap_reset_test.c

diff --git a/tools/testing/selftests/vfio/Makefile b/tools/testing/selftests/vfio/Makefile
index 2c32c48db509..17b3a2fe215c 100644
--- a/tools/testing/selftests/vfio/Makefile
+++ b/tools/testing/selftests/vfio/Makefile
@@ -13,6 +13,7 @@ TEST_GEN_PROGS += vfio_pci_device_test
 TEST_GEN_PROGS += vfio_pci_device_init_perf_test
 TEST_GEN_PROGS += vfio_pci_driver_test
 TEST_GEN_PROGS += vfio_pci_sriov_uapi_test
+TEST_GEN_PROGS += vfio_pci_mmap_reset_test
 
 TEST_FILES += scripts/cleanup.sh
 TEST_FILES += scripts/lib.sh
diff --git a/tools/testing/selftests/vfio/vfio_pci_mmap_reset_test.c b/tools/testing/selftests/vfio/vfio_pci_mmap_reset_test.c
new file mode 100644
index 000000000000..b004867905de
--- /dev/null
+++ b/tools/testing/selftests/vfio/vfio_pci_mmap_reset_test.c
@@ -0,0 +1,60 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <linux/pci_regs.h>
+#include <linux/vfio.h>
+
+#include <libvfio.h>
+
+#include "kselftest_harness.h"
+
+static const char *device_bdf;
+
+FIXTURE(vfio_pci_mmap_reset_test) {
+	struct iommu *iommu;
+	struct vfio_pci_device *device;
+};
+
+FIXTURE_SETUP(vfio_pci_mmap_reset_test)
+{
+	self->iommu = iommu_init(MODE_IOMMUFD);
+	self->device = vfio_pci_device_init(device_bdf, self->iommu);
+}
+
+FIXTURE_TEARDOWN(vfio_pci_mmap_reset_test)
+{
+	vfio_pci_device_cleanup(self->device);
+	iommu_cleanup(self->iommu);
+}
+
+TEST_F(vfio_pci_mmap_reset_test, mmap_fault_and_reset)
+{
+	volatile char dummy;
+	bool has_mmap = false;
+	int i;
+
+	if (!(self->device->info.flags & VFIO_DEVICE_FLAGS_RESET))
+		SKIP(return, "Device does not support reset\n");
+
+	for (i = 0; i < PCI_STD_NUM_BARS; i++) {
+		struct vfio_pci_bar *bar = &self->device->bars[i];
+
+		if (!bar->vaddr)
+			continue;
+
+		/* Touch BAR to trigger page fault under mmap_lock */
+		dummy = *(volatile char *)bar->vaddr;
+		(void)dummy;
+		has_mmap = true;
+	}
+
+	if (!has_mmap)
+		SKIP(return, "No mmapable BAR found on device\n");
+
+	/* Trigger device reset under memory_lock */
+	vfio_pci_device_reset(self->device);
+}
+
+int main(int argc, char *argv[])
+{
+	device_bdf = vfio_selftests_get_bdf(&argc, argv);
+	return test_harness_run(argc, argv);
+}
-- 
2.55.0.766.g2966f0265a-goog

Re: [RFC PATCH 1/1] selftests: vfio: Add mmap fault and device reset test

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] A circular locking dependency exists between `mmap_lock`, `vdev->memory_lock`, and `group->mutex`, leading to a potential deadlock.
--

--- Patch [1]: [RFC PATCH 1/1] selftests: vfio: Add mmap fault and device reset test ---
commit 3fb77a6912d6cc2309856fef1f82986ea816df96
Author: Vipin Sharma <vipinsh@google.com>
Subject: selftests: vfio: Add mmap fault and device reset test

This commit adds a selftest to verify VFIO PCI device reset on an mmapped
and faulted device. The test maps all available BARs, triggers a page
fault on them, and then initiates a device reset via the VFIO_DEVICE_RESET
ioctl to expose a circular locking dependency.

> diff --git a/tools/testing/selftests/vfio/vfio_pci_mmap_reset_test.c b/tools/testing/selftests/vfio/vfio_pci_mmap_reset_test.c
> new file mode 100644
> --- /dev/null
> +++ b/tools/testing/selftests/vfio/vfio_pci_mmap_reset_test.c
[ ... ]
> +TEST_F(vfio_pci_mmap_reset_test, mmap_fault_and_reset)
> +{
[ ... ]
> +	if (!has_mmap)
> +		SKIP(return, "No mmapable BAR found on device\n");
> +
> +	/* Trigger device reset under memory_lock */
> +	vfio_pci_device_reset(self->device);

[Severity: High]
This is a pre-existing issue, but does this test sequence reliably trigger a
host kernel deadlock by exposing a circular locking dependency between
mmap_lock, vdev->memory_lock, and group->mutex?

The lockdep cycle appears to start when a page fault holds mmap_lock and
acquires vdev->memory_lock:

drivers/vfio/pci/vfio_pci_core.c:vfio_pci_mmap_huge_fault()
    scoped_guard(rwsem_read, &vdev->memory_lock)

Separately, a device reset acquires vdev->memory_lock before initiating the
reset:

drivers/vfio/pci/vfio_pci_core.c:vfio_pci_ioctl_reset()
    vfio_pci_zap_and_down_write_memory_lock(vdev);

While held, it calls into the PCI reset functions which eventually prepare
the IOMMU and acquire group->mutex:

drivers/iommu/iommu.c:pci_dev_reset_iommu_prepare()
    guard(mutex)(&group->mutex);

The cycle completes when another kernel path holding group->mutex waits on
mmap_lock or memory_lock. Can a malicious guest VM or local user with VFIO
access trigger concurrent page faults and device resets to exploit this
cycle?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260821193502.92431-2-vipinsh@google.com?part=1