From: Josh Hilke <jrhilke@google.com>
To: Vipin Sharma <vipinsh@google.com>
Cc: kexec@lists.infradead.org, linux-kernel@vger.kernel.org,
linux-doc@vger.kernel.org, kvm@vger.kernel.org,
linux-mm@kvack.org, linux-kselftest@vger.kernel.org,
ajayachandra@nvidia.com, alex@shazbot.org, amastro@fb.com,
ankita@nvidia.com, apopple@nvidia.com, bhelgaas@google.com,
chrisl@kernel.org, christian.koenig@amd.com, corbet@lwn.net,
dmatlack@google.com, graf@amazon.com,
jacob.pan@linux.microsoft.com, jgg@nvidia.com, jgg@ziepe.ca,
julianr@linux.ibm.com, kees@kernel.org, kevin.tian@intel.com,
leon@kernel.org, leonro@nvidia.com, lukas@wunner.de,
mattev@meta.com, michal.winiarski@intel.com, parav@nvidia.com,
pasha.tatashin@soleen.com, praan@google.com,
pratyush@kernel.org, rananta@google.com, rientjes@google.com,
rodrigo.vivi@intel.com, rppt@kernel.org, saeedm@nvidia.com,
schnelle@linux.ibm.com, skhan@linuxfoundation.org,
skhawaja@google.com, vivek.kasireddy@intel.com, witu@nvidia.com,
yanjun.zhu@linux.dev, yi.l.liu@intel.com
Subject: Re: [PATCH v5 06/20] vfio/pci: Preserve vfio-pci device files across Live Update
Date: Tue, 21 Jul 2026 23:16:25 +0000 [thread overview]
Message-ID: <al_8v_I9Z2Dc_qjc@google.com> (raw)
In-Reply-To: <20260714151505.3466855-7-vipinsh@google.com>
On Tue, Jul 14, 2026 at 08:14:51AM -0700, Vipin Sharma wrote:
> +static int vfio_pci_liveupdate_freeze(struct liveupdate_file_op_args *args)
> +{
> + struct vfio_device *device = vfio_device_from_file(args->file);
> + struct vfio_pci_core_device *vdev;
> + struct pci_dev *pdev;
> +
> + vdev = container_of(device, struct vfio_pci_core_device, vdev);
> + pdev = vdev->pdev;
> +
> + guard(mutex)(&device->dev_set->lock);
> + guard(mutex)(&vdev->igate);
> + guard(rwsem_write)(&vdev->memory_lock);
> +
> + /*
> + * Userspace must disable interrupts on the device prior to freeze so
> + * that the device does not send any interrupts until new interrupt
> + * handlers have been established by the next kernel.
> + */
> + if (vdev->irq_type != VFIO_PCI_NUM_IRQS) {
> + pci_err(pdev, "Freeze failed! Interrupts are still enabled.\n");
> + return -EINVAL;
> + }
> +
> + if (pdev->current_state != PCI_D0) {
> + pci_err(pdev, "Freeze failed! Device not in D0 state.\n");
> + return -EINVAL;
> + }
> +
> + /*
> + * Reset is a temporary measure to provide kernel after kexec a clean
> + * device while VFIO live update work is under development and not
> + * fully supported. It will go away once continuous DMA support is
> + * added to device preservation.
> + */
> + vfio_pci_zap_bars(vdev);
> + vfio_pci_dma_buf_move(vdev, true);
> + vfio_pci_core_try_reset(vdev);
> + pci_write_config_word(pdev, PCI_COMMAND, PCI_COMMAND_INTX_DISABLE);
> + /*
> + * Userspace cannot use the FD correctly now irrespective of liveupdate
> + * freeze failing or succeeding. They will have to reinitialize the VFIO
> + * device to continue using it as reset might have loaded default PCI
> + * state. Disable ioctl, read, write and mmap access to the device.
> + */
> + smp_store_release(&vdev->liveupdate_frozen, true);
> + return 0;
> }
I'm seeing the same lockdep splat the David found in v4.
https://lore.kernel.org/all/ag-aFA1BJxdJMywr@google.com/#t
======================================================
WARNING: possible circular locking dependency detected
7.2.0-dbg-DEV #3 Tainted: G S
------------------------------------------------------
kexec/13355 is trying to acquire lock:
ff3f086c95315d08 (&group->mutex){+.+.}-{4:4}, at: pci_dev_reset_iommu_prepare+0x6e/0x200
but task is already holding lock:
ff3f082de7f399a8 (&vdev->memory_lock){++++}-{4:4}, at: vfio_pci_liveupdate_freeze+0x58/0x100
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #5 (&vdev->memory_lock){++++}-{4:4}:
down_read+0x3d/0x160
vfio_pci_mmap_huge_fault+0xb9/0x150
__do_fault+0x46/0x150
do_pte_missing+0x21a/0x1000
handle_mm_fault+0x7f8/0xb60
do_user_addr_fault+0x476/0x6c0
exc_page_fault+0x68/0xa0
asm_exc_page_fault+0x26/0x30
-> #4 (&mm->mmap_lock){++++}-{4:4}:
__might_fault+0x5e/0x80
_copy_to_user+0x23/0x60
perf_read+0x114/0x310
vfs_read+0xe7/0x360
ksys_read+0x73/0x100
do_syscall_64+0x15f/0x4e0
entry_SYSCALL_64_after_hwframe+0x77/0x7f
-> #3 (&cpuctx_mutex){+.+.}-{4:4}:
__mutex_lock+0x8c/0xd80
perf_event_ctx_lock_nested+0x15a/0x210
perf_event_enable+0x18/0xa0
lockup_detector_online_cpu+0x22/0x30
cpuhp_invoke_callback+0xfb/0x2c0
cpuhp_thread_fun+0x164/0x1e0
smpboot_thread_fn+0x17e/0x280
kthread+0x10c/0x140
ret_from_fork+0x16b/0x310
ret_from_fork_asm+0x1a/0x30
-> #2 (cpuhp_state-up){+.+.}-{0:0}:
cpuhp_thread_fun+0x95/0x1e0
smpboot_thread_fn+0x17e/0x280
kthread+0x10c/0x140
ret_from_fork+0x16b/0x310
ret_from_fork_asm+0x1a/0x30
-> #1 (cpu_hotplug_lock){++++}-{0:0}:
cpus_read_lock+0x3b/0xd0
__cpuhp_state_add_instance+0x19/0x40
iova_domain_init_rcaches+0x1ef/0x230
iommu_setup_dma_ops+0x18a/0x560
iommu_device_register+0x188/0x220
intel_iommu_init+0x35a/0x440
pci_iommu_init+0x16/0x40
do_one_initcall+0xf5/0x400
do_initcall_level+0x82/0xa0
do_initcalls+0x59/0xa0
kernel_init_freeable+0x152/0x1d0
kernel_init+0x1a/0x130
ret_from_fork+0x16b/0x310
ret_from_fork_asm+0x1a/0x30
-> #0 (&group->mutex){+.+.}-{4:4}:
__lock_acquire+0x14c8/0x2750
lock_acquire+0xd3/0x2c0
__mutex_lock+0x8c/0xd80
pci_dev_reset_iommu_prepare+0x6e/0x200
pcie_flr+0x32/0xc0
__pci_reset_function_locked+0x84/0x120
vfio_pci_core_try_reset+0xa4/0x110
vfio_pci_liveupdate_freeze+0x8a/0x100
luo_file_freeze+0xd1/0x290
luo_session_serialize+0xa6/0x240
liveupdate_reboot+0x19/0x30
kernel_kexec+0x39/0xb0
__se_sys_reboot+0xfd/0x210
do_syscall_64+0x15f/0x4e0
entry_SYSCALL_64_after_hwframe+0x77/0x7f
other info that might help us debug this:
Chain exists of:
&group->mutex --> &mm->mmap_lock --> &vdev->memory_lock
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&vdev->memory_lock);
lock(&mm->mmap_lock);
lock(&vdev->memory_lock);
lock(&group->mutex);
*** DEADLOCK ***
9 locks held by kexec/13355:
#0: ffffffff85a81270 (system_transition_mutex){+.+.}-{4:4}, at: __se_sys_reboot+0xe4/0x210
#1: ffffffff85e1cfa8 (luo_session_serialize_rwsem){++++}-{4:4}, at: luo_session_serialize+0x30/0x240
#2: ffffffff85e1d140 (luo_session_global.outgoing.rwsem){+.+.}-{4:4}, at: luo_session_serialize+0x43/0x240
#3: ff3f082dee702108 (&session->mutex){+.+.}-{4:4}, at: luo_session_serialize+0x99/0x240
#4: ff3f082db51c2588 (&luo_file->mutex){+.+.}-{4:4}, at: luo_file_freeze+0x81/0x290
#5: ff3f082d8af761a8 (&new_dev_set->lock){+.+.}-{4:4}, at: vfio_pci_liveupdate_freeze+0x38/0x100
#6: ff3f082de7f39780 (&vdev->igate){+.+.}-{4:4}, at: vfio_pci_liveupdate_freeze+0x49/0x100
#7: ff3f082de7f399a8 (&vdev->memory_lock){++++}-{4:4}, at: vfio_pci_liveupdate_freeze+0x58/0x100
#8: ff3f086c908911f8 (&dev->mutex){....}-{4:4}, at: pci_dev_trylock+0x25/0x60
stack backtrace:
CPU: 132 UID: 0 PID: 13355 Comm: kexec Tainted: G S 7.2.0-dbg-DEV #3 PREEMPTLAZY
Tainted: [S]=CPU_OUT_OF_SPEC
Hardware name: Google Izumi/izumi, BIOS 0.20260327.0-0 03/27/2026
Call Trace:
<TASK>
dump_stack_lvl+0x54/0x70
print_circular_bug+0x2e1/0x300
check_noncircular+0xf9/0x120
? __bfs+0x129/0x200
__lock_acquire+0x14c8/0x2750
? __lock_acquire+0x1223/0x2750
? check_noncircular+0xa5/0x120
? pci_dev_reset_iommu_prepare+0x6e/0x200
lock_acquire+0xd3/0x2c0
? pci_dev_reset_iommu_prepare+0x6e/0x200
? lock_is_held_type+0x76/0x100
? pci_dev_reset_iommu_prepare+0x6e/0x200
__mutex_lock+0x8c/0xd80
? pci_dev_reset_iommu_prepare+0x6e/0x200
? lockdep_hardirqs_on_prepare+0x152/0x220
? _raw_spin_unlock_irqrestore+0x35/0x50
pci_dev_reset_iommu_prepare+0x6e/0x200
pcie_flr+0x32/0xc0
__pci_reset_function_locked+0x84/0x120
vfio_pci_core_try_reset+0xa4/0x110
vfio_pci_liveupdate_freeze+0x8a/0x100
luo_file_freeze+0xd1/0x290
luo_session_serialize+0xa6/0x240
liveupdate_reboot+0x19/0x30
kernel_kexec+0x39/0xb0
__se_sys_reboot+0xfd/0x210
? check_object+0x1e8/0x390
? init_object+0x34/0x110
? lock_release+0xf0/0x330
? kmem_cache_free+0x1b5/0x520
? kmem_cache_free+0x1c9/0x520
? _raw_spin_unlock_irqrestore+0x35/0x50
? kmem_cache_free+0x1b5/0x520
? __x64_sys_close+0x3d/0x80
? entry_SYSCALL_64_after_hwframe+0x77/0x7f
? entry_SYSCALL_64_after_hwframe+0x77/0x7f
do_syscall_64+0x15f/0x4e0
entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f04e7233313
Code: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 89 fa b8 a9 00 00 00 bf ad de e1 fe be 69 19 12 28 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 f7 d8 48 8b 0d eb fb 06 00 64 89 01 48
RSP: 002b:00007ffe7c91b688 EFLAGS: 00000246 ORIG_RAX: 00000000000000a9
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f04e7233313
RDX: 0000000045584543 RSI: 0000000028121969 RDI: 00000000fee1dead
RBP: 00007ffe7c91b9b0 R08: 000000000000000a R09: 00007f04e72a4ff0
R10: 0000000000000011 R11: 0000000000000246 R12: 0000000000000001
R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000002
</TASK>
next prev parent reply other threads:[~2026-07-21 23:16 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-07-14 15:14 [PATCH v5 00/20] vfio/pci: Base Live Update support for VFIO Vipin Sharma
2026-07-14 15:14 ` [PATCH v5 01/20] vfio/pci: Factor out the reset logic in VFIO PCI device close path Vipin Sharma
2026-07-14 15:14 ` [PATCH v5 02/20] vfio: Export various helpers from VFIO Vipin Sharma
2026-07-14 15:14 ` [PATCH v5 03/20] vfio/pci: Export vfio_pci_dma_buf_move for vfio-pci module Vipin Sharma
2026-07-14 15:14 ` [PATCH v5 04/20] liveupdate: Export symbols needed by modules Vipin Sharma
2026-07-14 15:14 ` [PATCH v5 05/20] vfio/pci: Register a file handler with Live Update Orchestrator Vipin Sharma
2026-07-14 15:14 ` [PATCH v5 06/20] vfio/pci: Preserve vfio-pci device files across Live Update Vipin Sharma
2026-07-21 23:16 ` Josh Hilke [this message]
2026-07-14 15:14 ` [PATCH v5 07/20] vfio: Introduce vfio_find_device() helper Vipin Sharma
2026-07-14 15:14 ` [PATCH v5 08/20] vfio: Refactor vfio_device_fops_cdev_open() Vipin Sharma
2026-07-14 15:14 ` [PATCH v5 09/20] vfio: Add API to open cdev device for Live Update restore Vipin Sharma
2026-07-14 15:14 ` [PATCH v5 10/20] vfio/pci: Retrieve preserved device files after Live Update Vipin Sharma
2026-07-21 16:23 ` Josh Hilke
2026-07-14 15:14 ` [PATCH v5 11/20] vfio: Enforce preserved devices are retrieved via LIVEUPDATE_SESSION_RETRIEVE_FD Vipin Sharma
2026-07-14 15:14 ` [PATCH v5 12/20] docs: liveupdate: Add documentation for VFIO PCI Vipin Sharma
2026-07-14 15:14 ` [PATCH v5 13/20] vfio: selftests: Build liveupdate library in VFIO selftests Vipin Sharma
2026-07-14 15:14 ` [PATCH v5 14/20] vfio: selftests: Add vfio_pci_liveupdate_uapi_test Vipin Sharma
2026-07-14 15:15 ` [PATCH v5 15/20] vfio: selftests: Initialize vfio_pci_device using a VFIO cdev FD Vipin Sharma
2026-07-14 15:15 ` [PATCH v5 16/20] vfio: selftests: Add Makefile support for TEST_GEN_PROGS_EXTENDED Vipin Sharma
2026-07-14 15:15 ` [PATCH v5 17/20] vfio: selftests: Add vfio_pci_liveupdate_kexec_test Vipin Sharma
2026-07-14 15:15 ` [PATCH v5 18/20] vfio: selftests: Expose iommu_modes to tests Vipin Sharma
2026-07-14 15:15 ` [PATCH v5 19/20] vfio: selftests: Verify that opening VFIO device fails during Live Update Vipin Sharma
2026-07-14 15:15 ` [PATCH v5 20/20] vfio: selftests: Add continuous DMA to vfio_pci_liveupdate_kexec_test Vipin Sharma
2026-07-16 23:24 ` [PATCH v5 00/20] vfio/pci: Base Live Update support for VFIO Yanjun.Zhu
2026-07-17 1:22 ` Yanjun.Zhu
2026-07-20 17:08 ` Vipin Sharma
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=al_8v_I9Z2Dc_qjc@google.com \
--to=jrhilke@google.com \
--cc=ajayachandra@nvidia.com \
--cc=alex@shazbot.org \
--cc=amastro@fb.com \
--cc=ankita@nvidia.com \
--cc=apopple@nvidia.com \
--cc=bhelgaas@google.com \
--cc=chrisl@kernel.org \
--cc=christian.koenig@amd.com \
--cc=corbet@lwn.net \
--cc=dmatlack@google.com \
--cc=graf@amazon.com \
--cc=jacob.pan@linux.microsoft.com \
--cc=jgg@nvidia.com \
--cc=jgg@ziepe.ca \
--cc=julianr@linux.ibm.com \
--cc=kees@kernel.org \
--cc=kevin.tian@intel.com \
--cc=kexec@lists.infradead.org \
--cc=kvm@vger.kernel.org \
--cc=leon@kernel.org \
--cc=leonro@nvidia.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=lukas@wunner.de \
--cc=mattev@meta.com \
--cc=michal.winiarski@intel.com \
--cc=parav@nvidia.com \
--cc=pasha.tatashin@soleen.com \
--cc=praan@google.com \
--cc=pratyush@kernel.org \
--cc=rananta@google.com \
--cc=rientjes@google.com \
--cc=rodrigo.vivi@intel.com \
--cc=rppt@kernel.org \
--cc=saeedm@nvidia.com \
--cc=schnelle@linux.ibm.com \
--cc=skhan@linuxfoundation.org \
--cc=skhawaja@google.com \
--cc=vipinsh@google.com \
--cc=vivek.kasireddy@intel.com \
--cc=witu@nvidia.com \
--cc=yanjun.zhu@linux.dev \
--cc=yi.l.liu@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox