From: Vipin Sharma <vipinsh@google.com>
To: David Matlack <dmatlack@google.com>
Cc: kexec@lists.infradead.org, linux-kernel@vger.kernel.org,
linux-doc@vger.kernel.org, kvm@vger.kernel.org,
linux-mm@kvack.org, linux-kselftest@vger.kernel.org,
linux-pci@vger.kernel.org, ajayachandra@nvidia.com,
alex@shazbot.org, amastro@fb.com, ankita@nvidia.com,
apopple@nvidia.com, bhelgaas@google.com, chrisl@kernel.org,
christian.koenig@amd.com, corbet@lwn.net, graf@amazon.com,
jacob.pan@linux.microsoft.com, jgg@nvidia.com, jgg@ziepe.ca,
jrhilke@google.com, julianr@linux.ibm.com, kees@kernel.org,
kevin.tian@intel.com, leon@kernel.org, leonro@nvidia.com,
lukas@wunner.de, mattev@meta.com, michal.winiarski@intel.com,
parav@nvidia.com, pasha.tatashin@soleen.com, praan@google.com,
pratyush@kernel.org, rananta@google.com, rdunlap@infradead.org,
rientjes@google.com, rodrigo.vivi@intel.com, rppt@kernel.org,
saeedm@nvidia.com, schnelle@linux.ibm.com,
skhan@linuxfoundation.org, skhawaja@google.com,
vivek.kasireddy@intel.com, witu@nvidia.com,
yanjun.zhu@linux.dev, yi.l.liu@intel.com
Subject: Re: [PATCH v6 07/27] vfio/pci: Preserve vfio-pci device files across Live Update
Date: Tue, 17 Nov 2026 13:59:00 -0800 [thread overview]
Message-ID: <20261117003405.GA4153658.vipinsh@google.com> (raw)
In-Reply-To: <auzVEUpFEoWltpVg@google.com>
On Thu, Nov 05, 2026 at 07:12:17PM +0000, David Matlack wrote:
> On 2026-11-02 07:36 AM, Vipin Sharma wrote:
> > * TOCTOU synchronization: vfio_pci_dev_is_frozen() is checked locklessly
> > on entry to the ioctl, read/write and mmap paths, and re-checked under
> > a lock that freeze() holds before any operation that could change the
> > device or vconfig: under vdev->igate in vfio_pci_ioctl_set_irqs() and
> > around every config-space write in vfio_pci_config_rw_single(), under
> > vdev->memory_lock in vfio_pci_runtime_pm_entry(),
> > vfio_pci_ioctl_reset() and the SR-IOV enable path of
> > vfio_pci_core_sriov_configure(), and under dev_set->lock in
> > vfio_pci_dev_set_hot_reset(). A racing operation thus either completes
> > before freeze() takes its snapshot or fails, while existing BAR mmaps,
> > ioeventfds and dma-bufs keep working. Serializing config writes on
> > igate covers the PCI_COMMAND, INTx disable emulation, power state and
> > FLR paths at one point (the INTx disable emulation now uses the
> > igate-held __vfio_pci_intx_mask() and __vfio_pci_intx_unmask()
> > helpers), and also serializes the irq_type and virq_disabled accesses
> > of config writes with VFIO_DEVICE_SET_IRQS.
>
> I think there's a simpler way to do this. Checking the frozen flag
> locklessly and then re-checking it under whichever lock covers each
> path spreads the synchronization across igate, memory_lock and
> dev_set->lock. That makes it hard to convince yourself every path that
> can change the device is covered. It also makes every config space
> write take igate, for all vfio-pci users, even when
> CONFIG_VFIO_PCI_LIVEUPDATE=n.
>
> Instead, I suggest adding a per-device rw_semaphore to struct
> vfio_device in the VFIO core, e.g. liveupdate_rwsem, that protects a
> liveupdate_frozen flag:
>
> - vfio_device_fops_unl_ioctl(), vfio_device_fops_read() and
> vfio_device_fops_write() take liveupdate_rwsem for read and fail
> with -EIO if the device is frozen. That covers every device ioctl in
> one place, including the ones handled by the VFIO core
> (VFIO_DEVICE_FEATURE, VFIO_DEVICE_GET_REGION_INFO and
> VFIO_DEVICE_{ATTACH,DETACH}_IOMMUFD_PT). This patch doesn't block
> the last two at all today.
>
> - freeze() and unfreeze() take liveupdate_rwsem for write, before any
> other lock, and flip the flag. preserve() doesn't need it since it
> only serializes state that doesn't change after the device is
> opened.
>
> Putting this in the VFIO core also means other VFIO drivers can reuse
> it if they add Live Update support in the future.
>
> The lock order then becomes:
>
> liveupdate_rwsem -> dev_set->lock -> igate -> memory_lock
>
> freeze() can still take igate and memory_lock to check irq_type,
> pm_runtime_engaged and sriov_active. But the ioctl, read and write
> paths no longer need to re-check anything under those locks. So you
> can drop the igate change in vfio_pci_config_rw_single(), the
> open-coded vfio_pci_zap_and_down_write_memory_lock() in
> vfio_pci_runtime_pm_entry() and vfio_pci_ioctl_reset(), and the
> smp_load_acquire()/smp_store_release() pair.
>
> Two paths still need their own handling because they don't go through
> the device's file:
>
> - vfio_pci_core_sriov_configure() is reached through sysfs. Take
> liveupdate_rwsem for read around the frozen check and the
> sriov_active update.
>
> - VFIO_DEVICE_PCI_HOT_RESET issued on one device also resets the other
> devices in its dev_set. Keep the check in
> vfio_pci_dev_set_hot_reset() under dev_set->lock. freeze() sets the
> flag while holding both liveupdate_rwsem and dev_set->lock, so
> holding either one is enough to read it. Please don't take the other
> devices' liveupdate_rwsem there, since that would invert the lock
> order relative to freeze().
>
> I don't think mmap needs to be blocked at all. Creating a new mapping
> doesn't change the device or any of the state that gets preserved, and
> existing mappings already keep working while the device is frozen.
> Leaving mmap alone also avoids taking liveupdate_rwsem under mmap_lock,
> which could deadlock: the ioctl, read and write paths can fault on user
> memory (taking mmap_lock) while holding liveupdate_rwsem for read, and
> freeze() may be waiting for the write lock. Please update the comment
> in freeze() and the DOC section to match.
>
Thanks for suggestion, this is indeed simpler.
> > diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c
> > index 9914f3ac69ae..1f1ac7a798aa 100644
> > --- a/drivers/vfio/pci/vfio_pci_config.c
> > +++ b/drivers/vfio/pci/vfio_pci_config.c
> > @@ -644,10 +644,10 @@ static int vfio_basic_config_write(struct vfio_pci_core_device *vdev, int pos,
> >
> > if (virt_intx_disable && !vdev->virq_disabled) {
> > vdev->virq_disabled = true;
> > - vfio_pci_intx_mask(vdev);
> > + __vfio_pci_intx_mask(vdev);
> > } else if (!virt_intx_disable && vdev->virq_disabled) {
> > vdev->virq_disabled = false;
> > - vfio_pci_intx_unmask(vdev);
> > + __vfio_pci_intx_unmask(vdev);
> > }
> > }
>
> With the rwsem, config writes no longer need igate, so this change
> isn't needed. If config writes and VFIO_DEVICE_SET_IRQS really do race
> on irq_type and virq_disabled today, that's unrelated to Live Update.
> Please send it as a separate fix.
>
This will go away.
> Lastly, please add the freeze infrastructure and the checks in their
> own commit ahead of this one, e.g. "vfio: Add infrastructure to freeze
> devices for Live Update". Then this patch only has to implement
> freeze() and unfreeze() on top of it, which will make both easier to
> review.
Sounds good. Thanks for the review.
next prev parent reply other threads:[~2026-11-17 21:59 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-02 15:36 [PATCH v6 00/27] VFIO PCI Live Update (noiommu mode) Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 01/27] vfio/pci: Factor out the reset logic in VFIO PCI device close path Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 02/27] vfio: Export various helpers from VFIO Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 03/27] liveupdate: Export symbols needed by modules Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 04/27] vfio/pci: Register a file handler with Live Update Orchestrator Vipin Sharma
2026-11-06 22:29 ` Samiullah Khawaja
2026-11-10 19:02 ` Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 05/27] PCI: liveupdate: Add helpers to preserve struct pci_saved_state Vipin Sharma
2026-11-02 23:06 ` Bjorn Helgaas
2026-11-03 0:48 ` David Matlack
2026-11-04 22:55 ` Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 06/27] vfio: Add helpers to identify device files that can be preserved Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 07/27] vfio/pci: Preserve vfio-pci device files across Live Update Vipin Sharma
2026-11-05 19:12 ` David Matlack
2026-11-17 21:59 ` Vipin Sharma [this message]
2026-11-05 19:57 ` David Matlack
2026-11-06 22:35 ` Samiullah Khawaja
2026-11-02 15:36 ` [PATCH v6 08/27] vfio: Introduce vfio_find_device() helper Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 09/27] vfio: Refactor vfio_device_fops_cdev_open() Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 10/27] vfio: Add API to open cdev device for Live Update restore Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 11/27] vfio/pci: Retrieve preserved device files after Live Update Vipin Sharma
2026-11-06 17:19 ` David Matlack
2026-11-02 15:36 ` [PATCH v6 12/27] vfio: Enforce preserved devices are retrieved via LIVEUPDATE_SESSION_RETRIEVE_FD Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 13/27] vfio/pci: Protect incoming preserved devices until they finish Live Update Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 14/27] vfio/pci: Take over preserved devices without resetting them Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 15/27] docs: liveupdate: Add documentation for VFIO PCI Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 16/27] vfio: selftests: Build liveupdate library in VFIO selftests Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 17/27] vfio: selftests: Add vfio_pci_liveupdate_uapi_test Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 18/27] vfio: selftests: Initialize vfio_pci_device using a VFIO cdev FD Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 19/27] vfio: selftests: Add Makefile support for TEST_GEN_PROGS_EXTENDED Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 20/27] vfio: selftests: Add iommufd noiommu mode Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 21/27] vfio: selftests: Add helpers to map DMA memory at IOVA == PA in " Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 22/27] vfio: selftests: Add helpers for Live Update DMA memory and sysfs state Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 23/27] vfio: selftests: Add an attach() driver op Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 24/27] vfio: selftests: Test preserving noiommu devices in vfio_pci_liveupdate_uapi_test Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 25/27] vfio: selftests: Add vfio_pci_liveupdate_kexec_test Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 26/27] vfio: selftests: Verify that opening VFIO device fails during Live Update Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 27/27] vfio: selftests: Add continuous DMA to vfio_pci_liveupdate_kexec_test Vipin Sharma
2026-11-06 22:20 ` [PATCH v6 00/27] VFIO PCI Live Update (noiommu mode) David Matlack
2026-11-10 18:45 ` Vipin Sharma
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261117003405.GA4153658.vipinsh@google.com \
--to=vipinsh@google.com \
--cc=ajayachandra@nvidia.com \
--cc=alex@shazbot.org \
--cc=amastro@fb.com \
--cc=ankita@nvidia.com \
--cc=apopple@nvidia.com \
--cc=bhelgaas@google.com \
--cc=chrisl@kernel.org \
--cc=christian.koenig@amd.com \
--cc=corbet@lwn.net \
--cc=dmatlack@google.com \
--cc=graf@amazon.com \
--cc=jacob.pan@linux.microsoft.com \
--cc=jgg@nvidia.com \
--cc=jgg@ziepe.ca \
--cc=jrhilke@google.com \
--cc=julianr@linux.ibm.com \
--cc=kees@kernel.org \
--cc=kevin.tian@intel.com \
--cc=kexec@lists.infradead.org \
--cc=kvm@vger.kernel.org \
--cc=leon@kernel.org \
--cc=leonro@nvidia.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-pci@vger.kernel.org \
--cc=lukas@wunner.de \
--cc=mattev@meta.com \
--cc=michal.winiarski@intel.com \
--cc=parav@nvidia.com \
--cc=pasha.tatashin@soleen.com \
--cc=praan@google.com \
--cc=pratyush@kernel.org \
--cc=rananta@google.com \
--cc=rdunlap@infradead.org \
--cc=rientjes@google.com \
--cc=rodrigo.vivi@intel.com \
--cc=rppt@kernel.org \
--cc=saeedm@nvidia.com \
--cc=schnelle@linux.ibm.com \
--cc=skhan@linuxfoundation.org \
--cc=skhawaja@google.com \
--cc=vivek.kasireddy@intel.com \
--cc=witu@nvidia.com \
--cc=yanjun.zhu@linux.dev \
--cc=yi.l.liu@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox