Kernel KVM virtualization development
 help / color / mirror / Atom feed
From: "David Matlack" <dmatlack@google.com>
To: "Vipin Sharma" <vipinsh@google.com>, <kexec@lists.infradead.org>,
	<linux-kernel@vger.kernel.org>, <linux-doc@vger.kernel.org>,
	<kvm@vger.kernel.org>, <linux-mm@kvack.org>,
	<linux-kselftest@vger.kernel.org>, <linux-pci@vger.kernel.org>
Cc: <ajayachandra@nvidia.com>, <alex@shazbot.org>, <amastro@fb.com>,
	<ankita@nvidia.com>, <apopple@nvidia.com>, <bhelgaas@google.com>,
	<chrisl@kernel.org>, <christian.koenig@amd.com>, <corbet@lwn.net>,
	<dmatlack@google.com>, <graf@amazon.com>,
	<jacob.pan@linux.microsoft.com>, <jgg@nvidia.com>, <jgg@ziepe.ca>,
	<jrhilke@google.com>, <julianr@linux.ibm.com>, <kees@kernel.org>,
	<kevin.tian@intel.com>, <leon@kernel.org>, <leonro@nvidia.com>,
	<lukas@wunner.de>, <mattev@meta.com>,
	<michal.winiarski@intel.com>, <parav@nvidia.com>,
	<pasha.tatashin@soleen.com>, <praan@google.com>,
	<pratyush@kernel.org>, <rananta@google.com>,
	<rdunlap@infradead.org>, <rientjes@google.com>,
	<rodrigo.vivi@intel.com>, <rppt@kernel.org>, <saeedm@nvidia.com>,
	<schnelle@linux.ibm.com>, <skhan@linuxfoundation.org>,
	<skhawaja@google.com>, <vivek.kasireddy@intel.com>,
	<witu@nvidia.com>, <yanjun.zhu@linux.dev>, <yi.l.liu@intel.com>
Subject: Re: [PATCH v6 11/27] vfio/pci: Retrieve preserved device files after Live Update
Date: Thu, 19 Nov 2026 17:01:19 -0800	[thread overview]
Message-ID: <DMZMRTXS7OSC.M79XF39ZVEWE@google.com> (raw)
In-Reply-To: <20261102153656.1431701-12-vipinsh@google.com>

On Mon Nov 2, 2026 at 7:36 AM PST, Vipin Sharma wrote:
> Enable userspace to retrieve preserved VFIO device files after a Live
> Update by implementing the retrieve() and finish() file handler
> callbacks.
>
> During retrieve():
>  * Validate the handed-over struct vfio_pci_core_device_ser header,
>    sizes, and flags.
> [...]

Please make the commit messages in this series easier to read. They are
mostly long bulleted lists of everything the code does, which is hard to
follow and largely repeats the diff. A commit message should explain, in
prose, what problem the patch solves and why it solves it this way, and
only call out details that a reviewer can't easily get from the code.
This applies to most of the patches in the series (e.g. PATCH 07 and
PATCH 13 too).

> Do not implement can_finish(): closing a retrieved LUO session invokes
> finish, and refusing can_finish() would leak the session and leave
> unclaimed devices running. Instead, finish() always completes:
>  * If the retrieved device was claimed by opening it via
>    VFIO_DEVICE_BIND_IOMMUFD (which consumes vdev->liveupdate_incoming;
>    subsequent patches replace the open-time reset with non-reset
>    take-over, reject opening incoming devices other than via retrieve(),
>    and protect incoming kept-running devices across runtime PM, dev_set
>    resets, and non-noiommu boots), finish() simply completes its PCI
>    Live Update participation.
>  * If the device was not claimed (never retrieved, retrieve() failed, or
>    retrieved but never bound via VFIO_DEVICE_BIND_IOMMUFD),
>    vfio_pci_core_liveupdate_reset() logs one pci_warn() naming the
>    device, clears PCI_COMMAND_MASTER, resets the function, and restores
>    the inherited pci_saved_state just as closing the device would. If
>    the device has a function reset but is not reset, e.g. because
>    pci_dev_trylock() contends with a concurrent driver unbind, latch
>    vdev->liveupdate_reset_pending so vfio_pci_core_unregister_device()
>    completes the function reset and state restoration via
>    vfio_pci_core_liveupdate_reset_locked() under the device lock held by
>    device_release_driver(). Opening the device after finish() resets it
>    like any open, so it drops the latch.
>  * If the device was not retrieved, look it up under pci_dev_lock(),
>    which waits for a vfio-pci probe in progress. If it is bound to
>    vfio-pci, finish it as above. If it has no driver, clear bus
>    mastering and reset it. Otherwise leave it to the bound driver.
>  * Call pci_liveupdate_finish() and kho_restore_free() on every path.

This is a lot of complexity for one commit, and it's hard to review. I'd
like to start with the simplest approach and build from there: implement
can_finish() and have it return false until the device is fully
retrieved, i.e. retrieved and taken over by VFIO_DEVICE_BIND_IOMMUFD.
Then finish() only needs to free the incoming state and call
pci_liveupdate_finish().

That gets rid of vfio_pci_core_liveupdate_reset(),
vfio_pci_core_liveupdate_reset_locked(),
vfio_pci_liveupdate_finish_lookup() and its bridge and device locking,
liveupdate_reset_pending, and most of the finish documentation. If you
want to handle the other scenarios (devices that are never retrieved or
never bound, devices bound to other drivers, unbind racing with finish),
please add them in separate commits on top, each with its own
justification.

As for closing the session leaking it and leaving unclaimed devices
running: that only happens if userspace closes the session without
taking over all of its devices, and leaving the device running with its
memory still preserved is safe. Resetting the device in that case is
something that can be added later.

It also has a nice side effect: LUO calls can_finish() on every file in
a session before finishing any of them, so a memfd in the same session
that the device DMAs into can't be finished before the device has been
taken over.

> +static int
> +vfio_pci_liveupdate_attach_incoming(struct vfio_pci_core_device *vdev,
> +				    struct vfio_pci_liveupdate_incoming *in)
> +{
> +	guard(mutex)(&vdev->vdev.dev_set->lock);
> +
> +	if (vdev->liveupdate_incoming || vdev->liveupdate_claimed ||
> +	    !pci_liveupdate_is_incoming(vdev->pdev))
> +		return -EBUSY;
> +
> +	vdev->liveupdate_incoming = in;
> +	return 0;
> +}

The pci_liveupdate_is_incoming() check here is redundant with the one in
vfio_pci_liveupdate_retrieve(). Nothing can clear it in between, since
only finish() should do that (see below) and LUO holds the session mutex
across both retrieve and finish.

The liveupdate_claimed check is redundant too: LUO only lets a token be
retrieved successfully once, PATCH 12 prevents opening an incoming
device any other way, and the liveupdate_incoming check already prevents
attaching twice.

> +	if (!pci_liveupdate_is_incoming(pdev) ||
> +	    ser->cfg_size != pdev->cfg_size) {
> +		ret = -EINVAL;
> +		goto out_put_dev;
> +	}

If the device isn't incoming here, that's a kernel bug. The previous
kernel only preserves the file after pci_liveupdate_preserve()
succeeds, the PCI core marks the device incoming when it's enumerated,
and only VFIO's finish() should clear it. So this should be a
WARN_ON_ONCE().

> +	if (pci_liveupdate_is_incoming(pdev))
> +		pci_liveupdate_finish(pdev);
> +}

Likewise, the device must still be incoming here, so
pci_liveupdate_finish() can be called unconditionally.

> +static inline void
> +vfio_pci_liveupdate_unregister_dev(struct vfio_pci_core_device *vdev)
> +{
> +	struct pci_dev *pdev = vdev->pdev;
> +
> +	scoped_guard(mutex, &vdev->vdev.dev_set->lock) {
> +		/*
> +		 * Complete a reset that finish() could not do, see
> +		 * vfio_pci_liveupdate_finish_vdev().
> +		 */
> +		if (vdev->liveupdate_reset_pending) {
> +			vfio_pci_core_liveupdate_reset_locked(vdev);
> +			vdev->liveupdate_reset_pending = false;
> +		}
> +		vfio_pci_liveupdate_free_incoming(vdev);
> +	}
> +
> +	if (pci_liveupdate_is_incoming(pdev))
> +		pci_liveupdate_finish(pdev);
> +}

VFIO should only call pci_liveupdate_finish() from its finish()
callback (see the kernel-doc of pci_liveupdate_finish()).

If vfio-pci is unbound from an incoming device before finish(), it
should just leave the device and its Live Update state alone. That's
fine: if vfio-pci is bound to the device again, the device is still
incoming and can still be retrieved. And once the device has been
retrieved, LUO holds a reference to the device file until finish(), so
vfio_unregister_group_dev() already blocks unbinding it like it does for
any other open device file.

If userspace binds a different driver to the device instead, that's
userspace's problem. The PCI core documentation already says it is the
user's responsibility to ensure that incoming preserved devices are
bound to the correct driver.

So I don't think the unbind path needs any Live Update handling at all.

>  #ifdef CONFIG_VFIO_PCI_LIVEUPDATE
>  	bool			liveupdate_frozen;
>  	bool			liveupdate_intx_disabled;
> +	bool			liveupdate_claimed;
> +	bool			liveupdate_reset_pending;
> +	struct vfio_pci_liveupdate_incoming *liveupdate_incoming;
>  #endif

With the simpler finish(), I don't think liveupdate_claimed or
liveupdate_reset_pending are needed.

PATCH 13 later inverts liveupdate_claimed into liveupdate_kept_running
and uses it to block dev_set resets while a preserved device is still
running. That can use pci_liveupdate_is_incoming() instead. It blocks
resets until finish() rather than until take-over, but since finish()
now requires take-over, userspace controls that window. If some flag is
still needed later, please introduce it once with its final name and
meaning instead of adding it here and inverting it in PATCH 13.

  parent reply	other threads:[~2026-11-20  1:01 UTC|newest]

Thread overview: 47+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-02 15:36 [PATCH v6 00/27] VFIO PCI Live Update (noiommu mode) Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 01/27] vfio/pci: Factor out the reset logic in VFIO PCI device close path Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 02/27] vfio: Export various helpers from VFIO Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 03/27] liveupdate: Export symbols needed by modules Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 04/27] vfio/pci: Register a file handler with Live Update Orchestrator Vipin Sharma
2026-11-06 22:29   ` Samiullah Khawaja
2026-11-10 19:02     ` Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 05/27] PCI: liveupdate: Add helpers to preserve struct pci_saved_state Vipin Sharma
2026-11-02 23:06   ` Bjorn Helgaas
2026-11-03  0:48     ` David Matlack
2026-11-04 22:55       ` Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 06/27] vfio: Add helpers to identify device files that can be preserved Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 07/27] vfio/pci: Preserve vfio-pci device files across Live Update Vipin Sharma
2026-11-05 19:12   ` David Matlack
2026-11-17 21:59     ` Vipin Sharma
2026-11-05 19:57   ` David Matlack
2026-11-18  6:44     ` Vipin Sharma
2026-11-20  0:06       ` David Matlack
2026-11-06 22:35   ` Samiullah Khawaja
2026-11-18  6:55     ` Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 08/27] vfio: Introduce vfio_find_device() helper Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 09/27] vfio: Refactor vfio_device_fops_cdev_open() Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 10/27] vfio: Add API to open cdev device for Live Update restore Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 11/27] vfio/pci: Retrieve preserved device files after Live Update Vipin Sharma
2026-11-06 17:19   ` David Matlack
2026-11-18  7:51     ` Vipin Sharma
2026-11-20  0:16       ` David Matlack
2026-11-20  1:01   ` David Matlack [this message]
2026-11-02 15:36 ` [PATCH v6 12/27] vfio: Enforce preserved devices are retrieved via LIVEUPDATE_SESSION_RETRIEVE_FD Vipin Sharma
2026-11-20  0:19   ` David Matlack
2026-11-02 15:36 ` [PATCH v6 13/27] vfio/pci: Protect incoming preserved devices until they finish Live Update Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 14/27] vfio/pci: Take over preserved devices without resetting them Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 15/27] docs: liveupdate: Add documentation for VFIO PCI Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 16/27] vfio: selftests: Build liveupdate library in VFIO selftests Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 17/27] vfio: selftests: Add vfio_pci_liveupdate_uapi_test Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 18/27] vfio: selftests: Initialize vfio_pci_device using a VFIO cdev FD Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 19/27] vfio: selftests: Add Makefile support for TEST_GEN_PROGS_EXTENDED Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 20/27] vfio: selftests: Add iommufd noiommu mode Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 21/27] vfio: selftests: Add helpers to map DMA memory at IOVA == PA in " Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 22/27] vfio: selftests: Add helpers for Live Update DMA memory and sysfs state Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 23/27] vfio: selftests: Add an attach() driver op Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 24/27] vfio: selftests: Test preserving noiommu devices in vfio_pci_liveupdate_uapi_test Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 25/27] vfio: selftests: Add vfio_pci_liveupdate_kexec_test Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 26/27] vfio: selftests: Verify that opening VFIO device fails during Live Update Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 27/27] vfio: selftests: Add continuous DMA to vfio_pci_liveupdate_kexec_test Vipin Sharma
2026-11-06 22:20 ` [PATCH v6 00/27] VFIO PCI Live Update (noiommu mode) David Matlack
2026-11-10 18:45   ` Vipin Sharma

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=DMZMRTXS7OSC.M79XF39ZVEWE@google.com \
    --to=dmatlack@google.com \
    --cc=ajayachandra@nvidia.com \
    --cc=alex@shazbot.org \
    --cc=amastro@fb.com \
    --cc=ankita@nvidia.com \
    --cc=apopple@nvidia.com \
    --cc=bhelgaas@google.com \
    --cc=chrisl@kernel.org \
    --cc=christian.koenig@amd.com \
    --cc=corbet@lwn.net \
    --cc=graf@amazon.com \
    --cc=jacob.pan@linux.microsoft.com \
    --cc=jgg@nvidia.com \
    --cc=jgg@ziepe.ca \
    --cc=jrhilke@google.com \
    --cc=julianr@linux.ibm.com \
    --cc=kees@kernel.org \
    --cc=kevin.tian@intel.com \
    --cc=kexec@lists.infradead.org \
    --cc=kvm@vger.kernel.org \
    --cc=leon@kernel.org \
    --cc=leonro@nvidia.com \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=lukas@wunner.de \
    --cc=mattev@meta.com \
    --cc=michal.winiarski@intel.com \
    --cc=parav@nvidia.com \
    --cc=pasha.tatashin@soleen.com \
    --cc=praan@google.com \
    --cc=pratyush@kernel.org \
    --cc=rananta@google.com \
    --cc=rdunlap@infradead.org \
    --cc=rientjes@google.com \
    --cc=rodrigo.vivi@intel.com \
    --cc=rppt@kernel.org \
    --cc=saeedm@nvidia.com \
    --cc=schnelle@linux.ibm.com \
    --cc=skhan@linuxfoundation.org \
    --cc=skhawaja@google.com \
    --cc=vipinsh@google.com \
    --cc=vivek.kasireddy@intel.com \
    --cc=witu@nvidia.com \
    --cc=yanjun.zhu@linux.dev \
    --cc=yi.l.liu@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox