From: Vipin Sharma <vipinsh@google.com>
To: David Matlack <dmatlack@google.com>
Cc: kexec@lists.infradead.org, linux-kernel@vger.kernel.org,
linux-doc@vger.kernel.org, kvm@vger.kernel.org,
linux-mm@kvack.org, linux-kselftest@vger.kernel.org,
linux-pci@vger.kernel.org, ajayachandra@nvidia.com,
alex@shazbot.org, amastro@fb.com, ankita@nvidia.com,
apopple@nvidia.com, bhelgaas@google.com, chrisl@kernel.org,
christian.koenig@amd.com, corbet@lwn.net, graf@amazon.com,
jacob.pan@linux.microsoft.com, jgg@nvidia.com, jgg@ziepe.ca,
jrhilke@google.com, julianr@linux.ibm.com, kees@kernel.org,
kevin.tian@intel.com, leon@kernel.org, leonro@nvidia.com,
lukas@wunner.de, mattev@meta.com, michal.winiarski@intel.com,
parav@nvidia.com, pasha.tatashin@soleen.com, praan@google.com,
pratyush@kernel.org, rananta@google.com, rdunlap@infradead.org,
rientjes@google.com, rodrigo.vivi@intel.com, rppt@kernel.org,
saeedm@nvidia.com, schnelle@linux.ibm.com,
skhan@linuxfoundation.org, skhawaja@google.com,
vivek.kasireddy@intel.com, witu@nvidia.com,
yanjun.zhu@linux.dev, yi.l.liu@intel.com
Subject: Re: [PATCH v6 11/27] vfio/pci: Retrieve preserved device files after Live Update
Date: Tue, 17 Nov 2026 23:51:26 -0800 [thread overview]
Message-ID: <20261118065653.GE2250529.vipinsh@google.com> (raw)
In-Reply-To: <DMOAT4VV939P.2DI8CEOMMSKMY@google.com>
On Fri, Nov 06, 2026 at 05:19:28PM +0000, David Matlack wrote:
> On Mon Nov 2, 2026 at 3:36 PM UTC, Vipin Sharma wrote:
> > During retrieve():
> > * Validate the handed-over struct vfio_pci_core_device_ser header,
> > sizes, and flags.
> > * Locate the registered vfio-pci device matching the serialized domain
> > and BDF (requiring device->ops == &vfio_pci_ops) under the device
> > lock. vfio-pci registers the vfio device before its probe completes,
> > and the probe holds that lock, so a device that is still being probed
> > is not returned.
> [...]
> > * Open and return an anonymous VFIO cdev file via
> > vfio_device_liveupdate_cdev_open().
>
> Here's an idea for how the device file could be created on retrieve,
> which I'd like to hear your thoughts on.
>
> The patches from Pasha at [1] lets userspace pass a
> file-handler-specific argument to LIVEUPDATE_SESSION_RETRIEVE_FD. In the
> next version of the tmpfs series, I'm planning to use that to have
> userspace create the file and pass it in the argument, and the kernel
> then restores the preserved file into it.
>
> I wonder if VFIO should work the same way, e.g.:
>
> struct liveupdate_vfio_retrieve_arg {
> __u32 size;
> __s32 fd;
> };
>
> Userspace would open /dev/vfio/devices/noiommu-vfioX itself and pass
> the fd. retrieve() would check that it's an unbound vfio-pci cdev file
> for the preserved device (domain/BDF, noiommu, incoming, cfg_size),
> attach the incoming state to that file, and LUO would track that file
> as the retrieved file. VFIO_DEVICE_BIND_IOMMUFD would then take over the
> device as it does now. Since VFIO preservation is new, the argument
> could simply be required.
>
> I think this could get rid of some of the special handling:
>
> - There'd be no need to fake a cdev open with an anonymous inode, so
> PATCH 09 and PATCH 10 could probably be dropped.
>
> - retrieve() wouldn't need to look up the device by BDF, or reason
> about racing with probe under the device lock. If userspace was able
> to open the cdev, the device is registered and probed.
>
> - The device file would be opened through the normal cdev path, so the
> usual permission and LSM checks on the device node would apply.
>
> - It would be the same model across file types.
>
> A few things I think would need care if you go this way. The file check
> and the attach would need to happen under dev_set->lock so they can't
> race with a concurrent VFIO_DEVICE_BIND_IOMMUFD on the same fd. And the
> incoming state should probably be tied to that specific
> vfio_device_file rather than just the vdev, so only the file passed to
> RETRIEVE_FD can take over the device.
>
> It would also change PATCH 12. Opening the cdev of an incoming device
> would have to succeed, so the check would move to the first open of the
> device instead: fail with -EBUSY if the device is incoming and the file
> being bound doesn't have the incoming state attached. That one check in
> vfio-pci would also cover VFIO_GROUP_GET_DEVICE_FD, which opens the
> device right away, so vfio_liveupdate_incoming_is_preserved() and its
> dev_is_pci() special case could be dropped from the VFIO core.
>
> The next tmpfs version will also add a way for a file handler to say a
> failed retrieve can be retried, and a helper to claim an existing file
> before restoring into it (so a file LUO already manages, e.g. one that
> is preserved in an outgoing session, is rejected). Both seem useful
> here too, e.g. so that a mistake in the argument (wrong fd, an fd for
> another device, an already bound fd) doesn't make the device
> unretrievable. I'll Cc you when I send it, and I'm happy to share the
> branch before then.
>
> What do you think?
Overall, I agree with the strategy. I think VFIO cdev retrieval should
happen as the flow you mentioned above. I want to hightlight two things:
1. This will lead to lot of code changes happening in vfio core. This
means VFIO PCI live update code change will spill in vfio core.
Should I create a config VFIO_LIVEUPDATE for VFIO core code. It will
not be visible menuconfig option but selected when
VFIO_PCI_LIVEUPDATE is enabled.
2. List of other inflight series that this series will depend on is
increasing. It will be 3 (reset function, pci saved state, and
restore into fd).
For '2', should I send v7 without changes you are recommending in this
patch and future version of this series can transition to the approach
you are suggesting after LUO changes are landed. VFIO liveupdate series
will go through more iterations, so we will have opportunity to make the
change. WDYT?
>
> [1] https://lore.kernel.org/kexec/20261103010216.3447716-1-dmatlack@google.com/
next prev parent reply other threads:[~2026-11-18 7:51 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-02 15:36 [PATCH v6 00/27] VFIO PCI Live Update (noiommu mode) Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 01/27] vfio/pci: Factor out the reset logic in VFIO PCI device close path Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 02/27] vfio: Export various helpers from VFIO Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 03/27] liveupdate: Export symbols needed by modules Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 04/27] vfio/pci: Register a file handler with Live Update Orchestrator Vipin Sharma
2026-11-06 22:29 ` Samiullah Khawaja
2026-11-10 19:02 ` Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 05/27] PCI: liveupdate: Add helpers to preserve struct pci_saved_state Vipin Sharma
2026-11-02 23:06 ` Bjorn Helgaas
2026-11-03 0:48 ` David Matlack
2026-11-04 22:55 ` Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 06/27] vfio: Add helpers to identify device files that can be preserved Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 07/27] vfio/pci: Preserve vfio-pci device files across Live Update Vipin Sharma
2026-11-05 19:12 ` David Matlack
2026-11-17 21:59 ` Vipin Sharma
2026-11-05 19:57 ` David Matlack
2026-11-18 6:44 ` Vipin Sharma
2026-11-06 22:35 ` Samiullah Khawaja
2026-11-18 6:55 ` Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 08/27] vfio: Introduce vfio_find_device() helper Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 09/27] vfio: Refactor vfio_device_fops_cdev_open() Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 10/27] vfio: Add API to open cdev device for Live Update restore Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 11/27] vfio/pci: Retrieve preserved device files after Live Update Vipin Sharma
2026-11-06 17:19 ` David Matlack
2026-11-18 7:51 ` Vipin Sharma [this message]
2026-11-02 15:36 ` [PATCH v6 12/27] vfio: Enforce preserved devices are retrieved via LIVEUPDATE_SESSION_RETRIEVE_FD Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 13/27] vfio/pci: Protect incoming preserved devices until they finish Live Update Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 14/27] vfio/pci: Take over preserved devices without resetting them Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 15/27] docs: liveupdate: Add documentation for VFIO PCI Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 16/27] vfio: selftests: Build liveupdate library in VFIO selftests Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 17/27] vfio: selftests: Add vfio_pci_liveupdate_uapi_test Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 18/27] vfio: selftests: Initialize vfio_pci_device using a VFIO cdev FD Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 19/27] vfio: selftests: Add Makefile support for TEST_GEN_PROGS_EXTENDED Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 20/27] vfio: selftests: Add iommufd noiommu mode Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 21/27] vfio: selftests: Add helpers to map DMA memory at IOVA == PA in " Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 22/27] vfio: selftests: Add helpers for Live Update DMA memory and sysfs state Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 23/27] vfio: selftests: Add an attach() driver op Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 24/27] vfio: selftests: Test preserving noiommu devices in vfio_pci_liveupdate_uapi_test Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 25/27] vfio: selftests: Add vfio_pci_liveupdate_kexec_test Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 26/27] vfio: selftests: Verify that opening VFIO device fails during Live Update Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 27/27] vfio: selftests: Add continuous DMA to vfio_pci_liveupdate_kexec_test Vipin Sharma
2026-11-06 22:20 ` [PATCH v6 00/27] VFIO PCI Live Update (noiommu mode) David Matlack
2026-11-10 18:45 ` Vipin Sharma
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261118065653.GE2250529.vipinsh@google.com \
--to=vipinsh@google.com \
--cc=ajayachandra@nvidia.com \
--cc=alex@shazbot.org \
--cc=amastro@fb.com \
--cc=ankita@nvidia.com \
--cc=apopple@nvidia.com \
--cc=bhelgaas@google.com \
--cc=chrisl@kernel.org \
--cc=christian.koenig@amd.com \
--cc=corbet@lwn.net \
--cc=dmatlack@google.com \
--cc=graf@amazon.com \
--cc=jacob.pan@linux.microsoft.com \
--cc=jgg@nvidia.com \
--cc=jgg@ziepe.ca \
--cc=jrhilke@google.com \
--cc=julianr@linux.ibm.com \
--cc=kees@kernel.org \
--cc=kevin.tian@intel.com \
--cc=kexec@lists.infradead.org \
--cc=kvm@vger.kernel.org \
--cc=leon@kernel.org \
--cc=leonro@nvidia.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-pci@vger.kernel.org \
--cc=lukas@wunner.de \
--cc=mattev@meta.com \
--cc=michal.winiarski@intel.com \
--cc=parav@nvidia.com \
--cc=pasha.tatashin@soleen.com \
--cc=praan@google.com \
--cc=pratyush@kernel.org \
--cc=rananta@google.com \
--cc=rdunlap@infradead.org \
--cc=rientjes@google.com \
--cc=rodrigo.vivi@intel.com \
--cc=rppt@kernel.org \
--cc=saeedm@nvidia.com \
--cc=schnelle@linux.ibm.com \
--cc=skhan@linuxfoundation.org \
--cc=skhawaja@google.com \
--cc=vivek.kasireddy@intel.com \
--cc=witu@nvidia.com \
--cc=yanjun.zhu@linux.dev \
--cc=yi.l.liu@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox