Linux Kernel Selftest development
 help / color / mirror / Atom feed
From: "David Matlack" <dmatlack@google.com>
To: "Vipin Sharma" <vipinsh@google.com>,
	"David Matlack" <dmatlack@google.com>
Cc: <kexec@lists.infradead.org>, <linux-kernel@vger.kernel.org>,
	<linux-doc@vger.kernel.org>, <kvm@vger.kernel.org>,
	<linux-mm@kvack.org>, <linux-kselftest@vger.kernel.org>,
	<linux-pci@vger.kernel.org>, <ajayachandra@nvidia.com>,
	<alex@shazbot.org>, <amastro@fb.com>, <ankita@nvidia.com>,
	<apopple@nvidia.com>, <bhelgaas@google.com>, <chrisl@kernel.org>,
	<christian.koenig@amd.com>, <corbet@lwn.net>, <graf@amazon.com>,
	<jacob.pan@linux.microsoft.com>, <jgg@nvidia.com>, <jgg@ziepe.ca>,
	<jrhilke@google.com>, <julianr@linux.ibm.com>, <kees@kernel.org>,
	<kevin.tian@intel.com>, <leon@kernel.org>, <leonro@nvidia.com>,
	<lukas@wunner.de>, <mattev@meta.com>,
	<michal.winiarski@intel.com>, <parav@nvidia.com>,
	<pasha.tatashin@soleen.com>, <praan@google.com>,
	<pratyush@kernel.org>, <rananta@google.com>,
	<rdunlap@infradead.org>, <rientjes@google.com>,
	<rodrigo.vivi@intel.com>, <rppt@kernel.org>, <saeedm@nvidia.com>,
	<schnelle@linux.ibm.com>, <skhan@linuxfoundation.org>,
	<skhawaja@google.com>, <vivek.kasireddy@intel.com>,
	<witu@nvidia.com>, <yanjun.zhu@linux.dev>, <yi.l.liu@intel.com>
Subject: Re: [PATCH v6 07/27] vfio/pci: Preserve vfio-pci device files across Live Update
Date: Thu, 19 Nov 2026 16:06:38 -0800	[thread overview]
Message-ID: <DMZLLYOVIGOU.1RXWYJN6UQTM@google.com> (raw)
In-Reply-To: <20261117220059.GA2250529.vipinsh@google.com>

On Tue Nov 17, 2026 at 10:44 PM PST, Vipin Sharma wrote:
>> pci_config_map doesn't need to be preserved. PATCH 14 already rebuilds
>> it with vfio_config_init() in the next kernel and only uses the
>> preserved copy to check that the two are identical.
>> 
>> I assume the intent is to detect changes to vfio-pci's config emulation
>> between the two kernels, but the map isn't a good proxy for that. It
>> changes for differences that don't matter (e.g. hiding a capability or
>> learning the length of a new extended capability), which would fail
>> take-over and lose the Live Update on exactly the kernel upgrades Live
>> Update is meant for. And it doesn't change for differences that do
>> matter, like which bits are virtualized or writable within a capability.
>
> I think it does matter because reads and writes of unitialized
> capabilities in VFIO PCI config space goes to hardware directly. So, in
> the new kernel if pci_config_map changes it from PCI_CAP_ID_INVALID to
> others then those reads/writes can go to vconfig which will contain
> stale unitialized data from the old kernel's heap.
>
> This also changes behavior under a running user. Writes that reached
> hardware before Live Update can be silently dropped or redirected to
> vconfig. See the comment above struct perm_bits unassigned_perms{}
> in vfio_pci_config.c

That's true. Unassigned bytes passed through and vconfig is never
initialized for them, so hiding or exposing a capability does matter.
For example, 6467d0740a2b ("vfio/pci: Add DVSEC PCI Extended Config
Capability to user visible list") changed DVSEC from hidden (all bytes
raw) to exposed with a virtualized header. Across that boundary, v6
would have served DVSEC from uninitialized bytes in vconfig.

> How would a userspace opt-in? New kernel can change a capability visiblity
> or virt and write permission of the bits inside a capability. Kernel
> cannot know if userspace used the capability or not. Creating a VFIO
> device in the new kernel by default results in a opt-in and currently
> there is no way for userspace to opt-out unless we add a new
> functionality where userspace can disable/enable features something
> similar to KVM capabilities.
>
> For now, I think we don't need to worry about it. As this version will
> be a base and if any serialized data with flags set received in this
> kernel will lead to erroring out restore.

We don't need an opt-in mechanism in this series. Detecting mismatches
is enough to make Live Update safe in both directions: if the next
kernel would emulate the device in an incompatible way, retrieve should
fail.

But an opt-in is needed to make rollback possible once vfio-pci starts
changing its emulation. I imagine something like a VFIO_DEVICE_FEATURE
that userspace can use to enable new emulation behavior right after
opening the device, with the enabled bits recorded in a field appended
to struct vfio_pci_core_device_ser behind a required flag. Devices that
don't opt in keep the baseline emulation and stay compatible with older
kernels. But that can come in a follow-on series.

> In the next version, I will not preserve pci_config_map. Hopefully,
> anyone changing VFIO code doesn't forget to update flags if they change
> capability or permission bits (I will write comments at appropriate
> places.)
>
> If we want to be very careful, we can do two things in this series:
>
> 1. Create a per bit valid mask for the vconfig and compare this between
>    new and old kernel during restore. In the new kernel, VFIO then takes
>    each bit from the preserved copy only where the mask says it was
>    valid, and keeps its own freshly initialized value elsewhere.
> 2. Create a hash or store virt and writable bits in config space.
>    Compare it with the new kernel.
>
> Both of the above combined will safeguard against forgotten flag update
> in the code.
>
> We don't have to do it but it safeguards against mistakes. Let me know
> what you prefer.

I think we should do both in this series, as two per-bit masks over
config space, preserved in place of pci_config_map:

 - virt[]: the bits whose value vfio-pci keeps in vconfig and returns
   instead of the device's value. This says which bits of the preserved
   vconfig are valid.

 - write[]: the bits whose writes take effect. For bits in virt they
   update vconfig, and for other bits they are forwarded to the device
   (possibly after validation). Writes to other bits are ignored.

The masks should describe what vfio-pci actually does rather than copy
perm->virt and perm->write.

vfio-pci can keep changing its implementation in ways that don't break
Live Update, e.g. start intercepting writes to a register to sanity
check them before forwarding them to the device.

Something like:

  struct vfio_pci_core_device_ser {
          ...
          u8 virt[PCI_CFG_SPACE_EXP_SIZE];
          u8 write[PCI_CFG_SPACE_EXP_SIZE];
          u8 vconfig[PCI_CFG_SPACE_EXP_SIZE];
  } __packed;


[ Note: I'd probably store virt[], write[], and vconfig[] on their
  own folios to avoid multi-order folio allocations, but I am embedding
  them here for simplicity. ]

The outgoing kernel computes virt[] and write[] through a helper in
vfio_pci_config.c and saves vconfig[] during freeze().

The incoming kernel computes its own virt[] and write[] masks and
requires both to match exactly with what's in vfio_pci_core_device_ser,
otherwise it fails the retrieve. On take-over it uses the preserved
value for bits in virt and its own freshly initialized value for the
rest, like your option 1.

I'd store the full masks rather than a hash, so that a mismatch can be
pinpointed.

  reply	other threads:[~2026-11-20  0:06 UTC|newest]

Thread overview: 47+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-02 15:36 [PATCH v6 00/27] VFIO PCI Live Update (noiommu mode) Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 01/27] vfio/pci: Factor out the reset logic in VFIO PCI device close path Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 02/27] vfio: Export various helpers from VFIO Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 03/27] liveupdate: Export symbols needed by modules Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 04/27] vfio/pci: Register a file handler with Live Update Orchestrator Vipin Sharma
2026-11-06 22:29   ` Samiullah Khawaja
2026-11-10 19:02     ` Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 05/27] PCI: liveupdate: Add helpers to preserve struct pci_saved_state Vipin Sharma
2026-11-02 23:06   ` Bjorn Helgaas
2026-11-03  0:48     ` David Matlack
2026-11-04 22:55       ` Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 06/27] vfio: Add helpers to identify device files that can be preserved Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 07/27] vfio/pci: Preserve vfio-pci device files across Live Update Vipin Sharma
2026-11-05 19:12   ` David Matlack
2026-11-17 21:59     ` Vipin Sharma
2026-11-05 19:57   ` David Matlack
2026-11-18  6:44     ` Vipin Sharma
2026-11-20  0:06       ` David Matlack [this message]
2026-11-06 22:35   ` Samiullah Khawaja
2026-11-18  6:55     ` Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 08/27] vfio: Introduce vfio_find_device() helper Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 09/27] vfio: Refactor vfio_device_fops_cdev_open() Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 10/27] vfio: Add API to open cdev device for Live Update restore Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 11/27] vfio/pci: Retrieve preserved device files after Live Update Vipin Sharma
2026-11-06 17:19   ` David Matlack
2026-11-18  7:51     ` Vipin Sharma
2026-11-20  0:16       ` David Matlack
2026-11-20  1:01   ` David Matlack
2026-11-02 15:36 ` [PATCH v6 12/27] vfio: Enforce preserved devices are retrieved via LIVEUPDATE_SESSION_RETRIEVE_FD Vipin Sharma
2026-11-20  0:19   ` David Matlack
2026-11-02 15:36 ` [PATCH v6 13/27] vfio/pci: Protect incoming preserved devices until they finish Live Update Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 14/27] vfio/pci: Take over preserved devices without resetting them Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 15/27] docs: liveupdate: Add documentation for VFIO PCI Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 16/27] vfio: selftests: Build liveupdate library in VFIO selftests Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 17/27] vfio: selftests: Add vfio_pci_liveupdate_uapi_test Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 18/27] vfio: selftests: Initialize vfio_pci_device using a VFIO cdev FD Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 19/27] vfio: selftests: Add Makefile support for TEST_GEN_PROGS_EXTENDED Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 20/27] vfio: selftests: Add iommufd noiommu mode Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 21/27] vfio: selftests: Add helpers to map DMA memory at IOVA == PA in " Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 22/27] vfio: selftests: Add helpers for Live Update DMA memory and sysfs state Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 23/27] vfio: selftests: Add an attach() driver op Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 24/27] vfio: selftests: Test preserving noiommu devices in vfio_pci_liveupdate_uapi_test Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 25/27] vfio: selftests: Add vfio_pci_liveupdate_kexec_test Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 26/27] vfio: selftests: Verify that opening VFIO device fails during Live Update Vipin Sharma
2026-11-02 15:36 ` [PATCH v6 27/27] vfio: selftests: Add continuous DMA to vfio_pci_liveupdate_kexec_test Vipin Sharma
2026-11-06 22:20 ` [PATCH v6 00/27] VFIO PCI Live Update (noiommu mode) David Matlack
2026-11-10 18:45   ` Vipin Sharma

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=DMZLLYOVIGOU.1RXWYJN6UQTM@google.com \
    --to=dmatlack@google.com \
    --cc=ajayachandra@nvidia.com \
    --cc=alex@shazbot.org \
    --cc=amastro@fb.com \
    --cc=ankita@nvidia.com \
    --cc=apopple@nvidia.com \
    --cc=bhelgaas@google.com \
    --cc=chrisl@kernel.org \
    --cc=christian.koenig@amd.com \
    --cc=corbet@lwn.net \
    --cc=graf@amazon.com \
    --cc=jacob.pan@linux.microsoft.com \
    --cc=jgg@nvidia.com \
    --cc=jgg@ziepe.ca \
    --cc=jrhilke@google.com \
    --cc=julianr@linux.ibm.com \
    --cc=kees@kernel.org \
    --cc=kevin.tian@intel.com \
    --cc=kexec@lists.infradead.org \
    --cc=kvm@vger.kernel.org \
    --cc=leon@kernel.org \
    --cc=leonro@nvidia.com \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=lukas@wunner.de \
    --cc=mattev@meta.com \
    --cc=michal.winiarski@intel.com \
    --cc=parav@nvidia.com \
    --cc=pasha.tatashin@soleen.com \
    --cc=praan@google.com \
    --cc=pratyush@kernel.org \
    --cc=rananta@google.com \
    --cc=rdunlap@infradead.org \
    --cc=rientjes@google.com \
    --cc=rodrigo.vivi@intel.com \
    --cc=rppt@kernel.org \
    --cc=saeedm@nvidia.com \
    --cc=schnelle@linux.ibm.com \
    --cc=skhan@linuxfoundation.org \
    --cc=skhawaja@google.com \
    --cc=vipinsh@google.com \
    --cc=vivek.kasireddy@intel.com \
    --cc=witu@nvidia.com \
    --cc=yanjun.zhu@linux.dev \
    --cc=yi.l.liu@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox