From: David Matlack <dmatlack@google.com>
To: kexec@lists.infradead.org, linux-doc@vger.kernel.org,
linux-kernel@vger.kernel.org, linux-mm@kvack.org,
linux-pci@vger.kernel.org
Cc: Adithya Jayachandran <ajayachandra@nvidia.com>,
Alexander Graf <graf@amazon.com>,
Alex Williamson <alex@shazbot.org>,
Alex Williamson <alex.williamson@nvidia.com>,
Bjorn Helgaas <bhelgaas@google.com>,
Chris Down <chris@chrisdown.name>, Chris Li <chrisl@kernel.org>,
David Matlack <dmatlack@google.com>,
David Rientjes <rientjes@google.com>,
Jacob Pan <jacob.pan@linux.microsoft.com>,
Jason Gunthorpe <jgg@nvidia.com>,
Jonathan Corbet <corbet@lwn.net>,
Josh Hilke <jrhilke@google.com>,
Leon Romanovsky <leonro@nvidia.com>,
Lukas Wunner <lukas@wunner.de>, Mike Rapoport <rppt@kernel.org>,
Parav Pandit <parav@nvidia.com>,
Pasha Tatashin <pasha.tatashin@soleen.com>,
Pranjal Shrivastava <praan@google.com>,
Pratyush Yadav <pratyush@kernel.org>,
Randy Dunlap <rdunlap@infradead.org>,
Saeed Mahameed <saeedm@nvidia.com>,
Samiullah Khawaja <skhawaja@google.com>,
Shuah Khan <skhan@linuxfoundation.org>,
Vipin Sharma <vipinsh@google.com>, William Tu <witu@nvidia.com>,
Yi Liu <yi.l.liu@intel.com>
Subject: [PATCH v11 00/13] PCI: liveupdate: PCI core support for Live Update
Date: Wed, 18 Nov 2026 22:36:51 +0000 [thread overview]
Message-ID: <20261118223704.3286085-1-dmatlack@google.com> (raw)
This series can be found on GitHub:
https://github.com/dmatlack/linux/tree/liveupdate/pci/base/v11
This series introduces support in the PCI core for Live Update, enabling
drivers to preserve PCI devices across a kexec-based kernel update
without interrupting the device. This lets devices, such as ones
assigned to VMs through VFIO, keep running across a host kernel upgrade.
Preserved devices can keep doing DMA to system memory throughout the
Live Update. Devices may be behind bridges, but VFs, devices that need
device-specific ACS quirks, and CardBus are not supported. Support for
P2P and VF preservation will be addressed in future series.
Series Overview
---------------
This series implements the following to support PCI device preservation
across Live Update:
1. Set up a File-Lifecycle-Bound (FLB) handler to track and preserve
PCI-specific state (struct pci_ser) across Live Update using Kexec
Handover (KHO).
2. Add APIs for drivers to register devices for preservation
(outgoing) and for the PCI core to identify preserved devices
during enumeration (incoming).
3. Automatically preserve all upstream bridges for any preserved
endpoint. Use reference counting to ensure bridges remain preserved
as long as any downstream device is preserved.
4. Keep the Requester ID (bus, device, function) of preserved devices
stable for the duration of their preservation by adopting the
secondary bus numbers, subordinate bus numbers, and ARI Forwarding
Enable of preserved bridges. After a Live Update, pci=assign-busses
and pci=noari do not override this adopted state.
5. Save and restore the ACS Control register and Egress Control Vector
across resets for all devices (patch 8), so that adopted ACS
controls survive a reset.
6. Keep memory transactions to/from preserved devices routed
consistently by adopting Access Control Services (ACS) controls
across a Live Update, from the endpoint up to the root port.
7. Freeze each device's preservation status when it is shut down, and
avoid disabling bus mastering on preserved devices in the PCI
shutdown path, so preserved devices can keep doing DMA during
kexec.
8. Add Documentation/PCI/liveupdate.rst, built from the kernel-doc in
drivers/pci/liveupdate.c and include/linux/kho/abi/pci.h, describing
the FLB API, device tracking, and the division of responsibilities
between the PCI core, drivers, and userspace.
Merge Plan
----------
This series is intended to go through liveupdate.git, the tree listed in
the new PCI LIVE UPDATE MAINTAINERS entry.
v10 and a follow-up patch were applied to the lu-pci-preservation branch
of liveupdate.git. This version is meant to replace them, and applies on
v7.4-rc1, the base of that branch.
Testing
-------
VFIO
~~~~
This series was tested with v5 of the vfio-pci Live Update series,
rebased onto this series and the luo-internal-api branch of
liveupdate.git (which the VFIO series depends on):
https://lore.kernel.org/kvm/20260714151505.3466855-1-vipinsh@google.com/
The full set of patches used for testing can be found on GitHub.
https://github.com/dmatlack/linux/tree/liveupdate/pci/base/v11-with-vfio
Testing was performed using the VFIO selftests, including the new Live
Update selftests:
- vfio_pci_liveupdate_uapi_test
- vfio_pci_liveupdate_kexec_test
The tests were run in a QEMU-based VM environment (using an emulated
Intel IGB device connected to a PCIe Root Port to exercise bridge
support) and on bare metal using an Intel Emerald Rapids server with 8x
Intel DSA PCIe devices and 1x NVMe device, all of which were preserved
across a Live Update by vfio_pci_liveupdate_kexec_test.
PCI Selftests
~~~~~~~~~~~~~
This series was also tested with a new set of PCI Live Update
selftests:
https://github.com/dmatlack/linux/tree/liveupdate/pci/base/v11-with-selftests
Each test preserves a PCI device across a Live Update in a QEMU VM and
checks that the PCI core keeps the device, and every bridge above it,
running: same BDF and bus numbers, Bus Master Enable left set, ARI
Forwarding Enable and ACS controls adopted from the previous kernel, and
no PCI config writes that disrupt the device during the Live Update. The
tests cover endpoints on the root bus, below a Root Port, below a
switch, and below two levels of switches, as well as booting the next
kernel with command lines that would otherwise renumber buses or change
ARI and ACS. All tests pass on x86_64 (q35) and arm64 (virt), except
for the pci=assign-busses test, which is skipped on arm64 (the option is
x86-only).
Future Work
-----------
Please see the PCI core Live Update Roadmap:
https://lore.kernel.org/linux-pci/20261001232133.560284-1-dmatlack@google.com/
Changelog
---------
v11:
- Rebase onto v7.4-rc1, the base of lu-pci-preservation in
liveupdate.git
- Fold in "PCI: liveupdate: Factor out pci_dev_ser allocation and
release", which was posted and applied as a follow-up to v10 [1]:
introduce the struct pci_dev_ser allocation and release helpers in
"Track outgoing preserved PCI devices", and split
pci_liveupdate_preserve_device() into separate paths for new and
already-preserved devices in "Auto-preserve upstream bridges across
Live Update" (suggested by Bjorn; the follow-up was reviewed and
tested by Chris Down)
- Don't allocate an ACS save buffer for devices that rely on
device-specific ACS quirks (Alex)
- Adopt the ACS controls of preserved devices even if they match a
device-specific ACS quirk, logging a warning instead of programming
ACS from scratch, and panic if a preserved device has no ACS save
buffer (Alex)
- Clean up commit messages, comments and documentation: fix
inaccuracies, remove filler and duplicated text, document that
pci=noari does not override adopted ARI Forwarding Enable, and note
in the documentation that devices cannot be preserved or unpreserved
once they are shut down
- Panic if any outgoing device was never unpreserved when the outgoing
FLB is freed, instead of leaving dangling dev->liveupdate.outgoing
pointers, and share the check with the incoming side (Sashiko)
[1] https://lore.kernel.org/linux-pci/20261109174009.735309-1-dmatlack@google.com/
v10: https://lore.kernel.org/linux-pci/20261013162529.1928093-1-dmatlack@google.com/
v9: https://lore.kernel.org/linux-pci/20260918200640.887030-1-dmatlack@google.com/
v8: https://lore.kernel.org/linux-pci/20260728221007.2098560-1-dmatlack@google.com/
v7: https://lore.kernel.org/linux-pci/20260710212616.1351130-1-dmatlack@google.com/
v6: https://lore.kernel.org/linux-pci/20260522202410.3104264-1-dmatlack@google.com/
v5: https://lore.kernel.org/linux-pci/20260512184846.119396-1-dmatlack@google.com/
v4: https://lore.kernel.org/linux-pci/20260423212316.3431746-1-dmatlack@google.com/
v3: https://lore.kernel.org/kvm/20260323235817.1960573-1-dmatlack@google.com/
v2: https://lore.kernel.org/kvm/20260129212510.967611-1-dmatlack@google.com/
v1: https://lore.kernel.org/kvm/20251126193608.2678510-1-dmatlack@google.com/
rfc: https://lore.kernel.org/kvm/20251018000713.677779-1-vipinsh@google.com/
David Matlack (13):
PCI: liveupdate: Set up FLB handler for the PCI core
PCI: liveupdate: Track outgoing preserved PCI devices
PCI: liveupdate: Track incoming preserved PCI devices
PCI: liveupdate: Document driver binding responsibilities
PCI: liveupdate: Auto-preserve upstream bridges across Live Update
PCI: liveupdate: Preserve bus numbers during Live Update
PCI: Refactor matching logic for pci_dev_acs_ops
PCI: Save and restore the ACS Control register and Egress Control
Vector
PCI: liveupdate: Adopt ACS controls in incoming preserved devices
PCI: liveupdate: Adopt ARI Forwarding Enable on preserved bridges
PCI: liveupdate: Freeze preservation status during shutdown
PCI: liveupdate: Do not disable bus mastering on preserved devices
during kexec
Documentation: PCI: Add documentation for Live Update
Documentation/PCI/index.rst | 1 +
Documentation/PCI/liveupdate.rst | 35 +
.../admin-guide/kernel-parameters.txt | 11 +-
Documentation/core-api/liveupdate.rst | 5 +
MAINTAINERS | 15 +
drivers/pci/Kconfig | 14 +
drivers/pci/Makefile | 1 +
drivers/pci/liveupdate.c | 1011 +++++++++++++++++
drivers/pci/liveupdate.h | 72 ++
drivers/pci/pci-driver.c | 8 +-
drivers/pci/pci.c | 169 ++-
drivers/pci/pci.h | 5 +
drivers/pci/probe.c | 16 +-
drivers/pci/quirks.c | 58 +-
include/linux/kho/abi/pci.h | 96 ++
include/linux/pci.h | 7 +
include/linux/pci_liveupdate.h | 72 ++
17 files changed, 1556 insertions(+), 40 deletions(-)
create mode 100644 Documentation/PCI/liveupdate.rst
create mode 100644 drivers/pci/liveupdate.c
create mode 100644 drivers/pci/liveupdate.h
create mode 100644 include/linux/kho/abi/pci.h
create mode 100644 include/linux/pci_liveupdate.h
base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
--
2.56.1.703.gf848ed807c-goog
next reply other threads:[~2026-11-18 22:37 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 22:36 David Matlack [this message]
2026-11-18 22:36 ` [PATCH v11 01/13] PCI: liveupdate: Set up FLB handler for the PCI core David Matlack
2026-11-18 22:36 ` [PATCH v11 02/13] PCI: liveupdate: Track outgoing preserved PCI devices David Matlack
2026-11-18 22:36 ` [PATCH v11 03/13] PCI: liveupdate: Track incoming " David Matlack
2026-11-18 22:36 ` [PATCH v11 04/13] PCI: liveupdate: Document driver binding responsibilities David Matlack
2026-11-18 22:36 ` [PATCH v11 05/13] PCI: liveupdate: Auto-preserve upstream bridges across Live Update David Matlack
2026-11-18 22:36 ` [PATCH v11 06/13] PCI: liveupdate: Preserve bus numbers during " David Matlack
2026-11-18 22:36 ` [PATCH v11 07/13] PCI: Refactor matching logic for pci_dev_acs_ops David Matlack
2026-11-18 22:36 ` [PATCH v11 08/13] PCI: Save and restore the ACS Control register and Egress Control Vector David Matlack
2026-11-18 22:37 ` [PATCH v11 09/13] PCI: liveupdate: Adopt ACS controls in incoming preserved devices David Matlack
2026-11-18 22:37 ` [PATCH v11 10/13] PCI: liveupdate: Adopt ARI Forwarding Enable on preserved bridges David Matlack
2026-11-18 22:37 ` [PATCH v11 11/13] PCI: liveupdate: Freeze preservation status during shutdown David Matlack
2026-11-18 22:37 ` [PATCH v11 13/13] Documentation: PCI: Add documentation for Live Update David Matlack
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261118223704.3286085-1-dmatlack@google.com \
--to=dmatlack@google.com \
--cc=ajayachandra@nvidia.com \
--cc=alex.williamson@nvidia.com \
--cc=alex@shazbot.org \
--cc=bhelgaas@google.com \
--cc=chris@chrisdown.name \
--cc=chrisl@kernel.org \
--cc=corbet@lwn.net \
--cc=graf@amazon.com \
--cc=jacob.pan@linux.microsoft.com \
--cc=jgg@nvidia.com \
--cc=jrhilke@google.com \
--cc=kexec@lists.infradead.org \
--cc=leonro@nvidia.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-pci@vger.kernel.org \
--cc=lukas@wunner.de \
--cc=parav@nvidia.com \
--cc=pasha.tatashin@soleen.com \
--cc=praan@google.com \
--cc=pratyush@kernel.org \
--cc=rdunlap@infradead.org \
--cc=rientjes@google.com \
--cc=rppt@kernel.org \
--cc=saeedm@nvidia.com \
--cc=skhan@linuxfoundation.org \
--cc=skhawaja@google.com \
--cc=vipinsh@google.com \
--cc=witu@nvidia.com \
--cc=yi.l.liu@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox