Linux-mm Archive on lore.kernel.org
 help / color / mirror / Atom feed
From: "David Matlack" <dmatlack@google.com>
To: "Alex Williamson" <alex@shazbot.org>,
	"David Matlack" <dmatlack@google.com>
Cc: <kexec@lists.infradead.org>, <linux-doc@vger.kernel.org>,
	<linux-kernel@vger.kernel.org>, <linux-mm@kvack.org>,
	<linux-pci@vger.kernel.org>,
	"Adithya Jayachandran" <ajayachandra@nvidia.com>,
	"Alexander Graf" <graf@amazon.com>,
	"Bjorn Helgaas" <bhelgaas@google.com>,
	"Chris Li" <chrisl@kernel.org>,
	"David Rientjes" <rientjes@google.com>,
	"Jacob Pan" <jacob.pan@linux.microsoft.com>,
	"Jason Gunthorpe" <jgg@nvidia.com>,
	"Jonathan Corbet" <corbet@lwn.net>,
	"Josh Hilke" <jrhilke@google.com>,
	"Leon Romanovsky" <leonro@nvidia.com>,
	"Lukas Wunner" <lukas@wunner.de>,
	"Mike Rapoport" <rppt@kernel.org>,
	"Parav Pandit" <parav@nvidia.com>,
	"Pasha Tatashin" <pasha.tatashin@soleen.com>,
	"Pranjal Shrivastava" <praan@google.com>,
	"Pratyush Yadav" <pratyush@kernel.org>,
	"Randy Dunlap" <rdunlap@infradead.org>,
	"Saeed Mahameed" <saeedm@nvidia.com>,
	"Samiullah Khawaja" <skhawaja@google.com>,
	"Shuah Khan" <skhan@linuxfoundation.org>,
	"Vipin Sharma" <vipinsh@google.com>,
	"William Tu" <witu@nvidia.com>, "Yi Liu" <yi.l.liu@intel.com>
Subject: Re: [PATCH v10 09/13] PCI: liveupdate: Adopt ACS controls in incoming preserved devices
Date: Tue, 17 Nov 2026 13:16:21 -0800	[thread overview]
Message-ID: <DMXSQHZG5L27.2NIIT7F6FW0ME@google.com> (raw)
In-Reply-To: <20261116170942.720b06fd@shazbot.org>

On Mon Nov 16, 2026 at 5:09 PM PST, Alex Williamson wrote:
> On Tue, 13 Oct 2026 16:25:25 +0000
> David Matlack <dmatlack@google.com> wrote:
>> +int pci_liveupdate_adopt_acs(struct pci_dev *dev)
>> +{
>> +	/*
>> +	 * Check if the device was preserved over a previous Live Update (even
>> +	 * if it has already gone through pci_liveupdate_finish()). This ensures
>> +	 * that the device continues to use the ACS controls established by the
>> +	 * previous kernel.
>> +	 */
>> +	if (!dev->liveupdate.was_incoming)
>> +		return -EINVAL;
>> +
>> +	/*
>> +	 * The previous kernel should not have preserved any devices that
>> +	 * require device-specific quirks to enable ACS, but if such a device is
>> +	 * detected (e.g. new device-specific ACS quirk in the current kernel),
>> +	 * log a big warning and fall back to the normal enable ACS path.
>> +	 */
>> +	if (pci_need_dev_specific_enable_acs(dev)) {
>> +		pci_warn(dev, "Device-specific quirk required to enable ACS!\n");
>> +		WARN_ON_ONCE(true);
>> +		return -EINVAL;
>> +	}
>
> This looks pretty unsafe and the comment is only explaining how we
> might get here, not why we shouldn't panic.  IIUC, the IOMMU lho
> support requires singleton groups, but I'm not sure what it does if a
> group becomes non-singleton in the new kernel (such as via a new quirk).
>
> Is a warning related to what might be a parent bridge rather than the
> endpoint device, really sufficient here?  Thanks,

It's not necessarily safe. The new quirk could change how TLPs are being
routed while a downstream preserved device is still sending memory
transactions.

I think a better approach would be for a preserved device to always
adopt the ACS controls left by the previous kernel, and keep them
across resets through the ACS save state. So a new device-specific quirk
for the device is ignored, apart from logging a warning. IOW, new ACS
quirks only take effect on cold boot.

If a preserved device has an ACS capability but no save state (i.e. the
allocation failed), then we can panic, since its adopted controls would
be lost by the first reset..

Roughly:

	int pci_liveupdate_adopt_acs(struct pci_dev *dev)
	{
		if (!dev->liveupdate.was_incoming)
			return -EINVAL;

		if (pci_need_dev_specific_enable_acs(dev))
			pci_warn(dev, "Ignoring ACS quirk, adopting ACS controls from previous kernel\n");

		if (dev->acs_cap && !pci_find_saved_ext_cap(dev, PCI_EXT_CAP_ID_ACS))
			panic("%s: No ACS save state for preserved device\n",
			      pci_name(dev));

		return 0;
	}

The panic would only happen if allocating the save state fails. Ideally
I'd like save state allocation failures to cause the whole device setup
to fail. That's something I'm hoping to do in the future on top of the
pci_save_state rework [1], and then we can remove this panic. Until
then, the panic is constrained to Live Update, and the allocation
shouldn't fail in practice.

As for IOMMU groups: Sami's IOMMU Live Update series refuses to let any
device join a group that contains a preserved device, in both the
outgoing and incoming kernel (__iommu_probe_device() fails with -EBUSY)
[2][3]. So if the new kernel considers the path less isolated than the
previous kernel did, e.g. because of a new quirk, the group grows and
the IOMMU core refuses the second device rather than putting both in
one group. The system comes up with that device unavailable, which can
be debugged and recovered by rebooting, instead of panicking.

[1] https://lore.kernel.org/linux-pci/20261106004218.604799-1-dmatlack@google.com/
[2] https://lore.kernel.org/linux-iommu/20261110040828.1774343-6-skhawaja@google.com/
[3] https://lore.kernel.org/linux-iommu/20261110040828.1774343-13-skhawaja@google.com/

Does that sound reasonable?


  reply	other threads:[~2026-11-17 21:16 UTC|newest]

Thread overview: 31+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-10-13 16:25 [PATCH v10 00/13] PCI: liveupdate: PCI core support for Live Update David Matlack
2026-10-13 16:25 ` [PATCH v10 01/13] PCI: liveupdate: Set up FLB handler for the PCI core David Matlack
2026-11-06 16:52   ` Bjorn Helgaas
2026-10-13 16:25 ` [PATCH v10 02/13] PCI: liveupdate: Track outgoing preserved PCI devices David Matlack
2026-10-13 16:25 ` [PATCH v10 03/13] PCI: liveupdate: Track incoming " David Matlack
2026-11-06 16:53   ` Bjorn Helgaas
2026-10-13 16:25 ` [PATCH v10 04/13] PCI: liveupdate: Document driver binding responsibilities David Matlack
2026-11-06 16:54   ` Bjorn Helgaas
2026-10-13 16:25 ` [PATCH v10 05/13] PCI: liveupdate: Auto-preserve upstream bridges across Live Update David Matlack
2026-11-06 17:34   ` Bjorn Helgaas
2026-11-06 18:20     ` David Matlack
2026-11-06 18:40       ` Bjorn Helgaas
2026-11-06 18:40   ` Bjorn Helgaas
2026-10-13 16:25 ` [PATCH v10 06/13] PCI: liveupdate: Preserve bus numbers during " David Matlack
2026-11-06 17:43   ` Bjorn Helgaas
2026-10-13 16:25 ` [PATCH v10 07/13] PCI: Refactor matching logic for pci_dev_acs_ops David Matlack
2026-10-13 16:25 ` [PATCH v10 08/13] PCI: Save and restore the ACS Control register and Egress Control Vector David Matlack
2026-11-17  1:10   ` Alex Williamson
2026-11-17 21:24     ` David Matlack
2026-10-13 16:25 ` [PATCH v10 09/13] PCI: liveupdate: Adopt ACS controls in incoming preserved devices David Matlack
2026-11-17  1:09   ` Alex Williamson
2026-11-17 21:16     ` David Matlack [this message]
2026-10-13 16:25 ` [PATCH v10 10/13] PCI: liveupdate: Adopt ARI Forwarding Enable on preserved bridges David Matlack
2026-10-13 16:25 ` [PATCH v10 11/13] PCI: liveupdate: Freeze preservation status during shutdown David Matlack
2026-10-13 16:25 ` [PATCH v10 12/13] PCI: liveupdate: Do not disable bus mastering on preserved devices during kexec David Matlack
2026-10-13 16:25 ` [PATCH v10 13/13] Documentation: PCI: Add documentation for Live Update David Matlack
2026-11-03 19:21 ` [PATCH v10 00/13] PCI: liveupdate: PCI core support " Pasha Tatashin
2026-11-04 13:40   ` Mike Rapoport
2026-11-04 18:00 ` Pasha Tatashin
2026-11-04 18:06   ` David Matlack
2026-11-16 16:45     ` David Matlack

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=DMXSQHZG5L27.2NIIT7F6FW0ME@google.com \
    --to=dmatlack@google.com \
    --cc=ajayachandra@nvidia.com \
    --cc=alex@shazbot.org \
    --cc=bhelgaas@google.com \
    --cc=chrisl@kernel.org \
    --cc=corbet@lwn.net \
    --cc=graf@amazon.com \
    --cc=jacob.pan@linux.microsoft.com \
    --cc=jgg@nvidia.com \
    --cc=jrhilke@google.com \
    --cc=kexec@lists.infradead.org \
    --cc=leonro@nvidia.com \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=lukas@wunner.de \
    --cc=parav@nvidia.com \
    --cc=pasha.tatashin@soleen.com \
    --cc=praan@google.com \
    --cc=pratyush@kernel.org \
    --cc=rdunlap@infradead.org \
    --cc=rientjes@google.com \
    --cc=rppt@kernel.org \
    --cc=saeedm@nvidia.com \
    --cc=skhan@linuxfoundation.org \
    --cc=skhawaja@google.com \
    --cc=vipinsh@google.com \
    --cc=witu@nvidia.com \
    --cc=yi.l.liu@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox