[PATCH v2 0/2] PCI: Wait after FLR for devices that don't honor Immediate Readiness

[PATCH v2 0/2] PCI: Wait after FLR for devices that don't honor Immediate Readiness

[Alexander Gruhlke]

Since v7.2, VFIO passthrough of a Samsung 990 PRO NVMe [144d:a80c] fails
on the second VM start, and the device is inaccessible until reboot.  The
drive advertises Immediate Readiness but returns ~0 for up to 7 ms after
FLR, so since 10baa9b4df40 ("PCI: Drop unnecessary retries when restoring
BARs") the config restore after FLR is lost.

Patch 1 ignores Immediate Readiness on that drive.  A 990 EVO Plus
[144d:a80d] reports the same errors [1], but it isn't confirmed to
advertise Immediate Readiness, so it's not included.

Patch 2 is independent and keeps pci_dev_wait() from treating ~0 as
ready with RRS Software Visibility, which also affects the Intel
[8086:0a54] in the open d591f6804e7e regression.  Hui, Linux-Fan: testing
patch 2 on that drive would be appreciated.

Tested on v7.2.9 (Ryzen 9 7950X, X670E, drive firmware 1B2QJXD7): with
patch 1, three VM starts in a row work, and every FLR reports "ready 0ms
after FLR".  After updating the drive to firmware 8B2QJXD7, it still
advertises Immediate Readiness but responds right after FLR, and plain
v7.2.9 works, so only older firmware is affected.

Patch 2 in this form is only tested for regressions, as the drive no
longer shows the problem.  v1's variant, which read the Command register,
was tested against it, and the drive's Vendor ID read ~0 while it wasn't
ready.  Applies to pci/next.

Changes since v1:
- Replace the pcie_flr() change with a quirk for the 990 PRO (Lukas)
- pci_dev_wait(): check the Vendor ID for ~0 except on VFs instead of
  reading the Command register, and "may return" in the comment (Lukas)
- Swap the patch order, as the quirk alone fixes the regression

v1: https://lore.kernel.org/r/20261010160529.27974-1-gruhlke@mailbox.org

[1] https://bbs.archlinux.org/viewtopic.php?id=314833

Alexander Gruhlke (2):
  PCI: Ignore Immediate Readiness on Samsung 990 PRO NVMe
  PCI: Don't treat ~0 as ready in pci_dev_wait() with RRS SV

 drivers/pci/pci.c    |  8 +++++---
 drivers/pci/quirks.c | 11 +++++++++++
 2 files changed, 16 insertions(+), 3 deletions(-)

-- 
2.56.0

[PATCH v2 1/2] PCI: Ignore Immediate Readiness on Samsung 990 PRO NVMe

[Alexander Gruhlke]

The Samsung 990 PRO [144d:a80c] advertises Immediate Readiness, so
pcie_flr() doesn't wait after FLR, but the drive returns ~0 for up to
7 ms.  Since commit 10baa9b4df40 ("PCI: Drop unnecessary retries when
restoring BARs"), the config restore after FLR is lost, which breaks
VFIO passthrough of the drive.  Firmware 8B2QJXD7 responds right after
FLR, but older firmware such as 1B2QJXD7 doesn't, and the firmware
version isn't visible here.

Ignore Immediate Readiness on this device, so pcie_flr() waits.

Fixes: 10baa9b4df40 ("PCI: Drop unnecessary retries when restoring BARs")
Link: https://bbs.archlinux.org/viewtopic.php?id=314833
Link: https://lore.kernel.org/r/20261010160529.27974-1-gruhlke@mailbox.org
Suggested-by: Lukas Wunner <lukas@wunner.de>
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Alexander Gruhlke <gruhlke@mailbox.org>
---
 drivers/pci/quirks.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 0500d5fd4f..e8597d1209 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -5623,6 +5623,17 @@ static void quirk_no_flr_snet(struct pci_dev *dev)
 }
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_SOLIDRUN, 0x1000, quirk_no_flr_snet);
 
+/* Samsung 990 PRO advertises Immediate Readiness, but isn't ready after FLR */
+static void quirk_no_imm_ready(struct pci_dev *dev)
+{
+	if (!dev->imm_ready)
+		return;
+
+	pci_info(dev, "Ignoring incorrectly advertised Immediate Readiness support\n");
+	dev->imm_ready = 0;
+}
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_SAMSUNG, 0xa80c, quirk_no_imm_ready);
+
 static void quirk_no_ext_tags(struct pci_dev *pdev)
 {
 	struct pci_host_bridge *bridge = pci_find_host_bridge(pdev->bus);
-- 
2.56.0

Re: [PATCH v2 1/2] PCI: Ignore Immediate Readiness on Samsung 990 PRO NVMe

[sashiko-bot]

> The Samsung 990 PRO [144d:a80c] advertises Immediate Readiness, so
> pcie_flr() doesn't wait after FLR, but the drive returns ~0 for up to
> 7 ms.  Since commit 10baa9b4df40 ("PCI: Drop unnecessary retries when
> restoring BARs"), the config restore after FLR is lost, which breaks
> VFIO passthrough of the drive.  Firmware 8B2QJXD7 responds right after
> FLR, but older firmware such as 1B2QJXD7 doesn't, and the firmware
> version isn't visible here.
> 
> Ignore Immediate Readiness on this device, so pcie_flr() waits.
> 
> Fixes: 10baa9b4df40 ("PCI: Drop unnecessary retries when restoring BARs")
> Link: https://bbs.archlinux.org/viewtopic.php?id=314833
> Link: https://lore.kernel.org/r/20261010160529.27974-1-gruhlke@mailbox.org
> Suggested-by: Lukas Wunner <lukas@wunner.de>
> Assisted-by: Claude:claude-opus-5-5
> Signed-off-by: Alexander Gruhlke <gruhlke@mailbox.org>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261011163249.18479-1-gruhlke@mailbox.org?part=1

Re: [PATCH v2 1/2] PCI: Ignore Immediate Readiness on Samsung 990 PRO NVMe

[Lukas Wunner]

On Sun, Oct 11, 2026 at 06:32:48PM +0200, Alexander Gruhlke wrote:
> The Samsung 990 PRO [144d:a80c] advertises Immediate Readiness, so
> pcie_flr() doesn't wait after FLR, but the drive returns ~0 for up to
> 7 ms.  Since commit 10baa9b4df40 ("PCI: Drop unnecessary retries when
> restoring BARs"), the config restore after FLR is lost, which breaks
> VFIO passthrough of the drive.  Firmware 8B2QJXD7 responds right after
> FLR, but older firmware such as 1B2QJXD7 doesn't, and the firmware
> version isn't visible here.

If a firmware update is available from the vendor, we generally
refrain from adding a quirk and instead tell users to update.

OTOH, the offending patch breaks setups that worked before,
which is annoying.

It's also a bit of a bummer that the quirk can't check the
firmware version.  Has the upgrade changed the Revision ID
in the Config Space header?  If it did, that could be used
to identiy non-broken devices, but usually it's only incremented
for new silicon revisions.

Up to Bjorn what to do here.

FWIW,
Reviewed-by: Lukas Wunner <lukas@wunner.de>

Re: [PATCH v2 1/2] PCI: Ignore Immediate Readiness on Samsung 990 PRO NVMe

[Alexander Gruhlke]

On Sun, Oct 11, 2026 at 08:00:32PM +0200, Lukas Wunner wrote:
> It's also a bit of a bummer that the quirk can't check the
> firmware version.  Has the upgrade changed the Revision ID
> in the Config Space header?

No, it's 0x00 with both 1B2QJXD7 and 8B2QJXD7, and the Subsystem ID
(144d:a801) didn't change either.  On fixed firmware the quirk only
costs the 100 ms wait per FLR.

Fine with me either way, thanks for the review.

Alexander

[PATCH v2 2/2] PCI: Don't treat ~0 as ready in pci_dev_wait() with RRS SV

[Alexander Gruhlke]

With RRS Software Visibility, pci_dev_wait() considers a device ready as
soon as reading the Vendor ID doesn't return the RRS value.  Devices
aren't required to respond with RRS while they aren't ready, so the read
may return ~0 and pci_dev_wait() stops waiting too early.  This was seen
with Intel [8086:0a54] and Samsung [144d:a80c] NVMe SSDs.

Also wait for a Vendor ID other than ~0, except on VFs, whose Vendor ID
normally reads ~0.

Fixes: d591f6804e7e ("PCI: Wait for device readiness with Configuration RRS")
Link: https://lore.kernel.org/r/20250611101442.387378-1-hui.wang@canonical.com
Suggested-by: Lukas Wunner <lukas@wunner.de>
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Alexander Gruhlke <gruhlke@mailbox.org>
---
 drivers/pci/pci.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 350bae907e..fef8a799c1 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -1213,9 +1213,10 @@ static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout)
 	 * initialize.
 	 *
 	 * If the device is below a Root Port with Configuration RRS
-	 * Software Visibility enabled, reading the Vendor ID returns a
+	 * Software Visibility enabled, reading the Vendor ID may return a
 	 * special data value if the device responded with RRS.  Read the
-	 * Vendor ID until we get non-RRS status.
+	 * Vendor ID until we get non-RRS status and, unless it's a VF,
+	 * something other than ~0.
 	 *
 	 * If there's no Root Port or Configuration RRS Software Visibility
 	 * is not enabled, the device may still respond with RRS, but
@@ -1235,7 +1236,8 @@ static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout)
 
 		if (root && root->config_rrs_sv) {
 			pci_read_config_dword(dev, PCI_VENDOR_ID, &id);
-			if (!pci_bus_rrs_vendor_id(id))
+			if (!pci_bus_rrs_vendor_id(id) &&
+			    (!PCI_POSSIBLE_ERROR(id) || dev->is_virtfn))
 				break;
 		} else {
 			pci_read_config_dword(dev, PCI_COMMAND, &id);
-- 
2.56.0

Re: [PATCH v2 2/2] PCI: Don't treat ~0 as ready in pci_dev_wait() with RRS SV

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] pci: premature wait termination for VFs returning ~0 under RRS SV
--

commit 3101a5ea075ba619cd03d971d4a538bf4788ec75
Author: Alexander Gruhlke <gruhlke@mailbox.org>

PCI: Don't treat ~0 as ready in pci_dev_wait() with RRS SV

With RRS Software Visibility, pci_dev_wait() considers a device ready as
soon as reading the Vendor ID doesn't return the RRS value.  Devices
aren't required to respond with RRS while they aren't ready, so the read
may return ~0 and pci_dev_wait() stops waiting too early.

Link: https://lore.kernel.org/r/20250611101442.387378-1-hui.wang@canonical.com

> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index aa39b4dd45aea..cf429f64bce0f 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
[ ... ]
> @@ -1331,7 +1332,8 @@ static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout)
>  
>  		if (root && root->config_rrs_sv) {
>  			pci_read_config_dword(dev, PCI_VENDOR_ID, &id);
> -			if (!pci_bus_rrs_vendor_id(id))
> +			if (!pci_bus_rrs_vendor_id(id) &&
> +			    (!PCI_POSSIBLE_ERROR(id) || dev->is_virtfn))
>  				break;

[Severity: Medium]
Does this cause pci_dev_wait() to exit prematurely for Virtual Functions
(VFs) that return ~0 under Configuration RRS Software Visibility?

If a VF is not yet ready, its config space might return ~0 (for example, due
to a Request Timeout) instead of the RRS value. The dev->is_virtfn check
causes the wait loop to terminate immediately upon seeing ~0. This seems to
allow unready VFs to appear ready, defeating the reset delay.

Callers might then attempt to initialize the unready VF, leading to config
read/write failures or even triggering PCIe AERs if the unready VF later
responds with RRS to non-Vendor-ID accesses.

While VFs architecturally return ~0 for Vendor ID when they are ready, polling
PCI_COMMAND when the Vendor ID returns ~0 (as done in the non-RRS SV path
below) could resolve this without exiting prematurely.

>  		} else {
>  			pci_read_config_dword(dev, PCI_COMMAND, &id);

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261011163249.18479-1-gruhlke@mailbox.org?part=2

Re: [PATCH v2 2/2] PCI: Don't treat ~0 as ready in pci_dev_wait() with RRS SV

[Alexander Gruhlke]

On Sun, Oct 11, 2026 at 04:41:42PM +0000, sashiko-bot@kernel.org wrote:
> Does this cause pci_dev_wait() to exit prematurely for Virtual Functions
> (VFs) that return ~0 under Configuration RRS Software Visibility?

Yes, but that's the current behavior for all devices; the patch only
changes it for non-VFs.

VFs could be covered by reading the Command register when the Vendor ID
reads ~0 and waiting while that is ~0, too.  Devices that are ready would
still need only one read.

Lukas, Bjorn: should I do that in v3, or keep the current form?

Thanks,
Alexander

Re: [PATCH v2 2/2] PCI: Don't treat ~0 as ready in pci_dev_wait() with RRS SV

[Lukas Wunner]

On Sun, Oct 11, 2026 at 06:49:55PM +0200, Alexander Gruhlke wrote:
> On Sun, Oct 11, 2026 at 04:41:42PM +0000, sashiko-bot@kernel.org wrote:
> > Does this cause pci_dev_wait() to exit prematurely for Virtual Functions
> > (VFs) that return ~0 under Configuration RRS Software Visibility?
> 
> Yes, but that's the current behavior for all devices; the patch only
> changes it for non-VFs.
> 
> VFs could be covered by reading the Command register when the Vendor ID
> reads ~0 and waiting while that is ~0, too.  Devices that are ready would
> still need only one read.
> 
> Lukas, Bjorn: should I do that in v3, or keep the current form?

For a patch which is going to be backported to stable (due to the
Fixes tag), changing behavior is not a good idea, so I guess you
were right to read the Command Register on VFs if the return value
is 0xffff.  I apologize for my earlier misleading comments.

You could probably constrain the Command Register read operation
to VFs as those are the only ones that would need it, right?

I've started to wonder if VFs can actually be inaccessible and
thus cause a 0xffff response.  So I looked up where the spec
language in PCIe r7.1 sec 7.5.1.1.1 originates from:

It was introduced in PCIe r5.0 sec 9.3.4.1.2 and then r6.0 moved
spec language to sec 7.5.1.1.1.  As rationale, PCIe r5.0 only says:

   "This field in all VFs returns FFFFh when read. VI software should
    return the VF Device ID (see Section 9.3.3.11) value from the
    associated PF as the Device ID for the VF.
    
    IMPLEMENTATION NOTE: Legacy PCI Probing Software
    Returning FFFFh for Device ID and Vendor ID values allows some
    legacy software to ignore VFs. See Section 7.5.1.1.1."

So it's a little unclear to me whether inaccessibility of VFs can
actually occur, but again, for a patch with a Fixes tag, it's better
to play it safe.

Thanks,

Lukas