Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[Thomas Stegbauer]

In-Reply-To: <20260801201244.4421-1-andiwild@gmail.com>
References: <20260801201244.4421-1-andiwild@gmail.com>

Hi,

another data point for the regression from 72780f796468 ("PCI: Always
lift 2.5GT/s restriction in PCIe failed link retraining"), for the case
with a device present, which v3 does not appear to cover.

Hardware: Raspberry Pi 5 (BCM2712, pcie-brcmstb), ASMedia ASM1182e
[1b21:1182] Gen2 packet switch on a dual M.2 carrier, one NVMe SSD
behind each downstream port.

Downstream port 7 (0001:02:07.0) does not train at 5GT/s on this board.
Without the backport (v6.12.75) the quirk clamps it and the link comes
up at Gen1, in use for over a year:

  pci 0001:02:07.0: broken device, retraining non-functional downstream link at 2.5GT/s
  LnkSta:  Speed 2.5GT/s, Width x1, DLActive+
  LnkCtl2: Target Link Speed: 2.5GT/s

With the backport (v6.18.50, also v6.18.39; Raspberry Pi downstream
kernels, pcie-brcmstb identical between both 6.18 versions) the device
is lost:

  [0.518303] pci 0001:02:07.0: broken device, retraining non-functional downstream link at 2.5GT/s
  [0.538795] pci 0001:02:07.0: removing 2.5GT/s downstream link speed restriction
  [1.537936] pci 0001:02:07.0: retraining failed
  LnkSta:  Speed 5GT/s, Width x1, DLActive-, BWMgmt+
  LnkCtl2: Target Link Speed: 5GT/s

Same hardware, firmware and DT; only the kernel differs. Swapping the
SSDs moves the failure with the port, not with the drive.

To be fair, the Gen1 link is marginal: AER on the port shows RxErr
right after clearing, and BadTLP/BadDLLP under load. The replays are
absorbed, though: 8 GiB O_DIRECT read at 222 MB/s, no data errors.

So: the port is not clamped on entry, hence v3's new early return does
not trigger; the first block clamps and the link trains at 2.5GT/s; the
second block then lifts the clamp because LnkCap is 5GT/s; the 5GT/s
retrain fails as the initial training did, and the error path restores
the old 5GT/s target, leaving the link down (and adding ~2 s to boot).

Before 72780f796468 the lift was limited to the ASM2824 with the link
up, which is why this worked. Would it make sense to fall back to the
2.5GT/s clamp when lifting fails, rather than to the original target
speed, at least when the clamp had produced a working link?

Happy to test patches (v6.18.y based on the Raspberry Pi tree is
easiest for me). Downstream report with full logs:
https://github.com/raspberrypi/linux/issues/7635

Thanks,
Thomas S.

p.s. hopefully the mail get properly added, as I hade to add the in-reply and reference to the txt-body of the mail.

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[M M]

Thank you Armando for the detailed follow-up. It's "good" to see this problem is still active, though Proxmox is really slow with this. In the meantime we are 1,5 month ahead and without the much needed security updates related to the kernel.
I can only hope there will be some progress soon on their behalf, but it's pretty much quiet at Proxmox in general.

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[Thomas Lamprecht]

On 09/10/2026 12:53, M M wrote:
> Thank you Armando for the detailed follow-up. It's "good" to see this problem is still active, though Proxmox is really slow with this. In the meantime we are 1,5 month ahead and without the much needed security updates related to the kernel.
> I can only hope there will be some progress soon on their behalf, but it's pretty much quiet at Proxmox in general.

Please stop CC'ing random people and Proxmox email addresses that half
not even exist, that's just spamming, please stop that. Also not sure
what your talking w.r.t. security updates, we release fixes in our kernel
on a steady stream and are up-to-date here.

Besides, our downstream info about this was that one user tried the patch
and it did not work for them and you could not confirm this either as
requested in the same reply telling you [0], so we did not look at this
at all due to waiting on feedback. btw. did you actually test this now
confirming it fixes the issues on your system?
As v3 on its own would leave most of our reports broken, revert is more
likely to help here for the broader masses and will be the route we
look into.

[0]: https://bugzilla.proxmox.com/show_bug.cgi?id=7969#c8

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[SyncNOW.net - Armando Araujo Filho]

Hi Maciej, Bjorn, all,

Another active-link case for 72780f796468 ("PCI: Always lift 2.5GT/s
restriction in PCIe failed link retraining"). It answers two open
questions in this thread:

  - Maciej asked for HASD results. On this port HASD is clear
    (SpeedDis-), so the HASD check Andreas proposed would not help here.
  - This case never reaches the error path (no "retraining failed"),
    so falling back to 2.5GT/s on failure, as Thomas suggested, would
    not help either.

On a Huawei server the onboard BMC VGA controller, a Gen1-only
endpoint, sits behind a firmware-clamped PCH root port. After the quirk
lifts the clamp, the endpoint is missing from the bus scan, even though
the quirk reports no error and the link is up at 2.5GT/s afterwards.
A rescan finds the device again. Reproduced on Arch Linux 7.2.7. The
Proxmox bisection gives the same 7.0.14-6 -> 7.0.14-7 boundary that
Aoxtj and Mich found.

A note on how this report was produced: I am a network/infrastructure
engineer, not a kernel developer. I noticed the problem (loss of video
on a Proxmox upgrade), ran all the tests on the hardware, and collected
the logs, register dumps and kernel bisection below. The analysis was
done with an AI assistant (Anthropic's Claude): it compared the dmesg
and lspci output between kernels, identified the link-speed quirk and
the relevant commit, suggested the setpci recovery test, and drafted
this report. All data below was captured on the machine and I have
checked it against the raw logs. The interpretation in the "our guess"
paragraph and the suggestions at the end should be read as
hypotheses, not as kernel expertise on my part.

Kernels tested
--------------

Same hardware and configuration throughout:

  Good  Proxmox VE 7.0.2-6, 7.0.12-1, 7.0.14-1, 7.0.14-6
        (7.0.14-6 = Ubuntu-7.0.0-28.28i2, stable 6.18.38 / 7.1.3)
  Bad   Proxmox VE 7.0.14-7, 7.0.14-8
        (7.0.14-7 = Ubuntu-7.0.0-28.28i3, stable 6.18.39 / 7.1.4)
  Bad   Arch Linux 7.2.7-arch1-1 (official 2026.10.01 live ISO)

Hardware
--------

  DMI: Huawei Technologies Co., Ltd. Tecal RH2285 V2-12L/BC11SRSC1,
       BIOS RMISV055 02/02/2013
  CPU: 2x Xeon E5 v2 (Ivy Bridge-EP), PCH C600/X79

  Affected path:
    00:1c.3  Root Port 4  [8086:1d16] (rev b6)  LnkCap 5GT/s x1
             SltCap HotPlug- Surprise- (not hot-plug capable)
      03:00.0  XGI Z11/Z11M VGA [18ca:0027]
               Subsystem Huawei [19e5:2013]     LnkCap 2.5GT/s x1
               (PCIe v1 endpoint, Gen1-only; the BMC's video device)

Register state, good vs. bad
----------------------------

  00:1c.3 (root port)     good (7.0.14-6)     bad (7.0.14-7, 7.2.7)
  LnkCap                  5GT/s x1            5GT/s x1
  LnkCtl2 Target Speed    2.5GT/s (firmware)  5GT/s (lifted)
  LnkCtl2 HASD            SpeedDis-           SpeedDis-
  LnkSta                  2.5GT/s x1          2.5GT/s x1
  03:00.0 enumerated      yes                 NO

The link is up (x1) after the quirk too. It just comes back at
2.5GT/s, the most the endpoint supports.

dmesg, Arch Linux 7.2.7-arch1-1
-------------------------------

  Linux version 7.2.7-arch1-1 (linux@archlinux) ... #1 SMP
PREEMPT_DYNAMIC Mon, 21 Sep 2026
  ...
  [    1.019207] pci 0000:00:1c.0: removing 2.5GT/s downstream link
speed restriction
  [    2.019409] pci 0000:00:1c.0: retraining failed
  [    3.019695] pci 0000:00:1c.2: removing 2.5GT/s downstream link
speed restriction
  [    4.020409] pci 0000:00:1c.2: retraining failed
  [    5.021634] pci 0000:00:1c.3: [8086:1d16] type 01 class 0x060400
PCIe Root Port
  [    5.021666] pci 0000:00:1c.3: PCI bridge to [bus 03]
  [    5.021673] pci 0000:00:1c.3:   bridge window [io  0x3000-0x3fff]
  [    5.021678] pci 0000:00:1c.3:   bridge window [mem 0x94b00000-0x94bfffff]
  [    5.021690] pci 0000:00:1c.3:   bridge window [mem
0x90000000-0x93ffffff 64bit pref]
  [    5.021718] pci 0000:00:1c.3: removing 2.5GT/s downstream link
speed restriction
  [    5.021770] pci 0000:00:1c.3: PME# supported from D0 D3hot D3cold
  ...
  [    5.045412] pci 0000:00:1c.3: PCI bridge to [bus 03]
  [    5.076827] vgaarb: loaded                       <- no VGA device

00:1c.0 and 00:1c.2 are empty ports, the case v3 covers.

On the good kernels the same scan finds the endpoint right away:

  pci 0000:03:00.0: [18ca:0027] type 00 class 0x030000 PCIe Endpoint
  pci 0000:03:00.0: BAR 0 [mem 0x90000000-0x93ffffff pref]
  pci 0000:03:00.0: Video device with shadowed ROM at [mem
0x000c0000-0x000dffff]
  pci 0000:03:00.0: vgaarb: setting as boot VGA device

How this differs from the other active-link reports
---------------------------------------------------

In Aoxtj's, Mich's and Thomas's cases the retrain fails, the error
path runs, and the link is left down (x0 / DLLLA=0). Here:

  - there is no "retraining failed" message; the quirk returns about
    50 us after "removing 2.5GT/s ...";
  - the link is up afterwards (2.5GT/s x1);
  - the endpoint is healthy and comes back with a rescan (below).

So the device is only missing because bus 03 is scanned too early.

Our guess, not verified: the retrain takes the link down and back up
(the endpoint cannot do 5GT/s and falls back to 2.5GT/s). The wait in
the quirk may see DLLLA still set from before the link went down and
return at once. The bus below is then scanned while the link is still
retraining, and nothing answers. That would fit the 50 us return and
the missing error message. A one-line pci_info() of LNKSTA right after
pcie_set_target_speed() would show whether this is right. I can run
that, or Andreas's earlier diagnostic, if useful.

Recovery at runtime
-------------------

Restoring the firmware clamp, retraining and rescanning brings the
device back:

  # setpci -s 00:1c.3 CAP_EXP+0x30.w=0001:000f   # LnkCtl2 TLS = 2.5GT/s
  # setpci -s 00:1c.3 CAP_EXP+0x10.w=0020:0020   # LnkCtl Retrain Link
  # sleep 1
  # echo 1 > /sys/bus/pci/rescan

  pci 0000:03:00.0: [18ca:0027] type 00 class 0x030000 PCIe Endpoint
  pci 0000:03:00.0: vgaarb: VGA device added:
decodes=io+mem,owns=none,locks=none
  pcieport 0000:00:1c.3: ASPM: current common clock configuration is
inconsistent, reconfiguring
  pci 0000:03:00.0: BAR 0 [mem 0x90000000-0x93ffffff pref]: assigned
  pci 0000:03:00.0: BAR 1 [mem 0x94200000-0x9423ffff]: assigned
  pci 0000:03:00.0: BAR 2 [io  0x3000-0x307f]: assigned

By then the boot VGA console is already lost, so this only shows that
the device is still there.

Observations
------------

1. HASD is clear on this port, so a HASD-based check would not catch
   this case.

2. The failure is not in the error path, so a 2.5GT/s fallback on
   failure would not help either.

3. The retrain gains nothing here: the endpoint only supports 2.5GT/s
   and the link comes back at that speed. I realise the endpoint is
   not enumerated yet when the quirk runs on the port, so knowing its
   speed in advance may not be simple.

4. If an active link is retrained, waiting for it to actually go down
   and come back (or for the device to become ready) before the bus
   below is scanned might prevent the device going missing.

5. Like Mich's Dell and HP machines, this is an older Intel server
   platform where firmware clamps the ports on purpose. A command-line
   opt-out would give users of such machines a workaround other than
   staying on an older kernel.

I can test patches on this machine with a mainline or live kernel and
can provide full dmesg and lspci -vvv from good and bad kernels.

Thanks,
Armando Araujo Filho

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[blaat windows]

Another reproducible case of the active-link failure described in this thread.

Hardware:
Intel 4th-gen/9-series platform
Intel 82571EB quad-port NIC
Microsemi/PMC/IDT PES12N3A PCIe switch
Root port 00:1c.4, LnkCap 5GT/s x4
Working negotiated link: 2.5GT/s x4

Kernel results:
7.2-rc1 fail
7.1-rc7 succes

On failing kernels, the PES12N3A hierarchy does not enumerate and all four downstream 82571EB ports disappear.

I traced this to pcie_failed_link_retrain() and specifically the new generic clamp-removal code introduced by 72780f7964684939d7d2f69c348876213b184484 ("PCI: Always lift 2.5GT/s restriction in PCIe failed link retraining").

I tested 7.3.0-rc3+ with only this block commented out:

    pcie_capability_read_word(dev, PCI_EXP_LNKCTL2, &lnkctl2);
    if ((lnkctl2 & PCI_EXP_LNKCTL2_TLS) == PCI_EXP_LNKCTL2_TLS_2_5GT) {
            pci_info(dev, "removing 2.5GT/s downstream link speed restriction\n");
            ret = pcie_set_target_speed(dev, speed_cap, false);
            if (ret)
                    goto err;
    }

With that block disabled, 7.3.0-rc3+ boots normally, having all four NIC ports enumerate:

07:00.0 82571EB
07:00.1 82571EB
08:00.0 82571EB
08:00.1 82571EB

The important result is that the initial 2.5GT/s recovery is fine. Leaving the link at 2.5GT/s works. It is the subsequent:

    pcie_set_target_speed(dev, speed_cap, false);

which breaks this PES12N3A/82571EB link.

So this appears to be the same active-link failure mode, but with a PES12N3A switch rather than a direct 82571EB connection.

This was tested against vanilla 7.3.0-rc3+ with only the above local change.

I can provide full dmesg/lspci output and test a proposed fix if useful.

[PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[Andreas Wild]

From: "Maciej W. Rozycki" <macro@orcam.me.uk>

Since commit 72780f796468 ("PCI: Always lift 2.5GT/s restriction in PCIe
failed link retraining") the Target Speed quirk lifts a firmware-imposed
2.5GT/s restriction on any downstream port, without checking whether the
link is up.  Where nothing is plugged in, the retraining that follows can
never complete, so each attempt costs PCIE_LINK_RETRAIN_TIMEOUT_MS.  The
quirk makes two of them -- the initial one and the restore on the error
path -- adding a fixed 2 s to every boot.

On an MSI PRO Z690-A WIFI DDR4 (Intel 600 Series PCH) with one empty x1
slot, running v7.2-rc5:

 0.541 pci 0000:00:1c.0: removing 2.5GT/s downstream link speed restriction
 1.541 pci 0000:00:1c.0: retraining failed
 2.541 pci 0000:00:1c.2: [8086:7aba] type 01 class 0x060400

Where the Link Speed has already been clamped at 2.5GT/s and no link has
been established there is nothing worth doing, which is what the kerneldoc
for the quirk already describes: the restriction is to be lifted where
firmware arranged it "and the port reports its link already being up".
Bail out early in that case, before either the ASM2824 workaround or the
removal of the restriction is considered.

Ports whose link is up are unaffected, and so is the ASM2824 workaround,
which is reached with the Target Link Speed not clamped.

With this applied the quirk returns without touching the port: both
messages are gone, enumeration proceeds from 0000:00:1c.0 to 0000:00:1c.2
in 1 ms rather than 2 s, and the systemd "kernel" boot phase goes from
3.011 s to 1.036 s.

Fixes: 72780f796468 ("PCI: Always lift 2.5GT/s restriction in PCIe failed link retraining")
Cc: stable@vger.kernel.org
Reported-by: Andreas Wild <andiwild@gmail.com>
Closes: https://lore.kernel.org/lkml/20260801092152.5643-1-andiwild@gmail.com/
Tested-by: Andreas Wild <andiwild@gmail.com>
Signed-off-by: Maciej W. Rozycki <macro@orcam.me.uk>
Signed-off-by: Andreas Wild <andiwild@gmail.com>
---
v3:
 - Replaced with Maciej's much simpler approach: bail out of the quirk
   entirely when the Target Link Speed is already clamped at 2.5GT/s and no
   link has been established, rather than programming the speed and skipping
   only the retraining.  One function, no new API, no bwctrl changes.
 - Note this leaves the Target Link Speed clamped on such a port, where v2
   left it at the Port's maximum.  A device hot-plugged there later trains at
   2.5GT/s: pcie_wait_for_link_delay() only calls the quirk when
   pcie_wait_for_link_status() fails, so a link that comes up cleanly at
   2.5GT/s never re-runs it.  Flagging in case that matters; the clamp is
   firmware's, so honouring it on an unoccupied Port seems defensible.
 - v2: https://lore.kernel.org/lkml/20260801105441.6506-1-andiwild@gmail.com/
 - v1: https://lore.kernel.org/lkml/20260801092152.5643-1-andiwild@gmail.com/
 drivers/pci/quirks.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index b09f27f..9e407c4 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -108,7 +108,11 @@ int pcie_failed_link_retrain(struct pci_dev *dev)
 
 	pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &lnksta);
 	pcie_capability_read_word(dev, PCI_EXP_LNKCTL2, &oldlnkctl2);
-	if (!(lnksta & PCI_EXP_LNKSTA_DLLLA) && pcie_lbms_seen(dev, lnksta)) {
+	if (lnksta & PCI_EXP_LNKSTA_DLLLA) {
+		;
+	} else if (PCIE_LNKCTL2_TLS2SPEED(oldlnkctl2) == PCIE_SPEED_2_5GT) {
+		return ret;
+	} else if (pcie_lbms_seen(dev, lnksta)) {
 		pci_info(dev, "broken device, retraining non-functional downstream link at 2.5GT/s\n");
 		ret = pcie_set_target_speed(dev, PCIE_SPEED_2_5GT, false);
 		if (ret)
-- 
2.55.0

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[sashiko-bot]

> From: "Maciej W. Rozycki" <macro@orcam.me.uk>
> 
> Since commit 72780f796468 ("PCI: Always lift 2.5GT/s restriction in PCIe
> failed link retraining") the Target Speed quirk lifts a firmware-imposed
> 2.5GT/s restriction on any downstream port, without checking whether the
> link is up.  Where nothing is plugged in, the retraining that follows can
> never complete, so each attempt costs PCIE_LINK_RETRAIN_TIMEOUT_MS.  The
> quirk makes two of them -- the initial one and the restore on the error
> path -- adding a fixed 2 s to every boot.
> 
> On an MSI PRO Z690-A WIFI DDR4 (Intel 600 Series PCH) with one empty x1
> slot, running v7.2-rc5:
> 
>  0.541 pci 0000:00:1c.0: removing 2.5GT/s downstream link speed restriction
>  1.541 pci 0000:00:1c.0: retraining failed
> [ ... ]
> Reported-by: Andreas Wild <andiwild@gmail.com>
> Closes: https://lore.kernel.org/lkml/20260801092152.5643-1-andiwild@gmail.com/
> Tested-by: Andreas Wild <andiwild@gmail.com>
> Signed-off-by: Maciej W. Rozycki <macro@orcam.me.uk>
> Signed-off-by: Andreas Wild <andiwild@gmail.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260801201244.4421-1-andiwild@gmail.com?part=1

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[Thorsten Leemhuis]

On 8/1/26 22:11, Andreas Wild wrote:
> From: "Maciej W. Rozycki" <macro@orcam.me.uk>
> 
> Since commit 72780f796468 ("PCI: Always lift 2.5GT/s restriction in PCIe
> failed link retraining") the Target Speed quirk lifts a firmware-imposed
> 2.5GT/s restriction on any downstream port, without checking whether the
> link is up.  Where nothing is plugged in, the retraining that follows can
> never complete, so each attempt costs PCIE_LINK_RETRAIN_TIMEOUT_MS.  The
> quirk makes two of them -- the initial one and the restore on the error
> path -- adding a fixed 2 s to every boot.
> 
> On an MSI PRO Z690-A WIFI DDR4 (Intel 600 Series PCH) with one empty x1
> slot, running v7.2-rc5:
> 
>  0.541 pci 0000:00:1c.0: removing 2.5GT/s downstream link speed restriction
>  1.541 pci 0000:00:1c.0: retraining failed
>  2.541 pci 0000:00:1c.2: [8086:7aba] type 01 class 0x060400

TWIMC, this patch fixes a regression Edward reported here:
https://bugzilla.kernel.org/show_bug.cgi?id=221801

[side note: the report could be more specific, yes -- I wanted to ask
Edward to clarify a few things (like the actual slowdown) before
forwarding it, but it didn't came to that when I noticed this patch and
asked Edward to just check if it helped -- which is did.]

Ciao, Thorsten
> Where the Link Speed has already been clamped at 2.5GT/s and no link has
> been established there is nothing worth doing, which is what the kerneldoc
> for the quirk already describes: the restriction is to be lifted where
> firmware arranged it "and the port reports its link already being up".
> Bail out early in that case, before either the ASM2824 workaround or the
> removal of the restriction is considered.
> 
> Ports whose link is up are unaffected, and so is the ASM2824 workaround,
> which is reached with the Target Link Speed not clamped.
> 
> With this applied the quirk returns without touching the port: both
> messages are gone, enumeration proceeds from 0000:00:1c.0 to 0000:00:1c.2
> in 1 ms rather than 2 s, and the systemd "kernel" boot phase goes from
> 3.011 s to 1.036 s.
> 
> Fixes: 72780f796468 ("PCI: Always lift 2.5GT/s restriction in PCIe failed link retraining")
> Cc: stable@vger.kernel.org
> Reported-by: Andreas Wild <andiwild@gmail.com>
> Closes: https://lore.kernel.org/lkml/20260801092152.5643-1-andiwild@gmail.com/
> Tested-by: Andreas Wild <andiwild@gmail.com>
> Signed-off-by: Maciej W. Rozycki <macro@orcam.me.uk>
> Signed-off-by: Andreas Wild <andiwild@gmail.com>
> ---
> v3:
>  - Replaced with Maciej's much simpler approach: bail out of the quirk
>    entirely when the Target Link Speed is already clamped at 2.5GT/s and no
>    link has been established, rather than programming the speed and skipping
>    only the retraining.  One function, no new API, no bwctrl changes.
>  - Note this leaves the Target Link Speed clamped on such a port, where v2
>    left it at the Port's maximum.  A device hot-plugged there later trains at
>    2.5GT/s: pcie_wait_for_link_delay() only calls the quirk when
>    pcie_wait_for_link_status() fails, so a link that comes up cleanly at
>    2.5GT/s never re-runs it.  Flagging in case that matters; the clamp is
>    firmware's, so honouring it on an unoccupied Port seems defensible.
>  - v2: https://lore.kernel.org/lkml/20260801105441.6506-1-andiwild@gmail.com/
>  - v1: https://lore.kernel.org/lkml/20260801092152.5643-1-andiwild@gmail.com/
>  drivers/pci/quirks.c | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> index b09f27f..9e407c4 100644
> --- a/drivers/pci/quirks.c
> +++ b/drivers/pci/quirks.c
> @@ -108,7 +108,11 @@ int pcie_failed_link_retrain(struct pci_dev *dev)
>  
>  	pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &lnksta);
>  	pcie_capability_read_word(dev, PCI_EXP_LNKCTL2, &oldlnkctl2);
> -	if (!(lnksta & PCI_EXP_LNKSTA_DLLLA) && pcie_lbms_seen(dev, lnksta)) {
> +	if (lnksta & PCI_EXP_LNKSTA_DLLLA) {
> +		;
> +	} else if (PCIE_LNKCTL2_TLS2SPEED(oldlnkctl2) == PCIE_SPEED_2_5GT) {
> +		return ret;
> +	} else if (pcie_lbms_seen(dev, lnksta)) {
>  		pci_info(dev, "broken device, retraining non-functional downstream link at 2.5GT/s\n");
>  		ret = pcie_set_target_speed(dev, PCIE_SPEED_2_5GT, false);
>  		if (ret)

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[Maciej W. Rozycki]

On Mon, 3 Aug 2026, Thorsten Leemhuis wrote:

> > Since commit 72780f796468 ("PCI: Always lift 2.5GT/s restriction in PCIe
> > failed link retraining") the Target Speed quirk lifts a firmware-imposed
> > 2.5GT/s restriction on any downstream port, without checking whether the
> > link is up.  Where nothing is plugged in, the retraining that follows can
> > never complete, so each attempt costs PCIE_LINK_RETRAIN_TIMEOUT_MS.  The
> > quirk makes two of them -- the initial one and the restore on the error
> > path -- adding a fixed 2 s to every boot.
> 
> TWIMC, this patch fixes a regression Edward reported here:
> https://bugzilla.kernel.org/show_bug.cgi?id=221801

 Thank you for the pointer.

 As it happens attempts were made to cc me on that bug, which however went 
nowhere as messages were sent to my long-defunct <macro@linux-mips.org> 
e-mail address, a clear oversight of mine as I went, back in the day, 
through numerous sites to get this updated.

 I have now asked bugzilla admins to help me recover my account.

> [side note: the report could be more specific, yes -- I wanted to ask
> Edward to clarify a few things (like the actual slowdown) before
> forwarding it, but it didn't came to that when I noticed this patch and
> asked Edward to just check if it helped -- which is did.]

 Tested-by tags welcome!

 And thank you for your input overall, really useful and appreciated!

  Maciej

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[Thorsten Leemhuis]

On 8/4/26 00:07, Maciej W. Rozycki wrote:
> On Mon, 3 Aug 2026, Thorsten Leemhuis wrote:
>>> Since commit 72780f796468 ("PCI: Always lift 2.5GT/s restriction in PCIe
>>> failed link retraining") the Target Speed quirk lifts a firmware-imposed
>>> 2.5GT/s restriction on any downstream port, without checking whether the
>>> link is up.  Where nothing is plugged in, the retraining that follows can
>>> never complete, so each attempt costs PCIE_LINK_RETRAIN_TIMEOUT_MS.  The
>>> quirk makes two of them -- the initial one and the restore on the error
>>> path -- adding a fixed 2 s to every boot.
>>
>> TWIMC, this patch fixes a regression Edward reported here:
>> https://bugzilla.kernel.org/show_bug.cgi?id=221801
> 
>  Thank you for the pointer.

yw
>  As it happens attempts were made to cc me on that bug, which however went 
> nowhere as messages were sent to my long-defunct <macro@linux-mips.org> 
> e-mail address, a clear oversight of mine as I went, back in the day, 
> through numerous sites to get this updated.
> 
>  I have now asked bugzilla admins to help me recover my account.

FWIW, Artem does that and I'm not a fan of it at all -- I think it is
only appropriate when one knows the person occasionally interacts
directly (e.g. not via mail) with bugzilla (and thus I only do it in
those cases).
>> [side note: the report could be more specific, yes -- I wanted to ask
>> Edward to clarify a few things (like the actual slowdown) before
>> forwarding it, but it didn't came to that when I noticed this patch and
>> asked Edward to just check if it helped -- which is did.]
> 
>  Tested-by tags welcome!

FWIW, as I forgot to mention it easlier: Edward is CCed.

@Edward, if you'd like a record of your test result end up in the
history, reply with

 Tested-by: Full Name <email address>

to this thread, just omit the space in the beginning and change it
accordingly.

Ciao, Thorsten

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[Aoxtj]

Hi,

I'm the one who reported a PCIe/NVMe boot regression on the Proxmox forum:

  
https://forum.proxmox.com/threads/7-0-14-7-kernel-breaks-pci-link-training.185549/

A Proxmox dev pointed me here and asked me to try this patch. I'm not a 
kernel developer and did most of the debugging with AI help, so please 
bear with me.

TL;DR:
- Proxmox Kernel 7.0.14-6 to 7.0.14-8: SK hynix system drive disappears 
during boot.
- Narrowed down to commit 72780f796468 and verified by reverting it.
- 7.0.14-8 + revert 72780f796468: boots reliably.
- 7.0.14-8 + this v3 patch: only boots sometimes; failed boots lose the 
drive the same way.

Machine: ASRock B450M Pro4-F, SK hynix PE4010 NVMe system drive on a 
passive PCIe bifurcation card.

   AMD Renoir PCIe GPP Bridge 0000:00:02.4 [1022:1633]
   -> bus 07 -> SK hynix PE4010 0000:07:00.0 [1c5c:2527]

Initramfs dmesg from a failed v3 boot:

   [    0.555805] pci 0000:00:02.4: [1022:1633] type 01 class 0x060400 
PCIe Root Port
   [    0.555828] pci 0000:00:02.4: PCI bridge to [bus 07]
   [    0.555871] pci 0000:00:02.4: removing 2.5GT/s downstream link 
speed restriction
   [    1.636965] pci 0000:00:02.4: retraining failed

Nothing shows up on bus 07 afterwards, so my system drive is gone.

With the revert instead, the working link reports:

   Root Port 0000:00:02.4:
     LnkCap:  Speed 8GT/s, Width x4
     LnkSta:  Speed 2.5GT/s, Width x2, DLActive+
     LnkCtl2: Target Link Speed: 2.5GT/s, SpeedDis+
     LnkSta2: EqualizationComplete-, EqualizationPhase1/2/3-

   Endpoint 0000:07:00.0:
     LnkCap:  Speed 8GT/s, Width x4
     LnkSta:  Speed 2.5GT/s (downgraded), Width x2 (downgraded)
     LnkCtl2: Target Link Speed: 8GT/s
     LnkSta2: EqualizationComplete-, EqualizationPhase1/2/3-

My guess: this motherboard supports different PCIe generations depending 
on which CPU is installed, and with this CPU the link just can't run at 
the advertised 8GT/s — which would explain why the 8GT/s retrain fails.

Tested on a locally rebuilt Proxmox/Ubuntu 7.0.14-8 backport, not 
mainline. Also replied in kernel Bugzilla #221801:

   https://bugzilla.kernel.org/show_bug.cgi?id=221801

Happy to test a diagnostic patch.

Best regards,
Aoxtj

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[Andreas Wild]

On Sat, 1 Aug 2026, Aoxtj wrote:

> - 7.0.14-8 + revert 72780f796468: boots reliably.
> - 7.0.14-8 + this v3 patch: only boots sometimes; failed boots lose the
> drive the same way.

Thanks for testing it.  v3 only covers ports with no link at all, so it
does not address your case.

72780f796468 removed two guards at once: the Data Link Layer Link Active
check, and the device ID match against the ASMedia ASM2824.  My
regression comes from losing the first; yours looks like it comes from
losing the second.

v3 only bails out where DLLLA is clear.  Your link is up:

>      LnkSta:  Speed 2.5GT/s, Width x2, DLActive+

so the quirk runs past the v3 early return, lifts the restriction and
retrains at 8GT/s, exactly as it did before.  Reverting works for you
because your AMD Renoir Root Port didn't match the ASM2824 ID list that
used to gate this.

A possible workaround:

If your BIOS lets you pin that slot to Gen1, you could try that.  With
the Root Port advertising only 2.5GT/s the quirk never reaches the
retrain at all: on current mainline it returns at

	speed_cap = pcie_get_speed_cap(dev);
	if (speed_cap <= PCIE_SPEED_2_5GT)
		return ret;

and on the older code your 7.0.14 backport is based on, the
(lnkcap & PCI_EXP_LNKCAP_SLS) != PCI_EXP_LNKCAP_SLS_2_5GB condition is
false, so the restriction is never lifted.

Since the link already runs at 2.5GT/s x2, that should cost you nothing
- it just makes the Root Port advertise what the link can actually do.

Something that is different in your case:

Your Root Port reports

>      LnkCtl2: Target Link Speed: 2.5GT/s, SpeedDis+

where mine reports the same clamp with SpeedDis-.  SpeedDis is
PCI_EXP_LNKCTL2_HASD, and as far as I can tell the quirk never looks at
it; the only user in the tree is pcie-designware.c.  Firmware setting
both the clamp and Hardware Autonomous Speed Disable looks like a
deliberate pin rather than an incidental one.

I am aware that HASD is specified as disabling *hardware autonomous*
speed changes and so does not literally forbid a software-initiated
retrain, so this is just a heuristic about firmware intent.

About the drive disappearing:

The error path is

	err:
		pci_info(dev, "retraining failed\n");
		pcie_set_target_speed(dev, PCIE_LNKCTL2_TLS2SPEED(oldlnkctl2), true);

which reaches pcie_retrain_link(pdev, use_lt=true), whose last wait is

	rc = pcie_wait_for_link_status(pdev, use_lt, !use_lt);

With use_lt set that waits for the Link Training bit to clear (for
training to stop) and not for DLLLA to come back.  So the restore can
return having never confirmed the link recovered.  Since the quirk runs
from pci_device_add(), before pci_scan_bridge_extend() creates the
subordinate bus, a link that comes up shortly afterwards is never
scanned and the device behind it is simply not there.  That would fit
only booting sometimes better than anything about speed policy does.

I could easily be wrong about this - I am just reading the code, and I
came to the PCI core a few days ago via this one bug.

Since you offered to test a diagnostic, here is one.  It only touches the
err: block, which is identical in mainline and in the 7.0.14 code your
backport is based on, so it should apply to your tree (it applied to
v7.1.5 here with a -2 line offset):

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index b09f27f..e4fcdfb 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -126,7 +126,32 @@ int pcie_failed_link_retrain(struct pci_dev *dev)
 	return ret;
 err:
 	pci_info(dev, "retraining failed\n");
+	{
+		/* DIAGNOSTIC ONLY -- not for merging */
+		u16 sta = 0, ctl2 = 0;
+
+		pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &sta);
+		pcie_capability_read_word(dev, PCI_EXP_LNKCTL2, &ctl2);
+		pci_info(dev, "diag: pre ret=%d DLLLA=%d sta=%#06x ctl2=%#06x\n",
+			 ret, !!(sta & PCI_EXP_LNKSTA_DLLLA), sta, ctl2);
+	}
 	pcie_set_target_speed(dev, PCIE_LNKCTL2_TLS2SPEED(oldlnkctl2), true);
+	{
+		/* DIAGNOSTIC ONLY -- not for merging */
+		u16 sta = 0, ctl2 = 0;
+		int i;
+
+		for (i = 0; i <= 100; i++) {
+			pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &sta);
+			if (sta & PCI_EXP_LNKSTA_DLLLA)
+				break;
+			msleep(10);
+		}
+		pcie_capability_read_word(dev, PCI_EXP_LNKCTL2, &ctl2);
+		pci_info(dev, "diag: post DLLLA=%d after %dms sta=%#06x\n",
+			 !!(sta & PCI_EXP_LNKSTA_DLLLA), i * 10, sta);
+		pci_info(dev, "diag: post ctl2=%#06x\n", ctl2);
+	}
 	return ret;
 }


Expected output:

- "diag: pre" gives the error the retrain returned and whether the
  link was already down at that point.
- "diag: post DLLLA=1 after <n>ms" would mean the link does come back,
  just not before pci_device_add() returns and the bus below is
  scanned.  That would point at the error path needing to wait for
  DLLLA rather than only for the Link Training bit to clear.
- "diag: post DLLLA=0 after 1000ms" would mean the link is genuinely
  down and staying down, which is a different problem and
  probably a worse one.

One caveat: the poll loop waits up to a second in the error path, so it
changes timing.  If the diagnostic build happens to boot reliably where
the plain v3 build did not, that is also worth reporting.

Best regards,
Andreas Wild

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[Aoxtj]

On 2026/8/8 14:31, Andreas Wild wrote:
> Expected output:
>
> - "diag: pre" gives the error the retrain returned and whether the
>    link was already down at that point.
> - "diag: post DLLLA=1 after <n>ms" would mean the link does come back,
>    just not before pci_device_add() returns and the bus below is
>    scanned.  That would point at the error path needing to wait for
>    DLLLA rather than only for the Link Training bit to clear.
> - "diag: post DLLLA=0 after 1000ms" would mean the link is genuinely
>    down and staying down, which is a different problem and
>    probably a worse one.
>
> One caveat: the poll loop waits up to a second in the error path, so it
> changes timing.  If the diagnostic build happens to boot reliably where
> the plain v3 build did not, that is also worth reporting.
Diagnostic result: link is genuinely down, not recovering.

Tested 7.0.14-8-pve++ (v3 patch + diagnostic patch) on the same machine. 
Boot fails: SK hynix never enumerates on bus 07.

[0.542354] pci 0000:00:02.4: removing 2.5GT/s downstream link speed 
restriction
[1.623696] pci 0000:00:02.4: retraining failed
[1.623701] diag: pre  ret=-110 DLLLA=0 sta=0x9023 ctl2=0x0023
[3.734697] diag: post DLLLA=0 after 1010ms sta=0x9823
[3.734702] diag: post ctl2=0x0021

Sorry for taking so long to reply; I've been a little busy this week. 
Happy to test anything else.

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[Andreas Wild]

On Mon, 11 Aug 2026, Aoxtj wrote:

> Diagnostic result: link is genuinely down, not recovering.
>
> [1.623701] diag: pre  ret=-110 DLLLA=0 sta=0x9023 ctl2=0x0023
> [3.734697] diag: post DLLLA=0 after 1010ms sta=0x9823
> [3.734702] diag: post ctl2=0x0021

So the link really isn't coming back at all rather than too late for the
bus scan.

  after the failed 8GT/s retrain (ret=-110, -ETIMEDOUT):
    LNKSTA  0x9023   CLS 8.0GT/s  width x2  LT 0  DLLLA 0
    LNKCTL2 0x0023   TLS 8.0GT/s  HASD set

  after the restore, plus 1010 ms of polling:
    LNKSTA  0x9823   CLS 8.0GT/s  width x2  LT 1  DLLLA 0
    LNKCTL2 0x0021   TLS 2.5GT/s  HASD set

The register restore works, LNKCTL2 goes back to 2.5GT/s, but LT is still
asserted a second later, with DLLLA never returning, so the port is stuck
in link training rather than merely slow to recover.

On the HASD bit:

Your LNKCTL2 has bit 5 (PCI_EXP_LNKCTL2_HASD, "SpeedDis+") set in both
samples.  So firmware clamped the Target Link Speed to 2.5GT/s and also
disabled hardware autonomous speed changes, on a link that turns out not
to work at 8GT/s.

If you would like something to try, the change below skips lifting the
restriction if HASD is set.  It applies to your tree (I checked it against
v7.1.6 and on top of v3). Mainline would need a small adaptation, since
that block no longer has the LNKCAP test.

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -115,6 +115,11 @@ int pcie_failed_link_retrain(struct pci_dev *dev)
 	pcie_capability_read_dword(dev, PCI_EXP_LNKCAP, &lnkcap);
 	if ((lnkctl2 & PCI_EXP_LNKCTL2_TLS) == PCI_EXP_LNKCTL2_TLS_2_5GT &&
 	    (lnkcap & PCI_EXP_LNKCAP_SLS) != PCI_EXP_LNKCAP_SLS_2_5GB) {
+		if (lnkctl2 & PCI_EXP_LNKCTL2_HASD) {
+			pci_info(dev, "2.5GT/s restriction left, firmware set HASD\n");
+			return ret;
+		}
+
 		pci_info(dev, "removing 2.5GT/s downstream link speed restriction\n");
 		ret = pcie_set_target_speed(dev, PCIE_LNKCAP_SLS2SPEED(lnkcap), false);
 		if (ret)

Some things to consider:

I cannot verify the change myself - it compiles without warnings and that
is all I can say.

HASD is specified as disabling *hardware autonomous* speed changes, so it
does not strictly forbid a software-initiated retrain.  Interpreting it as
"firmware meant this" is just my guess.

As I said before, I have no background in this code.  I came to it through
one boot-time regression on my own machine, and that is the extent of it.

So please take it as "here is one thing that could help this particular
case", not as a view on how the quirk ought to work.  Maciej and Bjorn are
far better placed to judge whether the answer is this, or restoring the
device ID match, or something else entirely.

One further thought for people who know the code better than I do:

Since retraining apparently cannot recover the link once it is wedged,
would a secondary bus reset be the appropriate recovery in the error path?

Best regards,
Andreas

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[Thorsten Leemhuis]

On 8/11/26 08:42, Andreas Wild wrote:
> On Mon, 11 Aug 2026, Aoxtj wrote:
> 
>> Diagnostic result: link is genuinely down, not recovering.
>>
>> [1.623701] diag: pre  ret=-110 DLLLA=0 sta=0x9023 ctl2=0x0023
>> [3.734697] diag: post DLLLA=0 after 1010ms sta=0x9823
>> [3.734702] diag: post ctl2=0x0021
> 
> So the link really isn't coming back at all rather than too late for the
> bus scan.
> [...]
> If you would like something to try, the change below skips lifting the
> restriction if HASD is set.  It applies to your tree (I checked it against
> v7.1.6 and on top of v3). Mainline would need a small adaptation, since
> that block no longer has the LNKCAP test.

Side note: what's the status of all this? From the outside things looks
stalled, but maybe there was progress and I just missed it.

Are we waiting for a reply from Aoxtj before moving on with the fix at
the start of the thread? And whatever the answer is: would it make sense
to move on with that independently to get the other regression (boot
time delay) resolved?

Ciao, Thorsten
> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> --- a/drivers/pci/quirks.c
> +++ b/drivers/pci/quirks.c
> @@ -115,6 +115,11 @@ int pcie_failed_link_retrain(struct pci_dev *dev)
>  	pcie_capability_read_dword(dev, PCI_EXP_LNKCAP, &lnkcap);
>  	if ((lnkctl2 & PCI_EXP_LNKCTL2_TLS) == PCI_EXP_LNKCTL2_TLS_2_5GT &&
>  	    (lnkcap & PCI_EXP_LNKCAP_SLS) != PCI_EXP_LNKCAP_SLS_2_5GB) {
> +		if (lnkctl2 & PCI_EXP_LNKCTL2_HASD) {
> +			pci_info(dev, "2.5GT/s restriction left, firmware set HASD\n");
> +			return ret;
> +		}
> +
>  		pci_info(dev, "removing 2.5GT/s downstream link speed restriction\n");
>  		ret = pcie_set_target_speed(dev, PCIE_LNKCAP_SLS2SPEED(lnkcap), false);
>  		if (ret)
> 
> Some things to consider:
> 
> I cannot verify the change myself - it compiles without warnings and that
> is all I can say.
> 
> HASD is specified as disabling *hardware autonomous* speed changes, so it
> does not strictly forbid a software-initiated retrain.  Interpreting it as
> "firmware meant this" is just my guess.
> 
> As I said before, I have no background in this code.  I came to it through
> one boot-time regression on my own machine, and that is the extent of it.
> 
> So please take it as "here is one thing that could help this particular
> case", not as a view on how the quirk ought to work.  Maciej and Bjorn are
> far better placed to judge whether the answer is this, or restoring the
> device ID match, or something else entirely.
> 
> One further thought for people who know the code better than I do:
> 
> Since retraining apparently cannot recover the link once it is wedged,
> would a secondary bus reset be the appropriate recovery in the error path?
> 
> Best regards,
> Andreas
> 

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[Andreas Wild]

On Mon, 24 Aug 2026, Thorsten Leemhuis wrote:

> Side note: what's the status of all this? From the outside things looks
> stalled, but maybe there was progress and I just missed it.

Nothing has moved, so you have not missed anything.

> Are we waiting for a reply from Aoxtj before moving on with the fix at
> the start of the thread? And whatever the answer is: would it make sense
> to move on with that independently to get the other regression (boot
> time delay) resolved?

Not on my side.  Aoxtj tested v3 early on and it does not help them, which
is expected: their link is up at 2.5GT/s and v3 only skips ports with no
link at all.  The two are separate failure modes needing separate fixes,
so I see no reason to hold v3 back.

Best regards,
Andreas

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[Maciej W. Rozycki]

On Tue, 25 Aug 2026, Andreas Wild wrote:

> > Are we waiting for a reply from Aoxtj before moving on with the fix at
> > the start of the thread? And whatever the answer is: would it make sense
> > to move on with that independently to get the other regression (boot
> > time delay) resolved?
> 
> Not on my side.  Aoxtj tested v3 early on and it does not help them, which
> is expected: their link is up at 2.5GT/s and v3 only skips ports with no
> link at all.  The two are separate failure modes needing separate fixes,
> so I see no reason to hold v3 back.

 I'd be happy to hear any results of your posted HASD check though.

  Maciej

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[Thorsten Leemhuis]

On 8/25/26 12:18, Maciej W. Rozycki wrote:
> On Tue, 25 Aug 2026, Andreas Wild wrote:
> 
>>> Are we waiting for a reply from Aoxtj before moving on with the fix at
>>> the start of the thread? And whatever the answer is: would it make sense
>>> to move on with that independently to get the other regression (boot
>>> time delay) resolved?
>>
>> Not on my side.  Aoxtj tested v3 early on and it does not help them, which
>> is expected: their link is up at 2.5GT/s and v3 only skips ports with no
>> link at all.  The two are separate failure modes needing separate fixes,
>> so I see no reason to hold v3 back.

Thx for the clarification. Would afaics be really good to get that fix
in to solve that problem. Bjorn, is this somewhere on your todo list?
>  I'd be happy to hear any results of your posted HASD check though.
There is another report from Mich (now CCed). Mich, can you maybe test
that if you haven't yet (your report is really verbose, which makes
things somewhat hard to follow)?

FWIW, Mich's report can be found here:
https://bugzilla.kernel.org/show_bug.cgi?id=221919

Quoting from there:
"""
>  Mich 2026-08-25 09:56:05 UTC
> 
> Created attachment 310724 [details]
> Various collection of log files.
> 
> Overview
> ========
> After the Vendor:Device ID matching was removed from the
> pcie_failed_link_retrain() quirk, the kernel now lifts 2.5GT/s link speed
> clamps that platform firmware has deliberately applied. On a Dell PowerEdge
> R610 this causes the affected links to fail to come back up, the device
> behind them to disappear from the PCI bus, and a fatal PCIe error resulting
> in a kernel panic.
> 
> Kernel version
> ==============
> Failing : 7.0.14-12-pve (Proxmox, based on Ubuntu-7.0.0-31.31)
> Working : 7.0.14-5-pve  (Proxmox, based on an earlier Ubuntu base)
> 
> The Proxmox changelog shows 7.0.14-7 as the release that pulled in the large
> upstream update (Ubuntu-7.0.0-28.28i3); all later releases in that series are
> CVE cherry-picks only. Forum reports independently confirm -6 works and -7
> onwards do not.
> 
> Hardware
> ========
> Dell PowerEdge R610 (11G)
>   Chipset : Intel 5520/5500 I/O Hub
>   BIOS    : 6.4.0 (2013-07-23) on one machine, 6.6.0 (latest) on the other
>   NIC     : Intel 82571EB quad-port PCIe (the device that disappears)
>   RAID    : LSI MegaRAID SAS 2108 (PERC H700)
> 
> Reproduced on two separate R610 machines with different BIOS versions, so
> this is not a firmware revision issue.
> 
> What happens
> ============
> [    1.666183] pci 0000:00:07.0: removing 2.5GT/s downstream link speed restriction
>                <- exactly 1 second gap (link training timeout)
> [    2.666510] pci 0000:00:09.0: removing 2.5GT/s downstream link speed restriction
> [    2.673748] Uhhuh. NMI received for unknown reason 21 on CPU 0.
> 
> Resulting link state (captured while booted with ghes.disable=y):
>   00:07.0  LnkCap: Speed 5GT/s, Width x8  /  LnkSta: Speed 2.5GT/s, Width x0
>   00:09.0  LnkCap: Speed 5GT/s, Width x8  /  LnkSta: Speed 2.5GT/s, Width x0
> 
> Width x0 means the link is down. The Intel 82571EB installed behind one of
> these ports is not enumerated at all - it is completely absent from lspci.
> Only the onboard BCM5709 ports remain visible.
> 
> Without ghes.disable=y the machine panics before reaching userspace:
>   Kernel panic - not syncing: GHES: Fatal hardware error
> The call trace shows this happens during acpi_ghes_init:
>   acpi_ghes_init+0xd3/0x160
>   acpi_init+0x401/0x490
>   do_one_initcall+0x5f/0x350
>   kernel_init_freeable+0x243/0x2f0
> 
> The front panel LCD and iDRAC SEL both report "E171F PCIe fatal".
> 
> The ACPI BERT table is empty (48 bytes, header only), confirming this is a
> live error raised during PCI enumeration rather than a stale firmware record.
> 
> Suspected cause
> ===============
> Commit a89c82249c37 ("PCI: Work around PCIe link training failures")
> introduced pcie_failed_link_retrain(), originally limited to the ASMedia
> ASM2824 by Vendor:Device ID.
> 
> A later change - "PCI: Always lift 2.5GT/s restriction in PCIe failed link
> retraining" (Maciej W. Rozycki, Dec 2025) - removed that ID matching, so the
> quirk now lifts 2.5GT/s clamps on any downstream port, explicitly including
> clamps "the firmware may have already arranged".
> 
> On this platform the firmware sets that clamp intentionally and the hardware
> cannot negotiate a higher speed, so lifting it takes the link down entirely.
> 
> During review of that patch, concerns were raised on linux-pci that lifting
> firmware-set clamps on arbitrary hardware would cause regressions on real
> systems. This appears to be exactly such a case.
> 
> Workarounds tried
> =================
> ghes.disable=y : machine boots, but the NIC is still missing, so the node is
>                  unusable. Only suppresses the reporting.
> pci=noaer      : no effect (the link is already down before AER is involved).
> Pinning kernel 7.0.14-5-pve : works, and is what we are currently using.
> 
> Question
> ========
> Would it be possible to restrict this quirk back to the original ASMedia
> device IDs, or to skip lifting clamps that were set by firmware rather than
> by the quirk itself? On platforms where firmware deliberately limits a link,
> overriding that appears to be unsafe.
> 
> Attachments
> ===========
> Full dmesg, lspci -nnvvv, dmidecode, ACPI tables (BERT/HEST) attached.
> 
> Comment 1 The Linux kernel's regression tracker (Thorsten Leemhuis) 2026-08-26 04:07:17 UTC
> 
> Please use latest upstream kernels when reporting problem upstream, those downstream kernels are patched in various ways and thus might have issues that never even happened upstream or were fixed long ago. Nevertheless:
> 
> There is some discussion about an issue that might be the same here here: https://lore.kernel.org/all/20260801201244.4421-1-andiwild@gmail.com/
> 
> Would be best if you could try if the fix or one of the test-patches in that thread helps and report back there; alternatively, report back here please, ideally while allowing me to CC you on a mail about this (which would expose your email to the world)
> 
> Comment 2 Mich 2026-08-26 04:59:18 UTC
> 
> (In reply to The Linux kernel's regression tracker (Thorsten Leemhuis) from comment #1)
>> Please use latest upstream kernels when reporting problem upstream, those
>> downstream kernels are patched in various ways and thus might have issues
>> that never even happened upstream or were fixed long ago. Nevertheless:
>> 
>> There is some discussion about an issue that might be the same here here:
>> https://lore.kernel.org/all/20260801201244.4421-1-andiwild@gmail.com/
>> 
>> Would be best if you could try if the fix or one of the test-patches in that
>> thread helps and report back there; alternatively, report back here please,
>> ideally while allowing me to CC you on a mail about this (which would expose
>> your email to the world)
> 
> Thank you for the response, I will take a look at it!
> I tried adding my "spam" collection mailbox, but I am not allowed unfortunately. If you want to add this to CC "
> restafvalendergelijke@outlook.com" it's no problem. Thank you once again.
> 
> Comment 3 Mich 2026-09-01 09:46:23 UTC
> 
> Created attachment 310765 [details]
> Full diagnostics attached (dmesg, lspci -nnvvv, dmidecode, ACPI tables).
> 
> == Second platform, and a case the proposed v3 fix would NOT cover ==
> 
> Reproduced on a second machine from a different vendor:
> 
>   HP ProLiant DL360 G7, BIOS P68 (2015-08-16)
>   Intel 5520 I/O Hub (same chipset generation as the Dell R610)
>   Intel 82571EB quad-port NIC [8086:10bc] - same card as the R610s
>   Kernel 7.0.14-14-pve (Proxmox, Ubuntu-based). 7.0.14-5-pve works.
> 
> This rules out a Dell-specific firmware quirk: two vendors, different BIOS
> versions, same Intel 5520 chipset, identical failure mode.
> 
> == The important difference from my earlier R610 report ==
> 
> On the R610 the affected ports were EMPTY: LnkSta Width x0, DLActive- , no
> device behind them. That case is covered by the v3 approach (bail out when
> Target Link Speed is already 2.5GT/s and no link has been established).
> 
> On this HP machine one of the affected ports had an ACTIVE link with a
> device behind it before the quirk ran:
> 
>   00:08.0  before: LnkCap 5GT/s x4, LnkSta 2.5GT/s Width x4, DLActive+
>                    (Intel 82571EB, bus 04, working normally on 7.0.14-5)
>            after : LnkCap 5GT/s x4, LnkSta 2.5GT/s Width x0
>                    (bus 04 now empty, NIC absent from lspci)
> 
> Since DLLLA was set, pcie_failed_link_retrain() would take the first branch
> of the v3 patch and fall through to the clamp-removal block regardless. The
> clamp is then lifted on a port with a working device attached, the link
> fails to retrain, and the device is lost.
> 
> If that reading is correct, v3 fixes the empty-port case but not this one.
> I'm happy to be corrected - I'm reporting observed behaviour, not reading
> the patch as an expert.
> 
> == Full quirk trace on the HP ==
> 
> The quirk walks every root port, each with a one-second retrain timeout:
> 
>   [ 1.883010] pci 0000:00:02.0: removing 2.5GT/s downstream link speed restriction
>   [ 2.882097] pci 0000:00:02.0: retraining failed
>   [ 2.882422] pci 0000:00:03.0: removing 2.5GT/s downstream link speed restriction
>   [ 3.882097] pci 0000:00:03.0: retraining failed
>   [ 3.882441] pci 0000:00:04.0: removing 2.5GT/s downstream link speed restriction
>   [ 4.882097] pci 0000:00:04.0: retraining failed
>   [ 4.882417] pci 0000:00:05.0: removing 2.5GT/s downstream link speed restriction
>   [ 5.882097] pci 0000:00:05.0: retraining failed
>   [ 5.882409] pci 0000:00:06.0: removing 2.5GT/s downstream link speed restriction
>   [ 6.882097] pci 0000:00:06.0: retraining failed
>   [ 6.882425] pci 0000:00:07.0: removing 2.5GT/s downstream link speed restriction
>   [ 6.882746] pci 0000:00:08.0: removing 2.5GT/s downstream link speed restriction
>   [ 7.882097] pci 0000:00:08.0: retraining failed
>   [ 7.882415] pci 0000:00:09.0: removing 2.5GT/s downstream link speed restriction
>   [ 7.882742] pci 0000:00:0a.0: removing 2.5GT/s downstream link speed restriction
>   [ 7.906501] NMI: IOCK error (debug interrupt?) for reason 61 on CPU 0.
>   [ 8.882097] pci 0000:00:0a.0: retraining failed
> 
> Note 00:07.0 gets the same treatment but is NOT reported as failed - the
> onboard BCM5709 behind it survives at Width x4. So on the same machine, in
> the same boot, one device survives the retrain and another does not.
> 
> Also of note: roughly 7 seconds of boot time are spent on these one-second
> timeouts, on a machine where nothing needed retraining in the first place.
> 
> == Firmware clamps every port on both platforms ==
> 
> Dell R610 (working kernel, quirk not triggered):
>   00:01.0  LnkCap 5GT/s x4  /  LnkSta 2.5GT/s, Width x4   BCM5709, link up
>   00:03.0  LnkCap 5GT/s x4  /  LnkSta 2.5GT/s, Width x4   BCM5709, link up
>   00:07.0  LnkCap 5GT/s x8  /  LnkSta 2.5GT/s, Width x0   empty
>   00:09.0  LnkCap 5GT/s x8  /  LnkSta 2.5GT/s, Width x0   empty (Intel NIC on the failing node)
> 
> HP DL360 G7: same pattern across nine root ports.
> 
> Both platforms clamp every 5GT/s-capable port to 2.5GT/s, occupied or not.
> Moving the card to another slot is therefore not a workaround.
> 
> Full diagnostics attached (dmesg, lspci -nnvvv, dmidecode, ACPI tables).
> 
"""

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[Maciej W. Rozycki]

On Wed, 2 Sep 2026, Thorsten Leemhuis wrote:

> Thx for the clarification. Would afaics be really good to get that fix
> in to solve that problem. Bjorn, is this somewhere on your todo list?
> >  I'd be happy to hear any results of your posted HASD check though.
> There is another report from Mich (now CCed). Mich, can you maybe test
> that if you haven't yet (your report is really verbose, which makes
> things somewhat hard to follow)?

 Thank you for the update.  I'm back from holiday today and will review
the situation in detail in the coming few days.

  Maciej

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[Bjorn Helgaas]

[+cc M M, Thorsten]

On Sat, Aug 01, 2026 at 10:11:48PM +0200, Andreas Wild wrote:
> From: "Maciej W. Rozycki" <macro@orcam.me.uk>
> 
> Since commit 72780f796468 ("PCI: Always lift 2.5GT/s restriction in PCIe
> failed link retraining") the Target Speed quirk lifts a firmware-imposed
> 2.5GT/s restriction on any downstream port, without checking whether the
> link is up.  Where nothing is plugged in, the retraining that follows can
> never complete, so each attempt costs PCIE_LINK_RETRAIN_TIMEOUT_MS.  The
> quirk makes two of them -- the initial one and the restore on the error
> path -- adding a fixed 2 s to every boot.
> 
> On an MSI PRO Z690-A WIFI DDR4 (Intel 600 Series PCH) with one empty x1
> slot, running v7.2-rc5:
> 
>  0.541 pci 0000:00:1c.0: removing 2.5GT/s downstream link speed restriction
>  1.541 pci 0000:00:1c.0: retraining failed
>  2.541 pci 0000:00:1c.2: [8086:7aba] type 01 class 0x060400
> 
> Where the Link Speed has already been clamped at 2.5GT/s and no link has
> been established there is nothing worth doing, which is what the kerneldoc
> for the quirk already describes: the restriction is to be lifted where
> firmware arranged it "and the port reports its link already being up".
> Bail out early in that case, before either the ASM2824 workaround or the
> removal of the restriction is considered.
> 
> Ports whose link is up are unaffected, and so is the ASM2824 workaround,
> which is reached with the Target Link Speed not clamped.
> 
> With this applied the quirk returns without touching the port: both
> messages are gone, enumeration proceeds from 0000:00:1c.0 to 0000:00:1c.2
> in 1 ms rather than 2 s, and the systemd "kernel" boot phase goes from
> 3.011 s to 1.036 s.
> 
> Fixes: 72780f796468 ("PCI: Always lift 2.5GT/s restriction in PCIe failed link retraining")
> Cc: stable@vger.kernel.org
> Reported-by: Andreas Wild <andiwild@gmail.com>
> Closes: https://lore.kernel.org/lkml/20260801092152.5643-1-andiwild@gmail.com/
> Tested-by: Andreas Wild <andiwild@gmail.com>
> Signed-off-by: Maciej W. Rozycki <macro@orcam.me.uk>
> Signed-off-by: Andreas Wild <andiwild@gmail.com>

I provisionally applied this on pci/misc.  It seems to address at
least part of a v7.2 regression, so it probably should go on
pci/for-linus for v7.3.

But I'd like to have it in -next for a bit before asking Linus to pull
it.

I added:

  Closes: https://bugzilla.kernel.org/show_bug.cgi?id=221801

based on Edward's testing at
https://bugzilla.kernel.org/show_bug.cgi?id=221801#c13.  Edward, I
bcc'd you because I haven't seen a public email response, but if you
want your Tested-by to appear in the git commit, just respond to this
email thread.

But IIUC Aoxtj still sees issues with 72780f796468 ("PCI: Always lift
2.5GT/s restriction in PCIe failed link retraining") even with this
patch
(https://lore.kernel.org/all/331e97c7-e422-420d-9f3e-5d9f734464b3@axtjblog.cc)

And M M / Mich (BCC'd) also reports an issue that (IIUC) is not fixed by
this patch (https://bugzilla.kernel.org/show_bug.cgi?id=221919#c3)

So evidently there are still issues but maybe this patch fixes part of
them?

> ---
> v3:
>  - Replaced with Maciej's much simpler approach: bail out of the quirk
>    entirely when the Target Link Speed is already clamped at 2.5GT/s and no
>    link has been established, rather than programming the speed and skipping
>    only the retraining.  One function, no new API, no bwctrl changes.
>  - Note this leaves the Target Link Speed clamped on such a port, where v2
>    left it at the Port's maximum.  A device hot-plugged there later trains at
>    2.5GT/s: pcie_wait_for_link_delay() only calls the quirk when
>    pcie_wait_for_link_status() fails, so a link that comes up cleanly at
>    2.5GT/s never re-runs it.  Flagging in case that matters; the clamp is
>    firmware's, so honouring it on an unoccupied Port seems defensible.
>  - v2: https://lore.kernel.org/lkml/20260801105441.6506-1-andiwild@gmail.com/
>  - v1: https://lore.kernel.org/lkml/20260801092152.5643-1-andiwild@gmail.com/
>  drivers/pci/quirks.c | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> index b09f27f..9e407c4 100644
> --- a/drivers/pci/quirks.c
> +++ b/drivers/pci/quirks.c
> @@ -108,7 +108,11 @@ int pcie_failed_link_retrain(struct pci_dev *dev)
>  
>  	pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &lnksta);
>  	pcie_capability_read_word(dev, PCI_EXP_LNKCTL2, &oldlnkctl2);
> -	if (!(lnksta & PCI_EXP_LNKSTA_DLLLA) && pcie_lbms_seen(dev, lnksta)) {
> +	if (lnksta & PCI_EXP_LNKSTA_DLLLA) {
> +		;
> +	} else if (PCIE_LNKCTL2_TLS2SPEED(oldlnkctl2) == PCIE_SPEED_2_5GT) {
> +		return ret;
> +	} else if (pcie_lbms_seen(dev, lnksta)) {
>  		pci_info(dev, "broken device, retraining non-functional downstream link at 2.5GT/s\n");
>  		ret = pcie_set_target_speed(dev, PCIE_SPEED_2_5GT, false);
>  		if (ret)
> -- 
> 2.55.0
> 

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[Thorsten Leemhuis]

On 9/18/26 01:33, Bjorn Helgaas wrote:
> On Sat, Aug 01, 2026 at 10:11:48PM +0200, Andreas Wild wrote:
>> From: "Maciej W. Rozycki" <macro@orcam.me.uk>
>> Since commit 72780f796468 ("PCI: Always lift 2.5GT/s restriction in PCIe
>> failed link retraining") the Target Speed quirk lifts a firmware-imposed
>> 2.5GT/s restriction on any downstream port, without checking whether the
>> link is up.  Where nothing is plugged in, the retraining that follows can
>> never complete, so each attempt costs PCIE_LINK_RETRAIN_TIMEOUT_MS.  The
>> quirk makes two of them -- the initial one and the restore on the error
>> path -- adding a fixed 2 s to every boot.
>>
>> On an MSI PRO Z690-A WIFI DDR4 (Intel 600 Series PCH) with one empty x1
>> slot, running v7.2-rc5:
>> [...]
>> Fixes: 72780f796468 ("PCI: Always lift 2.5GT/s restriction in PCIe failed link retraining")
>> Cc: stable@vger.kernel.org
>> Reported-by: Andreas Wild <andiwild@gmail.com>
>> Closes: https://lore.kernel.org/lkml/20260801092152.5643-1-andiwild@gmail.com/
>> Tested-by: Andreas Wild <andiwild@gmail.com>
>> Signed-off-by: Maciej W. Rozycki <macro@orcam.me.uk>
>> Signed-off-by: Andreas Wild <andiwild@gmail.com>
> 
> I provisionally applied this on pci/misc.  It seems to address at
> least part of a v7.2 regression, so it probably should go on
> pci/for-linus for v7.3.
> 
> But I'd like to have it in -next for a bit before asking Linus to pull
> it. [...]

Great, thx -- and yes, -next sounds totally fine for me, as this fixes a
delay, not a hard breakage.
> But IIUC Aoxtj still sees issues with 72780f796468 ("PCI: Always lift
> 2.5GT/s restriction in PCIe failed link retraining") even with this
> patch
> (https://lore.kernel.org/all/331e97c7-e422-420d-9f3e-5d9f734464b3@axtjblog.cc)
> 
> And M M / Mich (BCC'd) also reports an issue that (IIUC) is not fixed by
> this patch (https://bugzilla.kernel.org/show_bug.cgi?id=221919#c3)
> 
> So evidently there are still issues but maybe this patch fixes part of
> them?
Yeah, Maciej ten days expressed that we to look into this. Anyway, I
suggest someone that is affected by that problem starts a new thread
(please CC all those that are affected by it and the regression list)
with summarizing the current state + this patch, as this thread got a
bit confusing... (please drop a link to that thread here afterwards)

Ciao, Thorsten

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[Maciej W. Rozycki]

On Fri, 18 Sep 2026, Thorsten Leemhuis wrote:

> > But IIUC Aoxtj still sees issues with 72780f796468 ("PCI: Always lift
> > 2.5GT/s restriction in PCIe failed link retraining") even with this
> > patch
> > (https://lore.kernel.org/all/331e97c7-e422-420d-9f3e-5d9f734464b3@axtjblog.cc)
> > 
> > And M M / Mich (BCC'd) also reports an issue that (IIUC) is not fixed by
> > this patch (https://bugzilla.kernel.org/show_bug.cgi?id=221919#c3)
> > 
> > So evidently there are still issues but maybe this patch fixes part of
> > them?
> Yeah, Maciej ten days expressed that we to look into this. Anyway, I
> suggest someone that is affected by that problem starts a new thread
> (please CC all those that are affected by it and the regression list)
> with summarizing the current state + this patch, as this thread got a
> bit confusing... (please drop a link to that thread here afterwards)

 It remains on my radar, no worries.  Hectic time here, a lot is going on.  
I'll do my best to look into it as soon as possible.  Also I'll be at LPC 
in Prague in a couple of weeks' time, so this may be an opportunity to 
discuss the issues in person.

  Maciej

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[Thorsten Leemhuis]

On 9/18/26 12:39, Maciej W. Rozycki wrote:
> On Fri, 18 Sep 2026, Thorsten Leemhuis wrote:
> 
>>> But IIUC Aoxtj still sees issues with 72780f796468 ("PCI: Always lift
>>> 2.5GT/s restriction in PCIe failed link retraining") even with this
>>> patch
>>> (https://lore.kernel.org/all/331e97c7-e422-420d-9f3e-5d9f734464b3@axtjblog.cc)
>>>
>>> And M M / Mich (BCC'd) also reports an issue that (IIUC) is not fixed by
>>> this patch (https://bugzilla.kernel.org/show_bug.cgi?id=221919#c3)
>>>
>>> So evidently there are still issues but maybe this patch fixes part of
>>> them?
>> Yeah, Maciej ten days expressed that we to look into this. Anyway, I
>> suggest someone that is affected by that problem starts a new thread
>> (please CC all those that are affected by it and the regression list)
>> with summarizing the current state + this patch, as this thread got a
>> bit confusing... (please drop a link to that thread here afterwards)
>
>  It remains on my radar, no worries.  Hectic time here, a lot is going on.  
> I'll do my best to look into it as soon as possible.  Also I'll be at LPC 
> in Prague in a couple of weeks' time, so this may be an opportunity to 
> discuss the issues in person.
Sorry, I didn't want to prod you; I mainly wrote what I wrote because
the reports about the two problems got mixed up, so things became
confusing -- which is why I think a new thread starting with a summary
might be best for all of us.

That being said and wrt to the "a lot going on": seems this causes some
devices to not work for iirc at least two reporters. Would it maybe be
better to revert the culprit and reapply it later once that is sorted
out? Or is this not an option or a bad one for some reason? Reminder,
normally Linus in a situation like this wants regressions ideally 'fixed
"within a week", preferably before the next rc':
https://lore.kernel.org/all/CAHk-%3Dwi86AosXs66-yi54%2BmpQjPu0upxB8ZAfG%2BLsMyJmcuMSA@mail.gmail.com/

Ciao, Thorsten

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[Maciej W. Rozycki]

On Fri, 18 Sep 2026, Thorsten Leemhuis wrote:

> That being said and wrt to the "a lot going on": seems this causes some
> devices to not work for iirc at least two reporters. Would it maybe be
> better to revert the culprit and reapply it later once that is sorted
> out? Or is this not an option or a bad one for some reason? Reminder,
> normally Linus in a situation like this wants regressions ideally 'fixed
> "within a week", preferably before the next rc':
> https://lore.kernel.org/all/CAHk-%3Dwi86AosXs66-yi54%2BmpQjPu0upxB8ZAfG%2BLsMyJmcuMSA@mail.gmail.com/

 Reverting will cause other configurations to regress, so this would be 
replacing one regression with another.  We seem to be in a situation where 
there are all kinds of breakage out there worked around by the system boot 
firmware by capping the link speed, and this quirk interferes with that.

 One way to handle this could be by having a list of downstream devices 
never to remove link capping for.  But that seems like a recipe for an
ongoing maintaince burden.

 Ideally we'd handle all the cases gracefully, even possibly out-of-spec 
ones, but it seems more and more evident that this may not be possible.  
It may be that the quirk will have to be dropped entirely after all.

 Thoughts?

  Maciej

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[Thorsten Leemhuis]

On 9/18/26 14:18, Maciej W. Rozycki wrote:
> On Fri, 18 Sep 2026, Thorsten Leemhuis wrote:
>> That being said and wrt to the "a lot going on": seems this causes some
>> devices to not work for iirc at least two reporters. Would it maybe be
>> better to revert the culprit and reapply it later once that is sorted
>> out? Or is this not an option or a bad one for some reason? Reminder,
>> normally Linus in a situation like this wants regressions ideally 'fixed
>> "within a week", preferably before the next rc':
>> https://lore.kernel.org/all/CAHk-%3Dwi86AosXs66-yi54%2BmpQjPu0upxB8ZAfG%2BLsMyJmcuMSA@mail.gmail.com/

Disclaimer/reminder: I'm an outsider trying to ensure regressions are
handled well, I'm neither affected by the problems nor involved in PCI
subsystem development.
>  Reverting will cause other configurations to regress, so this would be 
> replacing one regression with another.

Well, guess so, as regressions and fixes for them iirc have stacked
here, so there might be no good established base to go back to. But
*normally* going back to one is what Linus afaics wants to actually
prevent exactly a back-and-forth or an exchange of one bug for another.
He wants to ensure that people can always feel safe to update. See these
quotes from Linus for details:
https://www.kernel.org/doc/html/latest/process/handling-regressions.html#on-back-and-forth

So yes, not reverting might be the best at this point, as 7.2 is out for
a while already and so far there are only two reports iirc. You folks
are the best to judge here. That's why I'll also leave the other aspects
you raised to others.

Ciao, Thorsten

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[Thorsten Leemhuis]

On 9/18/26 12:39, Maciej W. Rozycki wrote:
> On Fri, 18 Sep 2026, Thorsten Leemhuis wrote:
>
>>> But IIUC Aoxtj still sees issues with 72780f796468 ("PCI: Always lift
>>> 2.5GT/s restriction in PCIe failed link retraining") even with this
>>> patch
>>> (https://lore.kernel.org/all/331e97c7-e422-420d-9f3e-5d9f734464b3@axtjblog.cc)
>>>
>>> And M M / Mich (BCC'd) also reports an issue that (IIUC) is not fixed by
>>> this patch (https://bugzilla.kernel.org/show_bug.cgi?id=221919#c3)
>>>
>>> So evidently there are still issues but maybe this patch fixes part of
>>> them?
>>>> Yeah, Maciej ten days expressed that we to look into this. Anyway, I
>> suggest someone that is affected by that problem starts a new thread
>> (please CC all those that are affected by it and the regression list)
>> with summarizing the current state + this patch, as this thread got a
>> bit confusing... (please drop a link to that thread here afterwards)
> 
>  It remains on my radar, no worries.  [...]

FWIW, another report from "blaat windows" (now CCed) can be found
out-of-thread here:

https://lore.kernel.org/all/AS2PR10MB7429335B6C8F4C7FA3F88638B5852@AS2PR10MB7429.EURPRD10.PROD.OUTLOOK.COM/

Quoting that below.
"""
>>>> Another reproducible case of the active-link failure described in this thread.
>>>> 
>>>> Hardware:
>>>> Intel 4th-gen/9-series platform
>>>> Intel 82571EB quad-port NIC
>>>> Microsemi/PMC/IDT PES12N3A PCIe switch
>>>> Root port 00:1c.4, LnkCap 5GT/s x4
>>>> Working negotiated link: 2.5GT/s x4
>>>> 
>>>> Kernel results:
>>>> 7.2-rc1 fail
>>>> 7.1-rc7 succes
>>>> 
>>>> On failing kernels, the PES12N3A hierarchy does not enumerate and all four downstream 82571EB ports disappear.
>>>> 
>>>> I traced this to pcie_failed_link_retrain() and specifically the new generic clamp-removal code introduced by 72780f7964684939d7d2f69c348876213b184484 ("PCI: Always lift 2.5GT/s restriction in PCIe failed link retraining").
>>>> 
>>>> I tested 7.3.0-rc3+ with only this block commented out:
>>>> 
>>>>     pcie_capability_read_word(dev, PCI_EXP_LNKCTL2, &lnkctl2);
>>>>     if ((lnkctl2 & PCI_EXP_LNKCTL2_TLS) == PCI_EXP_LNKCTL2_TLS_2_5GT) {
>>>>             pci_info(dev, "removing 2.5GT/s downstream link speed restriction\n");
>>>>             ret = pcie_set_target_speed(dev, speed_cap, false);
>>>>             if (ret)
>>>>                     goto err;
>>>>     }
>>>> 
>>>> With that block disabled, 7.3.0-rc3+ boots normally, having all four NIC ports enumerate:
>>>> 
>>>> 07:00.0 82571EB
>>>> 07:00.1 82571EB
>>>> 08:00.0 82571EB
>>>> 08:00.1 82571EB
>>>> 
>>>> The important result is that the initial 2.5GT/s recovery is fine. Leaving the link at 2.5GT/s works. It is the subsequent:
>>>> 
>>>>     pcie_set_target_speed(dev, speed_cap, false);
>>>> 
>>>> which breaks this PES12N3A/82571EB link.
>>>> 
>>>> So this appears to be the same active-link failure mode, but with a PES12N3A switch rather than a direct 82571EB connection.
>>>> 
>>>> This was tested against vanilla 7.3.0-rc3+ with only the above local change.
>>>> 
>>>> I can provide full dmesg/lspci output and test a proposed fix if useful.
"""

Ciao, Thorsten

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[Thorsten Leemhuis]

On 9/21/26 12:33, Thorsten Leemhuis wrote:
> On 9/18/26 12:39, Maciej W. Rozycki wrote:
>> On Fri, 18 Sep 2026, Thorsten Leemhuis wrote:
>>>> But IIUC Aoxtj still sees issues with 72780f796468 ("PCI: Always lift
>>>> 2.5GT/s restriction in PCIe failed link retraining") even with this
>>>> patch
>>>> (https://lore.kernel.org/all/331e97c7-e422-420d-9f3e-5d9f734464b3@axtjblog.cc)

@Andreas, @Bjorn: JFYI, Edward (now CCed) in the bugzilla thread and in
private pointed out that the partial fix this thread is about ("PCI:
Skip Target Speed quirk on clamped ports with no link") was applied by
CachyOS, but then reverted due to problems:
https://github.com/CachyOS/linux/commit/5c1239de195c6f4d26c3e4fc292eadab6feb41a0

No reason given, just a link to a diagnostic file that from a very quick
look showed an Oops from the Nvidia driver:
https://paste.cachyos.org/p/49a2487.log

So maybe it's a problem in that OOT driver, maybe not. Just wanted to
let you know about this. To quote:

"""
> Aug 30 21:56:16 cachyos kernel: [drm:nv_drm_dev_load [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00000100] Failed to allocate NvKmsKapiDevice
> Aug 30 21:56:17 <hostname-redacted> kernel: Oops: general protection fault, probably for non-canonical address 0xe95cb1b08fafee9c: 0000 [#1] SMP NOPTI
> Aug 30 21:56:17 <hostname-redacted> kernel: CPU: 0 UID: 0 PID: 503 Comm: modprobe Tainted: G           O        7.2.2-1-cachyos #1 PREEMPT(full)  e429b9d2132c094dcc3a5be7de95f46bd03f9bdb
> Aug 30 21:56:17 <hostname-redacted> kernel: Tainted: [O]=OOT_MODULE
> Aug 30 21:56:17 <hostname-redacted> kernel: Hardware name: LENOVO 82RF/LNVNB161216, BIOS J2CN40WW 04/15/2022
> Aug 30 21:56:17 <hostname-redacted> kernel: RIP: 0010:__refill_objects_node+0x283/0x390
> Aug 30 21:56:17 <hostname-redacted> kernel: Code: 48 ff c0 48 8b 5c 24 20 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 44 89 c1 4d 89 5c c2 f8 44 8b 43 28 48 8b 93 b0 00 00 00 <4b> 33 14 18 0f 1f 44 00 00 4d 01 d8 49 0f c8 4c 31 c2 48 39 e8 41
> Aug 30 21:56:17 <hostname-redacted> kernel: RSP: 0018:ffffccf440bef700 EFLAGS: 00010287
> Aug 30 21:56:17 <hostname-redacted> kernel: RAX: 0000000000000008 RBX: ffff8a2380045900 RCX: 0000000000000018
> Aug 30 21:56:17 <hostname-redacted> kernel: RDX: 690fa410ac2511e3 RSI: fffff1bb44805500 RDI: fffff1bb44805510
> Aug 30 21:56:17 <hostname-redacted> kernel: RBP: 0000000000000020 R08: 0000000000000080 R09: 0000000000200020
> Aug 30 21:56:17 <hostname-redacted> kernel: R10: ffff8a23819df220 R11: e95cb1b08fafee1c R12: ffff8a2380041e01
> Aug 30 21:56:17 <hostname-redacted> kernel: R13: ffffccf440bef748 R14: 0000000000000020 R15: 0000000000000007
> Aug 30 21:56:17 <hostname-redacted> kernel: FS:  0000000000000000(0000) GS:ffff8a276e465000(0000) knlGS:0000000000000000
> Aug 30 21:56:17 <hostname-redacted> kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> Aug 30 21:56:17 <hostname-redacted> kernel: CR2: 00007fbf1ad14270 CR3: 000000011ffe4002 CR4: 0000000000f70ef0
> Aug 30 21:56:17 <hostname-redacted> kernel: PKRU: 55555554
> Aug 30 21:56:17 <hostname-redacted> kernel: Call Trace:
> Aug 30 21:56:17 <hostname-redacted> kernel:  <TASK>
> Aug 30 21:56:17 <hostname-redacted> kernel:  refill_objects+0x3f/0x3c0
> Aug 30 21:56:17 <hostname-redacted> kernel:  kmem_cache_prefill_sheaf+0xe3/0x210
> Aug 30 21:56:17 <hostname-redacted> kernel:  mas_preallocate+0x4e7/0x730
> Aug 30 21:56:17 <hostname-redacted> kernel:  __split_vma+0xc5/0x550
> Aug 30 21:56:17 <hostname-redacted> kernel:  vms_gather_munmap_vmas+0x119/0x350
> Aug 30 21:56:17 <hostname-redacted> kernel:  mmap_region+0x10b8/0x1890
> Aug 30 21:56:17 <hostname-redacted> kernel:  do_mmap+0x2b0/0x580
> Aug 30 21:56:17 <hostname-redacted> kernel:  vm_mmap_pgoff+0xf7/0x190
> Aug 30 21:56:17 <hostname-redacted> kernel:  ? ksys_mmap_pgoff+0xbb/0x110
> Aug 30 21:56:17 <hostname-redacted> kernel:  ksys_mmap_pgoff+0xb0/0x110
> Aug 30 21:56:17 <hostname-redacted> kernel:  do_syscall_64+0xa6/0x3e0
> Aug 30 21:56:17 <hostname-redacted> kernel:  ? do_syscall_64+0x65/0x3e0
> Aug 30 21:56:17 <hostname-redacted> kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
> Aug 30 21:56:17 <hostname-redacted> kernel: RIP: 0033:0x7fbf1b7241f6
> Aug 30 21:56:17 <hostname-redacted> kernel: Code: 00 00 00 00 f3 0f 1e fa 41 f7 c1 ff 0f 00 00 75 2b 55 89 cd 53 48 89 fb 48 85 ff 74 4f 41 89 ea 48 89 df b8 09 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 22 5b 5d c3 0f 1f 80 00 00 00 00 c7 05 6e 71
> Aug 30 21:56:17 <hostname-redacted> kernel: RSP: 002b:00007ffcb1466808 EFLAGS: 00000206 ORIG_RAX: 0000000000000009
> Aug 30 21:56:17 <hostname-redacted> kernel: RAX: ffffffffffffffda RBX: 00007fbf1b4f5000 RCX: 00007fbf1b7241f6
> Aug 30 21:56:17 <hostname-redacted> kernel: RDX: 0000000000000005 RSI: 000000000000a000 RDI: 00007fbf1b4f5000
> Aug 30 21:56:17 <hostname-redacted> kernel: RBP: 0000000000000812 R08: 0000000000000000 R09: 0000000000001000
> Aug 30 21:56:17 <hostname-redacted> kernel: R10: 0000000000000812 R11: 0000000000000206 R12: 00007fbf1b504580
> Aug 30 21:56:17 <hostname-redacted> kernel: R13: 0000000000001000 R14: 000000000000b000 R15: 00007fbf1b4f4000
> Aug 30 21:56:17 <hostname-redacted> kernel:  </TASK>
> Aug 30 21:56:17 <hostname-redacted> kernel: Modules linked in: xt_LOG nf_log_syslog nft_limit xt_limit xt_addrtype xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat x_tables nf_tables dm_mod pkcs8_key_parser i2c_dev crypto_user ntsync zram 842_decompress 842_compress lz4hc_compress xe gpu_sched drm_gpuvm drm_exec drm_gpusvm_helper drm_suballoc_helper nvme nvme_core nvidia_drm(O) nvme_keyring nvidia_uvm(O) nvidia_modeset(O) nvme_auth i915 drm_buddy intel_gtt i2c_algo_bit intel_lpss_pci drm_display_helper spi_intel_pci intel_lpss spi_intel idma64 intel_vsec cec vmd serio_raw nvidia(O) drm_ttm_helper ttm video wmi aead
> Aug 30 21:56:17 <hostname-redacted> kernel: ---[ end trace 0000000000000000 ]---
> Aug 30 21:56:17 <hostname-redacted> kernel: RIP: 0010:__refill_objects_node+0x283/0x390
> Aug 30 21:56:17 <hostname-redacted> kernel: Code: 48 ff c0 48 8b 5c 24 20 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 44 89 c1 4d 89 5c c2 f8 44 8b 43 28 48 8b 93 b0 00 00 00 <4b> 33 14 18 0f 1f 44 00 00 4d 01 d8 49 0f c8 4c 31 c2 48 39 e8 41
> Aug 30 21:56:17 <hostname-redacted> kernel: RSP: 0018:ffffccf440bef700 EFLAGS: 00010287
> Aug 30 21:56:17 <hostname-redacted> kernel: RAX: 0000000000000008 RBX: ffff8a2380045900 RCX: 0000000000000018
> Aug 30 21:56:17 <hostname-redacted> kernel: RDX: 690fa410ac2511e3 RSI: fffff1bb44805500 RDI: fffff1bb44805510
> Aug 30 21:56:17 <hostname-redacted> kernel: RBP: 0000000000000020 R08: 0000000000000080 R09: 0000000000200020
> Aug 30 21:56:17 <hostname-redacted> kernel: R10: ffff8a23819df220 R11: e95cb1b08fafee1c R12: ffff8a2380041e01
> Aug 30 21:56:17 <hostname-redacted> kernel: R13: ffffccf440bef748 R14: 0000000000000020 R15: 0000000000000007
> Aug 30 21:56:17 <hostname-redacted> kernel: FS:  0000000000000000(0000) GS:ffff8a276e465000(0000) knlGS:0000000000000000
> Aug 30 21:56:17 <hostname-redacted> kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> Aug 30 21:56:17 <hostname-redacted> kernel: CR2: 00007fbf1ad14270 CR3: 000000011ffe4002 CR4: 0000000000f70ef0
> Aug 30 21:56:17 <hostname-redacted> kernel: PKRU: 55555554

"""

Ciao, Thorsten

>>>> And M M / Mich (BCC'd) also reports an issue that (IIUC) is not fixed by
>>>> this patch (https://bugzilla.kernel.org/show_bug.cgi?id=221919#c3)
>>>>
>>>> So evidently there are still issues but maybe this patch fixes part of
>>>> them?
>>>>> Yeah, Maciej ten days expressed that we to look into this. Anyway, I
>>> suggest someone that is affected by that problem starts a new thread
>>> (please CC all those that are affected by it and the regression list)
>>> with summarizing the current state + this patch, as this thread got a
>>> bit confusing... (please drop a link to that thread here afterwards)
>>
>>  It remains on my radar, no worries.  [...]
> 
> FWIW, another report from "blaat windows" (now CCed) can be found
> out-of-thread here:
> 
> https://lore.kernel.org/all/AS2PR10MB7429335B6C8F4C7FA3F88638B5852@AS2PR10MB7429.EURPRD10.PROD.OUTLOOK.COM/
> 
> Quoting that below.
> """
>>>>> Another reproducible case of the active-link failure described in this thread.
>>>>>
>>>>> Hardware:
>>>>> Intel 4th-gen/9-series platform
>>>>> Intel 82571EB quad-port NIC
>>>>> Microsemi/PMC/IDT PES12N3A PCIe switch
>>>>> Root port 00:1c.4, LnkCap 5GT/s x4
>>>>> Working negotiated link: 2.5GT/s x4
>>>>>
>>>>> Kernel results:
>>>>> 7.2-rc1 fail
>>>>> 7.1-rc7 succes
>>>>>
>>>>> On failing kernels, the PES12N3A hierarchy does not enumerate and all four downstream 82571EB ports disappear.
>>>>>
>>>>> I traced this to pcie_failed_link_retrain() and specifically the new generic clamp-removal code introduced by 72780f7964684939d7d2f69c348876213b184484 ("PCI: Always lift 2.5GT/s restriction in PCIe failed link retraining").
>>>>>
>>>>> I tested 7.3.0-rc3+ with only this block commented out:
>>>>>
>>>>>     pcie_capability_read_word(dev, PCI_EXP_LNKCTL2, &lnkctl2);
>>>>>     if ((lnkctl2 & PCI_EXP_LNKCTL2_TLS) == PCI_EXP_LNKCTL2_TLS_2_5GT) {
>>>>>             pci_info(dev, "removing 2.5GT/s downstream link speed restriction\n");
>>>>>             ret = pcie_set_target_speed(dev, speed_cap, false);
>>>>>             if (ret)
>>>>>                     goto err;
>>>>>     }
>>>>>
>>>>> With that block disabled, 7.3.0-rc3+ boots normally, having all four NIC ports enumerate:
>>>>>
>>>>> 07:00.0 82571EB
>>>>> 07:00.1 82571EB
>>>>> 08:00.0 82571EB
>>>>> 08:00.1 82571EB
>>>>>
>>>>> The important result is that the initial 2.5GT/s recovery is fine. Leaving the link at 2.5GT/s works. It is the subsequent:
>>>>>
>>>>>     pcie_set_target_speed(dev, speed_cap, false);
>>>>>
>>>>> which breaks this PES12N3A/82571EB link.
>>>>>
>>>>> So this appears to be the same active-link failure mode, but with a PES12N3A switch rather than a direct 82571EB connection.
>>>>>
>>>>> This was tested against vanilla 7.3.0-rc3+ with only the above local change.
>>>>>
>>>>> I can provide full dmesg/lspci output and test a proposed fix if useful.
> """
> 
> Ciao, Thorsten
> 

Re: [PATCH v3] PCI: Skip Target Speed quirk on clamped ports with no link

[Andreas Wild]

On Tue, 29 Sep 2026, Thorsten Leemhuis wrote:

> No reason given, just a link to a diagnostic file that from a very quick
> look showed an Oops from the Nvidia driver:
> https://paste.cachyos.org/p/49a2487.log
>
> So maybe it's a problem in that OOT driver, maybe not. [...]

I had a look at that log, and I don't think it points at the patch.

It covers two boots, both on the same 7.2.2-1-cachyos build, which
carries the patch.  The second one, from which the report was taken,
came up normally.

In the failing boot the NVIDIA driver fails to initialise the GPU
("RmInitAdapter failed! (0x24:0x72:1281)"), and about a second later
modprobe takes a general protection fault in __refill_objects_node(),
reached from mas_preallocate(): an object still on the maple_node free
list had a zeroed free pointer, as if it had been allocated or written
to while still free.  The journal ends a second after that, so this is
presumably the hang the revert mentions.

That looks like slab corruption, which usually shows up in a victim
rather than the culprit, so the lack of NVIDIA frames in the trace
doesn't clear the driver: its failed-init teardown reported leaked
objects just before.  Another candidate is 4a724bcf5d70 ("mm/slab: take
n->list_lock in __slab_try_return_freelist() to avoid race"), which only
reached stable in 7.2.7.

Nothing in the log implicates the patch, but it can't clear it either:
the patch's early return is silent, and the report only has warnings
and errors from the failing boot.  One unexplained difference between
the boots: the r8169 NIC is at 33:00.0 in the failing one and at
30:00.0 in the good one.

Best regards,
Andreas