[REGRESSION 6.16] r8169 RTL8168h/8111h fails to probe — "Unable to change power state from D3cold to D0" — bisected to 4d4c10f763d7

[REGRESSION 6.16] r8169 RTL8168h/8111h fails to probe — "Unable to change power state from D3cold to D0" — bisected to 4d4c10f763d7

[Josh Perry]

#regzbot introduced: 4d4c10f763d7

Since v6.16 one of two onboard RTL8168h/8111h NICs on this board fails 
to probe on boot; the device drops to D3cold and the driver can't bring 
it back:

   r8169 0000:02:00.0 eth0: RTL8168h/8111h, 00:2b:67:48:40:01, XID 541, 
IRQ 137
   r8169 0000:04:00.0: Unable to change power state from D3cold to D0, 
device inaccessible
   r8169 0000:04:00.0: Mem-Wr-Inval unavailable
   r8169 0000:04:00.0: error -EIO: PCI read failed
   r8169 0000:04:00.0: probe with driver r8169 failed with error -5

The board has two identical RTL8168h NICs (both XID 541): 0000:02:00.0 
and 0000:04:00.0. Only 04:00.0 fails — its sibling 02:00.0, on a 
different root port, probes and works normally on the very same kernel 
and boot. The failing NIC then does not appear (no enp4s0), taking the 
machine's WAN offline. This strongly suggests the problem is 
port/topology-specific rather than device- or driver-specific: the 
upstream port behind 04:00.0 is placed in D3cold and the endpoint cannot 
be resumed to D0.

Hardware: RTL8168h/8111h, XID 541, PCI 04:00.0 (onboard 1GbE).
Platform: Lenovo ThinkCentre M90n-1 (11AHS0B200), BIOS M2AKT49A 
(2026-03-25, latest available). Firmware is current, so this is not a 
platform-firmware issue.

Bisection: v6.15 good, v6.16 bad (verified by booting both). I then 
reverted 4d4c10f763d7 ("PCI: Explicitly put devices into D0 when 
initializing") together with its follow-up 907a7a2e5bf4 ("PCI/PM: Set up 
runtime PM even for devices without PCI PM") on top of 6.16.7: the NIC 
probes and links at 1Gbps/Full normally, with no workaround:

   r8169 0000:04:00.0 eth1: RTL8168h/8111h, 00:2b:67:48:40:02, XID 541, 
IRQ 138
   r8169 0000:04:00.0 enp4s0: Link is Up - 1Gbps/Full - flow control rx/tx

Workaround: booting an unmodified v6.16+ kernel with pcie_port_pm=off 
also restores the NIC, which is consistent with the upstream port being 
placed in D3cold and the device failing to resume to D0 after the 
explicit-D0 init change.

The follow-up 907a7a2e5bf4 does not fix this resume case: v6.18.33 is 
still affected (retested today on current firmware).

Happy to test patches or provide full dmesg / lspci.

Re: [REGRESSION 6.16] r8169 RTL8168h/8111h fails to probe — "Unable to change power state from D3cold to D0" — bisected to 4d4c10f763d7

[Thorsten Leemhuis]

On 6/12/26 03:07, Josh Perry wrote:
> #regzbot introduced: 4d4c10f763d7
> 
> Since v6.16 one of two onboard RTL8168h/8111h NICs on this board fails
> to probe on boot; the device drops to D3cold and the driver can't bring
> it back:

FWIW, that commit is 4d4c10f763d780 ("PCI: Explicitly put devices into
D0 when initializing") [v6.16-rc1] from Mario, who is already CCed, but
looks like might be on holiday or something due to inactivity on the
lists in the recent days. So it might take a few days before this moves on.

Josh, this is not my area of expertise, but there are two things I guess
might be helpful:

* retry with 7.1
* upload "dmesg" and "sudo lspci -vvv" output from working and broken
kernels somewhere (like bugzilla.kernel.org).

Ciao, Thorsten

>   r8169 0000:02:00.0 eth0: RTL8168h/8111h, 00:2b:67:48:40:01, XID 541,
> IRQ 137
>   r8169 0000:04:00.0: Unable to change power state from D3cold to D0,
> device inaccessible
>   r8169 0000:04:00.0: Mem-Wr-Inval unavailable
>   r8169 0000:04:00.0: error -EIO: PCI read failed
>   r8169 0000:04:00.0: probe with driver r8169 failed with error -5
> 
> The board has two identical RTL8168h NICs (both XID 541): 0000:02:00.0
> and 0000:04:00.0. Only 04:00.0 fails — its sibling 02:00.0, on a
> different root port, probes and works normally on the very same kernel
> and boot. The failing NIC then does not appear (no enp4s0), taking the
> machine's WAN offline. This strongly suggests the problem is port/
> topology-specific rather than device- or driver-specific: the upstream
> port behind 04:00.0 is placed in D3cold and the endpoint cannot be
> resumed to D0.
> 
> Hardware: RTL8168h/8111h, XID 541, PCI 04:00.0 (onboard 1GbE).
> Platform: Lenovo ThinkCentre M90n-1 (11AHS0B200), BIOS M2AKT49A
> (2026-03-25, latest available). Firmware is current, so this is not a
> platform-firmware issue.
> 
> Bisection: v6.15 good, v6.16 bad (verified by booting both). I then
> reverted 4d4c10f763d7 ("PCI: Explicitly put devices into D0 when
> initializing") together with its follow-up 907a7a2e5bf4 ("PCI/PM: Set up
> runtime PM even for devices without PCI PM") on top of 6.16.7: the NIC
> probes and links at 1Gbps/Full normally, with no workaround:
> 
>   r8169 0000:04:00.0 eth1: RTL8168h/8111h, 00:2b:67:48:40:02, XID 541,
> IRQ 138
>   r8169 0000:04:00.0 enp4s0: Link is Up - 1Gbps/Full - flow control rx/tx
> 
> Workaround: booting an unmodified v6.16+ kernel with pcie_port_pm=off
> also restores the NIC, which is consistent with the upstream port being
> placed in D3cold and the device failing to resume to D0 after the
> explicit-D0 init change.
> 
> The follow-up 907a7a2e5bf4 does not fix this resume case: v6.18.33 is
> still affected (retested today on current firmware).
> 
> Happy to test patches or provide full dmesg / lspci.
> 

Re: [REGRESSION 6.16] r8169 RTL8168h/8111h fails to probe — "Unable to change power state from D3cold to D0" — bisected to 4d4c10f763d7

[Mario Limonciello]

On 6/17/26 01:32, Thorsten Leemhuis wrote:
> On 6/12/26 03:07, Josh Perry wrote:
>> #regzbot introduced: 4d4c10f763d7
>>
>> Since v6.16 one of two onboard RTL8168h/8111h NICs on this board fails
>> to probe on boot; the device drops to D3cold and the driver can't bring
>> it back:
> 
> FWIW, that commit is 4d4c10f763d780 ("PCI: Explicitly put devices into
> D0 when initializing") [v6.16-rc1] from Mario, who is already CCed, but
> looks like might be on holiday or something due to inactivity on the
> lists in the recent days. So it might take a few days before this moves on.
> 
> Josh, this is not my area of expertise, but there are two things I guess
> might be helpful:
> 
> * retry with 7.1
> * upload "dmesg" and "sudo lspci -vvv" output from working and broken
> kernels somewhere (like bugzilla.kernel.org).

Yes; please retry with mainline.  We already had multiple regressions 
from that commit fixed, so if you bisected down to this commit then it's 
plausible that there is already a fix.

> 
> Ciao, Thorsten
> 
>>    r8169 0000:02:00.0 eth0: RTL8168h/8111h, 00:2b:67:48:40:01, XID 541,
>> IRQ 137
>>    r8169 0000:04:00.0: Unable to change power state from D3cold to D0,
>> device inaccessible
>>    r8169 0000:04:00.0: Mem-Wr-Inval unavailable
>>    r8169 0000:04:00.0: error -EIO: PCI read failed
>>    r8169 0000:04:00.0: probe with driver r8169 failed with error -5
>>
>> The board has two identical RTL8168h NICs (both XID 541): 0000:02:00.0
>> and 0000:04:00.0. Only 04:00.0 fails — its sibling 02:00.0, on a
>> different root port, probes and works normally on the very same kernel
>> and boot. The failing NIC then does not appear (no enp4s0), taking the
>> machine's WAN offline. This strongly suggests the problem is port/
>> topology-specific rather than device- or driver-specific: the upstream
>> port behind 04:00.0 is placed in D3cold and the endpoint cannot be
>> resumed to D0.
>>
>> Hardware: RTL8168h/8111h, XID 541, PCI 04:00.0 (onboard 1GbE).
>> Platform: Lenovo ThinkCentre M90n-1 (11AHS0B200), BIOS M2AKT49A
>> (2026-03-25, latest available). Firmware is current, so this is not a
>> platform-firmware issue.
>>
>> Bisection: v6.15 good, v6.16 bad (verified by booting both). I then
>> reverted 4d4c10f763d7 ("PCI: Explicitly put devices into D0 when
>> initializing") together with its follow-up 907a7a2e5bf4 ("PCI/PM: Set up
>> runtime PM even for devices without PCI PM") on top of 6.16.7: the NIC
>> probes and links at 1Gbps/Full normally, with no workaround:
>>
>>    r8169 0000:04:00.0 eth1: RTL8168h/8111h, 00:2b:67:48:40:02, XID 541,
>> IRQ 138
>>    r8169 0000:04:00.0 enp4s0: Link is Up - 1Gbps/Full - flow control rx/tx
>>
>> Workaround: booting an unmodified v6.16+ kernel with pcie_port_pm=off
>> also restores the NIC, which is consistent with the upstream port being
>> placed in D3cold and the device failing to resume to D0 after the
>> explicit-D0 init change.
>>
>> The follow-up 907a7a2e5bf4 does not fix this resume case: v6.18.33 is
>> still affected (retested today on current firmware).
>>
>> Happy to test patches or provide full dmesg / lspci.
>>
> 

Re: [REGRESSION 6.16] r8169 RTL8168h/8111h fails to probe — "Unable to change power state from D3cold to D0" — bisected to 4d4c10f763d7

[Bjorn Helgaas]

On Sun, Jun 21, 2026 at 12:24:42PM -0700, Mario Limonciello wrote:
> On 6/17/26 01:32, Thorsten Leemhuis wrote:
> > On 6/12/26 03:07, Josh Perry wrote:
> > > #regzbot introduced: 4d4c10f763d7
> > > 
> > > Since v6.16 one of two onboard RTL8168h/8111h NICs on this board fails
> > > to probe on boot; the device drops to D3cold and the driver can't bring
> > > it back:
> > 
> > FWIW, that commit is 4d4c10f763d780 ("PCI: Explicitly put devices into
> > D0 when initializing") [v6.16-rc1] from Mario, who is already CCed, but
> > looks like might be on holiday or something due to inactivity on the
> > lists in the recent days. So it might take a few days before this moves on.
> > 
> > Josh, this is not my area of expertise, but there are two things I guess
> > might be helpful:
> > 
> > * retry with 7.1
> > * upload "dmesg" and "sudo lspci -vvv" output from working and broken
> > kernels somewhere (like bugzilla.kernel.org).
> 
> Yes; please retry with mainline.  We already had multiple regressions from
> that commit fixed, so if you bisected down to this commit then it's
> plausible that there is already a fix.

Thanks a lot for your bisection and detailed report, Josh.  Can you
verify whether this is still a problem?

The only commit that claims to fix 4d4c10f763d7 ("PCI: Explicitly put
devices into D0 when initializing") is 907a7a2e5bf4 ("PCI/PM: Set up
runtime PM even for devices without PCI PM"), and you already verified
that it did not fix the problem (both appeared in v6.16).

c855c9921da7 ("PCI/ASPM: Don't reconfigure ASPM entering low-power
state"), which appeared in v7.2-rc1, mentions a similar symptom
("Unable to change power state from D3hot to D0, device inaccessible") 
so it's possible that it would fix the issue.

> > >    r8169 0000:02:00.0 eth0: RTL8168h/8111h, 00:2b:67:48:40:01, XID 541,
> > > IRQ 137
> > >    r8169 0000:04:00.0: Unable to change power state from D3cold to D0,
> > > device inaccessible
> > >    r8169 0000:04:00.0: Mem-Wr-Inval unavailable
> > >    r8169 0000:04:00.0: error -EIO: PCI read failed
> > >    r8169 0000:04:00.0: probe with driver r8169 failed with error -5
> > > 
> > > The board has two identical RTL8168h NICs (both XID 541): 0000:02:00.0
> > > and 0000:04:00.0. Only 04:00.0 fails — its sibling 02:00.0, on a
> > > different root port, probes and works normally on the very same kernel
> > > and boot. The failing NIC then does not appear (no enp4s0), taking the
> > > machine's WAN offline. This strongly suggests the problem is port/
> > > topology-specific rather than device- or driver-specific: the upstream
> > > port behind 04:00.0 is placed in D3cold and the endpoint cannot be
> > > resumed to D0.
> > > 
> > > Hardware: RTL8168h/8111h, XID 541, PCI 04:00.0 (onboard 1GbE).
> > > Platform: Lenovo ThinkCentre M90n-1 (11AHS0B200), BIOS M2AKT49A
> > > (2026-03-25, latest available). Firmware is current, so this is not a
> > > platform-firmware issue.
> > > 
> > > Bisection: v6.15 good, v6.16 bad (verified by booting both). I then
> > > reverted 4d4c10f763d7 ("PCI: Explicitly put devices into D0 when
> > > initializing") together with its follow-up 907a7a2e5bf4 ("PCI/PM: Set up
> > > runtime PM even for devices without PCI PM") on top of 6.16.7: the NIC
> > > probes and links at 1Gbps/Full normally, with no workaround:
> > > 
> > >    r8169 0000:04:00.0 eth1: RTL8168h/8111h, 00:2b:67:48:40:02, XID 541,
> > > IRQ 138
> > >    r8169 0000:04:00.0 enp4s0: Link is Up - 1Gbps/Full - flow control rx/tx
> > > 
> > > Workaround: booting an unmodified v6.16+ kernel with pcie_port_pm=off
> > > also restores the NIC, which is consistent with the upstream port being
> > > placed in D3cold and the device failing to resume to D0 after the
> > > explicit-D0 init change.
> > > 
> > > The follow-up 907a7a2e5bf4 does not fix this resume case: v6.18.33 is
> > > still affected (retested today on current firmware).
> > > 
> > > Happy to test patches or provide full dmesg / lspci.
> > > 
> > 
> 

Re: [REGRESSION 6.16] r8169 RTL8168h/8111h fails to probe — "Unable to change power state from D3cold to D0" — bisected to 4d4c10f763d7

[Josh Perry]

On 6/11/26 19:07, Josh Perry wrote:
> #regzbot introduced: 4d4c10f763d7
> 
> Since v6.16 one of two onboard RTL8168h/8111h NICs on this board fails 
> to probe on boot; the device drops to D3cold and the driver can't bring 
> it back:
> 
>    r8169 0000:02:00.0 eth0: RTL8168h/8111h, 00:2b:67:48:40:01, XID 541, 
> IRQ 137
>    r8169 0000:04:00.0: Unable to change power state from D3cold to D0, 
> device inaccessible
>    r8169 0000:04:00.0: Mem-Wr-Inval unavailable
>    r8169 0000:04:00.0: error -EIO: PCI read failed
>    r8169 0000:04:00.0: probe with driver r8169 failed with error -5
> 
> The board has two identical RTL8168h NICs (both XID 541): 0000:02:00.0 
> and 0000:04:00.0. Only 04:00.0 fails — its sibling 02:00.0, on a 
> different root port, probes and works normally on the very same kernel 
> and boot. The failing NIC then does not appear (no enp4s0), taking the 
> machine's WAN offline. This strongly suggests the problem is port/ 
> topology-specific rather than device- or driver-specific: the upstream 
> port behind 04:00.0 is placed in D3cold and the endpoint cannot be 
> resumed to D0.
> 
> Hardware: RTL8168h/8111h, XID 541, PCI 04:00.0 (onboard 1GbE).
> Platform: Lenovo ThinkCentre M90n-1 (11AHS0B200), BIOS M2AKT49A 
> (2026-03-25, latest available). Firmware is current, so this is not a 
> platform-firmware issue.
> 
> Bisection: v6.15 good, v6.16 bad (verified by booting both). I then 
> reverted 4d4c10f763d7 ("PCI: Explicitly put devices into D0 when 
> initializing") together with its follow-up 907a7a2e5bf4 ("PCI/PM: Set up 
> runtime PM even for devices without PCI PM") on top of 6.16.7: the NIC 
> probes and links at 1Gbps/Full normally, with no workaround:
> 
>    r8169 0000:04:00.0 eth1: RTL8168h/8111h, 00:2b:67:48:40:02, XID 541, 
> IRQ 138
>    r8169 0000:04:00.0 enp4s0: Link is Up - 1Gbps/Full - flow control rx/tx
> 
> Workaround: booting an unmodified v6.16+ kernel with pcie_port_pm=off 
> also restores the NIC, which is consistent with the upstream port being 
> placed in D3cold and the device failing to resume to D0 after the 
> explicit-D0 init change.
> 
> The follow-up 907a7a2e5bf4 does not fix this resume case: v6.18.33 is 
> still affected (retested today on current firmware).
> 
> Happy to test patches or provide full dmesg / lspci.


Hey guys,

Thanks, and apologies for the slow follow-up. Multiple of you asked me 
to test on the latest mainline.

I had a chance to do just that today with the latest mainline v7.3. 
Verdict is that it's still broken, same messages as in the original 
report. pcie_port_pm=off on the same build still works around it.

dmesg, "sudo lspci -vvv" and the sysfs PM state for both boots (same 
v7.3 build, only the command line differs) are attached here:

   https://bugzilla.kernel.org/show_bug.cgi?id=222179

New detail from the failing boot: the root port above the dead NIC 
(00:1d.3) is in D0 but its link shows DLActive-, and the endpoint's 
config space reads all-ones ("Unknown header type 7f"). With 
pcie_port_pm=off the port shows DLActive+ and the NIC is in D0 at 
2.5GT/s x1. So the port seems to come back but the link to 04:00.0 
doesn't retrain.

#regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=222179

Thanks again!

Josh

P.S. I'm relying on claude for formalisms like the bot tags and digging 
up the CC list, but I'm pretty noob so please let me know if I'm doing 
things wrong.

Re: [REGRESSION 6.16] r8169 RTL8168h/8111h fails to probe — "Unable to change power state from D3cold to D0" — bisected to 4d4c10f763d7

[Rafael J. Wysocki (Intel)]

On Thu, Oct 22, 2026 at 3:37 AM Josh Perry <josh@6bit.com> wrote:
>
> On 6/11/26 19:07, Josh Perry wrote:
> > #regzbot introduced: 4d4c10f763d7
> >
> > Since v6.16 one of two onboard RTL8168h/8111h NICs on this board fails
> > to probe on boot; the device drops to D3cold and the driver can't bring
> > it back:
> >
> >    r8169 0000:02:00.0 eth0: RTL8168h/8111h, 00:2b:67:48:40:01, XID 541,
> > IRQ 137
> >    r8169 0000:04:00.0: Unable to change power state from D3cold to D0,
> > device inaccessible
> >    r8169 0000:04:00.0: Mem-Wr-Inval unavailable
> >    r8169 0000:04:00.0: error -EIO: PCI read failed
> >    r8169 0000:04:00.0: probe with driver r8169 failed with error -5
> >
> > The board has two identical RTL8168h NICs (both XID 541): 0000:02:00.0
> > and 0000:04:00.0. Only 04:00.0 fails — its sibling 02:00.0, on a
> > different root port, probes and works normally on the very same kernel
> > and boot. The failing NIC then does not appear (no enp4s0), taking the
> > machine's WAN offline. This strongly suggests the problem is port/
> > topology-specific rather than device- or driver-specific: the upstream
> > port behind 04:00.0 is placed in D3cold and the endpoint cannot be
> > resumed to D0.
> >
> > Hardware: RTL8168h/8111h, XID 541, PCI 04:00.0 (onboard 1GbE).
> > Platform: Lenovo ThinkCentre M90n-1 (11AHS0B200), BIOS M2AKT49A
> > (2026-03-25, latest available). Firmware is current, so this is not a
> > platform-firmware issue.
> >
> > Bisection: v6.15 good, v6.16 bad (verified by booting both). I then
> > reverted 4d4c10f763d7 ("PCI: Explicitly put devices into D0 when
> > initializing") together with its follow-up 907a7a2e5bf4 ("PCI/PM: Set up
> > runtime PM even for devices without PCI PM") on top of 6.16.7: the NIC
> > probes and links at 1Gbps/Full normally, with no workaround:
> >
> >    r8169 0000:04:00.0 eth1: RTL8168h/8111h, 00:2b:67:48:40:02, XID 541,
> > IRQ 138
> >    r8169 0000:04:00.0 enp4s0: Link is Up - 1Gbps/Full - flow control rx/tx
> >
> > Workaround: booting an unmodified v6.16+ kernel with pcie_port_pm=off
> > also restores the NIC, which is consistent with the upstream port being
> > placed in D3cold and the device failing to resume to D0 after the
> > explicit-D0 init change.
> >
> > The follow-up 907a7a2e5bf4 does not fix this resume case: v6.18.33 is
> > still affected (retested today on current firmware).
> >
> > Happy to test patches or provide full dmesg / lspci.
>
>
> Hey guys,
>
> Thanks, and apologies for the slow follow-up. Multiple of you asked me
> to test on the latest mainline.
>
> I had a chance to do just that today with the latest mainline v7.3.
> Verdict is that it's still broken, same messages as in the original
> report. pcie_port_pm=off on the same build still works around it.
>
> dmesg, "sudo lspci -vvv" and the sysfs PM state for both boots (same
> v7.3 build, only the command line differs) are attached here:
>
>    https://bugzilla.kernel.org/show_bug.cgi?id=222179
>
> New detail from the failing boot: the root port above the dead NIC
> (00:1d.3) is in D0 but its link shows DLActive-, and the endpoint's
> config space reads all-ones ("Unknown header type 7f"). With
> pcie_port_pm=off the port shows DLActive+ and the NIC is in D0 at
> 2.5GT/s x1. So the port seems to come back but the link to 04:00.0
> doesn't retrain.
>
> #regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=222179

Please also try v7.3 with this patch applied:

https://lore.kernel.org/linux-pci/20261015020811.1132314-1-keng-yu.lin@hpe.com/raw

Re: [REGRESSION 6.16] r8169 RTL8168h/8111h fails to probe — "Unable to change power state from D3cold to D0" — bisected to 4d4c10f763d7

[Josh Perry]

On 10/22/26 04:50, Rafael J. Wysocki (Intel) wrote:
> On Thu, Oct 22, 2026 at 3:37 AM Josh Perry <josh@6bit.com> wrote:
>>
>> On 6/11/26 19:07, Josh Perry wrote:
>>> #regzbot introduced: 4d4c10f763d7
>>>
>>> Since v6.16 one of two onboard RTL8168h/8111h NICs on this board fails
>>> to probe on boot; the device drops to D3cold and the driver can't bring
>>> it back:
>>>
>>>     r8169 0000:02:00.0 eth0: RTL8168h/8111h, 00:2b:67:48:40:01, XID 541,
>>> IRQ 137
>>>     r8169 0000:04:00.0: Unable to change power state from D3cold to D0,
>>> device inaccessible
>>>     r8169 0000:04:00.0: Mem-Wr-Inval unavailable
>>>     r8169 0000:04:00.0: error -EIO: PCI read failed
>>>     r8169 0000:04:00.0: probe with driver r8169 failed with error -5
>>>
>>> The board has two identical RTL8168h NICs (both XID 541): 0000:02:00.0
>>> and 0000:04:00.0. Only 04:00.0 fails — its sibling 02:00.0, on a
>>> different root port, probes and works normally on the very same kernel
>>> and boot. The failing NIC then does not appear (no enp4s0), taking the
>>> machine's WAN offline. This strongly suggests the problem is port/
>>> topology-specific rather than device- or driver-specific: the upstream
>>> port behind 04:00.0 is placed in D3cold and the endpoint cannot be
>>> resumed to D0.
>>>
>>> Hardware: RTL8168h/8111h, XID 541, PCI 04:00.0 (onboard 1GbE).
>>> Platform: Lenovo ThinkCentre M90n-1 (11AHS0B200), BIOS M2AKT49A
>>> (2026-03-25, latest available). Firmware is current, so this is not a
>>> platform-firmware issue.
>>>
>>> Bisection: v6.15 good, v6.16 bad (verified by booting both). I then
>>> reverted 4d4c10f763d7 ("PCI: Explicitly put devices into D0 when
>>> initializing") together with its follow-up 907a7a2e5bf4 ("PCI/PM: Set up
>>> runtime PM even for devices without PCI PM") on top of 6.16.7: the NIC
>>> probes and links at 1Gbps/Full normally, with no workaround:
>>>
>>>     r8169 0000:04:00.0 eth1: RTL8168h/8111h, 00:2b:67:48:40:02, XID 541,
>>> IRQ 138
>>>     r8169 0000:04:00.0 enp4s0: Link is Up - 1Gbps/Full - flow control rx/tx
>>>
>>> Workaround: booting an unmodified v6.16+ kernel with pcie_port_pm=off
>>> also restores the NIC, which is consistent with the upstream port being
>>> placed in D3cold and the device failing to resume to D0 after the
>>> explicit-D0 init change.
>>>
>>> The follow-up 907a7a2e5bf4 does not fix this resume case: v6.18.33 is
>>> still affected (retested today on current firmware).
>>>
>>> Happy to test patches or provide full dmesg / lspci.
>>
>>
>> Hey guys,
>>
>> Thanks, and apologies for the slow follow-up. Multiple of you asked me
>> to test on the latest mainline.
>>
>> I had a chance to do just that today with the latest mainline v7.3.
>> Verdict is that it's still broken, same messages as in the original
>> report. pcie_port_pm=off on the same build still works around it.
>>
>> dmesg, "sudo lspci -vvv" and the sysfs PM state for both boots (same
>> v7.3 build, only the command line differs) are attached here:
>>
>>     https://bugzilla.kernel.org/show_bug.cgi?id=222179
>>
>> New detail from the failing boot: the root port above the dead NIC
>> (00:1d.3) is in D0 but its link shows DLActive-, and the endpoint's
>> config space reads all-ones ("Unknown header type 7f"). With
>> pcie_port_pm=off the port shows DLActive+ and the NIC is in D0 at
>> 2.5GT/s x1. So the port seems to come back but the link to 04:00.0
>> doesn't retrain.
>>
>> #regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=222179
> 
> Please also try v7.3 with this patch applied:
> 
> https://lore.kernel.org/linux-pci/20261015020811.1132314-1-keng-yu.lin@hpe.com/raw

No dice, unfortunately. Same issue:

   r8169 0000:04:00.0: Unable to change power state from D3cold to D0, 
device inaccessible
   r8169 0000:04:00.0: Mem-Wr-Inval unavailable
   r8169 0000:04:00.0: error -EIO: PCI read failed
   r8169 0000:04:00.0: probe with driver r8169 failed with error -5

Same state as unpatched v7.3: root port 00:1d.3 in D0 with LnkSta
DLActive- and ASPM L1 enabled, 04:00.0 config space reads all-ones
("Unknown header type 7f"), and the correctable RxErr on 00:1d.3 at
~9.7s. The PCI/r8169 dmesg lines match the unpatched boot line for line.

I attached the same info for the the patch run to the bugzilla ticket.

Josh

Re: [REGRESSION 6.16] r8169 RTL8168h/8111h fails to probe — "Unable to change power state from D3cold to D0" — bisected to 4d4c10f763d7

[Rafael J. Wysocki (Intel)]

On Thu, Oct 22, 2026 at 11:33 PM Josh Perry <josh@6bit.com> wrote:
>
> On 10/22/26 04:50, Rafael J. Wysocki (Intel) wrote:
> > On Thu, Oct 22, 2026 at 3:37 AM Josh Perry <josh@6bit.com> wrote:
> >>
> >> On 6/11/26 19:07, Josh Perry wrote:
> >>> #regzbot introduced: 4d4c10f763d7
> >>>
> >>> Since v6.16 one of two onboard RTL8168h/8111h NICs on this board fails
> >>> to probe on boot; the device drops to D3cold and the driver can't bring
> >>> it back:
> >>>
> >>>     r8169 0000:02:00.0 eth0: RTL8168h/8111h, 00:2b:67:48:40:01, XID 541,
> >>> IRQ 137
> >>>     r8169 0000:04:00.0: Unable to change power state from D3cold to D0,
> >>> device inaccessible
> >>>     r8169 0000:04:00.0: Mem-Wr-Inval unavailable
> >>>     r8169 0000:04:00.0: error -EIO: PCI read failed
> >>>     r8169 0000:04:00.0: probe with driver r8169 failed with error -5
> >>>
> >>> The board has two identical RTL8168h NICs (both XID 541): 0000:02:00.0
> >>> and 0000:04:00.0. Only 04:00.0 fails — its sibling 02:00.0, on a
> >>> different root port, probes and works normally on the very same kernel
> >>> and boot. The failing NIC then does not appear (no enp4s0), taking the
> >>> machine's WAN offline. This strongly suggests the problem is port/
> >>> topology-specific rather than device- or driver-specific: the upstream
> >>> port behind 04:00.0 is placed in D3cold and the endpoint cannot be
> >>> resumed to D0.
> >>>
> >>> Hardware: RTL8168h/8111h, XID 541, PCI 04:00.0 (onboard 1GbE).
> >>> Platform: Lenovo ThinkCentre M90n-1 (11AHS0B200), BIOS M2AKT49A
> >>> (2026-03-25, latest available). Firmware is current, so this is not a
> >>> platform-firmware issue.
> >>>
> >>> Bisection: v6.15 good, v6.16 bad (verified by booting both). I then
> >>> reverted 4d4c10f763d7 ("PCI: Explicitly put devices into D0 when
> >>> initializing") together with its follow-up 907a7a2e5bf4 ("PCI/PM: Set up
> >>> runtime PM even for devices without PCI PM") on top of 6.16.7: the NIC
> >>> probes and links at 1Gbps/Full normally, with no workaround:
> >>>
> >>>     r8169 0000:04:00.0 eth1: RTL8168h/8111h, 00:2b:67:48:40:02, XID 541,
> >>> IRQ 138
> >>>     r8169 0000:04:00.0 enp4s0: Link is Up - 1Gbps/Full - flow control rx/tx
> >>>
> >>> Workaround: booting an unmodified v6.16+ kernel with pcie_port_pm=off
> >>> also restores the NIC, which is consistent with the upstream port being
> >>> placed in D3cold and the device failing to resume to D0 after the
> >>> explicit-D0 init change.
> >>>
> >>> The follow-up 907a7a2e5bf4 does not fix this resume case: v6.18.33 is
> >>> still affected (retested today on current firmware).
> >>>
> >>> Happy to test patches or provide full dmesg / lspci.
> >>
> >>
> >> Hey guys,
> >>
> >> Thanks, and apologies for the slow follow-up. Multiple of you asked me
> >> to test on the latest mainline.
> >>
> >> I had a chance to do just that today with the latest mainline v7.3.
> >> Verdict is that it's still broken, same messages as in the original
> >> report. pcie_port_pm=off on the same build still works around it.
> >>
> >> dmesg, "sudo lspci -vvv" and the sysfs PM state for both boots (same
> >> v7.3 build, only the command line differs) are attached here:
> >>
> >>     https://bugzilla.kernel.org/show_bug.cgi?id=222179
> >>
> >> New detail from the failing boot: the root port above the dead NIC
> >> (00:1d.3) is in D0 but its link shows DLActive-, and the endpoint's
> >> config space reads all-ones ("Unknown header type 7f"). With
> >> pcie_port_pm=off the port shows DLActive+ and the NIC is in D0 at
> >> 2.5GT/s x1. So the port seems to come back but the link to 04:00.0
> >> doesn't retrain.
> >>
> >> #regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=222179
> >
> > Please also try v7.3 with this patch applied:
> >
> > https://lore.kernel.org/linux-pci/20261015020811.1132314-1-keng-yu.lin@hpe.com/raw
>
> No dice, unfortunately. Same issue:
>
>    r8169 0000:04:00.0: Unable to change power state from D3cold to D0,
> device inaccessible
>    r8169 0000:04:00.0: Mem-Wr-Inval unavailable
>    r8169 0000:04:00.0: error -EIO: PCI read failed
>    r8169 0000:04:00.0: probe with driver r8169 failed with error -5
>
> Same state as unpatched v7.3: root port 00:1d.3 in D0 with LnkSta
> DLActive- and ASPM L1 enabled, 04:00.0 config space reads all-ones
> ("Unknown header type 7f"), and the correctable RxErr on 00:1d.3 at
> ~9.7s. The PCI/r8169 dmesg lines match the unpatched boot line for line.
>
> I attached the same info for the patch run to the bugzilla ticket.

Thanks for the info!

Let's see what can be deduced from it.

First of all, the device is clearly accessible in pci_pm_init() and
then it somehow drops off the bus.  However, the complaint about the
failing power state change is not from the initial power-up.  It is
printed after setting the driver for the device, so most likely it
comes from pci_enable_device() triggered by rtl_init_one().
Interestingly enough, that message also says that the device's power
state before the attempted transition to D0 is D3cold, but that is not
the power state in which the initial power-up leaves it.  Had the
initial power-up left the device in D3cold, it would have printed an
"Unable to change power state" message which it doesn't do, so most
likely right after the initial power-up the device is actually in D0.

Since pcie_port_pm=off makes a difference, the problem is most likely
related to what happens to the failing device's parent port.  Namely,
pcie_port_pm=off simply prevents the port from going into D3cold which
indicates that the failing device drops off the bus because its parent
port goes into D3cold.  Indeed, that may be happening because the port
probes after the failing device has been initialized and if user space
flips the "power/control" attribute of the failing device from "on" to
"auto" after probing the port and before probing the device itself,
the port will suspend and go into D3cold.  Moreover, the device under
it is still in D0 when that happens because pci_pm_runtime_suspend()
does not change the power states of unbound devices (there is even a
comment in that function regarding the parent of an unbound device
possibly going into D3cold while the device is still in D0).  If that
happens, the pci_update_current_state() call in
pci_enable_device_flags() will update current_state for the device to
D3cold.

Of course, I'm not sure if this is what really happens and it still
needs to be figured out why things started to fail after introducing
the initial power-up.  I'd prefer to use the BZ ticket for
investigating it though.

Nevertheless, the above is a plausible scenario that may lead to the
observed symptoms.