RE: RTL8821CE on Intel N95 causes rare silent system hangs on Debian 13 (rtw88 / PCIe ASPM suspected)

[Ping-Ke Shih <pkshih@realtek.com>]

Sab1e <sab1e@qq.com> wrote:
> Hi,
> 
> I'd like to report a recurring full-system freeze. I want to be upfront that I
> am not certain this is a WiFi/rtw88 problem at all -- the RTL8821CE is
> currently just my leading suspect, and I have not confirmed it. I have already
> read the existing reports about 8821CE + ASPM L1 problems (Bugzilla #215131,
> the March 2026 "hard system freeze on RTL8821CE when power_save is enabled"
> thread, Ubuntu LP #1927808, Debian #1114930), and my symptom looks *similar but
> not identical* to those. I'm writing to ask whether you can tell if this is the
> same root cause or a different issue.
> 
> My main goal is to eliminate the source of this instability. I do not rely on
> this WiFi device (the machine uses Ethernet), so I am willing to simply disable
> or physically remove it. I mainly want to determine whether this card is indeed
> the root cause -- a fix is not necessarily required.

If you turn off WiFi from network manager, the symptom still presents?
This is the simplest way to bisect the problem. And it is worth to try
disabling ASPM from driver (disable_aspm=y; need cold reboot to take
effect).

If it still happens it'd be simple to physically remove it.

> 
> ----------------------------------------
> 
> Environment
> 
>   SoC:              Intel N95 (Alder Lake-N)
>   OS:               Debian 13 (trixie)
>   Kernel:           6.12.101+deb13-amd64 (Debian 6.12.101-1)
>   WiFi:             Realtek RTL8821CE, 10ec:c821, subsystem 10ec:c821
>                     PCI 0000:02:00.0, behind Root Port 0000:00:1d.0
>                     (Intel 8086:54b0)
>   Driver:           rtw_8821ce / rtw88_pci / rtw88_core (in-tree)
>   Firmware:         rtw88/rtw8821c_fw.bin v24.11.0, H2C version 12
>                     (139472 bytes, md5 2638a40042a27e98df2f7929c8eee760,
>                      shipped by Debian firmware-realtek 20250410-2)
>   Ethernet:         r8169 (RTL8111/8168/8411), PCI 0000:01:00.0
> 
>   Link state (Root Port 00:1d.0):
>     LnkCap:  Speed 8GT/s, Width x1, ASPM L0s L1
>     LnkCtl:  ASPM L0s L1 Enabled
>     LnkSta:  Speed 2.5GT/s (Gen1), Width x1
>     L1SubCtl1: PCI-PM_L1.1+ PCI-PM_L1.2+ ASPM_L1.1+ ASPM_L1.2+
>   Device 02:00.0:  ASPM L0s L1 Enabled, L1.1/L1.2 enabled, ClockPM+
> 
>   WiFi interface is NOT associated to any AP (wlp2s0 NO-CARRIER;
>   NetworkManager shows "unavailable"). NetworkManager periodically scans and
>   brings the interface inactive/disabled.

It still works in background, NetworkManager can disable WiFi only
(still keep your ethernet on).

> 
> ----------------------------------------
> 
> Symptoms
> 
> Two full-system silent freezes so far:
>   1) 2026-08-09 ~17:52 CST
>   2) 2026-08-11 ~19:15 CST
> 
> Both are *silent*: no panic, no soft lockup, no NMI watchdog output, and no
> automatic watchdog reboot (hardlockup_panic=1, softlockup_panic=1,
> panic_on_warn=1, nmi_watchdog=1, crashkernel=256M were all enabled; none
> fired). Recovery is only by hard power reset.
> 
> Both freezes happened *minutes after a WiFi scan completed*, during unrelated
> system activity (1st: 1panel doing heavy tar/disk I/O; 2nd: a burst of sshd /
> process activity). There is no rtw88 error message immediately preceding the
> freeze in the journal -- the last lines are ordinary system activity.

The log doesn't exactly point to rtw88, so it is hard to judge the cause...

> 
> ----------------------------------------
> 
> Observed errors (separate from the freeze, in the hours/days before it)
> 
> rtw_8821ce 0000:02:00.0: failed to download firmware
> rtw_8821ce 0000:02:00.0: mac power on failed
> rtw_8821ce 0000:02:00.0: failed to power on mac
> rtw_8821ce 0000:02:00.0: failed to send h2c command
> WARNING ... drivers/net/wireless/realtek/rtw88/pci.c ... failed to read DBI register
> 
> 
> AER on the Root Port (recurring at irregular intervals, does NOT hang by itself):
>   pcieport 0000:00:1d.0: AER: Correctable error message received from 0000:02:00.0
>   rtw_8821ce 0000:02:00.0: PCIe Bus Error: severity=Correctable,
>       type=Data Link Layer, (Transmitter ID)
>   rtw_8821ce 0000:02:00.0:   device [10ec:c821] error status/mask=00001000/0000e000
>   rtw_8821ce 0000:02:00.0:    [12] Timeout
> (also seen the same error reported by the root port itself, device [8086:54b0],
>  and occasionally BadDLLP.)

Normally, this is a PCI link becomes abnormal. As experiences, this might be
related to ASPM, so I'd suggest you can turn it off (I mentioned above).

> 
> ----------------------------------------
> 
> Relationship to the known issues (and why this may be different)
> 
> The known 8821CE reports (Bugzilla #215131, the 2026 LPS/ASPM thread) describe
> full freezes on *associated* laptops with a preamble of:
>     rtw_8821ce ...: pci bus timeout, check dma status
>     rtw_8821ce ...: firmware failed to leave lps state
> and the root cause is described as the chip failing to leave ASPM L1 (link
> stays in L1 -> RX DMA stalls).
> 
> My case is different in three ways:
>   1) The interface is *never associated*, so the driver does not enter
>      LPS/LPS-Deep (rtw_enter_lps requires an associated station). It instead
>      enters IPS (idle power save / MAC power-off) on IEEE80211_CONF_IDLE. The
>      errors I see are IPS-leave/power-on failures ("failed to download
>      firmware", "mac power on failed"), not the LPS signatures above.

ASPM is fully controlled by PCI MAC, so it can try to enable low power mode
if certain conditions are met.

>   2) The freeze is *silent* with no rtw88 error immediately before it.

I have ever seen some people reported this kind of symptom, but I don't
remember that exactly. It seems like power save related problem.

>   3) The 2021 NAPI workaround is *already active* here: commit 24f5e38a13b5
>      ("Disable PCIe ASPM while doing NAPI poll on 8821CE") sets rx_no_aspm
>      because chip id == 8821C and the upstream bridge vendor is Intel. So this
>      device freezes despite that workaround -- consistent with the 2026 report
>      noting the NAPI wrapper is insufficient for idle-state deadlocks, but here
>      it is idle *without association* (IPS, not LPS-Deep).

This is a workaround, so I'm not sure all platforms can leverage it.

> 
> ----------------------------------------
> 
> What I have tried (reproduction)
> 
> I could not reproduce the freeze on demand:
>   - netconsole to a remote host for ~40 h: no freeze.
>   - kdump + NMI/softlockup watchdog: did not trigger during the real freezes.
>   - Aggressive userspace triggers (scan loops, UDP flood, device enable/reset).
>   - Source-level timing injection in rtw_pci_link_ps()/rtw_pci_aspm_set()
>     (fixed 1-10 us delays): 0 AER / 0 DBI errors over 121 ASPM toggles + 1616
>     NAPI polls in 80 min.
>   - Rapidly toggling the L1.2 enable bits (root port 0x208 / device 0x168,
>     ASPM_L1_2_EN) via setpci: 39319 L0<->L1.2 transitions in 10 min, 0 AER,
>     no freeze.
> 
> My working hypothesis (unconfirmed) is that the freeze is a low-level
> link/power-state deadlock rather than a software timing race, since none of the
> software timing manipulation above triggered it.
> 
> Regarding the documented workaround: "options rtw88_pci disable_aspm=y" is
> *not effective* on this platform. 

You tried these already....
Please confirm you did cold reboot after setting "options rtw88_pci disable_aspm=y".

> ASPM (L0s/L1 and the L1.1/L1.2 substates) is
> forced enabled by the BIOS -- the kernel pcie_aspm policy is "default"
> (BIOS-controlled) and cannot override it. 

Can you manually enforce to disable it by `setpci` ?
Or try Linux kernel cmdline pcie_aspm.policy=performance?


> The disable_aspm module parameter
> only stops the driver from toggling Realtek's BIT_L1_SW_EN in the DBI register
> (RTK_PCIE_LINK_CFG), but the standard PCIe LNKCTL ASPM L1/L1.2 remains enabled
> by the BIOS, so the device cannot be kept out of L1 by software. A real
> driver/firmware fix is therefore needed here, not a workaround.
> 
> ----------------------------------------
> 
> Question
> 
> I found an existing issue whose symptoms resemble mine (the 8821CE ASPM L1
> freeze reports above), but my symptoms do not fully match them (unassociated /
> IPS path, and no rtw88 error message immediately before the freeze). Could you
> judge whether these are likely the same root cause or separate issues?

I can't judge if this is the same, but please try my suggestions.

> 
> If it helps, I can provide: kdump vmcore (from the forensics setup), netconsole
> captures, ftrace function traces of rtw_enter_ips / rtw_pci_link_ps /
> rtw_power_mode_change, and AER counters.

As your perspective, this is a low level problem. Without measuring hardware
signaling, it is hard to address... 

Ping-Ke