RE: [BUG] wifi: rtw88: Hard system freeze on RTL8821CE when power_save is enabled (LPS/ASPM conflict)

[Ping-Ke Shih <pkshih@realtek.com>]

LB F <goainwo@gmail.com> wrote:
> Hi Ping-Ke,
> 
> I am writing to formally report a critical bug that causes a hard
> system freeze on laptops equipped with the RTL8821CE WiFi module, and
> to propose solutions.
> 
> Description:
> On an HP laptop equipped with a Realtek RTL8821CE 802.11ac PCIe
> adapter (PCI ID: 10ec:c821), the system experiences a hard lockup
> (complete freeze of the UI and kernel, sysrq doesn't work, requires
> holding the power button) when the WiFi adapter enters the power
> saving state.
> 
> This issue occurs consistently across multiple Linux distributions and
> kernel versions (reproduced on upstream kernel 6.13 and 6.19-rc).
> 
> Steps to Reproduce:
> 1. Use a system with RTL8821CE (pci:10ec:c821).
> 2. Ensure NetworkManager is configured with wifi.powersave = 3 (or
> power saving is enabled via TLP/iw).
> 3. Connect to a WiFi network and let the system idle.
> 4. The system will eventually freeze completely.

Can you dig kernel log (by netconsole or ramoops) if something useful?
I'd like to know this is hardware level freeze or kernel can capture
something wrong. 

> 
> Workarounds that successfully prevent the freeze:
> * Passing disable_lps_deep=y to rtw88_core.
> * Passing disable_aspm=y to rtw88_pci (or pcie_aspm=off).
> * Disabling WiFi power save via NetworkManager.

Are these totally needed to workaround the problem? Or disable_aspm is
enough?

I'd list them in order of power consumption impact: (the topmost is lower impact)
  1. disable_aspm=y
  2. disable_lps_deep=y
  3. disable WiFi power save

If you can do experiments on your platform, we can be easier to decide
which workarounds are adopted.

> 
> Technical Analysis:
> The root cause appears to be an unhandled race condition or hardware
> bug between the adapter's Low Power State (LPS) Deep mode
> (LPS_DEEP_MODE_LCLK) and the PCIe Active State Power Management (ASPM
> L1) mechanism.
> 
> When the firmware drops into LPS_DEEP_MODE_LCLK concurrently with the
> PCIe bus entering ASPM L1, the chip fails to handle PCIe Wake
> signaling correctly. While there is an existing workaround in
> rtw_pci_napi_poll (pci.c:1806) that sets `rtwpci->rx_no_aspm = true`
> during NAPI poll for 8821CE, this polling wrapper is insufficient. The
> deadlock often occurs during idle states when polling isn't actively
> disabling ASPM, but the system suddenly needs to wake the radio.

`rtwpci->rx_no_aspm = true` was another workaround years ago on certain
platform. I'd say ASPM has many interoperability problems, even years ago.

But what does 'deadlock' mean? As I know NAPI poll is scheduled by ISR,
and going to receive packets. The rx_no_aspm workaround is to forcely turn
off ASPM during this period. 

> 
> Proposed Solutions:
> Given that LPS_DEEP_MODE_LCLK seems fundamentally unreliable on 8821ce
> PCIe variants when paired with standard Windows-era ASPM
> implementations on laptops (HP, Lenovo, ASUS are all affected), the
> most robust solution is to strip the unsupported deep sleep flag from
> the hardware spec.
> 
> ```diff
> --- a/drivers/net/wireless/realtek/rtw88/rtw8821c.c
> +++ b/drivers/net/wireless/realtek/rtw88/rtw8821c.c
> @@ -1999,7 +1999,7 @@ struct rtw_chip_info rtw8821c_hw_spec = {
> .bt_supported = true,
> .fbtc_has_ext_ctrl = true,
> .coex_info_hw_supported = true,
> - .lps_deep_mode_supported = BIT(LPS_DEEP_MODE_LCLK),
> + .lps_deep_mode_supported = 0, /* Disabled due to ASPM L1 hard locks */
> .dpk_supported = true,
> .pstdma_type = COEX_PSTDMA_FORCE_LPSOFF,
> .bfee_support = false,
> ```
> 
> Alternatively, a PCI Subsystem-based quirk should be introduced in
> rtw_pci_aspm_set() to refuse ASPM BIT_L1_SW_EN transitions for
> affected hardware IDs, similar to how CLKREQ issues are handled for
> 8822C via efuse->rfe_option.

I'd add a quirk to your platforms, so other platforms can still have
better power consumption. 

> 
> Cross-Reference Analysis of other RTL8821CE Bugs:
> After aggregating recent open bug reports for the 8821ce chip on
> Bugzilla (https://bugzilla.kernel.org), it is apparent that almost all
> of them are victims of the exact same underlying race condition.
> 1. Bug 215131: System freeze preceded by 'pci bus timeout, check dma
> status'. Workaround used: disable_aspm=1.
> 2. Bug 219830: Log shows 'firmware failed to leave lps state' and
> 'failed to send h2c command'. A direct smoking gun for LPS Deep mode
> freezing.
> 3. Bug 218697 & Bug 217491: Endless 'timed out to flush queue' floods.
> 4. Bug 217781 & Bug 216685: Random dropouts and low wireless speed.
> 
> Given the volume and age of these unresolved reports, disabling
> .lps_deep_mode_supported (or restricting ASPM L1) specifically for
> 10ec:c821 is desperately needed.
> 
> System Information:
> - Hardware: HP Notebook (SKU: P3S95EA#ACB, Family: 103C_5335KV)
> - CPU: Intel Core i3-5005U
> - WiFi PCI ID: 10ec:c821, Subsystem: 103c:831a
> - Kernel: 6.13 / 6.19
> - Driver module: rtw88_8821ce
> 
> I am happy to test any patches provided or formally submit the patch
> above if maintainers agree it is the right approach. Thank you!

We have not modified RTL8821CE for a long time, so I'd add workaround
to specific platform as mentioned above. 

Ping-Ke