[BUG] rtw88 8821ce: connection wedges (100% loss until reboot) with station power save

[BUG] rtw88 8821ce: connection wedges (100% loss until reboot) with station power save

[Abdurrahman Karadag]

Hardware:
  RTL8821CE [10ec:c821], subsystem AzureWave [1a3b:304a]
  PCIe root port Intel [8086:51be] (00:1c.6)
  ASUS Vivobook X1504ZA, firmware rtw8821c_fw.bin 24.11.0
  (feature word 0x7: SIG|LPS_C2H|LCLK, no TX_WAKE)
  Arch Linux, kernel 7.1.9 (also seen on 7.0.12).
  The same laptop had none of this under Windows.

Symptom:
  At some unpredictable point the connection wedges: traffic goes to 100%
  packet loss and STAYS there until a reboot. The interface still shows as
  associated (iwd/networkd report it connected, DHCP lease held), but nothing
  passes - gateway ARP goes INCOMPLETE and stays that way. It does not
  gradually degrade or self-heal; it is a hard stop that only a reboot clears.
  It tends to hit soon after boot / when joining a network rather than on a
  connection that has already been up for a long time.

  The one reliable workaround is disabling station power save:
      iw dev wlan0 set power_save off
  With power save off I have not hit the wedge; with it on it recurs. This is
  the strongest signal I have that the station-PS path is involved.

What I ruled out (each tested on this machine):
  - rtw88_pci disable_aspm=y      : no effect (the module param only gates the
                                    device DBI 0x719 bit; it does not call
                                    pci_disable_link_state, so host ASPM
                                    L1/L1ss stay on - link/l1* remain 1)
  - rtw88_core disable_lps_deep=y : no effect (firmware deep-PS off)
  - full cold power-off boot      : no effect
  - suspend/resume                : NOT the trigger - connectivity recovers
                                    fine after resume in my tests
  - The correctable PCIe "Physical Layer / RxErr" this card logs is DECOUPLED
    from the failure: I measured the link working perfectly both while RxErr is
    being logged and while it is absent. RxErr is not the cause.

Reproduction (honest):
  I could not reproduce the wedge deterministically. Controlled tests all
  passed on the stock driver: steady-state idle (minutes), long idle with an
  off-device pinger sending to the sleeping STA, and disconnect/reconnect
  loops. So it is intermittent and tied to station PS being active, but I have
  not found the exact trigger - which is why the reliable handle is
  "power_save off makes it stop".

Questions:
  - Is a hard wedge (RX/TX stops until reboot) with station power save a known
    failure mode on 8821ce? Does the driver have any watchdog/recovery for a
    firmware or RX-DMA stall on this chip, or does it rely on the firmware?
  - rtw_enter_lps_core() (ps.c) programs the same PS config for every chip
    (rlbm=1, smart_ps=2, awake_interval=1) with no chip-specific override.
    Given fw feature word 0x7 (no TX_WAKE) on 8821ce, could smart-PS mode 2
    be the problem, and would legacy PS-Poll (smart_ps=0) for
    RTW_CHIP_TYPE_8821C be worth trying?

  I am happy to test any patch and report back with Tested-by, and to capture
  logs, iw/AER dumps, rtw88 debugfs, or a PS/wake trace when it wedges.

Thanks.

Re: [BUG] rtw88 8821ce: connection wedges (100% loss until reboot) with station power save

[Abdurrahman Karadag]

Correction and new data.

First, a correction to my report: I have now hit the wedge with station
power save OFF, so power save is not the trigger. My earlier "power_save
off makes it stop" was coincidence on an intermittent bug. Please disregard
the PS/smart_ps angle.

Second, I captured a wedge with Wireshark on wlan0 (802.3 view, i.e. at the
netdev boundary), and the signature is much more specific than "connection
wedges":

  - RX is fully intact, including unicast: DHCP OFFER/ACK addressed to my
    MAC, an ICMP echo request from the router, TLS data from the router,
    and the gateway's ARP requests sent *unicast* to me were all received.
  - My DHCP DISCOVER/REQUEST (342/345 bytes, L2 broadcast) reach the AP and
    are answered within milliseconds - on two different APs (an Android
    hotspot and a MikroTik router).
  - 14-30 ms after those successful DHCP exchanges, my ARP requests for the
    gateway (42 bytes, L2 broadcast; 89 of them, 1/s) get zero replies on
    both networks.
  - The gateway ARPs *me* (unicast, 10 times at ~0.77 s intervals, then
    falls back to broadcast). I receive every request and reply immediately
    (42-byte unicast ARP reply) - yet it keeps asking, so my replies never
    reach it.
  - My TCP SYNs (78 bytes, unicast to the gateway MAC) get no SYN-ACK.

So the failing set is small STA->AP frames (42-byte ARP, both broadcast
and unicast; 78-byte SYN) and the working set is 342-byte broadcast DHCP,
with RX working throughout and no kernel/driver messages. Since DHCP
succeeds tens of milliseconds before ARP fails, this looks like a
per-frame property (frame size, or possibly ethertype) rather than a
temporal stall. Two unrelated APs show the identical pattern, so it is
not AP-specific. The interface stays associated; only a reboot clears it
(a live driver reload froze the machine once, so I avoid that).

Next time it wedges I will run a size probe (static ARP entry for the
gateway, then ping -s 8/56/200/400/1000) to confirm whether it is
size-dependent, plus station-dump tx-failed/retry deltas and a
neigh-flush -> reconnect -> link down/up ladder to see which layer holds
the wedge. If there is anything specific on the 8821c TX side you would
like me to dump (tx desc, debugfs, registers) while it is wedged, tell me
and I will capture it.

Capture available on request.

RE: [BUG] rtw88 8821ce: connection wedges (100% loss until reboot) with station power save

[Ping-Ke Shih]

Abdurrahman Karadag <abdurrahmankaradag19@gmail.com> wrote:
> Correction and new data.
> 
> First, a correction to my report: I have now hit the wedge with station
> power save OFF, so power save is not the trigger. My earlier "power_save
> off makes it stop" was coincidence on an intermittent bug. Please disregard
> the PS/smart_ps angle.
> 
> Second, I captured a wedge with Wireshark on wlan0 (802.3 view, i.e. at the
> netdev boundary), and the signature is much more specific than "connection
> wedges":
> 
>   - RX is fully intact, including unicast: DHCP OFFER/ACK addressed to my
>     MAC, an ICMP echo request from the router, TLS data from the router,
>     and the gateway's ARP requests sent *unicast* to me were all received.

Good to know RX is good. 

Can you setup another WiFi as monitor mode to capture 802.11 packets?

>   - My DHCP DISCOVER/REQUEST (342/345 bytes, L2 broadcast) reach the AP and
>     are answered within milliseconds - on two different APs (an Android
>     hotspot and a MikroTik router).
>   - 14-30 ms after those successful DHCP exchanges, my ARP requests for the
>     gateway (42 bytes, L2 broadcast; 89 of them, 1/s) get zero replies on
>     both networks.

Use another WiFi monitor to see if RTL8821CE actually transmitted the packets.

>   - The gateway ARPs *me* (unicast, 10 times at ~0.77 s intervals, then
>     falls back to broadcast). I receive every request and reply immediately
>     (42-byte unicast ARP reply) - yet it keeps asking, so my replies never
>     reach it.
>   - My TCP SYNs (78 bytes, unicast to the gateway MAC) get no SYN-ACK.
> 
> So the failing set is small STA->AP frames (42-byte ARP, both broadcast
> and unicast; 78-byte SYN) and the working set is 342-byte broadcast DHCP,
> with RX working throughout and no kernel/driver messages. Since DHCP
> succeeds tens of milliseconds before ARP fails, this looks like a
> per-frame property (frame size, or possibly ethertype) rather than a
> temporal stall. Two unrelated APs show the identical pattern, so it is
> not AP-specific. The interface stays associated; only a reboot clears it
> (a live driver reload froze the machine once, so I avoid that).
> 
> Next time it wedges I will run a size probe (static ARP entry for the
> gateway, then ping -s 8/56/200/400/1000) to confirm whether it is
> size-dependent, plus station-dump tx-failed/retry deltas and a
> neigh-flush -> reconnect -> link down/up ladder to see which layer holds
> the wedge. If there is anything specific on the 8821c TX side you would
> like me to dump (tx desc, debugfs, registers) while it is wedged, tell me
> and I will capture it.

I'm not sure why the size can affect the result. Normally large size
is harder to transmit basically though. 

Please fully turn off power save when you do the tests to reduce one
factor that can possibly cause TX slowly or stuck.

> 
> Capture available on request.

802.11 capture by another WiFi monitor is better. 

Re: [BUG] rtw88 8821ce: connection wedges (100% loss until reboot) with station power save

[Abdurrahman Karadag]

> Is there are version it works well on your platform?
> I'm thinking bisect is a method to find cause.

Yes - and I have to correct my first mail, which said "also seen on
7.0.12". Going back through the system journal and pacman log:

  - 7.0.12 (Arch) ran on this laptop from 2026-06-16 to 2026-08-23. The
    14 boots the journal holds from that period show no wedge episode
    (defined as: DHCP lease acquired, then the gateway unreachable / DNS
    failing continuously until reboot). The few candidates are either
    sessions caught in the separate 63 s regulatory reconnect loop, or
    sporadic DNS timeouts spread over hours-long otherwise-working
    sessions (14-27 timeouts in 0.8-3.5 h), plus one 176 s blip - none is
    the 100%-loss-until-reboot pattern.
  - On 2026-08-23 18:41 pacman upgraded linux 7.0.12 -> 7.1.9 (in the same
    transaction: linux-firmware-realtek 20260519 -> 20260810, whose
    rtw8821c_fw.bin is byte-identical by sha256, and systemd 260.2 ->
    261.2 a few minutes earlier).
  - The first boot on 7.1.9 (2026-08-23 23:37) logged 12 wedge episodes,
    and it has recurred on most days since.

So for a bisect: good = 7.0.12, bad = 7.1.9 (with the caveat that
systemd changed in the same upgrade; I don't think networkd can explain
a per-AC L2 TX failure, but I mention it for completeness). I still have
the 7.0.12 package and can confirm by running it again for a few days,
and I'm happy to bisect the rtw88/mac80211 range between the two if you
think that's the right next step. (Under Windows the same laptop has
never shown this.)

> The full cold power-off boot includes above two settings, right?

Yes. /etc/modprobe.d had "options rtw88_pci disable_aspm=y" and
"options rtw88_core disable_lps_deep=y", initramfs rebuilt, full power
off, and after boot both parameters read Y in /sys/module/. The wedge
still happened on that boot.

> How did you measure the link? CAT-C?

Nothing special: 30-second windows of ping to the gateway plus
"ip neigh" state and "iw dev wlan0 station dump" counters, while
recording the delta of /sys/bus/pci/devices/0000:02:00.0/aer_dev_correctable
and counting RxErr lines in the kernel log for the same window. Examples:
one window had 0% loss with 6 new RxErr; another had 0% loss with 0
RxErr; over that boot 329 RxErr accumulated while the link was fine. And
the wedge itself occurred in a window with almost no RxErr. So I could
not find any correlation in either direction.

> Can you setup another WiFi as monitor mode to capture 802.11 packets?
> Use another WiFi monitor to see if RTL8821CE actually transmitted the packets.

Yes, I will. I need a second adapter for that; I'll do it on a 2.4 GHz
20 MHz network so a simple monitor-mode dongle is enough, and filter on
the laptop's TA/RA. I'll report whether the ARP/BE frames appear on the
air at all, and if they do, whether the AP ACKs them.

> I'm not sure why the size can affect the result. Normally large size
> is harder to transmit basically though.

You are right, and I withdraw the size theory. Re-examining the same
capture by 802.11 access category instead of size explains every frame
with no exceptions:

  - every laptop TX frame that provably reached the AP carries IP DSCP
    0xc0 (CS6 -> UP 6 -> AC_VO): the DHCP DISCOVER/REQUEST frames
    (systemd-networkd's DHCP client marks them CS6) and IGMPv3 reports;
  - every laptop TX frame that provably never arrived is AC_BE:
    89 ARP requests (no IP header -> BE), 13 unicast ARP replies,
    83 DNS queries (0 answers), 2 TCP SYNs (no SYN-ACK), mDNS/LLMNR;
  - sizes overlap the wrong way for a size theory: BE frames of 42-201 B
    all fail, VO frames of 54-345 B all pass.

So the wedge looks like "AC_BE TX dead, AC_VO TX alive", RX intact, and
it persists across disconnect/reconnect and across AP changes (so not
per-association state), cleared only by a reboot. It happened with
power save fully off.

As far as I can tell from reading the driver, BE and VO take different
paths (separate PCIe TX rings per AC, and on 8821C BE/BK on the LOW
TX-FIFO queue vs VO/VI on NORMAL), so a per-AC stall - e.g. the BE ring's
mac80211 queue left stopped, or the LOW queue paused/page-starved - would
match what I see (BE frames silently aged out, VO flowing, no driver
message). Does that sound plausible to you, or is there a more likely
place for a per-AC TX stall on this chip? The monitor-mode capture
should at least show whether BE frames reach the air at all.

When it wedges, besides the air capture, I plan to dump before rebooting:
/sys/kernel/debug/ieee80211/phy0/queues (per-hw-queue stop reasons),
stations/<ap>/aqm (per-TID backlog), and via rtw88 debugfs read_reg:
REG_TXPAUSE (0x522), REG_TXDMA_STATUS (0x210), REG_FIFOPAGE_INFO_2/3
(0x234/0x238) and the BE/VO TXBD indices (0x3A8/0x3A0), compared with a
healthy baseline. If there are better registers or a debugfs page for
the BE ring / LOW queue state on 8821C, please tell me and I'll dump
those instead.

> Please fully turn off power save when you do the tests to reduce one
> factor that can possibly cause TX slowly or stuck.

Will do - power save stays off for all further tests.

Thanks a lot for looking at this.

RE: [BUG] rtw88 8821ce: connection wedges (100% loss until reboot) with station power save

[Ping-Ke Shih]

Abdurrahman Karadag <abdurrahmankaradag19@gmail.com> wrote:
> Hardware:
>   RTL8821CE [10ec:c821], subsystem AzureWave [1a3b:304a]
>   PCIe root port Intel [8086:51be] (00:1c.6)
>   ASUS Vivobook X1504ZA, firmware rtw8821c_fw.bin 24.11.0
>   (feature word 0x7: SIG|LPS_C2H|LCLK, no TX_WAKE)
>   Arch Linux, kernel 7.1.9 (also seen on 7.0.12).

Is there are version it works well on your platform?
I'm thinking bisect is a method to find cause. 

> 
> Symptom:
>   At some unpredictable point the connection wedges: traffic goes to 100%
>   packet loss and STAYS there until a reboot. The interface still shows as
>   associated (iwd/networkd report it connected, DHCP lease held), but nothing
>   passes - gateway ARP goes INCOMPLETE and stays that way. It does not
>   gradually degrade or self-heal; it is a hard stop that only a reboot clears.
>   It tends to hit soon after boot / when joining a network rather than on a
>   connection that has already been up for a long time.

I'd reply this by the latter mail of yours.

> 
>   The one reliable workaround is disabling station power save:
>       iw dev wlan0 set power_save off
>   With power save off I have not hit the wedge; with it on it recurs. This is
>   the strongest signal I have that the station-PS path is involved.

(Asked to disregard this.)

> 
> What I ruled out (each tested on this machine):
>   - rtw88_pci disable_aspm=y      : no effect (the module param only gates the
>                                     device DBI 0x719 bit; it does not call
>                                     pci_disable_link_state, so host ASPM
>                                     L1/L1ss stay on - link/l1* remain 1)
>   - rtw88_core disable_lps_deep=y : no effect (firmware deep-PS off)

The full cold power-off boot includes above two settings, right?

>   - full cold power-off boot      : no effect
>   - suspend/resume                : NOT the trigger - connectivity recovers
>                                     fine after resume in my tests
>   - The correctable PCIe "Physical Layer / RxErr" this card logs is DECOUPLED
>     from the failure: I measured the link working perfectly both while RxErr is
>     being logged and while it is absent. RxErr is not the cause.

How did you measure the link? CAT-C?

> 
> Reproduction (honest):
>   I could not reproduce the wedge deterministically. Controlled tests all
>   passed on the stock driver: steady-state idle (minutes), long idle with an
>   off-device pinger sending to the sleeping STA, and disconnect/reconnect
>   loops. So it is intermittent and tied to station PS being active, but I have
>   not found the exact trigger - which is why the reliable handle is
>   "power_save off makes it stop".
> 
> Questions:
>   - Is a hard wedge (RX/TX stops until reboot) with station power save a known
>     failure mode on 8821ce? Does the driver have any watchdog/recovery for a
>     firmware or RX-DMA stall on this chip, or does it rely on the firmware?

The latter mail of yours explain RX is fully intact.

For RX path, if beacon gets loss, it will disconnect, so I think RX still works
for your case.

>   - rtw_enter_lps_core() (ps.c) programs the same PS config for every chip
>     (rlbm=1, smart_ps=2, awake_interval=1) with no chip-specific override.
>     Given fw feature word 0x7 (no TX_WAKE) on 8821ce, could smart-PS mode 2
>     be the problem, and would legacy PS-Poll (smart_ps=0) for
>     RTW_CHIP_TYPE_8821C be worth trying?

It looks like it still happened if you entirely turn off power save, so
ignore this...