Re: [BUG] rtw88 8821ce: connection wedges (100% loss until reboot) with station power save

[Abdurrahman Karadag <abdurrahmankaradag19@gmail.com>]

> Is there are version it works well on your platform?
> I'm thinking bisect is a method to find cause.

Yes - and I have to correct my first mail, which said "also seen on
7.0.12". Going back through the system journal and pacman log:

  - 7.0.12 (Arch) ran on this laptop from 2026-06-16 to 2026-08-23. The
    14 boots the journal holds from that period show no wedge episode
    (defined as: DHCP lease acquired, then the gateway unreachable / DNS
    failing continuously until reboot). The few candidates are either
    sessions caught in the separate 63 s regulatory reconnect loop, or
    sporadic DNS timeouts spread over hours-long otherwise-working
    sessions (14-27 timeouts in 0.8-3.5 h), plus one 176 s blip - none is
    the 100%-loss-until-reboot pattern.
  - On 2026-08-23 18:41 pacman upgraded linux 7.0.12 -> 7.1.9 (in the same
    transaction: linux-firmware-realtek 20260519 -> 20260810, whose
    rtw8821c_fw.bin is byte-identical by sha256, and systemd 260.2 ->
    261.2 a few minutes earlier).
  - The first boot on 7.1.9 (2026-08-23 23:37) logged 12 wedge episodes,
    and it has recurred on most days since.

So for a bisect: good = 7.0.12, bad = 7.1.9 (with the caveat that
systemd changed in the same upgrade; I don't think networkd can explain
a per-AC L2 TX failure, but I mention it for completeness). I still have
the 7.0.12 package and can confirm by running it again for a few days,
and I'm happy to bisect the rtw88/mac80211 range between the two if you
think that's the right next step. (Under Windows the same laptop has
never shown this.)

> The full cold power-off boot includes above two settings, right?

Yes. /etc/modprobe.d had "options rtw88_pci disable_aspm=y" and
"options rtw88_core disable_lps_deep=y", initramfs rebuilt, full power
off, and after boot both parameters read Y in /sys/module/. The wedge
still happened on that boot.

> How did you measure the link? CAT-C?

Nothing special: 30-second windows of ping to the gateway plus
"ip neigh" state and "iw dev wlan0 station dump" counters, while
recording the delta of /sys/bus/pci/devices/0000:02:00.0/aer_dev_correctable
and counting RxErr lines in the kernel log for the same window. Examples:
one window had 0% loss with 6 new RxErr; another had 0% loss with 0
RxErr; over that boot 329 RxErr accumulated while the link was fine. And
the wedge itself occurred in a window with almost no RxErr. So I could
not find any correlation in either direction.

> Can you setup another WiFi as monitor mode to capture 802.11 packets?
> Use another WiFi monitor to see if RTL8821CE actually transmitted the packets.

Yes, I will. I need a second adapter for that; I'll do it on a 2.4 GHz
20 MHz network so a simple monitor-mode dongle is enough, and filter on
the laptop's TA/RA. I'll report whether the ARP/BE frames appear on the
air at all, and if they do, whether the AP ACKs them.

> I'm not sure why the size can affect the result. Normally large size
> is harder to transmit basically though.

You are right, and I withdraw the size theory. Re-examining the same
capture by 802.11 access category instead of size explains every frame
with no exceptions:

  - every laptop TX frame that provably reached the AP carries IP DSCP
    0xc0 (CS6 -> UP 6 -> AC_VO): the DHCP DISCOVER/REQUEST frames
    (systemd-networkd's DHCP client marks them CS6) and IGMPv3 reports;
  - every laptop TX frame that provably never arrived is AC_BE:
    89 ARP requests (no IP header -> BE), 13 unicast ARP replies,
    83 DNS queries (0 answers), 2 TCP SYNs (no SYN-ACK), mDNS/LLMNR;
  - sizes overlap the wrong way for a size theory: BE frames of 42-201 B
    all fail, VO frames of 54-345 B all pass.

So the wedge looks like "AC_BE TX dead, AC_VO TX alive", RX intact, and
it persists across disconnect/reconnect and across AP changes (so not
per-association state), cleared only by a reboot. It happened with
power save fully off.

As far as I can tell from reading the driver, BE and VO take different
paths (separate PCIe TX rings per AC, and on 8821C BE/BK on the LOW
TX-FIFO queue vs VO/VI on NORMAL), so a per-AC stall - e.g. the BE ring's
mac80211 queue left stopped, or the LOW queue paused/page-starved - would
match what I see (BE frames silently aged out, VO flowing, no driver
message). Does that sound plausible to you, or is there a more likely
place for a per-AC TX stall on this chip? The monitor-mode capture
should at least show whether BE frames reach the air at all.

When it wedges, besides the air capture, I plan to dump before rebooting:
/sys/kernel/debug/ieee80211/phy0/queues (per-hw-queue stop reasons),
stations/<ap>/aqm (per-TID backlog), and via rtw88 debugfs read_reg:
REG_TXPAUSE (0x522), REG_TXDMA_STATUS (0x210), REG_FIFOPAGE_INFO_2/3
(0x234/0x238) and the BE/VO TXBD indices (0x3A8/0x3A0), compared with a
healthy baseline. If there are better registers or a debugfs page for
the BE ring / LOW queue state on 8821C, please tell me and I'll dump
those instead.

> Please fully turn off power save when you do the tests to reduce one
> factor that can possibly cause TX slowly or stuck.

Will do - power save stays off for all further tests.

Thanks a lot for looking at this.