[BUG] rtw88 8821ce: connection wedges (100% loss until reboot) with station power save

[BUG] rtw88 8821ce: connection wedges (100% loss until reboot) with station power save

[Abdurrahman Karadag]

Hardware:
  RTL8821CE [10ec:c821], subsystem AzureWave [1a3b:304a]
  PCIe root port Intel [8086:51be] (00:1c.6)
  ASUS Vivobook X1504ZA, firmware rtw8821c_fw.bin 24.11.0
  (feature word 0x7: SIG|LPS_C2H|LCLK, no TX_WAKE)
  Arch Linux, kernel 7.1.9 (also seen on 7.0.12).
  The same laptop had none of this under Windows.

Symptom:
  At some unpredictable point the connection wedges: traffic goes to 100%
  packet loss and STAYS there until a reboot. The interface still shows as
  associated (iwd/networkd report it connected, DHCP lease held), but nothing
  passes - gateway ARP goes INCOMPLETE and stays that way. It does not
  gradually degrade or self-heal; it is a hard stop that only a reboot clears.
  It tends to hit soon after boot / when joining a network rather than on a
  connection that has already been up for a long time.

  The one reliable workaround is disabling station power save:
      iw dev wlan0 set power_save off
  With power save off I have not hit the wedge; with it on it recurs. This is
  the strongest signal I have that the station-PS path is involved.

What I ruled out (each tested on this machine):
  - rtw88_pci disable_aspm=y      : no effect (the module param only gates the
                                    device DBI 0x719 bit; it does not call
                                    pci_disable_link_state, so host ASPM
                                    L1/L1ss stay on - link/l1* remain 1)
  - rtw88_core disable_lps_deep=y : no effect (firmware deep-PS off)
  - full cold power-off boot      : no effect
  - suspend/resume                : NOT the trigger - connectivity recovers
                                    fine after resume in my tests
  - The correctable PCIe "Physical Layer / RxErr" this card logs is DECOUPLED
    from the failure: I measured the link working perfectly both while RxErr is
    being logged and while it is absent. RxErr is not the cause.

Reproduction (honest):
  I could not reproduce the wedge deterministically. Controlled tests all
  passed on the stock driver: steady-state idle (minutes), long idle with an
  off-device pinger sending to the sleeping STA, and disconnect/reconnect
  loops. So it is intermittent and tied to station PS being active, but I have
  not found the exact trigger - which is why the reliable handle is
  "power_save off makes it stop".

Questions:
  - Is a hard wedge (RX/TX stops until reboot) with station power save a known
    failure mode on 8821ce? Does the driver have any watchdog/recovery for a
    firmware or RX-DMA stall on this chip, or does it rely on the firmware?
  - rtw_enter_lps_core() (ps.c) programs the same PS config for every chip
    (rlbm=1, smart_ps=2, awake_interval=1) with no chip-specific override.
    Given fw feature word 0x7 (no TX_WAKE) on 8821ce, could smart-PS mode 2
    be the problem, and would legacy PS-Poll (smart_ps=0) for
    RTW_CHIP_TYPE_8821C be worth trying?

  I am happy to test any patch and report back with Tested-by, and to capture
  logs, iw/AER dumps, rtw88 debugfs, or a PS/wake trace when it wedges.

Thanks.

Re: [BUG] rtw88 8821ce: connection wedges (100% loss until reboot) with station power save

[Abdurrahman Karadag]

Correction and new data.

First, a correction to my report: I have now hit the wedge with station
power save OFF, so power save is not the trigger. My earlier "power_save
off makes it stop" was coincidence on an intermittent bug. Please disregard
the PS/smart_ps angle.

Second, I captured a wedge with Wireshark on wlan0 (802.3 view, i.e. at the
netdev boundary), and the signature is much more specific than "connection
wedges":

  - RX is fully intact, including unicast: DHCP OFFER/ACK addressed to my
    MAC, an ICMP echo request from the router, TLS data from the router,
    and the gateway's ARP requests sent *unicast* to me were all received.
  - My DHCP DISCOVER/REQUEST (342/345 bytes, L2 broadcast) reach the AP and
    are answered within milliseconds - on two different APs (an Android
    hotspot and a MikroTik router).
  - 14-30 ms after those successful DHCP exchanges, my ARP requests for the
    gateway (42 bytes, L2 broadcast; 89 of them, 1/s) get zero replies on
    both networks.
  - The gateway ARPs *me* (unicast, 10 times at ~0.77 s intervals, then
    falls back to broadcast). I receive every request and reply immediately
    (42-byte unicast ARP reply) - yet it keeps asking, so my replies never
    reach it.
  - My TCP SYNs (78 bytes, unicast to the gateway MAC) get no SYN-ACK.

So the failing set is small STA->AP frames (42-byte ARP, both broadcast
and unicast; 78-byte SYN) and the working set is 342-byte broadcast DHCP,
with RX working throughout and no kernel/driver messages. Since DHCP
succeeds tens of milliseconds before ARP fails, this looks like a
per-frame property (frame size, or possibly ethertype) rather than a
temporal stall. Two unrelated APs show the identical pattern, so it is
not AP-specific. The interface stays associated; only a reboot clears it
(a live driver reload froze the machine once, so I avoid that).

Next time it wedges I will run a size probe (static ARP entry for the
gateway, then ping -s 8/56/200/400/1000) to confirm whether it is
size-dependent, plus station-dump tx-failed/retry deltas and a
neigh-flush -> reconnect -> link down/up ladder to see which layer holds
the wedge. If there is anything specific on the 8821c TX side you would
like me to dump (tx desc, debugfs, registers) while it is wedged, tell me
and I will capture it.

Capture available on request.

Re: [BUG] rtw88 8821ce: connection wedges (100% loss until reboot) with station power save

[Abdurrahman Karadag]

> Is there are version it works well on your platform?
> I'm thinking bisect is a method to find cause.

Yes - and I have to correct my first mail, which said "also seen on
7.0.12". Going back through the system journal and pacman log:

  - 7.0.12 (Arch) ran on this laptop from 2026-06-16 to 2026-08-23. The
    14 boots the journal holds from that period show no wedge episode
    (defined as: DHCP lease acquired, then the gateway unreachable / DNS
    failing continuously until reboot). The few candidates are either
    sessions caught in the separate 63 s regulatory reconnect loop, or
    sporadic DNS timeouts spread over hours-long otherwise-working
    sessions (14-27 timeouts in 0.8-3.5 h), plus one 176 s blip - none is
    the 100%-loss-until-reboot pattern.
  - On 2026-08-23 18:41 pacman upgraded linux 7.0.12 -> 7.1.9 (in the same
    transaction: linux-firmware-realtek 20260519 -> 20260810, whose
    rtw8821c_fw.bin is byte-identical by sha256, and systemd 260.2 ->
    261.2 a few minutes earlier).
  - The first boot on 7.1.9 (2026-08-23 23:37) logged 12 wedge episodes,
    and it has recurred on most days since.

So for a bisect: good = 7.0.12, bad = 7.1.9 (with the caveat that
systemd changed in the same upgrade; I don't think networkd can explain
a per-AC L2 TX failure, but I mention it for completeness). I still have
the 7.0.12 package and can confirm by running it again for a few days,
and I'm happy to bisect the rtw88/mac80211 range between the two if you
think that's the right next step. (Under Windows the same laptop has
never shown this.)

> The full cold power-off boot includes above two settings, right?

Yes. /etc/modprobe.d had "options rtw88_pci disable_aspm=y" and
"options rtw88_core disable_lps_deep=y", initramfs rebuilt, full power
off, and after boot both parameters read Y in /sys/module/. The wedge
still happened on that boot.

> How did you measure the link? CAT-C?

Nothing special: 30-second windows of ping to the gateway plus
"ip neigh" state and "iw dev wlan0 station dump" counters, while
recording the delta of /sys/bus/pci/devices/0000:02:00.0/aer_dev_correctable
and counting RxErr lines in the kernel log for the same window. Examples:
one window had 0% loss with 6 new RxErr; another had 0% loss with 0
RxErr; over that boot 329 RxErr accumulated while the link was fine. And
the wedge itself occurred in a window with almost no RxErr. So I could
not find any correlation in either direction.

> Can you setup another WiFi as monitor mode to capture 802.11 packets?
> Use another WiFi monitor to see if RTL8821CE actually transmitted the packets.

Yes, I will. I need a second adapter for that; I'll do it on a 2.4 GHz
20 MHz network so a simple monitor-mode dongle is enough, and filter on
the laptop's TA/RA. I'll report whether the ARP/BE frames appear on the
air at all, and if they do, whether the AP ACKs them.

> I'm not sure why the size can affect the result. Normally large size
> is harder to transmit basically though.

You are right, and I withdraw the size theory. Re-examining the same
capture by 802.11 access category instead of size explains every frame
with no exceptions:

  - every laptop TX frame that provably reached the AP carries IP DSCP
    0xc0 (CS6 -> UP 6 -> AC_VO): the DHCP DISCOVER/REQUEST frames
    (systemd-networkd's DHCP client marks them CS6) and IGMPv3 reports;
  - every laptop TX frame that provably never arrived is AC_BE:
    89 ARP requests (no IP header -> BE), 13 unicast ARP replies,
    83 DNS queries (0 answers), 2 TCP SYNs (no SYN-ACK), mDNS/LLMNR;
  - sizes overlap the wrong way for a size theory: BE frames of 42-201 B
    all fail, VO frames of 54-345 B all pass.

So the wedge looks like "AC_BE TX dead, AC_VO TX alive", RX intact, and
it persists across disconnect/reconnect and across AP changes (so not
per-association state), cleared only by a reboot. It happened with
power save fully off.

As far as I can tell from reading the driver, BE and VO take different
paths (separate PCIe TX rings per AC, and on 8821C BE/BK on the LOW
TX-FIFO queue vs VO/VI on NORMAL), so a per-AC stall - e.g. the BE ring's
mac80211 queue left stopped, or the LOW queue paused/page-starved - would
match what I see (BE frames silently aged out, VO flowing, no driver
message). Does that sound plausible to you, or is there a more likely
place for a per-AC TX stall on this chip? The monitor-mode capture
should at least show whether BE frames reach the air at all.

When it wedges, besides the air capture, I plan to dump before rebooting:
/sys/kernel/debug/ieee80211/phy0/queues (per-hw-queue stop reasons),
stations/<ap>/aqm (per-TID backlog), and via rtw88 debugfs read_reg:
REG_TXPAUSE (0x522), REG_TXDMA_STATUS (0x210), REG_FIFOPAGE_INFO_2/3
(0x234/0x238) and the BE/VO TXBD indices (0x3A8/0x3A0), compared with a
healthy baseline. If there are better registers or a debugfs page for
the BE ring / LOW queue state on 8821C, please tell me and I'll dump
those instead.

> Please fully turn off power save when you do the tests to reduce one
> factor that can possibly cause TX slowly or stuck.

Will do - power save stays off for all further tests.

Thanks a lot for looking at this.

RE: [BUG] rtw88 8821ce: connection wedges (100% loss until reboot) with station power save

[Ping-Ke Shih]

Abdurrahman Karadag <abdurrahmankaradag19@gmail.com> wrote:
> Hardware:
>   RTL8821CE [10ec:c821], subsystem AzureWave [1a3b:304a]
>   PCIe root port Intel [8086:51be] (00:1c.6)
>   ASUS Vivobook X1504ZA, firmware rtw8821c_fw.bin 24.11.0
>   (feature word 0x7: SIG|LPS_C2H|LCLK, no TX_WAKE)
>   Arch Linux, kernel 7.1.9 (also seen on 7.0.12).

Is there are version it works well on your platform?
I'm thinking bisect is a method to find cause. 

> 
> Symptom:
>   At some unpredictable point the connection wedges: traffic goes to 100%
>   packet loss and STAYS there until a reboot. The interface still shows as
>   associated (iwd/networkd report it connected, DHCP lease held), but nothing
>   passes - gateway ARP goes INCOMPLETE and stays that way. It does not
>   gradually degrade or self-heal; it is a hard stop that only a reboot clears.
>   It tends to hit soon after boot / when joining a network rather than on a
>   connection that has already been up for a long time.

I'd reply this by the latter mail of yours.

> 
>   The one reliable workaround is disabling station power save:
>       iw dev wlan0 set power_save off
>   With power save off I have not hit the wedge; with it on it recurs. This is
>   the strongest signal I have that the station-PS path is involved.

(Asked to disregard this.)

> 
> What I ruled out (each tested on this machine):
>   - rtw88_pci disable_aspm=y      : no effect (the module param only gates the
>                                     device DBI 0x719 bit; it does not call
>                                     pci_disable_link_state, so host ASPM
>                                     L1/L1ss stay on - link/l1* remain 1)
>   - rtw88_core disable_lps_deep=y : no effect (firmware deep-PS off)

The full cold power-off boot includes above two settings, right?

>   - full cold power-off boot      : no effect
>   - suspend/resume                : NOT the trigger - connectivity recovers
>                                     fine after resume in my tests
>   - The correctable PCIe "Physical Layer / RxErr" this card logs is DECOUPLED
>     from the failure: I measured the link working perfectly both while RxErr is
>     being logged and while it is absent. RxErr is not the cause.

How did you measure the link? CAT-C?

> 
> Reproduction (honest):
>   I could not reproduce the wedge deterministically. Controlled tests all
>   passed on the stock driver: steady-state idle (minutes), long idle with an
>   off-device pinger sending to the sleeping STA, and disconnect/reconnect
>   loops. So it is intermittent and tied to station PS being active, but I have
>   not found the exact trigger - which is why the reliable handle is
>   "power_save off makes it stop".
> 
> Questions:
>   - Is a hard wedge (RX/TX stops until reboot) with station power save a known
>     failure mode on 8821ce? Does the driver have any watchdog/recovery for a
>     firmware or RX-DMA stall on this chip, or does it rely on the firmware?

The latter mail of yours explain RX is fully intact.

For RX path, if beacon gets loss, it will disconnect, so I think RX still works
for your case.

>   - rtw_enter_lps_core() (ps.c) programs the same PS config for every chip
>     (rlbm=1, smart_ps=2, awake_interval=1) with no chip-specific override.
>     Given fw feature word 0x7 (no TX_WAKE) on 8821ce, could smart-PS mode 2
>     be the problem, and would legacy PS-Poll (smart_ps=0) for
>     RTW_CHIP_TYPE_8821C be worth trying?

It looks like it still happened if you entirely turn off power save, so
ignore this...

RE: [BUG] rtw88 8821ce: connection wedges (100% loss until reboot) with station power save

[Ping-Ke Shih]

Abdurrahman Karadag <abdurrahmankaradag19@gmail.com> wrote:
> Correction and new data.
> 
> First, a correction to my report: I have now hit the wedge with station
> power save OFF, so power save is not the trigger. My earlier "power_save
> off makes it stop" was coincidence on an intermittent bug. Please disregard
> the PS/smart_ps angle.
> 
> Second, I captured a wedge with Wireshark on wlan0 (802.3 view, i.e. at the
> netdev boundary), and the signature is much more specific than "connection
> wedges":
> 
>   - RX is fully intact, including unicast: DHCP OFFER/ACK addressed to my
>     MAC, an ICMP echo request from the router, TLS data from the router,
>     and the gateway's ARP requests sent *unicast* to me were all received.

Good to know RX is good. 

Can you setup another WiFi as monitor mode to capture 802.11 packets?

>   - My DHCP DISCOVER/REQUEST (342/345 bytes, L2 broadcast) reach the AP and
>     are answered within milliseconds - on two different APs (an Android
>     hotspot and a MikroTik router).
>   - 14-30 ms after those successful DHCP exchanges, my ARP requests for the
>     gateway (42 bytes, L2 broadcast; 89 of them, 1/s) get zero replies on
>     both networks.

Use another WiFi monitor to see if RTL8821CE actually transmitted the packets.

>   - The gateway ARPs *me* (unicast, 10 times at ~0.77 s intervals, then
>     falls back to broadcast). I receive every request and reply immediately
>     (42-byte unicast ARP reply) - yet it keeps asking, so my replies never
>     reach it.
>   - My TCP SYNs (78 bytes, unicast to the gateway MAC) get no SYN-ACK.
> 
> So the failing set is small STA->AP frames (42-byte ARP, both broadcast
> and unicast; 78-byte SYN) and the working set is 342-byte broadcast DHCP,
> with RX working throughout and no kernel/driver messages. Since DHCP
> succeeds tens of milliseconds before ARP fails, this looks like a
> per-frame property (frame size, or possibly ethertype) rather than a
> temporal stall. Two unrelated APs show the identical pattern, so it is
> not AP-specific. The interface stays associated; only a reboot clears it
> (a live driver reload froze the machine once, so I avoid that).
> 
> Next time it wedges I will run a size probe (static ARP entry for the
> gateway, then ping -s 8/56/200/400/1000) to confirm whether it is
> size-dependent, plus station-dump tx-failed/retry deltas and a
> neigh-flush -> reconnect -> link down/up ladder to see which layer holds
> the wedge. If there is anything specific on the 8821c TX side you would
> like me to dump (tx desc, debugfs, registers) while it is wedged, tell me
> and I will capture it.

I'm not sure why the size can affect the result. Normally large size
is harder to transmit basically though. 

Please fully turn off power save when you do the tests to reduce one
factor that can possibly cause TX slowly or stuck.

> 
> Capture available on request.

802.11 capture by another WiFi monitor is better. 

RE: [BUG] rtw88 8821ce: connection wedges (100% loss until reboot) with station power save

[Ping-Ke Shih]

Abdurrahman Karadag <abdurrahmankaradag19@gmail.com> wrote:
> > Is there are version it works well on your platform?
> > I'm thinking bisect is a method to find cause.
> 
> Yes - and I have to correct my first mail, which said "also seen on
> 7.0.12". Going back through the system journal and pacman log:
> 
>   - 7.0.12 (Arch) ran on this laptop from 2026-06-16 to 2026-08-23. The
>     14 boots the journal holds from that period show no wedge episode
>     (defined as: DHCP lease acquired, then the gateway unreachable / DNS
>     failing continuously until reboot). The few candidates are either
>     sessions caught in the separate 63 s regulatory reconnect loop, or
>     sporadic DNS timeouts spread over hours-long otherwise-working
>     sessions (14-27 timeouts in 0.8-3.5 h), plus one 176 s blip - none is
>     the 100%-loss-until-reboot pattern.
>   - On 2026-08-23 18:41 pacman upgraded linux 7.0.12 -> 7.1.9 (in the same
>     transaction: linux-firmware-realtek 20260519 -> 20260810, whose
>     rtw8821c_fw.bin is byte-identical by sha256, and systemd 260.2 ->
>     261.2 a few minutes earlier).
>   - The first boot on 7.1.9 (2026-08-23 23:37) logged 12 wedge episodes,
>     and it has recurred on most days since.
> 
> So for a bisect: good = 7.0.12, bad = 7.1.9 (with the caveat that
> systemd changed in the same upgrade; I don't think networkd can explain
> a per-AC L2 TX failure, but I mention it for completeness). I still have
> the 7.0.12 package and can confirm by running it again for a few days,
> and I'm happy to bisect the rtw88/mac80211 range between the two if you
> think that's the right next step. (Under Windows the same laptop has
> never shown this.)

Checking commits between 7.0.12 and 7.1.9, the only related commit might be
   c95323ea9dfb ("wifi: rtw88: coex: Solve LE-HID lag & update coex version to 26020420")

You can revert the patch from 7.1.9 to see if it becomes normal.

Another simple way is to use 7.0.12 kernel + 7.1.9 rtw88 driver and
opposite combination to address the cause, like

     kernel         rtw88 driver         result
     ------         ------------         ------
     7.0.12         (built-in)           Good
     7.0.12         7.1.9
     7.1.9          (built-in)           NG
     7.1.9          7.0.12 

This can also bisect if the cause is driver or mac80211.

> 
> > I'm not sure why the size can affect the result. Normally large size
> > is harder to transmit basically though.
> 
> You are right, and I withdraw the size theory. Re-examining the same
> capture by 802.11 access category instead of size explains every frame
> with no exceptions:
> 
>   - every laptop TX frame that provably reached the AP carries IP DSCP
>     0xc0 (CS6 -> UP 6 -> AC_VO): the DHCP DISCOVER/REQUEST frames
>     (systemd-networkd's DHCP client marks them CS6) and IGMPv3 reports;
>   - every laptop TX frame that provably never arrived is AC_BE:
>     89 ARP requests (no IP header -> BE), 13 unicast ARP replies,
>     83 DNS queries (0 answers), 2 TCP SYNs (no SYN-ACK), mDNS/LLMNR;
>   - sizes overlap the wrong way for a size theory: BE frames of 42-201 B
>     all fail, VO frames of 54-345 B all pass.
> 
> So the wedge looks like "AC_BE TX dead, AC_VO TX alive", RX intact, and
> it persists across disconnect/reconnect and across AP changes (so not
> per-association state), cleared only by a reboot. It happened with
> power save fully off.
> 
> As far as I can tell from reading the driver, BE and VO take different
> paths (separate PCIe TX rings per AC, and on 8821C BE/BK on the LOW
> TX-FIFO queue vs VO/VI on NORMAL), so a per-AC stall - e.g. the BE ring's
> mac80211 queue left stopped, or the LOW queue paused/page-starved - would
> match what I see (BE frames silently aged out, VO flowing, no driver
> message). Does that sound plausible to you, or is there a more likely
> place for a per-AC TX stall on this chip? The monitor-mode capture
> should at least show whether BE frames reach the air at all.

I think BE and VO should almost the same, but as your perspective
BE and VO go via different paths. It is worth to do an experiment
to let all packets (by driver modification) go via VO queue.

> 
> When it wedges, besides the air capture, I plan to dump before rebooting:
> /sys/kernel/debug/ieee80211/phy0/queues (per-hw-queue stop reasons),
> stations/<ap>/aqm (per-TID backlog), and via rtw88 debugfs read_reg:
> REG_TXPAUSE (0x522), REG_TXDMA_STATUS (0x210), REG_FIFOPAGE_INFO_2/3
> (0x234/0x238) and the BE/VO TXBD indices (0x3A8/0x3A0), compared with a
> healthy baseline. If there are better registers or a debugfs page for
> the BE ring / LOW queue state on 8821C, please tell me and I'll dump
> those instead.

Before checking these values, let's bisect the code, check sniffer, and 
do experiment first.

Re: [BUG] rtw88 8821ce: connection wedges (100% loss until reboot) with station power save

[Abdurrahman Karadag]

> Checking commits between 7.0.12 and 7.1.9, the only related commit might be
>    c95323ea9dfb ("wifi: rtw88: coex: Solve LE-HID lag & update coex version
>    to 26020420")
> You can revert the patch from 7.1.9 to see if it becomes normal.

I reverted it from 7.1.9 and the wedge still happened, about an hour after
boot, so that commit is not the cause. BT is disabled on this machine
anyway (coex_info reports "BT disabled", BT status non-conn).

Rather than continue bisecting, I set up a watchdog that dumps
queues/aqm/TXBD indices at the moment of the wedge, before anything
touches the interface. I have five such dumps now (I discount one of
them, taken on a captive portal network where the ping probes are
unreliable), plus healthy baselines, and they show a consistent
driver-level failure. I have also been able to reproduce that failure
deterministically and to recover from it. Details below; I am
sending a patch as a separate mail.

1. What the wedge looks like from inside the driver
---------------------------------------------------

This is the dump from 2026-08-28 22:05, taken with power save off, on
5 GHz, 91 minutes into the association:

  /sys/kernel/debug/ieee80211/phy0/queues
    00: 0x00000000/0   (VO)
    01: 0x00000000/0   (VI)
    02: 0x00000001/0   (BE)  <- stopped, reason bit 0 = DRIVER
    03: 0x00000000/0   (BK)

  stations/<ap>/aqm tid 0 (BE): backlog 20941 -> 26960 bytes,
    94 -> 144 packets, flags DIRTY, still growing while sampled

  iw station dump, 5 s apart:
    tx packets  116017 -> 116017   (frozen)
    rx packets  244885 -> 244992
    beacon rx    53304 ->  53353

  TXBD_IDX_BEQ (0x3A8) = 0x003c003a in all four samples, taken over
    about 15 seconds

  dmesg, shortly before: "failed to get tx report from firmware"

The BE queue is stopped, and rtw_pci_tx_write() only does that when the
ring has no room left:

    if (avail_desc(ring->r.wp, ring->r.rp, ring->r.len) < 2) {
            ieee80211_stop_queue(rtwdev->hw, skb_get_queue_mapping(skb));
            ring->queue_stopped = true;
    }

The only ieee80211_wake_queue() for that ring is inside the completion
loop of rtw_pci_tx_isr():

    count = cur_rp - ring->r.rp;
    while (count--) {
            ...
            if (ring->queue_stopped && avail_desc(...) > 4)
                    ieee80211_wake_queue(hw, q_map);
            ...
    }

so it can only run if the hardware read index has advanced. In the dump
that index is identical in every sample, the ring stays full, and the
queue is therefore never woken again. Nothing is logged, no counter
moves, and there is no watchdog for this state, which is why the
interface stays associated with RX working while TX is silently dead.

A second dump (2026-08-30 09:12, this one with power save on) shows the
same thing: BE stopped with reason DRIVER, backlog growing, tx packets
frozen at 115902 while rx packets and beacon counters keep increasing,
TXBD_IDX_BEQ = 0x001c001a identical at +2 s, +7 s and +2 min.

Two other dumps caught an earlier stage of the same failure: the read
index was already frozen while the ring still had room, so the queue was
not stopped yet. In the 2026-08-28 20:32 dump the read index stays at
0x45 across four samples while the write index moves 0x19 -> 0x1b ->
0x20 -> 0x22, i.e. frames were still being written into a ring that the
hardware had stopped draining.

2. Reproducing it deterministically
------------------------------------

Since I could not wait for another occurrence, I broke the TX path on
purpose: a temporary module parameter that skips the doorbell
write in rtw_pci_tx_kick_off_queue() for the BE queue, simulating a lost
kick-off. With BE traffic running, this reproduces the failure in about
12 seconds:

    BE ring 0x000e000e   q02 0x00000001   100% loss

that is, the same observable state as the dumps above: BE queue stopped
with reason DRIVER, ring full, read index not moving, TX dead, RX
unaffected. A Wireshark capture on wlan0 during the artificial stall
looks like the ones taken during real wedges: our ARP requests leave the
netdev and no replies arrive, while RX keeps working (artificial: 40 TX
/ 51 RX, 9 ARP requests, 0 replies; real: 193 / 300, 186 requests, 0
replies).

3. Recovery
-----------

I then added a TX stall check to rtw_watch_dog_work(): for each TX ring,
if there is outstanding work (queue stopped, or cur_rp != ring->r.wp)
and the hardware read index has not moved for three consecutive rounds
(about 6 s), run the kick-off for that queue again. Controlled run, with
the recovery selectable at runtime so both arms use the same build:

  13:01:16.293  TX queue 1 stalled (rp 14 wp 170), recovery DISABLED
  13:01:22.309  TX queue 1 stalled (rp 14 wp 5),   recovery DISABLED
  13:01:28.325  TX queue 1 stalled (rp 14 wp 12),  recovery DISABLED
  13:01:34.341  TX queue 1 stalled (rp 14 wp 12),  recovery DISABLED
  13:01:40.293  TX queue 1 stalled (rp 14 wp 12),  recovery DISABLED
  13:01:46.309  TX queue 1 stalled (rp 14 wp 12, stopped), kicking

With recovery disabled the stall persisted for 24 s across five
detection rounds and did not clear by itself. One kick-off was enough:
the ring went from 0x000e000e to 0x00470047, the queue stop flag
cleared, and ping returned to 0% loss within a second of the kick. The
wake up then happens through the existing path, because once the
hardware starts consuming again rtw_pci_tx_isr() runs with a non-zero
count.

4. What this does and does not prove
-------------------------------------

The artificial stall simulates a lost doorbell, so I cannot claim the
root cause in the field is the same. I looked for a register-level
difference between the two and did not find one. I had suspected the
real dumps were special because the hardware read index sits a couple of
slots ahead of the last announced write index, but after instrumenting
the driver I see the same relationship during perfectly healthy
operation, when the ring fills up under load and the queue is stopped
and woken again within milliseconds. The only thing that distinguishes
the failure is that the read index stops advancing at all.

What the experiment does show is that once a TX ring stops draining, the
driver has no way back, and that a re-kick recovers at least the
lost-doorbell case.

I still do not know why the hardware stops consuming descriptors in the
field, and it is not easy to catch: since the dumps above I have gone
about 74 hours without a single occurrence, across five driver builds
(including the stock one), five networks and both bands, with the
association living well past the ages at which it used to happen. It
seems to come in bursts - three occurrences in four hours on one
evening, one two days later, then nothing.

The 802.11 monitor capture you asked for is still on my list (the second
adapter is on the way), and my watchdog now also re-writes the doorbell
during a real wedge so that I can tell whether a re-kick is enough there
too. I will report that separately.

5. Corrections to my earlier mails
-----------------------------------

  - "persists across disconnect/reconnect ... cleared only by a reboot":
    only true once the ring is full. Two of the wedges were cleared by a
    reconnect, and the 2026-08-30 one by ip link set wlan0 down/up,
    which reinitialises the rings.
  - "AC_BE TX dead, AC_VO TX alive": that was the early stage. In the
    two dumps where the ring is full, ping -Q 0xc0 fails as well, so by
    then TX is dead on VO too.
  - "it tends to hit soon after boot": wrong. The association ages at
    the moment of the four wedges were 95, 91, 36 and 61 minutes.
  - Power save: three of the five dumps were taken with PS on, because
    my udev rule only ran on ACTION=="add" and the setting reverts after
    a link down/up. It is now forced off every 20 s. The two dumps I
    rely on most, 2026-08-28 20:32 and 22:05, were both taken with PS
    off, and they show the same state as the PS-on ones, so PS does not
    look like a factor.

I will send the patch separately so that it lands in patchwork on its
own. It only adds the stall detection and the re-kick; it does not touch
the normal TX path, it skips rings with nothing in flight so an idle
device is never poked, and it warns once per stall rather than on every
retry.

One question, in case a re-kick turns out not to be enough in the field.
The out-of-tree rtw88 driver has a pci_old.c for the older PCIe
generation, where the PCIe DMA is reset after a TRX hang using the two
status bits at REG_DBI_CTRL + 3 (bit 0 TX, bit 1 RX), with bit 2
enabling the detection. Is that status bit valid on 8821CE as well? If
it is, I can dump it during the next wedge and, if it confirms a DMA
hang, a reset could be added as a second stage.

Full dumps (five wedges, healthy baselines, the controlled run) and the
captures are available on request.

RE: [BUG] rtw88 8821ce: connection wedges (100% loss until reboot) with station power save

[Ping-Ke Shih]

Abdurrahman Karadag <abdurrahmankaradag19@gmail.com> wrote:
> > Checking commits between 7.0.12 and 7.1.9, the only related commit might be
> >    c95323ea9dfb ("wifi: rtw88: coex: Solve LE-HID lag & update coex version
> >    to 26020420")
> > You can revert the patch from 7.1.9 to see if it becomes normal.
> 
> I reverted it from 7.1.9 and the wedge still happened, about an hour after
> boot, so that commit is not the cause. BT is disabled on this machine
> anyway (coex_info reports "BT disabled", BT status non-conn).

I wonder even you use driver of 7.0.12 (on 7.1.9). It will still happen.
Not sure if host side change something?
Can you 100% ensure 7.0.12 is fine?

> 
> Rather than continue bisecting, I set up a watchdog that dumps
> queues/aqm/TXBD indices at the moment of the wedge, before anything
> touches the interface. I have five such dumps now (I discount one of
> them, taken on a captive portal network where the ping probes are
> unreliable), plus healthy baselines, and they show a consistent
> driver-level failure. I have also been able to reproduce that failure
> deterministically and to recover from it. Details below; I am
> sending a patch as a separate mail.
> 
> 1. What the wedge looks like from inside the driver
> ---------------------------------------------------
> 
> This is the dump from 2026-08-28 22:05, taken with power save off, on
> 5 GHz, 91 minutes into the association:
> 
>   /sys/kernel/debug/ieee80211/phy0/queues
>     00: 0x00000000/0   (VO)
>     01: 0x00000000/0   (VI)
>     02: 0x00000001/0   (BE)  <- stopped, reason bit 0 = DRIVER
>     03: 0x00000000/0   (BK)
> 
>   stations/<ap>/aqm tid 0 (BE): backlog 20941 -> 26960 bytes,
>     94 -> 144 packets, flags DIRTY, still growing while sampled
> 
>   iw station dump, 5 s apart:
>     tx packets  116017 -> 116017   (frozen)
>     rx packets  244885 -> 244992
>     beacon rx    53304 ->  53353
> 
>   TXBD_IDX_BEQ (0x3A8) = 0x003c003a in all four samples, taken over
>     about 15 seconds

Hardware read index is 0x3c, and host write index is 0x3a.
So, it reaches the limit of stop queue. 

[...]

> 
> I will send the patch separately so that it lands in patchwork on its
> own. It only adds the stall detection and the re-kick; it does not touch
> the normal TX path, it skips rings with nothing in flight so an idle
> device is never poked, and it warns once per stall rather than on every
> retry.

As your experiments, the cause is hardware never reads TX buffer and gets
stuck, right?

I quickly check the patch. It looks like the way to unlock this state is
to call rtw_pci_tx_kick_off_queue() again?

Which means not a driver side bug (stop queue but not restart queue properly),
right? 

> 
> One question, in case a re-kick turns out not to be enough in the field.
> The out-of-tree rtw88 driver has a pci_old.c for the older PCIe
> generation, where the PCIe DMA is reset after a TRX hang using the two
> status bits at REG_DBI_CTRL + 3 (bit 0 TX, bit 1 RX), with bit 2
> enabling the detection. Is that status bit valid on 8821CE as well? If
> it is, I can dump it during the next wedge and, if it confirms a DMA
> hang, a reset could be added as a second stage.

I checked vendor driver. It only does this at initial step, not to recover
it at runtime.