[PATCH rtw-next v2] wifi: rtw88: pci: wake the TX queues when the rings are reset

[PATCH rtw-next v2] wifi: rtw88: pci: wake the TX queues when the rings are reset

[Abdurrahman Karadag]

When a TX ring fills, rtw_pci_tx_write() stops the mac80211 queue and
records that in ring->queue_stopped, which is cleared only by the
completion loop in rtw_pci_tx_isr(). Resetting the rings rewinds the
indices, so nothing that was in flight can complete and run that loop,
and the queue stays stopped over a ring that is now empty.
ieee80211_restart_hw() does not release that stop either, because
ieee80211_reconfig() wakes only IEEE80211_QUEUE_STOP_REASON_SUSPEND.

Release those queues where the reset happens. The stop side is the one
that matters: rtw_fw_recovery() reaches rtw_pci_dma_release() through
rtw_enter_ips() before it calls ieee80211_restart_hw(), so that is
where a queue stopped before the crash is still stopped. By the time
rtw_pci_setup() runs the flag is already clear.

Wake all the queues, but only if at least one ring had filled. This
cannot tell one DRIVER stop from another: rtw_pci_io_err_detected() and
the hardware scan paths in fw.c stop every queue with the same
IEEE80211_QUEUE_STOP_REASON_DRIVER, which mac80211 keeps as a single
non-refcounted slot per queue. Gating on ring->queue_stopped at least
keeps the resets that stopped nothing out of it.

On an RTL8821CE, a BE ring held until rtw_pci_tx_write() stopped its
queue did not come back from the restart. The ring itself was reset -
0x3a8 read 0x00000000 and REG_TXPAUSE was clear again - but BE's
mac80211 stop reason still read 0x1 three minutes later and TX never
resumed. With this patch it reads 0x0 and TX resumes within 2 s. The
ring was filled by writing REG_TXPAUSE 0x0f from debugfs under load,
and rtw_fw_recovery() was called from a debug build.

Signed-off-by: Abdurrahman Karadag <abdurrahmankaradag19@gmail.com>
---
v2:
 - wake with ieee80211_wake_queues() instead of recording and replaying
   per-ring queue mappings (Ping-Ke Shih)
 - drop the Fixes tag: the bug only bites once a recovery path exists,
   which the blamed commit predates (Ping-Ke Shih)
 - cut the commit message down to why and how (Ping-Ke Shih)
 - say which of the two reset call sites fires during recovery, and
   that the measured stall was induced from debugfs
 - patch 2 of the v1 series is dropped rather than resent, while the
   cause of the stall is still open

v1: https://lore.kernel.org/linux-wireless/20261002231013.11792-1-abdurrahmankaradag19@gmail.com/

 drivers/net/wireless/realtek/rtw88/pci.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/drivers/net/wireless/realtek/rtw88/pci.c b/drivers/net/wireless/realtek/rtw88/pci.c
index 66d2e5f..4d3524e 100644
--- a/drivers/net/wireless/realtek/rtw88/pci.c
+++ b/drivers/net/wireless/realtek/rtw88/pci.c
@@ -475,7 +475,29 @@ static void rtw_pci_reset_buf_desc(struct rtw_dev *rtwdev)
 
 static void rtw_pci_reset_trx_ring(struct rtw_dev *rtwdev)
 {
+	struct rtw_pci *rtwpci = (struct rtw_pci *)rtwdev->priv;
+	enum rtw_tx_queue_type queue;
+	bool wake = false;
+
 	rtw_pci_reset_buf_desc(rtwdev);
+
+	/*
+	 * The indices are back at zero, so nothing that was in flight can
+	 * complete and reach the wake in rtw_pci_tx_isr(); a queue stopped
+	 * when its ring filled would stay stopped for good. Only wake when
+	 * that happened: the stop reason is shared with the AER and scan
+	 * paths, which wake the queues again themselves.
+	 */
+	for (queue = 0; queue < RTK_MAX_TX_QUEUE_NUM; queue++) {
+		if (!rtwpci->tx_rings[queue].queue_stopped)
+			continue;
+
+		rtwpci->tx_rings[queue].queue_stopped = false;
+		wake = true;
+	}
+
+	if (wake)
+		ieee80211_wake_queues(rtwdev->hw);
 }
 
 static void rtw_pci_enable_interrupt(struct rtw_dev *rtwdev,
-- 
2.55.0

base-commit: 83de3a16c7b37064a59305040ba9b0f93b832794

RE: [PATCH rtw-next v2] wifi: rtw88: pci: wake the TX queues when the rings are reset

[Ping-Ke Shih]

Abdurrahman Karadag <abdurrahmankaradag19@gmail.com> wrote:
> When a TX ring fills, rtw_pci_tx_write() stops the mac80211 queue and
> records that in ring->queue_stopped, which is cleared only by the
> completion loop in rtw_pci_tx_isr(). Resetting the rings rewinds the
> indices, so nothing that was in flight can complete and run that loop,
> and the queue stays stopped over a ring that is now empty.
> ieee80211_restart_hw() does not release that stop either, because
> ieee80211_reconfig() wakes only IEEE80211_QUEUE_STOP_REASON_SUSPEND.

The main reason why we need this patch is ieee80211_restart_hw(), so
please point out this in subject. 

> 
> Release those queues where the reset happens. The stop side is the one
> that matters: rtw_fw_recovery() reaches rtw_pci_dma_release() through
> rtw_enter_ips() before it calls ieee80211_restart_hw(), so that is
> where a queue stopped before the crash is still stopped. By the time
> rtw_pci_setup() runs the flag is already clear.
> 
> Wake all the queues, but only if at least one ring had filled. This
> cannot tell one DRIVER stop from another: rtw_pci_io_err_detected() and
> the hardware scan paths in fw.c stop every queue with the same
> IEEE80211_QUEUE_STOP_REASON_DRIVER, which mac80211 keeps as a single
> non-refcounted slot per queue. Gating on ring->queue_stopped at least
> keeps the resets that stopped nothing out of it.
> 
> On an RTL8821CE, a BE ring held until rtw_pci_tx_write() stopped its
> queue did not come back from the restart. The ring itself was reset -
> 0x3a8 read 0x00000000 and REG_TXPAUSE was clear again - but BE's
> mac80211 stop reason still read 0x1 three minutes later and TX never
> resumed. With this patch it reads 0x0 and TX resumes within 2 s. The
> ring was filled by writing REG_TXPAUSE 0x0f from debugfs under load,
> and rtw_fw_recovery() was called from a debug build.

This isn't related to this patch, but it is still digging, so no need.

As I see, the commit message can be very simple -- wake up queues for
ieee80211_restart_hw() case.

(For the analysis in another mail, I'll read it later.)

> 
> Signed-off-by: Abdurrahman Karadag <abdurrahmankaradag19@gmail.com>
> ---
> v2:
>  - wake with ieee80211_wake_queues() instead of recording and replaying
>    per-ring queue mappings (Ping-Ke Shih)
>  - drop the Fixes tag: the bug only bites once a recovery path exists,
>    which the blamed commit predates (Ping-Ke Shih)
>  - cut the commit message down to why and how (Ping-Ke Shih)
>  - say which of the two reset call sites fires during recovery, and
>    that the measured stall was induced from debugfs
>  - patch 2 of the v1 series is dropped rather than resent, while the
>    cause of the stall is still open
> 
> v1: https://lore.kernel.org/linux-wireless/20261002231013.11792-1-abdurrahmankaradag19@gmail.com/
> 
>  drivers/net/wireless/realtek/rtw88/pci.c | 22 ++++++++++++++++++++++
>  1 file changed, 22 insertions(+)
> 
> diff --git a/drivers/net/wireless/realtek/rtw88/pci.c b/drivers/net/wireless/realtek/rtw88/pci.c
> index 66d2e5f..4d3524e 100644
> --- a/drivers/net/wireless/realtek/rtw88/pci.c
> +++ b/drivers/net/wireless/realtek/rtw88/pci.c
> @@ -475,7 +475,29 @@ static void rtw_pci_reset_buf_desc(struct rtw_dev *rtwdev)
> 
>  static void rtw_pci_reset_trx_ring(struct rtw_dev *rtwdev)

I'm thinking if here is too early to wake up queues. Should we do this in
rtw_reconfig_complete()?

Or we check flag RTW_FLAG_RESTARTING of rtwdev->flags to wake up queues.
This will be clear why we need to wake up. The comment isn't necessary
by the way.

>  {
> +       struct rtw_pci *rtwpci = (struct rtw_pci *)rtwdev->priv;
> +       enum rtw_tx_queue_type queue;
> +       bool wake = false;
> +
>         rtw_pci_reset_buf_desc(rtwdev);
> +
> +       /*
> +        * The indices are back at zero, so nothing that was in flight can
> +        * complete and reach the wake in rtw_pci_tx_isr(); a queue stopped
> +        * when its ring filled would stay stopped for good. Only wake when
> +        * that happened: the stop reason is shared with the AER and scan
> +        * paths, which wake the queues again themselves.
> +        */
> +       for (queue = 0; queue < RTK_MAX_TX_QUEUE_NUM; queue++) {
> +               if (!rtwpci->tx_rings[queue].queue_stopped)
> +                       continue;
> +
> +               rtwpci->tx_rings[queue].queue_stopped = false;
> +               wake = true;
> +       }
> +
> +       if (wake)
> +               ieee80211_wake_queues(rtwdev->hw);
>  }
> 
>  static void rtw_pci_enable_interrupt(struct rtw_dev *rtwdev,
> --
> 2.55.0
> 
> base-commit: 83de3a16c7b37064a59305040ba9b0f93b832794

Re: [PATCH rtw-next v2] wifi: rtw88: pci: wake the TX queues when the rings are reset

[Abdurrahman Karadag]

> I'm thinking if here is too early to wake up queues. Should we do this in
> rtw_reconfig_complete()?

It is too early, and rtw_reconfig_complete() is the right place. v3 does
that.

On the reset path rtw_core_stop() has already cleared RTW_FLAG_RUNNING by
the time rtw_pci_reset_trx_ring() runs, so anything mac80211 hands the
driver between that wake and the end of the restart is freed again by
rtw_ops_tx(). reconfig_complete() is called from the wake_up label in
ieee80211_reconfig(), where the comment reads "If this is for hw restart
things are still running". It also makes the fix bus-independent instead
of PCI-only.

I measured it at the new hook before sending, with two builds of current
rtw-next that differ by nothing but this patch, using mac80211's own
tracepoints instead of a register read. Under load the BE queue is
stopped by the ring-full path, then rtw_fw_recovery() is called:

  without the patch
    511.499278  stop_queue  queue:2 reason:0      DRIVER, left dangling
    511.501625  stop_queue  queue:0-3 reason:4    SUSPEND, the restart
    511.957404  drv_reconfig_complete reconfig_type:0
    511.957411  wake_queue  queue:0-3 reason:4    only SUSPEND is woken
    AC_BE stays 0x00000001, 100% loss

  with the patch
    475.930915  drv_reconfig_complete reconfig_type:0
    475.930925  wake_queue  queue:2 reason:0      10 us later
    475.930926  wake_queue  queue:0-3 reason:4
    AC_BE reads 0x00000000, 0% loss

So the wake lands after the restart has completed and before mac80211
wakes the SUSPEND reason itself. No TXPAUSE write and no injection this
time - the ring fills on its own under load, which is also why I am happy
to leave the measurement out of the commit message.

One consequence I should flag, since v2 covered it and v3 does not. A
plain "ip link set wlan0 down/up" also reaches rtw_pci_reset_trx_ring(),
and mac80211 never clears a DRIVER stop on interface down - nothing
outside the stop/wake helpers touches local->q_stop_reasons[]. So with v3
a queue stopped before a down/up stays stopped. Is the restart case the
one you want fixed, or should a down/up release it too?

I also left ring->queue_stopped alone. It stays set until the first
completion after the restart, which clears it, and rtw_pci_tx_write()
does not read it before stopping a queue, so a stale value costs nothing.
I can clear it in rtw_pci_reset_trx_ring() if you would rather the driver
state did not claim a stop that is gone.

> As I see, the commit message can be very simple -- wake up queues for
> ieee80211_restart_hw() case.

Done, and the subject names it now. The measurement paragraph is out -
you are right that the stall belongs elsewhere.