[PATCH 0/2] introduce mt76 tx hang watchdog

[PATCH 0/2] introduce mt76 tx hang watchdog

[Lorenzo Bianconi]

Port mt76x02_check_tx_hang watchdog from vendor driver in order to
reset the card when tx mac/dma engine is stucked. Tx mac/dma stuck
occurs when the device is heavily loaded or in a noisy environment.
In that condition tx queues are not drained since the device does not
generate tx interrupts.
This series is based on "add EDCCA support to mt76x{0,2}e drivers"
https://patchwork.kernel.org/cover/10727149/

Lorenzo Bianconi (2):
  mt76: run mac work every 100ms
  mt76: introduce mt76x02_check_tx_hang watchdog

 drivers/net/wireless/mediatek/mt76/mt76.h     |  1 +
 .../net/wireless/mediatek/mt76/mt76x0/pci.c   |  2 +-
 .../net/wireless/mediatek/mt76/mt76x0/usb.c   |  2 +-
 drivers/net/wireless/mediatek/mt76/mt76x02.h  |  7 ++
 .../wireless/mediatek/mt76/mt76x02_debugfs.c  |  2 +
 .../net/wireless/mediatek/mt76/mt76x02_mac.c  | 90 ++++++++++++++++++-
 .../net/wireless/mediatek/mt76/mt76x02_mmio.c | 23 +++++
 .../wireless/mediatek/mt76/mt76x2/pci_main.c  |  2 +-
 .../wireless/mediatek/mt76/mt76x2/usb_main.c  |  2 +-
 9 files changed, 125 insertions(+), 6 deletions(-)

-- 
2.20.1

[PATCH 1/2] mt76: run mac work every 100ms

[Lorenzo Bianconi]

Introduce MT_WATCHDOG_TIME macro in order to run mac work every 100ms.
This is a preliminary patch to implement a proper tx hang watchdog

Signed-off-by: Lorenzo Bianconi <lorenzo.bianconi@redhat.com>
---
 drivers/net/wireless/mediatek/mt76/mt76x0/pci.c      | 2 +-
 drivers/net/wireless/mediatek/mt76/mt76x0/usb.c      | 2 +-
 drivers/net/wireless/mediatek/mt76/mt76x02.h         | 1 +
 drivers/net/wireless/mediatek/mt76/mt76x02_mac.c     | 2 +-
 drivers/net/wireless/mediatek/mt76/mt76x2/pci_main.c | 2 +-
 drivers/net/wireless/mediatek/mt76/mt76x2/usb_main.c | 2 +-
 6 files changed, 6 insertions(+), 5 deletions(-)

diff --git a/drivers/net/wireless/mediatek/mt76/mt76x0/pci.c b/drivers/net/wireless/mediatek/mt76/mt76x0/pci.c
index 720b05e325a5..d2005632101f 100644
--- a/drivers/net/wireless/mediatek/mt76/mt76x0/pci.c
+++ b/drivers/net/wireless/mediatek/mt76/mt76x0/pci.c
@@ -30,7 +30,7 @@ static int mt76x0e_start(struct ieee80211_hw *hw)
 	mt76x02_mac_start(dev);
 	mt76x0_phy_calibrate(dev, true);
 	ieee80211_queue_delayed_work(dev->mt76.hw, &dev->mac_work,
-				     MT_CALIBRATE_INTERVAL);
+				     MT_WATCHDOG_TIME);
 	ieee80211_queue_delayed_work(dev->mt76.hw, &dev->cal_work,
 				     MT_CALIBRATE_INTERVAL);
 	set_bit(MT76_STATE_RUNNING, &dev->mt76.state);
diff --git a/drivers/net/wireless/mediatek/mt76/mt76x0/usb.c b/drivers/net/wireless/mediatek/mt76/mt76x0/usb.c
index 0e6b43bb4678..b7ace310ca72 100644
--- a/drivers/net/wireless/mediatek/mt76/mt76x0/usb.c
+++ b/drivers/net/wireless/mediatek/mt76/mt76x0/usb.c
@@ -118,7 +118,7 @@ static int mt76x0u_start(struct ieee80211_hw *hw)
 
 	mt76x0_phy_calibrate(dev, true);
 	ieee80211_queue_delayed_work(dev->mt76.hw, &dev->mac_work,
-				     MT_CALIBRATE_INTERVAL);
+				     MT_WATCHDOG_TIME);
 	ieee80211_queue_delayed_work(dev->mt76.hw, &dev->cal_work,
 				     MT_CALIBRATE_INTERVAL);
 	set_bit(MT76_STATE_RUNNING, &dev->mt76.state);
diff --git a/drivers/net/wireless/mediatek/mt76/mt76x02.h b/drivers/net/wireless/mediatek/mt76/mt76x02.h
index 5f5831d18af5..17e13e373f53 100644
--- a/drivers/net/wireless/mediatek/mt76/mt76x02.h
+++ b/drivers/net/wireless/mediatek/mt76/mt76x02.h
@@ -26,6 +26,7 @@
 #include "mt76x02_dfs.h"
 #include "mt76x02_dma.h"
 
+#define MT_WATCHDOG_TIME	(HZ / 10)
 #define MT_CALIBRATE_INTERVAL	HZ
 
 #define MT_MAX_CHAINS		2
diff --git a/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c b/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c
index aed108e8761f..a4ff5a7a219f 100644
--- a/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c
+++ b/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c
@@ -888,7 +888,7 @@ void mt76x02_mac_work(struct work_struct *work)
 	mt76_tx_status_check(&dev->mt76, NULL, false);
 
 	ieee80211_queue_delayed_work(mt76_hw(dev), &dev->mac_work,
-				     MT_CALIBRATE_INTERVAL);
+				     MT_WATCHDOG_TIME);
 }
 
 void mt76x02_mac_set_bssid(struct mt76x02_dev *dev, u8 idx, const u8 *addr)
diff --git a/drivers/net/wireless/mediatek/mt76/mt76x2/pci_main.c b/drivers/net/wireless/mediatek/mt76/mt76x2/pci_main.c
index b54a32397486..ec936002a286 100644
--- a/drivers/net/wireless/mediatek/mt76/mt76x2/pci_main.c
+++ b/drivers/net/wireless/mediatek/mt76/mt76x2/pci_main.c
@@ -33,7 +33,7 @@ mt76x2_start(struct ieee80211_hw *hw)
 		goto out;
 
 	ieee80211_queue_delayed_work(mt76_hw(dev), &dev->mac_work,
-				     MT_CALIBRATE_INTERVAL);
+				     MT_WATCHDOG_TIME);
 
 	set_bit(MT76_STATE_RUNNING, &dev->mt76.state);
 
diff --git a/drivers/net/wireless/mediatek/mt76/mt76x2/usb_main.c b/drivers/net/wireless/mediatek/mt76/mt76x2/usb_main.c
index 2b48cc51a30d..77f95688a41b 100644
--- a/drivers/net/wireless/mediatek/mt76/mt76x2/usb_main.c
+++ b/drivers/net/wireless/mediatek/mt76/mt76x2/usb_main.c
@@ -28,7 +28,7 @@ static int mt76x2u_start(struct ieee80211_hw *hw)
 		goto out;
 
 	ieee80211_queue_delayed_work(mt76_hw(dev), &dev->mac_work,
-				     MT_CALIBRATE_INTERVAL);
+				     MT_WATCHDOG_TIME);
 	set_bit(MT76_STATE_RUNNING, &dev->mt76.state);
 
 out:
-- 
2.20.1

[PATCH 2/2] mt76: introduce mt76x02_check_tx_hang watchdog

[Lorenzo Bianconi]

Port mt76x02_check_tx_hang watchdog from vendor driver in order to
perform a device reset when tx mac/dma logic hangs. Tx mac/dma stuck
has been observed when the device is heavily loaded or in a noisy
environment.

Signed-off-by: Lorenzo Bianconi <lorenzo.bianconi@redhat.com>
---
 drivers/net/wireless/mediatek/mt76/mt76.h     |  1 +
 drivers/net/wireless/mediatek/mt76/mt76x02.h  |  6 ++
 .../wireless/mediatek/mt76/mt76x02_debugfs.c  |  2 +
 .../net/wireless/mediatek/mt76/mt76x02_mac.c  | 88 ++++++++++++++++++-
 .../net/wireless/mediatek/mt76/mt76x02_mmio.c | 23 +++++
 5 files changed, 119 insertions(+), 1 deletion(-)

diff --git a/drivers/net/wireless/mediatek/mt76/mt76.h b/drivers/net/wireless/mediatek/mt76/mt76.h
index 5cd508a68609..8ef430d8afc4 100644
--- a/drivers/net/wireless/mediatek/mt76/mt76.h
+++ b/drivers/net/wireless/mediatek/mt76/mt76.h
@@ -421,6 +421,7 @@ struct mt76_dev {
 	struct mt76_queue q_tx[__MT_TXQ_MAX];
 	struct mt76_queue q_rx[__MT_RXQ_MAX];
 	const struct mt76_queue_ops *queue_ops;
+	int tx_dma_idx[4];
 
 	wait_queue_head_t tx_wait;
 	struct sk_buff_head status_list;
diff --git a/drivers/net/wireless/mediatek/mt76/mt76x02.h b/drivers/net/wireless/mediatek/mt76/mt76x02.h
index 17e13e373f53..09ae4f246131 100644
--- a/drivers/net/wireless/mediatek/mt76/mt76x02.h
+++ b/drivers/net/wireless/mediatek/mt76/mt76x02.h
@@ -29,6 +29,8 @@
 #define MT_WATCHDOG_TIME	(HZ / 10)
 #define MT_CALIBRATE_INTERVAL	HZ
 
+#define MT_TX_HANG_TH		10
+
 #define MT_MAX_CHAINS		2
 struct mt76x02_rx_freq_cal {
 	s8 high_gain[MT_MAX_CHAINS];
@@ -91,6 +93,9 @@ struct mt76x02_dev {
 	u8 tbtt_count;
 	u16 beacon_int;
 
+	u32 tx_hang_reset;
+	u8 tx_hang_check;
+
 	struct mt76x02_calibration cal;
 
 	s8 target_power;
@@ -144,6 +149,7 @@ s8 mt76x02_tx_get_max_txpwr_adj(struct mt76x02_dev *dev,
 				const struct ieee80211_tx_rate *rate);
 s8 mt76x02_tx_get_txpwr_adj(struct mt76x02_dev *dev, s8 txpwr,
 			    s8 max_txpwr_adj);
+bool mt76x02_tx_hang(struct mt76x02_dev *dev);
 void mt76x02_tx_set_txpwr_auto(struct mt76x02_dev *dev, s8 txpwr);
 void mt76x02_set_tx_ackto(struct mt76x02_dev *dev);
 void mt76x02_set_coverage_class(struct ieee80211_hw *hw,
diff --git a/drivers/net/wireless/mediatek/mt76/mt76x02_debugfs.c b/drivers/net/wireless/mediatek/mt76/mt76x02_debugfs.c
index a9d52ba1e270..7580c5c986ff 100644
--- a/drivers/net/wireless/mediatek/mt76/mt76x02_debugfs.c
+++ b/drivers/net/wireless/mediatek/mt76/mt76x02_debugfs.c
@@ -133,5 +133,7 @@ void mt76x02_init_debugfs(struct mt76x02_dev *dev)
 				    read_txpower);
 
 	debugfs_create_devm_seqfile(dev->mt76.dev, "agc", dir, read_agc);
+
+	debugfs_create_u32("tx_hang_reset", 0400, dir, &dev->tx_hang_reset);
 }
 EXPORT_SYMBOL_GPL(mt76x02_init_debugfs);
diff --git a/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c b/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c
index a4ff5a7a219f..402944e81e5e 100644
--- a/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c
+++ b/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c
@@ -778,6 +778,90 @@ static void mt76x02_check_mac_err(struct mt76x02_dev *dev)
 		MT_MAC_SYS_CTRL_ENABLE_TX | MT_MAC_SYS_CTRL_ENABLE_RX);
 }
 
+static void mt76x02_mac_watchdog_reset(struct mt76x02_dev *dev)
+{
+	u32 mask = dev->mt76.mmio.irqmask;
+	int i;
+
+	ieee80211_stop_queues(dev->mt76.hw);
+	set_bit(MT76_RESET, &dev->mt76.state);
+
+	if (dev->beacon_mask) {
+		tasklet_disable(&dev->pre_tbtt_tasklet);
+		mt76_clear(dev, MT_BEACON_TIME_CFG,
+			   MT_BEACON_TIME_CFG_BEACON_TX |
+			   MT_BEACON_TIME_CFG_TBTT_EN);
+	}
+
+	tasklet_disable(&dev->tx_tasklet);
+
+	for (i = 0; i < ARRAY_SIZE(dev->mt76.napi); i++)
+		napi_disable(&dev->mt76.napi[i]);
+
+	mt76x02_irq_disable(dev, mask);
+
+	/* perform device reset */
+	mt76_clear(dev, MT_TXOP_CTRL_CFG, MT_TXOP_ED_CCA_EN);
+	mt76_wr(dev, MT_MAC_SYS_CTRL, 0);
+	mt76_clear(dev, MT_WPDMA_GLO_CFG,
+		   MT_WPDMA_GLO_CFG_TX_DMA_EN | MT_WPDMA_GLO_CFG_RX_DMA_EN);
+	usleep_range(5000, 10000);
+	mt76_wr(dev, MT_INT_SOURCE_CSR, 0xffffffff);
+
+	/* let fw reset DMA */
+	mt76_set(dev, 0x734, 0x3);
+
+	for (i = 0; i < ARRAY_SIZE(dev->mt76.q_tx); i++)
+		mt76_queue_tx_cleanup(dev, i, true);
+
+	mt76_wr(dev, MT_MAC_SYS_CTRL,
+		MT_MAC_SYS_CTRL_ENABLE_TX | MT_MAC_SYS_CTRL_ENABLE_RX);
+	mt76_set(dev, MT_WPDMA_GLO_CFG,
+		 MT_WPDMA_GLO_CFG_TX_DMA_EN | MT_WPDMA_GLO_CFG_RX_DMA_EN);
+	if (dev->ed_monitor)
+		mt76_set(dev, MT_TXOP_CTRL_CFG, MT_TXOP_ED_CCA_EN);
+
+	clear_bit(MT76_RESET, &dev->mt76.state);
+
+	mt76x02_irq_enable(dev, mask);
+
+	tasklet_enable(&dev->tx_tasklet);
+	tasklet_schedule(&dev->tx_tasklet);
+
+	for (i = 0; i < ARRAY_SIZE(dev->mt76.napi); i++) {
+		napi_enable(&dev->mt76.napi[i]);
+		napi_schedule(&dev->mt76.napi[i]);
+	}
+
+	if (dev->beacon_mask) {
+		tasklet_enable(&dev->pre_tbtt_tasklet);
+		mt76_set(dev, MT_BEACON_TIME_CFG,
+			 MT_BEACON_TIME_CFG_BEACON_TX |
+			 MT_BEACON_TIME_CFG_TBTT_EN);
+	}
+
+	ieee80211_wake_queues(dev->mt76.hw);
+
+	mt76_txq_schedule_all(&dev->mt76);
+}
+
+static void mt76x02_check_tx_hang(struct mt76x02_dev *dev)
+{
+	if (mt76x02_tx_hang(dev)) {
+		if (++dev->tx_hang_check < MT_TX_HANG_TH)
+			return;
+
+		mt76x02_mac_watchdog_reset(dev);
+
+		dev->tx_hang_reset++;
+		dev->tx_hang_check = 0;
+		memset(dev->mt76.tx_dma_idx, 0xff,
+		       sizeof(dev->mt76.tx_dma_idx));
+	} else {
+		dev->tx_hang_check = 0;
+	}
+}
+
 static void
 mt76x02_edcca_tx_enable(struct mt76x02_dev *dev, bool enable)
 {
@@ -876,7 +960,9 @@ void mt76x02_mac_work(struct work_struct *work)
 		dev->aggr_stats[idx++] += val >> 16;
 	}
 
-	/* XXX: check beacon stuck for ap mode */
+	if (mt76_is_mmio(dev))
+		mt76x02_check_tx_hang(dev);
+
 	if (!dev->beacon_mask)
 		mt76x02_check_mac_err(dev);
 
diff --git a/drivers/net/wireless/mediatek/mt76/mt76x02_mmio.c b/drivers/net/wireless/mediatek/mt76/mt76x02_mmio.c
index 6974acc75e2b..89d9cb1f45bf 100644
--- a/drivers/net/wireless/mediatek/mt76/mt76x02_mmio.c
+++ b/drivers/net/wireless/mediatek/mt76/mt76x02_mmio.c
@@ -385,3 +385,26 @@ void mt76x02_mac_start(struct mt76x02_dev *dev)
 			   MT_INT_TX_STAT);
 }
 EXPORT_SYMBOL_GPL(mt76x02_mac_start);
+
+bool mt76x02_tx_hang(struct mt76x02_dev *dev)
+{
+	u32 dma_idx, prev_dma_idx;
+	struct mt76_queue *q;
+	int i;
+
+	for (i = 0; i < 4; i++) {
+		q = &dev->mt76.q_tx[i];
+
+		if (!q->queued)
+			continue;
+
+		prev_dma_idx = dev->mt76.tx_dma_idx[i];
+		dma_idx = ioread32(&q->regs->dma_idx);
+		dev->mt76.tx_dma_idx[i] = dma_idx;
+
+		if (prev_dma_idx == dma_idx)
+			break;
+	}
+
+	return i < 4;
+}
-- 
2.20.1

Re: [PATCH 1/2] mt76: run mac work every 100ms

[Stanislaw Gruszka]

On Mon, Dec 24, 2018 at 05:24:41PM +0100, Lorenzo Bianconi wrote:
> Introduce MT_WATCHDOG_TIME macro in order to run mac work every 100ms.
> This is a preliminary patch to implement a proper tx hang watchdog
<snip>
> diff --git a/drivers/net/wireless/mediatek/mt76/mt76x2/pci_main.c b/drivers/net/wireless/mediatek/mt76/mt76x2/pci_main.c
> index b54a32397486..ec936002a286 100644
> --- a/drivers/net/wireless/mediatek/mt76/mt76x2/pci_main.c
> +++ b/drivers/net/wireless/mediatek/mt76/mt76x2/pci_main.c
> @@ -33,7 +33,7 @@ mt76x2_start(struct ieee80211_hw *hw)
>  		goto out;
>  
>  	ieee80211_queue_delayed_work(mt76_hw(dev), &dev->mac_work,
> -				     MT_CALIBRATE_INTERVAL);
> +				     MT_WATCHDOG_TIME);

There are other procedures that are performed on mac_work , which most
likely don't have to be done so frequently. I think watchdog should
be placed it it's own work.

Regards
Stanislaw

Re: [PATCH 2/2] mt76: introduce mt76x02_check_tx_hang watchdog

[Stanislaw Gruszka]

On Mon, Dec 24, 2018 at 05:24:42PM +0100, Lorenzo Bianconi wrote:
> Port mt76x02_check_tx_hang watchdog from vendor driver in order to
> perform a device reset when tx mac/dma logic hangs. Tx mac/dma stuck
> has been observed when the device is heavily loaded or in a noisy
> environment.

On what hardware this happen ? Is this only for AP mode?
 
> diff --git a/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c b/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c
> index a4ff5a7a219f..402944e81e5e 100644
> --- a/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c
> +++ b/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c
> @@ -778,6 +778,90 @@ static void mt76x02_check_mac_err(struct mt76x02_dev *dev)
>  		MT_MAC_SYS_CTRL_ENABLE_TX | MT_MAC_SYS_CTRL_ENABLE_RX);
>  }
>  
> +static void mt76x02_mac_watchdog_reset(struct mt76x02_dev *dev)

This all is MMIO specific, it should be moved to mt76x02_mmio.c

Regards 
Stanislaw

Re: [PATCH 2/2] mt76: introduce mt76x02_check_tx_hang watchdog

[Lorenzo Bianconi]

>
> On Mon, Dec 24, 2018 at 05:24:42PM +0100, Lorenzo Bianconi wrote:
> > Port mt76x02_check_tx_hang watchdog from vendor driver in order to
> > perform a device reset when tx mac/dma logic hangs. Tx mac/dma stuck
> > has been observed when the device is heavily loaded or in a noisy
> > environment.
>
> On what hardware this happen ? Is this only for AP mode?

Hi Stanislaw,

I an able to trigger the issue using a mt7662 AP device till now, not
running STA or other modes yet.
I will respin this patch adding tx watchdog just on mt76x2 devices in
AP mode for the moment.

Regards,
Lorenzo

>
> > diff --git a/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c b/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c
> > index a4ff5a7a219f..402944e81e5e 100644
> > --- a/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c
> > +++ b/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c
> > @@ -778,6 +778,90 @@ static void mt76x02_check_mac_err(struct mt76x02_dev *dev)
> >               MT_MAC_SYS_CTRL_ENABLE_TX | MT_MAC_SYS_CTRL_ENABLE_RX);
> >  }
> >
> > +static void mt76x02_mac_watchdog_reset(struct mt76x02_dev *dev)
>
> This all is MMIO specific, it should be moved to mt76x02_mmio.c
>
> Regards
> Stanislaw

Re: [PATCH 1/2] mt76: run mac work every 100ms

[Lorenzo Bianconi]

>
> On Mon, Dec 24, 2018 at 05:24:41PM +0100, Lorenzo Bianconi wrote:
> > Introduce MT_WATCHDOG_TIME macro in order to run mac work every 100ms.
> > This is a preliminary patch to implement a proper tx hang watchdog
> <snip>
> > diff --git a/drivers/net/wireless/mediatek/mt76/mt76x2/pci_main.c b/drivers/net/wireless/mediatek/mt76/mt76x2/pci_main.c
> > index b54a32397486..ec936002a286 100644
> > --- a/drivers/net/wireless/mediatek/mt76/mt76x2/pci_main.c
> > +++ b/drivers/net/wireless/mediatek/mt76/mt76x2/pci_main.c
> > @@ -33,7 +33,7 @@ mt76x2_start(struct ieee80211_hw *hw)
> >               goto out;
> >
> >       ieee80211_queue_delayed_work(mt76_hw(dev), &dev->mac_work,
> > -                                  MT_CALIBRATE_INTERVAL);
> > +                                  MT_WATCHDOG_TIME);
>
> There are other procedures that are performed on mac_work , which most
> likely don't have to be done so frequently. I think watchdog should
> be placed it it's own work.

I guess it will be useful to have a dedicated work for it if we add
new watchdogs (like mt7603 driver).
I will fix it in v2.

Regards,
Lorenzo

>
> Regards
> Stanislaw
>