[RFC net-next 2/2] bnxt_en: recover a failed TX RING_FREE with a ring reset

[Joe Damato <joe@dama.to>]

When HWRM_RING_FREE is sent with a completion ring set and the completion
never arrives, hwrm_ring_free_send_msg() logs

  hwrm_ring_free type 1 failed. rc:fffffff0 err:0
  Resp cmpl intr err msg: 0x51

and returns -EIO. bnxt_hwrm_tx_ring_free() clears fw_ring_id anyway and
__bnxt_close_nic() calls bnxt_free_mem(), unmapping ring memory
the firmware has not been told to stop using.

On a host with an IOMMU the result is an IO_PAGE_FAULT or a DMAR fault
against the freed pages seconds later.

A TX ring whose completions have stopped will trigger the netdev
watchdog, the reset closes the device, and every RING_FREE routed
through that dead completion ring times out.

Try to recover from this by adding bnxt_tx_ring_reset_and_free and
bnxt_hwrm_tx_ring_reset.

The intent here is that bnxt_tx_ring_reset_and_free will try a TX ring reset
in polled mode (assuming the completion ring is dead) and, if that succeeds,
retry the RING_FREE in polled mode as well.

Bump tx_resets in this path. A non-zero ethtool tx_resets value on
a device that never tripped the netdev watchdog would identify a
firmware that stopped answering RING_FREE on a completion ring.

Signed-off-by: Joe Damato <joe@dama.to>
---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 41 +++++++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
index ac7716dbf88d..c47f6f24dfa4 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
@@ -7660,6 +7660,44 @@ static int hwrm_ring_free_send_msg(struct bnxt *bp,
 	return 0;
 }
 
+static int bnxt_hwrm_tx_ring_reset(struct bnxt *bp,
+				   struct bnxt_tx_ring_info *txr)
+{
+	struct bnxt_ring_struct *ring = &txr->tx_ring_struct;
+	struct hwrm_ring_reset_input *req;
+	int rc;
+
+	rc = hwrm_req_init(bp, req, HWRM_RING_RESET);
+	if (rc)
+		return rc;
+
+	req->ring_type = RING_RESET_REQ_RING_TYPE_TX;
+	req->ring_id = cpu_to_le16(ring->fw_ring_id);
+	return hwrm_req_send_silent(bp, req);
+}
+
+static int bnxt_tx_ring_reset_and_free(struct bnxt *bp,
+				       struct bnxt_tx_ring_info *txr)
+{
+	struct bnxt_ring_struct *ring = &txr->tx_ring_struct;
+	struct bnxt_cp_ring_info *cpr;
+	int rc;
+
+	rc = bnxt_hwrm_tx_ring_reset(bp, txr);
+	if (rc) {
+		netdev_err(bp->dev,
+			   "TX ring %d reset failed after RING_FREE failed, rc: %d\n",
+			   txr->txq_index, rc);
+		return rc;
+	}
+
+	cpr = &txr->bnapi->cp_ring;
+	cpr->sw_stats->tx.tx_resets++;
+
+	return hwrm_ring_free_send_msg(bp, ring, RING_FREE_REQ_RING_TYPE_TX,
+				       INVALID_HW_RING_ID);
+}
+
 static int bnxt_hwrm_tx_ring_free(struct bnxt *bp,
 				  struct bnxt_tx_ring_info *txr,
 				  bool close_path)
@@ -7675,6 +7713,9 @@ static int bnxt_hwrm_tx_ring_free(struct bnxt *bp,
 		       INVALID_HW_RING_ID;
 	rc = hwrm_ring_free_send_msg(bp, ring, RING_FREE_REQ_RING_TYPE_TX,
 				     cmpl_ring_id);
+	if (rc && cmpl_ring_id != INVALID_HW_RING_ID)
+		rc = bnxt_tx_ring_reset_and_free(bp, txr);
+
 	ring->fw_ring_id = INVALID_HW_RING_ID;
 	return rc;
 }
-- 
2.53.0-Meta