[PATCH net 2/2] bnxt_en: Wait for the PF in bnxt_resume()

[Michael Chan <michael.chan@broadcom.com>]

From: Keegan Freyhof <keegan.freyhof@broadcom.com>

When bnxt_resume() is called on a VF with the PF running in the
same kernel, the initial FW message may timeout if the PF has not
been initialized.  Use the more appropriate bnxt_hwrm_poll() with
the silent flag to suppress warning messages and loop up to 10 times
if the error code warrants additional polling.  This initial polling
is done without the rtnl_lock() to allow the PF running in the same
kernel (which will also require the rtnl_lock) to finish its
initialization.

This fixes a failure found in the lab when running system hibernate
on a BCM57608 NIC with 64 VFs created.  The FW version is 238.1.59.0.

bnxt_en 0000:23:08.5 eno4v5: hwrm req_type 0x0 seq id 0xb0 error 0xa
bnxt_en 0000:23:08.5: PM: dpm_run_callback(): pci_pm_restore returns -19
bnxt_en 0000:23:08.5: PM: failed to restore async: error -19
bnxt_en 0000:23:08.3 eno4v3: hwrm req_type 0x0 seq id 0xb0 error 0xa
bnxt_en 0000:23:08.4 eno4v4: hwrm req_type 0x0 seq id 0xb0 error 0xa
bnxt_en 0000:23:08.7 eno4v7: hwrm req_type 0x0 seq id 0xb0 error 0xa
bnxt_en 0000:23:08.3: PM: dpm_run_callback(): pci_pm_restore returns -19
bnxt_en 0000:23:08.4: PM: dpm_run_callback(): pci_pm_restore returns -19

Tested on dual-port BCM57608 with 64 VFs on each port.  The polling
loop provides adequate margins.

Fixes: f65a2044a8c9 ("bnxt_en: Add suspend/resume callbacks.")
Reviewed-by: Pavan Chebbi <pavan.chebbi@broadcom.com>
Signed-off-by: Keegan Freyhof <keegan.freyhof@broadcom.com>
Signed-off-by: Michael Chan <michael.chan@broadcom.com>
---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 37 ++++++++++++++++++++---
 1 file changed, 33 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
index 9d232d835758..03682d9182c6 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
@@ -17567,21 +17567,50 @@ static int bnxt_suspend(struct device *device)
 	return rc;
 }
 
+#define BNXT_RESUME_MAX_RETRY	10
+
 static int bnxt_resume(struct device *device)
 {
 	struct net_device *dev = dev_get_drvdata(device);
 	struct bnxt *bp = netdev_priv(dev);
-	int rc = 0;
+	int i, rc;
 
-	rtnl_lock();
-	netdev_lock(dev);
 	rc = pci_enable_device(bp->pdev);
 	if (rc) {
 		netdev_err(dev, "Cannot re-enable PCI device during resume, err = %d\n",
 			   rc);
-		goto resume_exit;
+		return rc;
 	}
 	pci_set_master(bp->pdev);
+	if (BNXT_VF(bp)) {
+		for (i = 0; i < BNXT_RESUME_MAX_RETRY; i++) {
+			rc = bnxt_hwrm_poll(bp);
+			if (rc != -EAGAIN && rc != -EBUSY && rc != -ECOMM)
+				break;
+
+			/* Wait briefly for -EAGAIN (FW busy), a bit longer for
+			 * -ECOMM (PF down) to allow the PF to fully initialize.
+			 * For -EBUSY, the message typically has timed out, so
+			 * try again immediately.
+			 */
+			if (i < BNXT_RESUME_MAX_RETRY - 1) {
+				if (rc == -EAGAIN)
+					msleep(250);
+				else if (rc == -ECOMM)
+					msleep(750);
+			}
+		}
+		if (rc) {
+			netdev_err(dev,
+				   "Firmware not responding during resume, rc = %d\n",
+				   rc);
+			return -ENODEV;
+		}
+	}
+
+	rtnl_lock();
+	netdev_lock(dev);
+
 	if (bnxt_hwrm_ver_get(bp)) {
 		rc = -ENODEV;
 		goto resume_exit;
-- 
2.51.0