DPDK-dev Archive on lore.kernel.org
 help / color / mirror / Atom feed
From: Long Li <longli@microsoft.com>
To: dev@dpdk.org, Wei Hu <weh@microsoft.com>,
	Stephen Hemminger <stephen@networkplumber.org>
Cc: Long Li <longli@microsoft.com>, stable@dpdk.org
Subject: [PATCH v3 1/7] net/netvsc: retry VF hotplug indefinitely until PCI device disappears
Date: Fri, 15 May 2026 12:28:35 -0700	[thread overview]
Message-ID: <20260515192843.552762-2-longli@microsoft.com> (raw)
In-Reply-To: <20260515192843.552762-1-longli@microsoft.com>

After PCI rescan on Azure, the MANA kernel driver can take over 100
seconds to probe and create the /sys/bus/pci/devices/<dev>/net directory.
The previous fixed retry limit (NETVSC_MAX_HOTADD_RETRY=10, ~12 seconds)
was insufficient, causing VF re-attach to fail with 'Failed to parse PCI
device' on systems with slow MANA driver initialization.

Replace the fixed retry limit with an indefinite retry that only gives up
when the PCI device itself disappears from sysfs. This is safe because:

- The retry uses rte_eal_alarm callbacks which are serialized on the EAL
  interrupt thread, preventing races with VF remove or device close paths.
- Device close (eth_hn_dev_uninit) explicitly cancels all pending hotplug
  alarms via rte_eal_alarm_cancel and frees the context.
- If the PCI device is removed while retrying, access() detects the
  missing sysfs path and stops immediately.

A periodic NOTICE log every 30 retries (~30s) provides visibility into
long waits without flooding the log at DEBUG level.

Fixes: a2a23a794b3a ("net/netvsc: support VF device hot add/remove")
Cc: stable@dpdk.org
Signed-off-by: Long Li <longli@microsoft.com>
---
v3:
- Wrapped rte_eal_alarm_set lines to stay within 100 columns
  (checkpatch line-length warning)
v2:
- Added detailed comment explaining why indefinite retry is
  safe (PCI sysfs disappearance is the termination condition)

 drivers/net/netvsc/hn_ethdev.c | 41 +++++++++++++++++++++++++---------
 1 file changed, 31 insertions(+), 10 deletions(-)

diff --git a/drivers/net/netvsc/hn_ethdev.c b/drivers/net/netvsc/hn_ethdev.c
index b8880edb4c..85e500c178 100644
--- a/drivers/net/netvsc/hn_ethdev.c
+++ b/drivers/net/netvsc/hn_ethdev.c
@@ -89,8 +89,8 @@ struct netvsc_mp_param {
 #define NETVSC_ARG_TXBREAK "tx_copybreak"
 #define NETVSC_ARG_RX_EXTMBUF_ENABLE "rx_extmbuf_enable"
 
-/* The max number of retry when hot adding a VF device */
-#define NETVSC_MAX_HOTADD_RETRY 10
+/* Retry interval for hot-add VF device (microseconds) */
+#define NETVSC_HOTADD_RETRY_INTERVAL 1000000
 
 struct hn_xstats_name_off {
 	char name[RTE_ETH_XSTATS_NAME_SIZE];
@@ -622,19 +622,39 @@ static void netvsc_hotplug_retry(void *args)
 	PMD_DRV_LOG(DEBUG, "%s: retry count %d",
 		    __func__, hot_ctx->eal_hot_plug_retry);
 
-	if (hot_ctx->eal_hot_plug_retry++ > NETVSC_MAX_HOTADD_RETRY) {
-		PMD_DRV_LOG(NOTICE, "Failed to parse PCI device retry=%d",
-			    hot_ctx->eal_hot_plug_retry);
+	hot_ctx->eal_hot_plug_retry++;
+
+	/* Check if PCI device still exists — if it disappeared, give up.
+	 * Otherwise keep retrying indefinitely until the net directory
+	 * appears.  This is safe because:
+	 * - MANA driver probe can take >100s after PCI rescan
+	 * - The retry uses rte_eal_alarm callbacks serialized on the
+	 *   EAL interrupt thread, preventing races with device close
+	 * - Device close cancels pending alarms and frees the context
+	 * - If the PCI device is removed, the access() check below
+	 *   detects the missing sysfs path and stops immediately
+	 */
+	snprintf(buf, sizeof(buf), "/sys/bus/pci/devices/%s", d->name);
+	if (access(buf, F_OK) != 0) {
+		PMD_DRV_LOG(NOTICE,
+			    "PCI device %s no longer exists, giving up after %d retries",
+			    d->name, hot_ctx->eal_hot_plug_retry);
 		goto free_hotadd_ctx;
 	}
 
 	snprintf(buf, sizeof(buf), "/sys/bus/pci/devices/%s/net", d->name);
 	di = opendir(buf);
 	if (!di) {
-		PMD_DRV_LOG(DEBUG, "%s: can't open directory %s, "
-			    "retrying in 1 second", __func__, buf);
-		/* The device is still being initialized, retry after 1 second */
-		rte_eal_alarm_set(1000000, netvsc_hotplug_retry, hot_ctx);
+		if (hot_ctx->eal_hot_plug_retry % 30 == 0)
+			PMD_DRV_LOG(NOTICE,
+				    "%s: waiting for %s (retry %d, %ds elapsed)",
+				    __func__, buf, hot_ctx->eal_hot_plug_retry,
+				    hot_ctx->eal_hot_plug_retry);
+		else
+			PMD_DRV_LOG(DEBUG, "%s: can't open directory %s, "
+				    "retrying in 1 second", __func__, buf);
+		rte_eal_alarm_set(NETVSC_HOTADD_RETRY_INTERVAL,
+				  netvsc_hotplug_retry, hot_ctx);
 		return;
 	}
 
@@ -758,7 +778,8 @@ netvsc_hotadd_callback(const char *device_name, enum rte_dev_event_type type,
 			rte_spinlock_lock(&hv->hotadd_lock);
 			LIST_INSERT_HEAD(&hv->hotadd_list, hot_ctx, list);
 			rte_spinlock_unlock(&hv->hotadd_lock);
-			rte_eal_alarm_set(1000000, netvsc_hotplug_retry, hot_ctx);
+			rte_eal_alarm_set(NETVSC_HOTADD_RETRY_INTERVAL,
+					  netvsc_hotplug_retry, hot_ctx);
 			return;
 		}
 
-- 
2.43.0


  reply	other threads:[~2026-05-15 19:28 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-05-06  2:05 [PATCH v2 1/7] net/netvsc: retry VF hotplug indefinitely until PCI device disappears Long Li
2026-05-06  2:05 ` [PATCH v2 2/7] net/netvsc: retry on SIOCGIFHWADDR failure during VF hotplug Long Li
2026-05-06  2:05 ` [PATCH v2 3/7] net/netvsc: retry full probe when IB device not ready during hotplug Long Li
2026-05-06  2:05 ` [PATCH v2 4/7] net/netvsc: add debug logging for VF hotplug retry Long Li
2026-05-06  2:05 ` [PATCH v2 5/7] net/netvsc: retry when no matching MAC found in net directory Long Li
2026-05-06  2:05 ` [PATCH v2 6/7] net/netvsc: forward per-queue stats from VF device Long Li
2026-05-06  2:05 ` [PATCH v2 7/7] net/netvsc: handle VF recovery events for service reset Long Li
2026-05-07  2:49 ` [PATCH v2 1/7] net/netvsc: retry VF hotplug indefinitely until PCI device disappears Stephen Hemminger
2026-05-15 19:45   ` [EXTERNAL] " Long Li
2026-05-15 19:28 ` [PATCH v3 0/7] net/netvsc: fix VF hotplug and service reset handling Long Li
2026-05-15 19:28   ` Long Li [this message]
2026-05-15 19:28   ` [PATCH v3 2/7] net/netvsc: retry on SIOCGIFHWADDR failure during VF hotplug Long Li
2026-05-15 19:28   ` [PATCH v3 3/7] net/netvsc: retry full probe when IB device not ready during hotplug Long Li
2026-05-15 19:28   ` [PATCH v3 4/7] net/netvsc: add debug logging for VF hotplug retry Long Li
2026-05-15 19:28   ` [PATCH v3 5/7] net/netvsc: retry when no matching MAC found in net directory Long Li
2026-05-15 19:28   ` [PATCH v3 6/7] net/netvsc: forward per-queue stats from VF device Long Li
2026-05-15 19:28   ` [PATCH v3 7/7] net/netvsc: handle VF recovery events for service reset Long Li

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20260515192843.552762-2-longli@microsoft.com \
    --to=longli@microsoft.com \
    --cc=dev@dpdk.org \
    --cc=stable@dpdk.org \
    --cc=stephen@networkplumber.org \
    --cc=weh@microsoft.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox