[PATCH v2] net/mana: do not block Rx interrupt calls during a reset

[Orgad Shaneh <orgads@gmail.com>]

Rx interrupt enable and disable wait for reset_ops_lock and then
return -EBUSY if a reset is in progress. The reset thread holds that
lock across the teardown (mana_dev_stop(), the RESET_ENTER request
to secondary processes, mana_dev_close()) and across the re-probe,
and each request to the secondaries may take up to
MANA_MP_REQ_TIMEOUT_SEC. An lcore that calls rx_intr_enable or
rx_intr_disable during one of those steps blocks for that long,
only to get -EBUSY.

Retry the trylock while the device is active, yielding the CPU in
between, and return -EBUSY as soon as it is not. Outside a reset the
lock is held by another queue's Rx interrupt call or by a control
operation. The data path must already be stopped before
dev_stop/dev_close, so the wait is short in a correct application.

Suggested-by: Wei Hu <weh@microsoft.com>
Signed-off-by: Orgad Shaneh <orgads@gmail.com>
---
v2:
- Yield with sched_yield() instead of rte_pause(), so a preempted
  lock holder on the same CPU can run (Stephen).
- Fix the commit message and comment: outside a reset the lock is
  also held by control operations, so the wait is short only in an
  application that stops the data path before dev_stop/dev_close.

Applies on next-net for-main, on top of efa36b47334d ("net/mana: fix
Rx interrupt enable failing on lock contention"). Squash it into that
commit if you prefer.

Builds with meson -Dwerror=true (gcc 14, rdma-core 56). A MANA reset
cannot be triggered on demand on Azure, so the reset path is untested.

The rwlock (tryrdlock in rx_intr, write lock in control ops and the
reset thread) fits the burst_state follow-up for main; I'll look at
it there.

 drivers/net/mana/mana.c | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)

diff --git a/drivers/net/mana/mana.c b/drivers/net/mana/mana.c
index f624cde6b9..71a3fd20d7 100644
--- a/drivers/net/mana/mana.c
+++ b/drivers/net/mana/mana.c
@@ -2,6 +2,7 @@
  * Copyright 2022 Microsoft Corporation
  */
 
+#include <sched.h>
 #include <unistd.h>
 #include <dirent.h>
 #include <fcntl.h>
@@ -451,14 +452,24 @@ mana_reset_trylock(struct mana_priv *priv)
 }
 
 /*
- * Like mana_reset_trylock(), but wait for the lock. Used by Rx interrupt
- * enable/disable, which every queue calls on each sleep and wakeup: they
- * must not fail just because another queue holds the lock for a moment.
+ * Like mana_reset_trylock(), but wait while the device is active. Used by
+ * Rx interrupt enable/disable, which every queue calls on each sleep and
+ * wakeup: they must not fail just because another queue holds the lock
+ * for a moment. Outside a reset the lock is held by another queue's Rx
+ * interrupt call or by a control operation; the data path must be stopped
+ * before dev_stop/dev_close, so in a correct application the wait is
+ * short. Once a reset has started, fail at once instead of waiting for
+ * its teardown or re-probe step.
  */
 static int
 mana_reset_lock(struct mana_priv *priv)
 {
-	pthread_mutex_lock(&priv->reset_ops_lock);
+	while (pthread_mutex_trylock(&priv->reset_ops_lock)) {
+		if (rte_atomic_load_explicit(&priv->dev_state,
+		    rte_memory_order_relaxed) != MANA_DEV_ACTIVE)
+			return -EBUSY;
+		sched_yield();
+	}
 
 	if (rte_atomic_load_explicit(&priv->dev_state,
 	    rte_memory_order_acquire) != MANA_DEV_ACTIVE) {
-- 
2.52.0