Linux-HyperV List
 help / color / mirror / Atom feed
From: netdev-bot+sashiko@kernel.org
To: sahilchandna@linux.microsoft.com
Cc: haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com,
	andrew+netdev@lunn.ch, davem@davemloft.net, edumazet@kernel.org,
	kuba@kernel.org, pabeni@redhat.com, kotaranov@microsoft.com,
	horms@kernel.org, ernis@linux.microsoft.com,
	gargaditya@linux.microsoft.com, mawasthi@linux.microsoft.com,
	kees+treewide@kernel.org, schakrabarti@linux.microsoft.com,
	leitao@debian.org, linux-hyperv@vger.kernel.org,
	netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
	linux-rdma@vger.kernel.org, stable@vger.kernel.org
Subject: Re: [PATCH net v3] net: mana: Avoid waiting on dead HWC after device reset
Date: Mon, 16 Nov 2026 13:54:27 +0000	[thread overview]
Message-ID: <179483726776.3136.399784833950391727@kernel.org> (raw)
In-Reply-To: <20261112085226.878780-1-sahilchandna@linux.microsoft.com>

Thank you for your contribution! Sashiko AI review found 6 potential
issue(s) to consider.

Critical: 0 · High: 1 · Medium: 3 · Low: 2

If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).

- [High] If mana_gd_dev_reset() fails, mana_dealloc_queues() lowers
  hwc_timeout to 1 ms (cmpxchg at mana_en.c:3889) before any teardown…
- [Medium] The wait before a function-wide device reset drops from 120 s
  to about 300 ms.
- [Medium] The patch claims to bound how long mana_dealloc_queues() holds
  RTNL when the hardware is unresponsive ('This replaces the 120s drain…
- [Medium] net: mana: premature TX NAPI deletion in mana_dealloc_queues()
- [Low] mana_hwc_destroy_channel() now skips mana_smc_teardown_hwc() when
  hwc_dead is set, but its comment still says 'If it's non-zero, the HWC…
- [Low] The comment above mana_need_log() still says '/* Suppress logging
  when we set timeout to zero */'.

--

net: mana: Avoid waiting on dead HWC after device reset

This patch replaces the 120 s sleep-based TX drain in mana_dealloc_queues()
with synchronous CQ polling of up to about 300 ms per queue, then resets the
device if sends are still pending. A successful reset sets a new hwc_dead
flag, which makes mana_hwc_send_request() fail fast and skips
mana_fence_rqs() and the SHM teardown handshake; a failed reset lowers
hwc_timeout to 1 ms.

Link to v2: https://lore.kernel.org/all/20261030125009.3758592-1-sahilchandna@linux.microsoft.com/

> diff --git a/drivers/net/ethernet/microsoft/mana/gdma_main.c b/drivers/net/ethernet/microsoft/mana/gdma_main.c
> index ab942d5bfe058..94ee3b61527d6 100644
> --- a/drivers/net/ethernet/microsoft/mana/gdma_main.c
> +++ b/drivers/net/ethernet/microsoft/mana/gdma_main.c
> @@ -153,7 +153,9 @@ bool mana_need_log(struct gdma_context *gc, int err)
>  		return true;
>  
>  	hwc = gc->hwc.driver_data;
> -	if (hwc && hwc->hwc_timeout == 0)
> +	if (hwc &&
> +	    (READ_ONCE(hwc->hwc_timeout) == 0 ||
> +	     READ_ONCE(hwc->hwc_dead)))
>  		return false;

[Severity: Low]
This isn't a bug, but the comment above mana_need_log() still reads:

/* Suppress logging when we set timeout to zero */

With this change, every -ETIMEDOUT is also suppressed once hwc_dead is set,
even when hwc_timeout is non-zero. That includes the new fail-fast returns
from mana_hwc_send_request(). So callers such as mana_destroy_wq_obj(),
mana_pf_deregister_filter(), mana_gd_destroy_dma_region() and the "Failed
to disable vPort" message in mana_dealloc_queues() log nothing.

Could the comment be updated to cover the hwc_dead case too?

> diff --git a/drivers/net/ethernet/microsoft/mana/hw_channel.c b/drivers/net/ethernet/microsoft/mana/hw_channel.c
> index fb9f38f65911c..ca9dcbfea9e46 100644
> --- a/drivers/net/ethernet/microsoft/mana/hw_channel.c
> +++ b/drivers/net/ethernet/microsoft/mana/hw_channel.c

[ ... ]

> @@ -819,7 +819,11 @@ void mana_hwc_destroy_channel(struct gdma_context *gc)
>  	 * non-zero, the HWC worked and we should tear down the HWC here.
>  	 */
>  	if (gc->max_num_cqs > 0) {
> -		mana_smc_teardown_hwc(&gc->shm_channel, false);
> +		if (READ_ONCE(hwc->hwc_dead))
> +			dev_warn(hwc->dev,
> +				 "HWC unavailable, skip SHM teardown handshake\n");
> +		else
> +			mana_smc_teardown_hwc(&gc->shm_channel, false);
>  		WRITE_ONCE(gc->max_num_cqs, 0);
>  	}

[Severity: Low]
The comment just above still says the HWC should be torn down here whenever
max_num_cqs is non-zero. mana_smc_teardown_hwc() also describes the
handshake as required:

drivers/net/ethernet/microsoft/mana/shm_channel.c:mana_smc_teardown_hwc() {
    ...
	/* Read shmem response (polling for VF possession) and validate.
	 * For teardown, waiting for response is required to ensure hardware
	 * invalidates MST entries before software frees memory.
	 */
    ...
}

Right after this branch, mana_hwc_destroy_channel() frees the HWC queues,
caller_ctx and the hwc itself.

Skipping the handshake is only safe if a successful mana_gd_dev_reset()
guarantees the device has dropped those MST entries. Is that assumption
written down anywhere?

The pcie_flr() kernel-doc says it initiates FLR "unconditionally on @dev
without checking any flags and DEVCAP". A 0 return shows that the device's
config space is readable again, not that the reset actually happened. If
the FLR were ignored, hwc_dead would be set on a live device and the
handshake would be skipped. I couldn't check from the tree whether MANA
always honours FLR.

Could these comments explain why the handshake can be skipped after a
reset?

[ ... ]

> diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/ethernet/microsoft/mana/mana_en.c
> index 38249e092090a..739021aa09b38 100644
> --- a/drivers/net/ethernet/microsoft/mana/mana_en.c
> +++ b/drivers/net/ethernet/microsoft/mana/mana_en.c

[ ... ]

> @@ -3799,42 +3809,110 @@ static int mana_dealloc_queues(struct net_device *ndev)

[ ... ]

> +				tx_cq->budget = 0;
> +				for (retry = 0; retry <= MANA_TX_DRAIN_RETRIES;
> +				     retry++) {
> +					do {
> +						tx_cq->work_done = 0;
> +						mana_poll_tx_cq(tx_cq);
> +					} while (tx_cq->work_done);
> +
> +					if (!atomic_read(&txq->pending_sends) ||
> +					    trigger_flr ||
> +					    READ_ONCE(hwc->hwc_dead))
> +						break;
> +					if (retry < MANA_TX_DRAIN_RETRIES)
> +						msleep(100);
>  				}
> -				break;
> +			}
> +
> +			pending = atomic_read(&txq->pending_sends);
> +
> +			if (!trigger_flr && pending > 0) {
> +				trigger_flr = true;
> +				flr_txq_id = txq->gdma_txq_id;
> +				flr_pending = pending;
> +			}
> +		}

[Severity: Medium]
This cuts the wait before a device reset from 120 s to about 300 ms. If any
queue still has pending_sends after the first poll and three more 100 ms
apart, trigger_flr is set and mana_gd_dev_reset() runs.

mana_gd_dev_reset() calls pcie_flr() on PCI and cdx_dev_reset() on CDX. So
it resets the whole function, including the HWC and every vPort, not just
the stalled queue.

On success, hwc_dead is set. Nothing clears it until
mana_hwc_destroy_channel() frees the hwc. Until then, every HWC request
from any port or from mana_ib fails with -ETIMEDOUT in
mana_hwc_send_request().

Recovery depends on mana_gf_stats_work_handler() getting -ETIMEDOUT and
calling mana_schedule_serv_work(). mana_cdx_bus_ops does not set
schedule_serv_work. Does the device stay dead on CDX?

mana_dealloc_queues() also runs on every ndo_stop and mana_detach() (MTU,
channel, ring and XDP changes), not only during recovery. A device that is
just slow or briefly paused would have finished draining within the old
120 s window. Now it gets a function-level reset.

Is 300 ms known to be long enough before a reset this disruptive? It would
help if the commit message explained why 300 ms was chosen. It could also
say that the reset is function-wide and that the device stays unusable
until a service reset.

> +		if (trigger_flr && !READ_ONCE(hwc->hwc_dead)) {
> +			/* Completions never arrived: the device is
> +			 * unresponsive. Issue device reset and
> +			 * mark the HWC dead so subsequent HWC commands
> +			 * fail fast instead of each waiting the full
> +			 * hwc_timeout.
> +			 */
> +			err = mana_gd_dev_reset(gd->gdma_context);

[Severity: Medium]
The commit message says:

  "This replaces the 120s drain timeout with at most 300 ms of sleep per
  queue."

However, mana_gd_dev_reset() still runs synchronously here, with RTNL and
the netdev instance lock held. On PCI this calls pcie_flr():

drivers/pci/pci.c:pcie_flr() {
    ...
	if (!pci_wait_for_pending_transaction(dev))
		pci_err(dev, "timed out waiting for pending transaction; performing function level reset anyway\n");
    ...
	msleep(100);

	ret = pci_dev_wait(dev, "FLR", PCIE_RESET_READY_POLL_MS);
    ...
}

pci_dev_wait() keeps sleeping, doubling the delay each time, until the
Command register reads back valid. It gives up only after about 60 s.

This reset is issued exactly when the driver has decided the device is
unresponsive. If the device does not come back from FLR, can RTNL be held
for about another 60 s here?

After that, the reset-failure path also reaches mana_fence_rqs(), which
sleeps 100 ms per RXQ when the fence fails.

> +			if (err) {
> +				netdev_err(ndev, "device reset failed %d with %d pkts pending in txq %u\n",
> +					   err, flr_pending, flr_txq_id);
> +
> +				/* reduce timeout if device reset failed */
> +				timeout = READ_ONCE(hwc->hwc_timeout);
> +				if (timeout > 1)
> +					cmpxchg(&hwc->hwc_timeout, timeout, 1);
> +			} else {
> +				WRITE_ONCE(hwc->hwc_dead, true);
>  			}
>  		}

[Severity: High]
If mana_gd_dev_reset() fails, hwc_timeout drops to 1 ms here, before any
teardown command has been sent. The commit message explains why the
commands are still posted:

  "If the reset fails, the device may not have been reset and may still
  access memory, so teardown commands are still posted"

hwc_dead stays false, so mana_disable_vport_rx(), mana_fence_rqs(),
mana_destroy_wq_obj() and mana_gd_destroy_dma_region() are all posted.
But mana_hwc_send_request() now waits only 1 ms for each reply:

	timeout = READ_ONCE(hwc->hwc_timeout);
	if (!wait_for_completion_timeout(&ctx->comp_event,
					 (msecs_to_jiffies(timeout)))) {

Any firmware reply slower than 1 ms gives -ETIMEDOUT, and the callers
ignore it.

The pending TX SKBs are unmapped and freed in the loop just below. After
that, mana_destroy_rxqs(), mana_destroy_txq() and mana_destroy_eq() free
the RX buffers, queue rings and DMA-region memory. mana_gd_destroy_queue()
frees the memory whether or not the destroy succeeded:

drivers/net/ethernet/microsoft/mana/gdma_main.c:mana_gd_destroy_queue() {
    ...
	mana_gd_destroy_dma_region(gc, gmi->dma_region_handle);
	mana_gd_free_memory(gmi);
	kfree(queue);
}

Can the device then DMA into memory that has already been freed?

Before this patch, the first teardown command on this path waited the full
hwc_timeout. If the HWC still worked, every teardown command was
acknowledged before memory was freed. This path is also much easier to
reach now, since the reset follows a 300 ms stall instead of 120 s.

The 1 ms value also stays in place until the HWC is rebuilt. A timed-out
msg_id slot is freed and handed out again right away through
find_first_zero_bit(), and ctx->comp_event is initialised only once, when
the HWC is set up. mana_hwc_handle_resp() matches a reply to its caller
only by msg_id:

drivers/net/ethernet/microsoft/mana/hw_channel.c:mana_hwc_handle_resp() {
    ...
	ctx = hwc->caller_ctx + msg_id;
	err = mana_hwc_verify_resp_msg(ctx, resp_msg, resp_len);
    ...
	memcpy(ctx->output_buf, resp_msg, resp_len);
    ...
	complete(&ctx->comp_event);
}

So a late reply to one request can complete a later request that got the
same slot.

The mana_en commands catch this with the msg_type check in
mana_verify_resp_hdr(), so mana_destroy_wq_obj() won't accept a stale
fence reply. mana_gd_destroy_dma_region() checks only resp.hdr.status:

	err = mana_gd_send_request(gc, sizeof(req), &req, sizeof(resp), &resp);
	if (err || resp.hdr.status) {

Could a stale reply with status 0 be accepted here as a successful
GDMA_DESTROY_DMA_REGION?

Would it be safer to keep the normal hwc_timeout for the teardown commands
when the reset fails?

[ ... ]

> @@ -3799,42 +3809,110 @@ static int mana_dealloc_queues(struct net_device *ndev)
>  	 * a txq because it may not timely see apc->port_is_up being cleared
>  	 * to false, but it doesn't matter since mana_start_xmit() drops any
>  	 * new packets due to apc->port_is_up being false.
> -	 *
> -	 * Drain all the in-flight TX packets.
> -	 * A timeout of 120 seconds for all the queues is used.
> -	 * This will break the while loop when h/w is not responding.
> -	 * This value of 120 has been decided here considering max
> -	 * number of queues.
> +	 * Drain the in-flight TX completions for each queue. TX is already
> +	 * stopped, so pending_sends can no longer grow. For each queue,
> +	 * disable NAPI so it cannot run at the same time as this drain,
> +	 * then poll its CQ for up to 300 ms. If sends are still missing,
> +	 * the hardware is not responding: reset the device and mark the
> +	 * HWC dead so that later HWC commands fail fast.
>  	 */
>
>  	if (apc->tx_qp) {
>  		for (i = 0; i < apc->num_queues; i++) {
> +			struct napi_struct *napi;
> +			struct mana_cq *tx_cq;
> +			int pending;
> +
> +			if (!apc->tx_qp[i])
> +				continue;
> +
> +			tx_cq = &apc->tx_qp[i]->tx_cq;
>  			txq = &apc->tx_qp[i]->txq;
> -			tsleep = 1000;
> -			while (atomic_read(&txq->pending_sends) > 0 &&
> -			       time_before(jiffies, timeout)) {
> -				usleep_range(tsleep, tsleep + 1000);
> -				tsleep <<= 1;
> -			}
> -			if (atomic_read(&txq->pending_sends)) {
> -				err = mana_gd_dev_reset(gd->gdma_context);
> -				if (err) {
> -					netdev_err(ndev, "device reset failed %d with %d pkts pending in txq %u\n",
> -						   err,
> -					    atomic_read(&txq->pending_sends),
> -					    txq->gdma_txq_id);
> +			napi = &tx_cq->napi;
> +
> +			if (txq->napi_initialized) {
> +				/* Stop the NAPI poll so it cannot run
> +				 * concurrently with the synchronous drain.
> +				 */
> +				napi_synchronize(napi);
> +				napi_disable_locked(napi);
> +				cancel_work_sync(&tx_cq->dim.work);
> +
> +				/* Poll the CQ until no sends are outstanding,
> +				 * sleeping 100 ms between polls. Poll once
> +				 * if earlier queue was stuck or device
> +				 * has already been reset.
> +				 * A zero budget frees skbs via the non-NAPI
> +				 * path, which is required outside softirq.
> +				 */
> +				tx_cq->budget = 0;
> +				for (retry = 0; retry <= MANA_TX_DRAIN_RETRIES;
> +				     retry++) {
> +					do {
> +						tx_cq->work_done = 0;
> +						mana_poll_tx_cq(tx_cq);
> +					} while (tx_cq->work_done);
> +
> +					if (!atomic_read(&txq->pending_sends) ||
> +					    trigger_flr ||
> +					    READ_ONCE(hwc->hwc_dead))
> +						break;
> +					if (retry < MANA_TX_DRAIN_RETRIES)
> +						msleep(100);
>  				}
> -				break;
> +			}
> +
> +			pending = atomic_read(&txq->pending_sends);
> +
> +			if (!trigger_flr && pending > 0) {
> +				trigger_flr = true;
> +				flr_txq_id = txq->gdma_txq_id;
> +				flr_pending = pending;
> +			}
> +		}
> +
> +		if (trigger_flr && !READ_ONCE(hwc->hwc_dead)) {
> +			/* Completions never arrived: the device is
> +			 * unresponsive. Issue device reset and
> +			 * mark the HWC dead so subsequent HWC commands
> +			 * fail fast instead of each waiting the full
> +			 * hwc_timeout.
> +			 */
> +			err = mana_gd_dev_reset(gd->gdma_context);
> +			if (err) {
> +				netdev_err(ndev, "device reset failed %d with %d pkts pending in txq %u\n",
> +					   err, flr_pending, flr_txq_id);
> +
> +				/* reduce timeout if device reset failed */
> +				timeout = READ_ONCE(hwc->hwc_timeout);
> +				if (timeout > 1)
> +					cmpxchg(&hwc->hwc_timeout, timeout, 1);
> +			} else {
> +				WRITE_ONCE(hwc->hwc_dead, true);
>  			}
>  		}
>
>  		for (i = 0; i < apc->num_queues; i++) {
> +			struct napi_struct *napi;
> +			struct mana_cq *tx_cq;
> +
> +			if (!apc->tx_qp[i])
> +				continue;
> +
> +			tx_cq = &apc->tx_qp[i]->tx_cq;
> +			napi = &tx_cq->napi;
> +
>  			txq = &apc->tx_qp[i]->txq;
>  			while ((skb = skb_dequeue(&txq->pending_skbs))) {
>  				mana_unmap_skb(skb, apc);
>  				dev_kfree_skb_any(skb);
>  			}
>  			atomic_set(&txq->pending_sends, 0);
> +
> +			if (txq->napi_initialized) {
> +				netif_napi_del_locked(napi);
> +				txq->napi_initialized = false;
> +			}
>  		}
>  	}
>

[Severity: Medium]
This deletes the TX NAPI and clears txq->napi_initialized long before
the SQ object is destroyed and the TX CQ is detached.

mana_destroy_txq() was arranged so that mana_gd_detach_cq(),
netif_napi_del_locked() and clearing napi_initialized all happen
together, after mana_destroy_wq_obj(GDMA_SQ). Its comment says this is
so a completion EQE that arrives before the WQ is destroyed still finds
the CQ, and so one grace period covers both mana_gd_intr() and NAPI
users.

With this change, the TX CQ is still published in gc->cq_table[], and
the EQ is still live, while mana_disable_vport_rx(), mana_fence_rqs(),
mana_destroy_rxqs() and mana_destroy_wq_obj() run. A completion EQE for
this CQ in that window reaches mana_schedule_napi() on a napi_struct
that has already been through netif_napi_del_locked().

The memory stays valid until kvfree() in mana_destroy_txq(). The
NAPI_STATE_SCHED bit left set by napi_disable_locked() probably keeps
this from doing real harm. Still, this is exactly the ordering
mana_destroy_txq() was written to avoid.

When mana_destroy_txq() runs later, napi_initialized is already false
for every queue, so its detach/del block is skipped. The CQ is then
detached from mana_deinit_cq() -> mana_gd_destroy_queue() ->
mana_gd_destroy_cq(), which calls synchronize_net() again. That gives
two grace periods per TX queue under RTNL, where there used to be one.

I assume napi_initialized is cleared here so that mana_destroy_txq()
does not call napi_synchronize() and napi_disable_locked() a second
time. Both would spin forever on the NAPI_STATE_SCHED bit left by the
first disable.

Could that be avoided without moving the deletion? For example, the
disabled state could be tracked separately from napi_initialized, so
that mana_destroy_txq() skips only the synchronize/disable step. The
detach and netif_napi_del_locked() would then still happen together
after mana_destroy_wq_obj().

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261112085226.878780-1-sahilchandna%40linux.microsoft.com

      parent reply	other threads:[~2026-11-16 13:54 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-12  8:52 [PATCH net v3] net: mana: Avoid waiting on dead HWC after device reset Sahil Chandna
2026-11-13  8:52 ` sashiko-bot
2026-11-16 13:54 ` netdev-bot+sashiko [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=179483726776.3136.399784833950391727@kernel.org \
    --to=netdev-bot+sashiko@kernel.org \
    --cc=andrew+netdev@lunn.ch \
    --cc=davem@davemloft.net \
    --cc=decui@microsoft.com \
    --cc=edumazet@kernel.org \
    --cc=ernis@linux.microsoft.com \
    --cc=gargaditya@linux.microsoft.com \
    --cc=haiyangz@microsoft.com \
    --cc=horms@kernel.org \
    --cc=kees+treewide@kernel.org \
    --cc=kotaranov@microsoft.com \
    --cc=kuba@kernel.org \
    --cc=leitao@debian.org \
    --cc=linux-hyperv@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=mawasthi@linux.microsoft.com \
    --cc=netdev@vger.kernel.org \
    --cc=pabeni@redhat.com \
    --cc=sahilchandna@linux.microsoft.com \
    --cc=schakrabarti@linux.microsoft.com \
    --cc=stable@vger.kernel.org \
    --cc=wei.liu@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox