Netdev List
 help / color / mirror / Atom feed
From: netdev-bot+sashiko@kernel.org
To: sahilchandna@linux.microsoft.com
Cc: haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com,
	andrew+netdev@lunn.ch, davem@davemloft.net, edumazet@kernel.org,
	kuba@kernel.org, pabeni@redhat.com, kotaranov@microsoft.com,
	horms@kernel.org, ernis@linux.microsoft.com,
	gargaditya@linux.microsoft.com, mawasthi@linux.microsoft.com,
	kees+treewide@kernel.org, schakrabarti@linux.microsoft.com,
	leitao@debian.org, linux-hyperv@vger.kernel.org,
	netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
	linux-rdma@vger.kernel.org, stable@vger.kernel.org
Subject: Re: [PATCH net v4] net: mana: Avoid waiting on dead HWC after device reset
Date: Sun, 22 Nov 2026 10:20:52 +0000	[thread overview]
Message-ID: <179534285205.3135.17134166045415765008@kernel.org> (raw)
In-Reply-To: <20261118091436.3985155-1-sahilchandna@linux.microsoft.com>

Thank you for your contribution! Sashiko AI review found 2 potential
issue(s) to consider.

Critical: 0 · High: 0 · Medium: 1 · Low: 1

If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).

- [Medium] In mana_dealloc_queues()
  (drivers/net/ethernet/microsoft/mana/mana_en.c), the patch cuts the…
- [Low] When mana_gd_dev_reset() fails in mana_dealloc_queues(), hwc_dead
  is deliberately left clear, and v4 dropped the hwc_timeout reduction…

--

net: mana: Avoid waiting on dead HWC after device reset

This commit replaces the 120 second sleep-based TX drain in
mana_dealloc_queues() with synchronous CQ polling. Each queue gets about
300 ms, and the device is reset if sends are still outstanding after that.
After a successful reset a new hwc_dead flag is set, so that
mana_hwc_send_request() fails fast with -ETIMEDOUT, and mana_fence_rqs() and
the SHM teardown handshake are skipped.

> diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/ethernet/microsoft/mana/mana_en.c
> index 7e9223143fc75..5e4379107b457 100644
> --- a/drivers/net/ethernet/microsoft/mana/mana_en.c
> +++ b/drivers/net/ethernet/microsoft/mana/mana_en.c

[ ... ]

> @@ -3774,19 +3780,24 @@ int mana_attach(struct net_device *ndev)
>  	return 0;
>  }
>  
> +#define MANA_TX_DRAIN_RETRIES	3
> +
>  static int mana_dealloc_queues(struct net_device *ndev)
>  {

[ ... ]

> @@ -3799,31 +3810,79 @@ static int mana_dealloc_queues(struct net_device *ndev)
>  	 * to false, but it doesn't matter since mana_start_xmit() drops any
>  	 * new packets due to apc->port_is_up being false.
>  	 *
> -	 * Drain all the in-flight TX packets.
> -	 * A timeout of 120 seconds for all the queues is used.
> -	 * This will break the while loop when h/w is not responding.
> -	 * This value of 120 has been decided here considering max
> -	 * number of queues.
> +	 * Drain the in-flight TX completions for each queue. TX is already
> +	 * stopped, so pending_sends can no longer grow. For each queue,
> +	 * disable NAPI so it cannot run at the same time as this drain,
> +	 * then poll its CQ for up to 300 ms. If sends are still missing,
> +	 * the hardware is not responding: reset the device and mark the
> +	 * HWC dead so that later HWC commands fail fast.
>  	 */
>  
>  	if (apc->tx_qp) {
>  		for (i = 0; i < apc->num_queues; i++) {

[ ... ]

> +				tx_cq->budget = 0;
> +				for (retry = 0; retry <= MANA_TX_DRAIN_RETRIES;
> +				     retry++) {
> +					do {
> +						tx_cq->work_done = 0;
> +						mana_poll_tx_cq(tx_cq);
> +					} while (tx_cq->work_done);
> +
> +					if (!atomic_read(&txq->pending_sends) ||
> +					    trigger_flr ||
> +					    READ_ONCE(hwc->hwc_dead))
> +						break;
> +					if (retry < MANA_TX_DRAIN_RETRIES)
> +						msleep(100);
>  				}
> -				break;
> +			}
> +
> +			pending = atomic_read(&txq->pending_sends);
> +
> +			if (!trigger_flr && pending > 0) {
> +				trigger_flr = true;
> +				flr_txq_id = txq->gdma_txq_id;
> +				flr_pending = pending;
> +			}
> +		}

[Severity: Medium]
Before, the driver waited up to 120 seconds for completions before deciding
the device was unresponsive. Now it waits about 300 ms per queue. Is 300 ms
a safe threshold for a function-wide reset?

mana_dealloc_queues() runs during routine operations. These include
mana_close(), mana_change_mtu(), mana_set_channels(), mana_set_ringparam(),
mana_set_priv_flags(), mana_xdp_set() and the per-port queue reset work.

Suppose a device works but is slow, and a TX completion is still outstanding
after the fourth poll. Then trigger_flr is set and mana_gd_dev_reset() runs:

drivers/net/ethernet/microsoft/mana/gdma_pci.c:mana_pci_dev_reset() {
	return pcie_flr(to_pci_dev(gc->dev));
}

On CDX this is cdx_dev_reset() instead. Both reset the whole function. That
hits every vPort on the gdma_context and any mana_ib user, not just the port
being torn down. Also, pcie_flr() here does not save and restore config
space.

A successful reset sets hwc->hwc_dead. Nothing clears it until
mana_hwc_destroy_channel() frees the hwc and a new one is allocated. Until
then mana_hwc_send_request() fails at once:

	if (READ_ONCE(hwc->hwc_dead))
		return -ETIMEDOUT;

Wouldn't the mana_attach() that follows mana_detach() then fail and leave
the interface down?

On PCI, recovery seems to happen only when mana_gf_stats_work_handler()
sees -ETIMEDOUT and schedules a service reset.

The CDX bus_ops have no schedule_serv_work callback, so
mana_schedule_serv_work() returns -EOPNOTSUPP there. Would a CDX device stay
down until the driver is rebound?

The commit message says the budget "is not based on a hardware limit". It
also says that in testing the late completions had all arrived "by its
second poll", which comes after the first 100 ms sleep. That is about a 3x
margin for a decision that resets the whole device, in a patch marked for
stable.

> +
> +		if (trigger_flr && !READ_ONCE(hwc->hwc_dead)) {
> +			/* Completions never arrived: the device is
> +			 * unresponsive. Issue device reset and
> +			 * mark the HWC dead so subsequent HWC commands
> +			 * fail fast instead of each waiting the full
> +			 * hwc_timeout.
> +			 */
> +			err = mana_gd_dev_reset(gd->gdma_context);
> +			if (err) {
> +				netdev_err(ndev, "device reset failed %d with %d pkts pending in txq %u\n",
> +					   err, flr_pending, flr_txq_id);
> +			} else {
> +				WRITE_ONCE(hwc->hwc_dead, true);
>  			}
>  		}

[Severity: Low]
Two comments here say that missing sends lead to a reset and to the HWC
being marked dead so later commands fail fast. One is the block comment at
the top of the drain, the other is the comment just above
mana_gd_dev_reset(). But hwc_dead is only set when the reset succeeds. Do
these comments match the error path?

If mana_gd_dev_reset() fails, hwc_dead stays false and hwc_timeout is not
changed (v4 dropped the step that lowered it). The next call is
mana_disable_vport_rx()->mana_hwc_send_request(). It passes both hwc_dead
checks, posts MANA_CONFIG_VPORT_RX, and then waits:

	timeout = READ_ONCE(hwc->hwc_timeout);
	if (!wait_for_completion_timeout(&ctx->comp_event,
					 (msecs_to_jiffies(timeout)))) {

That wait uses the full firmware value, 60s on the tested system.
mana_fence_rqs() also still runs on this path and sleeps 100 ms per RX queue
on error. pcie_flr() can itself block in pci_dev_wait() before it fails.

This is still shorter than the old 120s drain plus 60s wait. Item 2 of the
commit message also says that posting teardown commands after a failed reset
is intentional.

Could the comments and the commit message say that RTNL can still be held
for the full hwc_timeout on this path?

[ ... ]

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261118091436.3985155-1-sahilchandna%40linux.microsoft.com

      parent reply	other threads:[~2026-11-22 10:20 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18  9:14 [PATCH net v4] net: mana: Avoid waiting on dead HWC after device reset Sahil Chandna
2026-11-19 21:33 ` Haiyang Zhang
2026-11-22 10:20 ` netdev-bot+sashiko [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=179534285205.3135.17134166045415765008@kernel.org \
    --to=netdev-bot+sashiko@kernel.org \
    --cc=andrew+netdev@lunn.ch \
    --cc=davem@davemloft.net \
    --cc=decui@microsoft.com \
    --cc=edumazet@kernel.org \
    --cc=ernis@linux.microsoft.com \
    --cc=gargaditya@linux.microsoft.com \
    --cc=haiyangz@microsoft.com \
    --cc=horms@kernel.org \
    --cc=kees+treewide@kernel.org \
    --cc=kotaranov@microsoft.com \
    --cc=kuba@kernel.org \
    --cc=leitao@debian.org \
    --cc=linux-hyperv@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=mawasthi@linux.microsoft.com \
    --cc=netdev@vger.kernel.org \
    --cc=pabeni@redhat.com \
    --cc=sahilchandna@linux.microsoft.com \
    --cc=schakrabarti@linux.microsoft.com \
    --cc=stable@vger.kernel.org \
    --cc=wei.liu@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox