From: netdev-bot+sashiko@kernel.org
To: sahilchandna@linux.microsoft.com
Cc: haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com,
andrew+netdev@lunn.ch, davem@davemloft.net, edumazet@kernel.org,
kuba@kernel.org, pabeni@redhat.com, weh@microsoft.com,
kotaranov@microsoft.com, horms@kernel.org,
ernis@linux.microsoft.com, gargaditya@linux.microsoft.com,
leitao@debian.org, linux-hyperv@vger.kernel.org,
netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-rdma@vger.kernel.org
Subject: Re: [PATCH net-next] net: mana: Add CQE polling as first-level TX timeout recovery
Date: Mon, 16 Nov 2026 13:54:58 +0000 [thread overview]
Message-ID: <179483729831.3136.3233216496914201382@kernel.org> (raw)
In-Reply-To: <20261112105752.978446-1-sahilchandna@linux.microsoft.com>
Thank you for your contribution! Sashiko AI review found 4 potential
issue(s) to consider.
Critical: 0 · High: 0 · Medium: 1 · Low: 3
If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).
- [Medium] Level 1 of the new recovery only calls napi_schedule() on the
TX CQ that stalled, in mana_tx_timeout()…
- [Low] The commit message says the patch makes TX timeout recovery
'three-tiered' and lists 'Level 3: device reset(FLR)' as the step…
- [Low] apc->last_tx_timeout is cleared in only one place: the post-
attach tail of mana_per_port_queue_reset_work_handler()…
- [Low] The new comment in mana_tx_timeout() (mana_en.c:990-996) says
escalation happens because 'the earlier CQE poll did not help and…
--
net: mana: Add CQE polling as first-level TX timeout recovery
This commit changes mana_tx_timeout() so that the first TX timeout, or
one arriving more than 2 minutes after the previous one, schedules NAPI on
the stalled queue's TX CQ instead of resetting the port. A further timeout
on any queue within 2 minutes escalates to the existing per-port
detach/attach reset, and a new tx_cqe_poll_recovery ethtool counter tracks
the CQE poll attempts.
> This makes the TX timeout recovery three-tiered:
>
> Level 1: CQE poll via napi_schedule() (this patch)
> Level 2: Port queue reset (detach/attach)
> Level 3: device reset(FLR)
[Severity: Low]
Is the Level 3 step accurate for the TX timeout path? When
mana_pre_alloc_rxbufs(), mana_detach() or mana_attach() fails in
mana_per_port_queue_reset_work_handler(), the handler only logs:
err = mana_attach(ndev);
if (err)
netdev_err(ndev, "mana_attach failed: %d\n", err);
It doesn't call mana_schedule_serv_work() or set GC_IN_SERVICE. In
mana_en.c, mana_schedule_serv_work() is only called from
mana_gf_stats_work_handler() on an HWC timeout. gdma_main.c calls it for
hardware service EQEs.
So when a port reset fails or doesn't help, the next timeout triggers
another port reset, or after 2 minutes goes back to Level 1. It never
reaches an FLR.
Could the commit message describe the FLR as a separate existing
mechanism, rather than as the third tier of this escalation?
> diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/ethernet/microsoft/mana/mana_en.c
> index e6d494df22e4d..0d8c8b8fbc83f 100644
> --- a/drivers/net/ethernet/microsoft/mana/mana_en.c
> +++ b/drivers/net/ethernet/microsoft/mana/mana_en.c
> @@ -336,6 +336,9 @@ static void mana_per_port_queue_reset_work_handler(struct work_struct *work)
> if (err)
> netdev_err(ndev, "mana_attach failed: %d\n", err);
>
> + /* Let the next TX timeout try a CQE poll recovery first */
> + WRITE_ONCE(apc->last_tx_timeout, 0);
> +
[Severity: Low]
Is this the only place last_tx_timeout gets cleared? The other paths
that tear down and rebuild the TX queues don't touch it. These include
mana_close()/mana_open(), the other mana_detach()/mana_attach() callers
(channel, MTU and XDP changes), and the service/FLR resume through
mana_probe().
Suppose a Level 1 CQE poll ran at T0 and one of those paths then rebuilt
the queues. A TX timeout on the new queues before
T0 + MANA_TX_TIMEOUT_RESET_THRESHOLD still sees a recent, non-zero last
in mana_tx_timeout():
if (last &&
time_after(last, jiffies - MANA_TX_TIMEOUT_RESET_THRESHOLD)) {
queue_work(ac->per_port_queue_reset_wq, &apc->queue_reset_work);
return;
}
It would then go straight to a port reset, without a CQE poll on queues
that were never polled.
Should the timestamp also be cleared when queues are rebuilt outside this
worker, for example in mana_open() or mana_attach()?
> dealloc_pre_rxbufs:
> mana_pre_dealloc_rxbufs(apc);
> clear_flag:
[ ... ]
> @@ -980,11 +984,30 @@ static void mana_tx_timeout(struct net_device *netdev, unsigned int txqueue)
> if (test_bit(GC_IN_SERVICE, &gc->flags))
> return;
>
> - /* Note: If there are pending queue reset work for this port(apc),
> - * subsequent request queued up from here are ignored. This is because
> - * we are using the same work instance per port(apc).
> + if (!apc->port_is_up)
> + return;
> +
> + /* If the previous timeout fired less than
> + * MANA_TX_TIMEOUT_RESET_THRESHOLD ago, the earlier CQE poll did not
> + * help and queue stalled again : escalate to a port reset.
[Severity: Low]
This isn't a bug, but there's a stray tab between "again" and the colon.
The comment also suggests the same queue stalled again. However,
last_tx_timeout is a single per-port field in struct mana_port_context,
so a timeout on any queue within the threshold escalates. That includes
a queue that never had a CQE poll.
For example, queue A might be polled, and queue B might time out about
15 seconds later (watchdog_timeo). B then goes straight to a port reset.
Could the comment say that a timeout on any queue of the port escalates?
> + * Note: If there is pending queue reset work for this port (apc),
> + * subsequent requests queued up from here are ignored, because the
> + * same work instance is used per port (apc).
> */
> - queue_work(ac->per_port_queue_reset_wq, &apc->queue_reset_work);
> + if (last &&
> + time_after(last, jiffies - MANA_TX_TIMEOUT_RESET_THRESHOLD)) {
> + queue_work(ac->per_port_queue_reset_wq, &apc->queue_reset_work);
> + return;
> + }
> +
> + /* First timeout, or the previous one was long ago: try to recover by
> + * polling the TX CQ for completions before escalating to a reset.
> + */
> + if (txqueue < apc->num_queues && apc->tx_qp && apc->tx_qp[txqueue]) {
> + WRITE_ONCE(apc->last_tx_timeout, jiffies);
> + apc->eth_stats.tx_cqe_poll_recovery++;
> + napi_schedule(&apc->tx_qp[txqueue]->tx_cq.napi);
[Severity: Medium]
Does this recover the case where the MSI-X interrupt for the queue's EQ
was lost? This only schedules NAPI on the TX CQ, and the NAPI path
re-arms only the CQ in mana_cq_handler():
if (w < cq->budget) {
mana_gd_ring_cq(gdma_queue, SET_ARM_BIT);
The only EQ re-arm is at the end of mana_gd_process_eq_events(), which
runs from the EQ interrupt:
mana_gd_ring_doorbell(gc, eq->gdma_dev->doorbell, eq->type, eq->id,
head, SET_ARM_BIT, 0);
The TX CQ and the RX CQ of queue i share the same parent EQ:
mana_create_txq():
spec.cq.parent_eq = apc->eqs[i].eq;
mana_create_rxq() (called with &apc->eqs[i]):
spec.cq.parent_eq = eq->eq;
Suppose the EQ interrupt was lost and the EQ stays un-armed. TX would
make progress briefly. New TX CQEs would then post EQEs into an un-armed
EQ, no interrupt would fire, and TX would stall again. This path never
polls the RX CQ on the same EQ, so RX on queue i would stay stalled.
Before this patch, the first watchdog firing did a detach/attach, which
rebuilt and re-armed the EQ. Now escalation only happens if the next TX
timeout arrives within 2 minutes.
With light TX load on that queue, timeouts could be more than 2 minutes
apart. Each one would then be handled as another Level 1 poll, and RX on
queue i might never recover.
The driver code doesn't show the exact hardware semantics of the GDMA EQ
arm bit, so this depends on whether an interrupt consumes the arm state.
Does the commit message claim of "recovering missed-interrupt scenarios
without queue reset" apply only to CQ-level misses? Or should Level 1
also handle the EQ and the RX CQ that shares it?
> + }
> }
>
> static int mana_shaper_set(struct net_shaper_binding *binding,
[ ... ]
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261112105752.978446-1-sahilchandna%40linux.microsoft.com
prev parent reply other threads:[~2026-11-16 13:54 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-12 10:57 [PATCH net-next] net: mana: Add CQE polling as first-level TX timeout recovery Sahil Chandna
2026-11-12 12:07 ` Breno Leitao
2026-11-13 8:34 ` Sahil Chandna
2026-11-13 10:16 ` Breno Leitao
2026-11-13 10:58 ` sashiko-bot
2026-11-13 12:11 ` Sahil Chandna
2026-11-16 13:54 ` netdev-bot+sashiko [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=179483729831.3136.3233216496914201382@kernel.org \
--to=netdev-bot+sashiko@kernel.org \
--cc=andrew+netdev@lunn.ch \
--cc=davem@davemloft.net \
--cc=decui@microsoft.com \
--cc=edumazet@kernel.org \
--cc=ernis@linux.microsoft.com \
--cc=gargaditya@linux.microsoft.com \
--cc=haiyangz@microsoft.com \
--cc=horms@kernel.org \
--cc=kotaranov@microsoft.com \
--cc=kuba@kernel.org \
--cc=leitao@debian.org \
--cc=linux-hyperv@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=sahilchandna@linux.microsoft.com \
--cc=weh@microsoft.com \
--cc=wei.liu@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox