* [PATCH net-next] net: mana: Add CQE polling as first-level TX timeout recovery @ 2026-11-12 10:57 Sahil Chandna 2026-11-12 12:07 ` Breno Leitao 2026-11-16 13:54 ` netdev-bot+sashiko 0 siblings, 2 replies; 5+ messages in thread From: Sahil Chandna @ 2026-11-12 10:57 UTC (permalink / raw) To: haiyangz, wei.liu, decui, andrew+netdev, davem, edumazet, kuba, pabeni, weh, kotaranov, horms, ernis, gargaditya, leitao, sahilchandna, linux-hyperv, netdev, linux-kernel, linux-rdma The current mana_tx_timeout() handler unconditionally schedules a full port queue reset (detach/attach) whenever the TX watchdog fires. This is unnecessary for transient stalls where completions are simply delayed or an interrupt was missed. Implement a graduated recovery approach: on the first TX timeout (or when the previous timeout was more than 2 minutes ago), schedule NAPI on the stalled queue's TX CQ via napi_schedule(). This triggers mana_poll_tx_cq() through the normal NAPI path, processing any pending completions without relying on the interrupt and recovering missed-interrupt scenarios without queue reset. If any subsequent TX timeout fires on any queue, within 2 minutes of the previous one, escalate to the existing port-level detach/attach reset, as the CQE poll did not recover the queue. This makes the TX timeout recovery three-tiered: Level 1: CQE poll via napi_schedule() (this patch) Level 2: Port queue reset (detach/attach) Level 3: device reset(FLR) Co-developed-by: Dipayaan Roy <dipayanroy@linux.microsoft.com> Signed-off-by: Dipayaan Roy <dipayanroy@linux.microsoft.com> Signed-off-by: Sahil Chandna <sahilchandna@linux.microsoft.com> --- drivers/net/ethernet/microsoft/mana/mana_en.c | 31 ++++++++++++++++--- .../ethernet/microsoft/mana/mana_ethtool.c | 2 ++ include/net/mana/mana.h | 4 +++ 3 files changed, 33 insertions(+), 4 deletions(-) diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/ethernet/microsoft/mana/mana_en.c index e6d494df22e4..0d8c8b8fbc83 100644 --- a/drivers/net/ethernet/microsoft/mana/mana_en.c +++ b/drivers/net/ethernet/microsoft/mana/mana_en.c @@ -336,6 +336,9 @@ static void mana_per_port_queue_reset_work_handler(struct work_struct *work) if (err) netdev_err(ndev, "mana_attach failed: %d\n", err); + /* Let the next TX timeout try a CQE poll recovery first */ + WRITE_ONCE(apc->last_tx_timeout, 0); + dealloc_pre_rxbufs: mana_pre_dealloc_rxbufs(apc); clear_flag: @@ -964,6 +967,7 @@ static void mana_tx_timeout(struct net_device *netdev, unsigned int txqueue) struct mana_port_context *apc = netdev_priv(netdev); struct mana_context *ac = apc->ac; struct gdma_context *gc = ac->gdma_dev->gdma_context; + unsigned long last = READ_ONCE(apc->last_tx_timeout); /* Debug knob for bringup/qualification: when set, log the timeout and * skip the reset so the failing state is preserved for telemetry. @@ -980,11 +984,30 @@ static void mana_tx_timeout(struct net_device *netdev, unsigned int txqueue) if (test_bit(GC_IN_SERVICE, &gc->flags)) return; - /* Note: If there are pending queue reset work for this port(apc), - * subsequent request queued up from here are ignored. This is because - * we are using the same work instance per port(apc). + if (!apc->port_is_up) + return; + + /* If the previous timeout fired less than + * MANA_TX_TIMEOUT_RESET_THRESHOLD ago, the earlier CQE poll did not + * help and queue stalled again : escalate to a port reset. + * Note: If there is pending queue reset work for this port (apc), + * subsequent requests queued up from here are ignored, because the + * same work instance is used per port (apc). */ - queue_work(ac->per_port_queue_reset_wq, &apc->queue_reset_work); + if (last && + time_after(last, jiffies - MANA_TX_TIMEOUT_RESET_THRESHOLD)) { + queue_work(ac->per_port_queue_reset_wq, &apc->queue_reset_work); + return; + } + + /* First timeout, or the previous one was long ago: try to recover by + * polling the TX CQ for completions before escalating to a reset. + */ + if (txqueue < apc->num_queues && apc->tx_qp && apc->tx_qp[txqueue]) { + WRITE_ONCE(apc->last_tx_timeout, jiffies); + apc->eth_stats.tx_cqe_poll_recovery++; + napi_schedule(&apc->tx_qp[txqueue]->tx_cq.napi); + } } static int mana_shaper_set(struct net_shaper_binding *binding, diff --git a/drivers/net/ethernet/microsoft/mana/mana_ethtool.c b/drivers/net/ethernet/microsoft/mana/mana_ethtool.c index a91eb12badca..ff4799c61a6c 100644 --- a/drivers/net/ethernet/microsoft/mana/mana_ethtool.c +++ b/drivers/net/ethernet/microsoft/mana/mana_ethtool.c @@ -22,6 +22,8 @@ static const struct mana_stats_desc mana_eth_stats[] = { tx_linear_pkt_cnt)}, {"rx_cqe_unknown_type", offsetof(struct mana_ethtool_stats, rx_cqe_unknown_type)}, + {"tx_cqe_poll_recovery", offsetof(struct mana_ethtool_stats, + tx_cqe_poll_recovery)}, }; static const struct mana_stats_desc mana_hc_stats[] = { diff --git a/include/net/mana/mana.h b/include/net/mana/mana.h index 09725e3256fa..5de6eaa02102 100644 --- a/include/net/mana/mana.h +++ b/include/net/mana/mana.h @@ -94,6 +94,8 @@ enum mana_priv_flag_bits { #define MANA_RX_FRAG_ALIGNMENT 64 +#define MANA_TX_TIMEOUT_RESET_THRESHOLD (2 * 60 * HZ) + struct mana_stats_rx { u64 packets; u64 bytes; @@ -447,6 +449,7 @@ struct mana_tx_qp { struct mana_ethtool_stats { u64 stop_queue; u64 wake_queue; + u64 tx_cqe_poll_recovery; u64 tx_cqe_err; u64 tx_cqe_unknown_type; u64 tx_linear_pkt_cnt; @@ -566,6 +569,7 @@ struct mana_port_context { struct net_device *ndev; struct work_struct queue_reset_work; + unsigned long last_tx_timeout; /* Debug knob to log TX timeout but skip recovery reset */ bool tx_timeout_skip_reset; -- 2.53.0 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH net-next] net: mana: Add CQE polling as first-level TX timeout recovery 2026-11-12 10:57 [PATCH net-next] net: mana: Add CQE polling as first-level TX timeout recovery Sahil Chandna @ 2026-11-12 12:07 ` Breno Leitao 2026-11-13 8:34 ` Sahil Chandna 2026-11-16 13:54 ` netdev-bot+sashiko 1 sibling, 1 reply; 5+ messages in thread From: Breno Leitao @ 2026-11-12 12:07 UTC (permalink / raw) To: Sahil Chandna Cc: haiyangz, wei.liu, decui, andrew+netdev, davem, edumazet, kuba, pabeni, weh, kotaranov, horms, ernis, gargaditya, linux-hyperv, netdev, linux-kernel, linux-rdma Hello Sahil, On Thu, Nov 12, 2026 at 04:57:18AM -0600, Sahil Chandna wrote: > Implement a graduated recovery approach: on the first TX timeout (or > when the previous timeout was more than 2 minutes ago), schedule NAPI on > the stalled queue's TX CQ via napi_schedule(). This triggers > mana_poll_tx_cq() through the normal NAPI path, processing any pending > completions without relying on the interrupt and recovering > missed-interrupt scenarios without queue reset. Which interrupt is being missed here? > If any subsequent TX timeout fires on any queue, within 2 minutes of > the previous one, escalate to the existing port-level detach/attach > reset, as the CQE poll did not recover the queue. How was 2 minutes chosen? mana sets watchdog_timeo to 15 * HZ, no? ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH net-next] net: mana: Add CQE polling as first-level TX timeout recovery 2026-11-12 12:07 ` Breno Leitao @ 2026-11-13 8:34 ` Sahil Chandna 2026-11-13 10:16 ` Breno Leitao 0 siblings, 1 reply; 5+ messages in thread From: Sahil Chandna @ 2026-11-13 8:34 UTC (permalink / raw) To: Breno Leitao Cc: haiyangz, wei.liu, decui, andrew+netdev, davem, edumazet, kuba, pabeni, weh, kotaranov, horms, ernis, gargaditya, linux-hyperv, netdev, linux-kernel, linux-rdma On 12-11-2026 17:37, Breno Leitao wrote: > Hello Sahil, > > On Thu, Nov 12, 2026 at 04:57:18AM -0600, Sahil Chandna wrote: >> Implement a graduated recovery approach: on the first TX timeout (or >> when the previous timeout was more than 2 minutes ago), schedule NAPI on >> the stalled queue's TX CQ via napi_schedule(). This triggers >> mana_poll_tx_cq() through the normal NAPI path, processing any pending >> completions without relying on the interrupt and recovering >> missed-interrupt scenarios without queue reset. > > Which interrupt is being missed here? > Hi Breno, It is the TX CQ completion interrupt. After NAPI finishes, it re-arms the TX CQ. When a new CQE arrives, the hardware post EQE, which raises the MSI-X and schedules NAPI. If that notification for the TX CQ is lost , NAPI never runs for the TX CQ and pending_sends never drains and the queue stays stopped, so napi_schedule() polls the CQ, reaps the CQEs and re-arms it. >> If any subsequent TX timeout fires on any queue, within 2 minutes of >> the previous one, escalate to the existing port-level detach/attach >> reset, as the CQE poll did not recover the queue. > > How was 2 minutes chosen? mana sets watchdog_timeo to 15 * HZ, no? The watchdog_timeo is TX stall timeout setting i.e if a queue remains stalled for this much time, it trigger watchdog timeout. Here, 2 minute is not timeout/wait. The 2 minutes here is for deciding whether a new queue stall event is a repeat of the previous incident or a new, unrelated one that should get the cheap poll first. For e.g. if a queue gets stalled, 15 * HZ sec later, timeout occurs and this patch trigger manual NAPI, if another stall occurs (it can be on any queue) within 2 minutes of last one, the existing recovery mechanism of mana detach/attach will be tried.The patch is adding a first level less destructive recovery first. For selecting 2 minutes, I do not have any hardware basis for it afaik, but I can confirm and check with our host team as well. Hope this helps ! Regards, Sahil ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH net-next] net: mana: Add CQE polling as first-level TX timeout recovery 2026-11-13 8:34 ` Sahil Chandna @ 2026-11-13 10:16 ` Breno Leitao 0 siblings, 0 replies; 5+ messages in thread From: Breno Leitao @ 2026-11-13 10:16 UTC (permalink / raw) To: Sahil Chandna Cc: haiyangz, wei.liu, decui, andrew+netdev, davem, edumazet, kuba, pabeni, weh, kotaranov, horms, ernis, gargaditya, linux-hyperv, netdev, linux-kernel, linux-rdma On Fri, Nov 13, 2026 at 02:04:08PM +0530, Sahil Chandna wrote: > On 12-11-2026 17:37, Breno Leitao wrote: > > Hello Sahil, > > > > On Thu, Nov 12, 2026 at 04:57:18AM -0600, Sahil Chandna wrote: > >> Implement a graduated recovery approach: on the first TX timeout (or > >> when the previous timeout was more than 2 minutes ago), schedule NAPI on > >> the stalled queue's TX CQ via napi_schedule(). This triggers > >> mana_poll_tx_cq() through the normal NAPI path, processing any pending > >> completions without relying on the interrupt and recovering > >> missed-interrupt scenarios without queue reset. > > > > Which interrupt is being missed here? > > > Hi Breno, > > It is the TX CQ completion interrupt. After NAPI finishes, it re-arms > the TX CQ. When a new CQE arrives, the hardware post EQE, which raises > the MSI-X and schedules NAPI. If that notification for the TX CQ is lost > , NAPI never runs for the TX CQ and pending_sends never drains and the > queue stays stopped, so napi_schedule() polls the CQ, reaps the CQEs > and re-arms it. Thanks. Which one gets lost, the EQE or the MSI-X? > >> If any subsequent TX timeout fires on any queue, within 2 minutes of > >> the previous one, escalate to the existing port-level detach/attach > >> reset, as the CQE poll did not recover the queue. > > > > How was 2 minutes chosen? mana sets watchdog_timeo to 15 * HZ, no? > The watchdog_timeo is TX stall timeout setting i.e if a queue remains > stalled for this much time, it trigger watchdog timeout. Here, 2 minute > is not timeout/wait. The 2 minutes here is for deciding whether a new > queue stall event is a repeat of the previous incident or a new, > unrelated one that should get the cheap poll first. For e.g. if a queue > gets stalled, 15 * HZ sec later, timeout occurs and this patch trigger > manual NAPI, if another stall occurs (it can be on any queue) within 2 > minutes of last one, the existing recovery mechanism of mana > detach/attach will be tried.The patch is adding a first level less > destructive recovery first. > For selecting 2 minutes, I do not have any hardware basis for it afaik, > but I can confirm and check with our host team as well. So the goal is to tell an isolated stall from a wider problem. A port-wide timestamp only approximates that. It cannot tell the same queue from another one, and since dev_watchdog() reports one queue per run, two queues that stall together show up ~15s apart: the first gets a kick, the second gets the port reset anyway, no? Could mana_tx_timeout() check this directly? Look at the other TX queues the way dev_watchdog() does (stopped, with trans_start older than watchdog_timeo). If more than one is stalled, reset right away. Otherwise kick this queue, and reset if the same queue times out again within a couple of watchdog periods, as gve_tx_timeout_try_q_kick() does. That also gets rid of the 2 minutes. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH net-next] net: mana: Add CQE polling as first-level TX timeout recovery 2026-11-12 10:57 [PATCH net-next] net: mana: Add CQE polling as first-level TX timeout recovery Sahil Chandna 2026-11-12 12:07 ` Breno Leitao @ 2026-11-16 13:54 ` netdev-bot+sashiko 1 sibling, 0 replies; 5+ messages in thread From: netdev-bot+sashiko @ 2026-11-16 13:54 UTC (permalink / raw) To: sahilchandna Cc: haiyangz, wei.liu, decui, andrew+netdev, davem, edumazet, kuba, pabeni, weh, kotaranov, horms, ernis, gargaditya, leitao, linux-hyperv, netdev, linux-kernel, linux-rdma Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider. Critical: 0 · High: 0 · Medium: 1 · Low: 3 If you believe none of the issues are legitimate, please reply to each issue of Medium severity or higher and explain why the AI is wrong. If any of them are legitimate and you plan to respin, please let patchwork know by including "pw-bot: cr" as a separate line at the end of your reply (one such reply per series is enough). - [Medium] Level 1 of the new recovery only calls napi_schedule() on the TX CQ that stalled, in mana_tx_timeout()… - [Low] The commit message says the patch makes TX timeout recovery 'three-tiered' and lists 'Level 3: device reset(FLR)' as the step… - [Low] apc->last_tx_timeout is cleared in only one place: the post- attach tail of mana_per_port_queue_reset_work_handler()… - [Low] The new comment in mana_tx_timeout() (mana_en.c:990-996) says escalation happens because 'the earlier CQE poll did not help and… -- net: mana: Add CQE polling as first-level TX timeout recovery This commit changes mana_tx_timeout() so that the first TX timeout, or one arriving more than 2 minutes after the previous one, schedules NAPI on the stalled queue's TX CQ instead of resetting the port. A further timeout on any queue within 2 minutes escalates to the existing per-port detach/attach reset, and a new tx_cqe_poll_recovery ethtool counter tracks the CQE poll attempts. > This makes the TX timeout recovery three-tiered: > > Level 1: CQE poll via napi_schedule() (this patch) > Level 2: Port queue reset (detach/attach) > Level 3: device reset(FLR) [Severity: Low] Is the Level 3 step accurate for the TX timeout path? When mana_pre_alloc_rxbufs(), mana_detach() or mana_attach() fails in mana_per_port_queue_reset_work_handler(), the handler only logs: err = mana_attach(ndev); if (err) netdev_err(ndev, "mana_attach failed: %d\n", err); It doesn't call mana_schedule_serv_work() or set GC_IN_SERVICE. In mana_en.c, mana_schedule_serv_work() is only called from mana_gf_stats_work_handler() on an HWC timeout. gdma_main.c calls it for hardware service EQEs. So when a port reset fails or doesn't help, the next timeout triggers another port reset, or after 2 minutes goes back to Level 1. It never reaches an FLR. Could the commit message describe the FLR as a separate existing mechanism, rather than as the third tier of this escalation? > diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/ethernet/microsoft/mana/mana_en.c > index e6d494df22e4d..0d8c8b8fbc83f 100644 > --- a/drivers/net/ethernet/microsoft/mana/mana_en.c > +++ b/drivers/net/ethernet/microsoft/mana/mana_en.c > @@ -336,6 +336,9 @@ static void mana_per_port_queue_reset_work_handler(struct work_struct *work) > if (err) > netdev_err(ndev, "mana_attach failed: %d\n", err); > > + /* Let the next TX timeout try a CQE poll recovery first */ > + WRITE_ONCE(apc->last_tx_timeout, 0); > + [Severity: Low] Is this the only place last_tx_timeout gets cleared? The other paths that tear down and rebuild the TX queues don't touch it. These include mana_close()/mana_open(), the other mana_detach()/mana_attach() callers (channel, MTU and XDP changes), and the service/FLR resume through mana_probe(). Suppose a Level 1 CQE poll ran at T0 and one of those paths then rebuilt the queues. A TX timeout on the new queues before T0 + MANA_TX_TIMEOUT_RESET_THRESHOLD still sees a recent, non-zero last in mana_tx_timeout(): if (last && time_after(last, jiffies - MANA_TX_TIMEOUT_RESET_THRESHOLD)) { queue_work(ac->per_port_queue_reset_wq, &apc->queue_reset_work); return; } It would then go straight to a port reset, without a CQE poll on queues that were never polled. Should the timestamp also be cleared when queues are rebuilt outside this worker, for example in mana_open() or mana_attach()? > dealloc_pre_rxbufs: > mana_pre_dealloc_rxbufs(apc); > clear_flag: [ ... ] > @@ -980,11 +984,30 @@ static void mana_tx_timeout(struct net_device *netdev, unsigned int txqueue) > if (test_bit(GC_IN_SERVICE, &gc->flags)) > return; > > - /* Note: If there are pending queue reset work for this port(apc), > - * subsequent request queued up from here are ignored. This is because > - * we are using the same work instance per port(apc). > + if (!apc->port_is_up) > + return; > + > + /* If the previous timeout fired less than > + * MANA_TX_TIMEOUT_RESET_THRESHOLD ago, the earlier CQE poll did not > + * help and queue stalled again : escalate to a port reset. [Severity: Low] This isn't a bug, but there's a stray tab between "again" and the colon. The comment also suggests the same queue stalled again. However, last_tx_timeout is a single per-port field in struct mana_port_context, so a timeout on any queue within the threshold escalates. That includes a queue that never had a CQE poll. For example, queue A might be polled, and queue B might time out about 15 seconds later (watchdog_timeo). B then goes straight to a port reset. Could the comment say that a timeout on any queue of the port escalates? > + * Note: If there is pending queue reset work for this port (apc), > + * subsequent requests queued up from here are ignored, because the > + * same work instance is used per port (apc). > */ > - queue_work(ac->per_port_queue_reset_wq, &apc->queue_reset_work); > + if (last && > + time_after(last, jiffies - MANA_TX_TIMEOUT_RESET_THRESHOLD)) { > + queue_work(ac->per_port_queue_reset_wq, &apc->queue_reset_work); > + return; > + } > + > + /* First timeout, or the previous one was long ago: try to recover by > + * polling the TX CQ for completions before escalating to a reset. > + */ > + if (txqueue < apc->num_queues && apc->tx_qp && apc->tx_qp[txqueue]) { > + WRITE_ONCE(apc->last_tx_timeout, jiffies); > + apc->eth_stats.tx_cqe_poll_recovery++; > + napi_schedule(&apc->tx_qp[txqueue]->tx_cq.napi); [Severity: Medium] Does this recover the case where the MSI-X interrupt for the queue's EQ was lost? This only schedules NAPI on the TX CQ, and the NAPI path re-arms only the CQ in mana_cq_handler(): if (w < cq->budget) { mana_gd_ring_cq(gdma_queue, SET_ARM_BIT); The only EQ re-arm is at the end of mana_gd_process_eq_events(), which runs from the EQ interrupt: mana_gd_ring_doorbell(gc, eq->gdma_dev->doorbell, eq->type, eq->id, head, SET_ARM_BIT, 0); The TX CQ and the RX CQ of queue i share the same parent EQ: mana_create_txq(): spec.cq.parent_eq = apc->eqs[i].eq; mana_create_rxq() (called with &apc->eqs[i]): spec.cq.parent_eq = eq->eq; Suppose the EQ interrupt was lost and the EQ stays un-armed. TX would make progress briefly. New TX CQEs would then post EQEs into an un-armed EQ, no interrupt would fire, and TX would stall again. This path never polls the RX CQ on the same EQ, so RX on queue i would stay stalled. Before this patch, the first watchdog firing did a detach/attach, which rebuilt and re-armed the EQ. Now escalation only happens if the next TX timeout arrives within 2 minutes. With light TX load on that queue, timeouts could be more than 2 minutes apart. Each one would then be handled as another Level 1 poll, and RX on queue i might never recover. The driver code doesn't show the exact hardware semantics of the GDMA EQ arm bit, so this depends on whether an interrupt consumes the arm state. Does the commit message claim of "recovering missed-interrupt scenarios without queue reset" apply only to CQ-level misses? Or should Level 1 also handle the EQ and the RX CQ that shares it? > + } > } > > static int mana_shaper_set(struct net_shaper_binding *binding, [ ... ] -- Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261112105752.978446-1-sahilchandna%40linux.microsoft.com ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-11-16 13:54 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-11-12 10:57 [PATCH net-next] net: mana: Add CQE polling as first-level TX timeout recovery Sahil Chandna 2026-11-12 12:07 ` Breno Leitao 2026-11-13 8:34 ` Sahil Chandna 2026-11-13 10:16 ` Breno Leitao 2026-11-16 13:54 ` netdev-bot+sashiko
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox