From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4201A3ACA7E for ; Thu, 9 Jul 2026 21:19:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783631951; cv=none; b=P6B1jJZ6cy4UJUq51HJ3USpocZ09L8yCxJY/w4DL/y8jnM5nDLQgwUWDIYJHLUYQKzq1VIVeS/rEhe4X3BfIjHBVs/GgjoshsvtDPUwlB7lK6gTQURiVJVYSC2wjjzHkC12Kf1Hg/1nVqqlJtIhYA7nPffif17WNJ48J3i7yLCc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783631951; c=relaxed/simple; bh=0g/6zbiEcbIQq0OxAyV2Q08ui9MvxNfKFpl/ZYmlVHA=; h=Date:Mime-Version:Message-ID:Subject:From:To:Cc:Content-Type; b=YuImeJWaLgwG+ENItuhPFhZjNA0A+P2DPIfP0l+MURwr3Fv2OuBi1s2KAaVWw2zex6c/c3UGbSCuD+6BnCARb6e9tCXWZtSKXwdXf43N8ItL2IqlGWeV2TfVMB3GQtYw1bGmxzGE+RpdyW/w1Gyl9ukV7NfADxkboYOe+3N+d/Q= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--hramamurthy.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=eMIjiuvk; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--hramamurthy.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="eMIjiuvk" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-c85a298cd62so245220a12.0 for ; Thu, 09 Jul 2026 14:19:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1783631948; x=1784236748; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:mime-version:date:from :to:cc:subject:date:message-id:reply-to:content-type; bh=+T7RxQXDP6VavQCmW76B4jR7UdEawZb+uP1OIxEgxgI=; b=eMIjiuvkE3IVTxMZ20PuOooAc4sFD5pV6hePdv9stUH26y1b+b3mm73F3oWr610V72 0Qufj1l0VtSSpx9xxC0BjBdge0kHZXMmnxgDd8lzX5D9e44Br1lQ/uCpHIiGhkKBDoHN OXRRcgJGid93Addkp8BDWC7JPx0+vNhwTe+5GBl4JAZHW+tcBB6DT9bAtY4FLoDf/cqB KXb4G3DialODL5XsI3Bxh44UELcvPjIYuQ4VqlVVyBzZQSqNEymltQdmCs5itSJDDDDs Ib7A5QWo0Lm7a046sutrQDjQnGz/6a0o0hLWrRHKeh2E4aEHJmFWLYgYmGVXNvn5LPqo 7bug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783631948; x=1784236748; h=content-type:cc:to:from:subject:message-id:mime-version:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=+T7RxQXDP6VavQCmW76B4jR7UdEawZb+uP1OIxEgxgI=; b=T8vQsXQ2MRRimbBnouGZCHOMazfxd9twrS61+ZqRfa9O7GDwM9JVT67XJT9Iu2CASs bkLKNC5kFxRAElK1qvFFy+N5K9BwFlZ2qdf8V/nx6X+0M5nnZczUZ+8/teBNlfZg3z4L M88bKHO1oXO09zh4Pu3cjU4wT2lkAeI5r92mDm1V6UTljNsMLHnz1ZDOrl0gvikuwsZY SCjT4s11pl+8HWTPbmiFkN9C54EO7sBZpoLQ/bGQkEs2oL471Q8bJQuo2w2nH6noYtNJ +qRQXyNKgEhMA14b0Nu7gA8BWdFNrvL44Gupk7Xbnlv1uvsbB+oDiJzTK2dm3+TpGYOR RgxQ== X-Forwarded-Encrypted: i=1; AHgh+RqYOTDcZRpRticoM4OW0BxNdB3nNms/xutnMQg7CQBJxQLNG7FgEpT2HSSgcO8fYUZ+b0DadTbqExwUvg4=@vger.kernel.org X-Gm-Message-State: AOJu0YxjxjQ7iGCth7Q60y6bX8z6Cc3aO5pdrWgYJiMGF3Ao0EIyGyuQ BcKoswnodTZZQVDmtjkT9N4146bSbrqR1XSFf481a+/KLKgYvHDCE92Lgok26N6G+hFOLt6hTuq +y62tFhCS5jOITEEAf/ioCSieDw== X-Received: from pgmj15.prod.google.com ([2002:a63:594f:0:b0:c96:9e6a:6261]) (user=hramamurthy job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:918c:b0:3bf:63af:859 with SMTP id adf61e73a8af0-3c0bd0f8fd3mr10854302637.45.1783631948380; Thu, 09 Jul 2026 14:19:08 -0700 (PDT) Date: Thu, 9 Jul 2026 21:19:06 +0000 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Mailer: git-send-email 2.55.0.795.g602f6c329a-goog Message-ID: <20260709211906.3322883-1-hramamurthy@google.com> Subject: [PATCH net v2] gve: fix Rx queue stall on alloc failure From: Harshitha Ramamurthy To: netdev@vger.kernel.org Cc: joshwash@google.com, hramamurthy@google.com, andrew+netdev@lunn.ch, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, willemb@google.com, jordanrhee@google.com, nktgrg@google.com, maolson@google.com, thostet@google.com, csully@google.com, bcf@google.com, maciej.fijalkowski@intel.com, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Eddie Phillips Content-Type: text/plain; charset="UTF-8" From: Eddie Phillips When the system is under extreme memory pressure, page allocations can fail during the Rx buffer refill loop. If the number of buffers posted to hardware falls below a critical low threshold and the refill loop exits due to allocation failures, the queue can stall: 1. The device drops incoming packets because there are no descriptors. 2. Since no packets are processed, no Rx completions are generated. 3. Because no completions occur, NAPI is never scheduled, preventing the refill loop from running again even after memory is freed. This results in a permanent queue stall. Resolve this by introducing a starvation recovery timer for each Rx queue. If the number of buffers posted to hardware falls below a critical low threshold, start a timer to periodically reschedule NAPI. Once NAPI runs and successfully refills the queue above the threshold, the timer is not rescheduled. The threshold is set to 32 because a single maximum-sized Receive Segment Coalescing (RSC) packet can consume up to 19 descriptors in the Rx path. Lower thresholds (such as 8 or 16) would be insufficient to process a complete maximum-sized RSC packet, risking packet drops or unexpected hardware behavior under memory pressure. Setting the threshold to 32 guarantees a safe margin to handle at least one full RSC packet. Cc: stable@vger.kernel.org Fixes: 9b8dd5e5ea48 ("gve: DQO: Add RX path") Reviewed-by: Jordan Rhee Signed-off-by: Eddie Phillips Signed-off-by: Harshitha Ramamurthy --- Changes in v2: - Link to v1: https://lore.kernel.org/netdev/20260701005341.3699161-1-hramamurthy@google.com/ - Relocated the starvation timer to the end of gve_rx_ring to avoid polluting hotpath cachelines - Decoupled timer lifecycle from allocation cycles by moving initialization and shutdown to start/stop pathways instead of setup/remove pathways. - Added explicit rationale for the 32-descriptor threshold (GVE_RX_BUF_THRESH_DQO) ensuring it is safe for maximum-sized RSC packets. - Removed addition of a stat tracking critical low buffer events drivers/net/ethernet/google/gve/gve.h | 3 +++ drivers/net/ethernet/google/gve/gve_rx_dqo.c | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/drivers/net/ethernet/google/gve/gve.h b/drivers/net/ethernet/google/gve/gve.h index 2f7bd330..bdd53d08 100644 --- a/drivers/net/ethernet/google/gve/gve.h +++ b/drivers/net/ethernet/google/gve/gve.h @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -41,6 +42,7 @@ /* Interval to schedule a stats report update, 20000ms. */ #define GVE_STATS_REPORT_TIMER_PERIOD 20000 +#define GVE_RX_NAPI_RESCHED_MS 20 /* msecs */ /* Numbers of NIC tx/rx stats in stats report. */ #define NIC_TX_STATS_REPORT_NUM 0 @@ -341,6 +343,7 @@ struct gve_rx_ring { struct xdp_rxq_info xdp_rxq; struct xsk_buff_pool *xsk_pool; struct page_frag_cache page_cache; /* Page cache to allocate XDP frames */ + struct timer_list starvation_timer; /* for queue starvation recovery */ }; /* A TX desc ring entry */ diff --git a/drivers/net/ethernet/google/gve/gve_rx_dqo.c b/drivers/net/ethernet/google/gve/gve_rx_dqo.c index 02cba280..8271f731 100644 --- a/drivers/net/ethernet/google/gve/gve_rx_dqo.c +++ b/drivers/net/ethernet/google/gve/gve_rx_dqo.c @@ -18,6 +18,16 @@ #include #include +static void gve_rx_starvation_timer(struct timer_list *t) +{ + struct gve_rx_ring *rx = timer_container_of(rx, t, starvation_timer); + struct gve_priv *priv = rx->gve; + struct gve_notify_block *block; + + block = &priv->ntfy_blocks[rx->ntfy_id]; + napi_schedule(&block->napi); +} + static void gve_rx_free_hdr_bufs(struct gve_priv *priv, struct gve_rx_ring *rx) { struct device *hdev = &priv->pdev->dev; @@ -120,6 +130,7 @@ void gve_rx_stop_ring_dqo(struct gve_priv *priv, int idx) if (rx->dqo.page_pool) page_pool_disable_direct_recycling(rx->dqo.page_pool); + timer_shutdown_sync(&rx->starvation_timer); gve_remove_napi(priv, ntfy_idx); gve_rx_remove_from_block(priv, idx); gve_rx_reset_ring_dqo(priv, idx); @@ -208,8 +219,10 @@ static int gve_rx_alloc_hdr_bufs(struct gve_priv *priv, struct gve_rx_ring *rx, void gve_rx_start_ring_dqo(struct gve_priv *priv, int idx) { int ntfy_idx = gve_rx_idx_to_ntfy(priv, idx); + struct gve_rx_ring *rx = &priv->rx[idx]; gve_rx_add_to_block(priv, idx); + timer_setup(&rx->starvation_timer, gve_rx_starvation_timer, 0); gve_add_napi(priv, ntfy_idx, gve_napi_poll_dqo); } @@ -365,6 +378,7 @@ void gve_rx_post_buffers_dqo(struct gve_rx_ring *rx) struct gve_rx_compl_queue_dqo *complq = &rx->dqo.complq; struct gve_rx_buf_queue_dqo *bufq = &rx->dqo.bufq; struct gve_priv *priv = rx->gve; + u32 num_bufs_avail_to_hw; u32 num_avail_slots; u32 num_full_slots; u32 num_posted = 0; @@ -400,6 +414,26 @@ void gve_rx_post_buffers_dqo(struct gve_rx_ring *rx) } rx->fill_cnt += num_posted; + + /* If the queue has fewer than GVE_RX_BUF_THRESH_DQO descriptors + * visible to the hardware, the hardware is in danger of starving + * and cannot trigger interrupts. + * + * We use a threshold of 32 because a single maximum-sized RSC + * packet can consume up to 19 descriptors in the Rx path. Lower + * thresholds (e.g., 8 or 16) would be unsafe as they could cause + * the device to drop/stall on a maximum-sized RSC packet. + * + * Start the timer to periodically reschedule NAPI and recover. + */ + num_bufs_avail_to_hw = + ((bufq->tail & ~(GVE_RX_BUF_THRESH_DQO - 1)) - + bufq->head) & bufq->mask; + + if (num_bufs_avail_to_hw < GVE_RX_BUF_THRESH_DQO) { + mod_timer(&rx->starvation_timer, + jiffies + msecs_to_jiffies(GVE_RX_NAPI_RESCHED_MS)); + } } static void gve_rx_skb_csum(struct sk_buff *skb, -- 2.55.0.795.g602f6c329a-goog