[PATCH 0/4] net/gve: fix TX queue stalls and sync DQO RDA with Linux driver

[PATCH 0/4] net/gve: fix TX queue stalls and sync DQO RDA with Linux driver

[Tathagat Priyadarshi]

This patch series aligns the Google Virtual NIC (gve) DQO RDA transmit path
with the upstream Linux kernel driver (drivers/net/ethernet/google/gve/) to
resolve transmit queue stalls and hardware validation issues:

1. net/gve: fix DQO descriptor cleaning loop bound
   Fixes the off-by-one error in gve_tx_clean_descs_dqo()
   ("while (--nb_descs)" -> "while (nb_descs--)") and avoids stalling
   the completion queue on unrecognized descriptor types.

2. net/gve: fix DQO TX queue stall on multi-seg and TSO
   Tracks descriptor intervals with last_re_idx per packet against
   GVE_TX_MIN_RE_INTERVAL (32) instead of counting packets at burst end,
   preventing descriptor ring starvation and queue deadlocks under
   multi-segment and TSO traffic.

3. net/gve: add cacheline overlap margin to DQO TX ring
   Enforces GVE_TX_MIN_DESC_PREVENT_CACHE_OVERLAP (4 descriptors) margin
   in ring space checks, avoiding CPU and device DMA cacheline collisions.

4. net/gve: add packet and TSO hardware constraints for DQO
   Augments gve_tx_validate_descs() (which enforces the 10-descriptor
   limit per packet or MTU segment) with the remaining hardware length
   constraints: [17, 9728] bytes for non-TSO packets, maximum MSS limit
   (9728 bytes) to avoid 14-bit field wrap-around, and validates header
   contiguity and payload presence on TSO packets.

Tathagat Priyadarshi (4):
  net/gve: fix DQO descriptor cleaning loop bound
  net/gve: fix DQO TX queue stall on multi-seg and TSO
  net/gve: add cacheline overlap margin to DQO TX ring
  net/gve: add packet and TSO hardware constraints for DQO

 drivers/net/gve/base/gve_desc_dqo.h |  3 ++
 drivers/net/gve/gve_ethdev.h        |  2 +-
 drivers/net/gve/gve_tx_dqo.c        | 80 +++++++++++++++++++++--------
 3 files changed, 62 insertions(+), 23 deletions(-)

-- 
2.43.0

[PATCH 1/4] net/gve: fix DQO descriptor cleaning loop bound

[Tathagat Priyadarshi]

In gve_tx_clean_descs_dqo(), the loop condition "while (--nb_descs)"
causes an off-by-one error:
- When nb_descs == 1, --nb_descs decrements to 0 on the first check,
  executing 0 iterations instead of 1.
- When nb_descs == 0, --nb_descs underflows uint16_t to 65535,
  executing 65535 iterations.

Change the condition to "while (nb_descs--)" so that exactly nb_descs
iterations are executed for any input value.

Additionally, in gve_tx_clean_dqo(), handle unrecognized completion
types by breaking from the switch rather than returning immediately,
allowing complq_tail to advance past unknown descriptors rather than
stalling the completion ring at the same entry.

Fixes: 4022f9999f56 ("net/gve: support basic Tx data path for DQO")
Cc: stable@dpdk.org

Acked-by: Joshua Washington <joshwash@google.com>
Signed-off-by: Tathagat Priyadarshi <tathagat.dpdk@gmail.com>
---
 drivers/net/gve/gve_tx_dqo.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/net/gve/gve_tx_dqo.c b/drivers/net/gve/gve_tx_dqo.c
index 5c03d3d1a2..c864d722d9 100644
--- a/drivers/net/gve/gve_tx_dqo.c
+++ b/drivers/net/gve/gve_tx_dqo.c
@@ -108,8 +108,8 @@ gve_tx_clean_dqo(struct gve_tx_queue *txq)
 		PMD_DRV_DP_LOG(DEBUG, "GVE_COMPL_TYPE_DQO_MISS ignored !!!");
 		break;
 	default:
-		PMD_DRV_DP_LOG(ERR, "unknown completion type.");
-		return;
+		PMD_DRV_DP_LOG(ERR, "unknown completion type %u", compl_desc->type);
+		break;
 	}
 
 	next++;
@@ -122,7 +122,7 @@ gve_tx_clean_dqo(struct gve_tx_queue *txq)
 
 static inline void
 gve_tx_clean_descs_dqo(struct gve_tx_queue *txq, uint16_t nb_descs) {
-	while (--nb_descs)
+	while (nb_descs--)
 		gve_tx_clean_dqo(txq);
 }
 
-- 
2.43.0

[PATCH 2/4] net/gve: fix DQO TX queue stall on multi-seg and TSO

[Tathagat Priyadarshi]

The DPDK GVE driver previously tracked transmitted packet count (re_cnt)
at the end of each burst rather than tracking descriptor intervals per
packet against GVE_TX_MIN_RE_INTERVAL (32).

In multi-segment and TSO transmissions, each packet consumes multiple
descriptors. The descriptor ring could exhaust before 32 packets were
sent, resulting in zero descriptors requesting report_event and
preventing the hardware DMA engine from returning GVE_COMPL_TYPE_DQO_DESC
completions. Without these completions, txq->nb_free was never
replenished, permanently stalling the TX queue.

Align with the Linux kernel driver (gve_tx_update_tail()) by tracking
descriptor indices with last_re_idx:
- Replace re_cnt with last_re_idx in struct gve_tx_queue.
- Reset last_re_idx = 0 in gve_reset_tx_ring_state_dqo().
- On each packet transmission, check descriptor interval
  (last_desc_idx - last_re_idx) & mask >= GVE_TX_MIN_RE_INTERVAL.
  When met, set report_event on the packet's last descriptor and
  update last_re_idx.

Fixes: 4022f9999f56 ("net/gve: support basic Tx data path for DQO")
Fixes: 403c671a46b6 ("net/gve: support TSO in DQO RDA")
Cc: stable@dpdk.org

Acked-by: Joshua Washington <joshwash@google.com>
Signed-off-by: Tathagat Priyadarshi <tathagat.dpdk@gmail.com>
---
 drivers/net/gve/gve_ethdev.h |  2 +-
 drivers/net/gve/gve_tx_dqo.c | 18 ++++++++++--------
 2 files changed, 11 insertions(+), 9 deletions(-)

diff --git a/drivers/net/gve/gve_ethdev.h b/drivers/net/gve/gve_ethdev.h
index c9a176ff17..2033f49444 100644
--- a/drivers/net/gve/gve_ethdev.h
+++ b/drivers/net/gve/gve_ethdev.h
@@ -193,8 +193,8 @@ struct gve_tx_queue {
 	uint16_t sw_size;
 	uint8_t cur_gen_bit;
 	uint32_t last_desc_cleaned;
+	uint32_t last_re_idx;
 	void **txqs;
-	uint16_t re_cnt;
 
 	/* Only valid for DQO_RDA queue format */
 	struct gve_tx_queue *complq;
diff --git a/drivers/net/gve/gve_tx_dqo.c b/drivers/net/gve/gve_tx_dqo.c
index c864d722d9..a8382ff3c4 100644
--- a/drivers/net/gve/gve_tx_dqo.c
+++ b/drivers/net/gve/gve_tx_dqo.c
@@ -264,6 +264,8 @@ gve_tx_burst_dqo(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 	uint16_t nb_descs;
 	bool csum, tso;
 	uint16_t tx_id;
+	uint32_t last_desc_idx;
+	uint32_t last_re_interval;
 
 	pkts = txq->pkt_ring_dqo;
 	txr = txq->tx_ring;
@@ -331,20 +333,19 @@ gve_tx_burst_dqo(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 		txd = &txr[(tx_id - 1) & mask];
 		txd->pkt.end_of_packet = 1;
 
+		last_desc_idx = (tx_id - 1) & mask;
+		last_re_interval = (last_desc_idx - txq->last_re_idx) & mask;
+		if (unlikely(last_re_interval >= GVE_TX_MIN_RE_INTERVAL)) {
+			txd->pkt.report_event = true;
+			txq->last_re_idx = last_desc_idx;
+		}
+
 		txq->nb_free -= nb_descs;
 		txq->nb_used += nb_descs;
 	}
 
 	/* update the tail pointer if any packets were processed */
 	if (nb_tx > 0) {
-		/* Request a descriptor completion on the last descriptor */
-		txq->re_cnt += nb_tx;
-		if (txq->re_cnt >= GVE_TX_MIN_RE_INTERVAL) {
-			txd = &txr[(tx_id - 1) & mask];
-			txd->pkt.report_event = true;
-			txq->re_cnt = 0;
-		}
-
 		rte_write32(tx_id, txq->qtx_tail);
 		txq->tx_tail = tx_id;
 
@@ -445,6 +446,7 @@ gve_reset_tx_ring_state_dqo(struct gve_tx_queue *txq)
 	txq->nb_used = 0;
 
 	txq->last_desc_cleaned = 0;
+	txq->last_re_idx = 0;
 	txq->nb_free = txq->nb_tx_desc - 1;
 
 	txq->complq_tail = 0;
-- 
2.43.0

[PATCH 3/4] net/gve: add cacheline overlap margin to DQO TX ring

[Tathagat Priyadarshi]

The GVNIC DQO hardware DMA engine prefetches descriptors in cacheline
units (64 bytes = 4 descriptors of 16 bytes each). When producer tail
and consumer head indices share a cacheline, CPU and device DMA memory
collisions can occur.

Align with the Linux driver (gve_has_tx_slots_available()) by requiring
GVE_TX_MIN_DESC_PREVENT_CACHE_OVERLAP (4 descriptors) headroom when
checking descriptor availability before enqueueing packets.

Fixes: 4022f9999f56 ("net/gve: support basic Tx data path for DQO")
Cc: stable@dpdk.org

Acked-by: Joshua Washington <joshwash@google.com>
Signed-off-by: Tathagat Priyadarshi <tathagat.dpdk@gmail.com>
---
 drivers/net/gve/gve_tx_dqo.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/drivers/net/gve/gve_tx_dqo.c b/drivers/net/gve/gve_tx_dqo.c
index a8382ff3c4..9210b02eb1 100644
--- a/drivers/net/gve/gve_tx_dqo.c
+++ b/drivers/net/gve/gve_tx_dqo.c
@@ -261,6 +261,7 @@ gve_tx_burst_dqo(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 	uint16_t nb_tx = 0;
 	uint64_t bytes = 0;
 	uint64_t ol_flags;
+	uint16_t req_descs;
 	uint16_t nb_descs;
 	bool csum, tso;
 	uint16_t tx_id;
@@ -291,14 +292,17 @@ gve_tx_burst_dqo(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 
 		nb_descs = gve_tx_pkt_nb_data_descs(tx_pkt);
 		nb_descs += tso;
+		req_descs = nb_descs + GVE_TX_MIN_DESC_PREVENT_CACHE_OVERLAP;
 
-		/* Clean if there aren't enough descriptors to send the packet. */
-		if (unlikely(!gve_can_tx(txq, nb_descs, 1))) {
+		/* Clean if there aren't enough descriptors to send the packet
+		 * while maintaining the cacheline overlap margin.
+		 */
+		if (unlikely(!gve_can_tx(txq, req_descs, 1))) {
 			int nb_to_clean = RTE_MAX(DQO_TX_MULTIPLIER * txq->rs_thresh,
-						  nb_descs);
+						  (int)req_descs);
 
 			gve_tx_clean_descs_dqo(txq, nb_to_clean);
-			if (!gve_can_tx(txq, nb_descs, 1))
+			if (!gve_can_tx(txq, req_descs, 1))
 				break;
 		}
 
-- 
2.43.0

[PATCH 4/4] net/gve: add packet and TSO hardware constraints for DQO

[Tathagat Priyadarshi]

The GVNIC DQO hardware restricts each packet (or MTU segment in TSO)
to at most 10 data descriptors (GVE_TX_MAX_DATA_DESCS), beyond which
the hardware drops traffic. In addition to this 10-descriptor limit,
the hardware imposes strict packet sizing and header constraints.

Enforce these hardware packet and TSO validation constraints in DQO
RDA mode within gve_tx_validate_descs() matching the upstream Linux
kernel driver:
- Define GVE_TX_MAX_TSO_MSS_DQO (9728 bytes) to avoid 14-bit MSS
  field overflow in struct gve_tx_tso_context_desc_dqo.
- Define GVE_TX_MIN_PKT_SIZE_DQO (17 bytes) and GVE_TX_MAX_PKT_SIZE_DQO
  (9728 bytes) to reject runts (which cause hardware DMA parse errors)
  and oversized non-TSO packets.
- In gve_tx_validate_descs():
  * Reject zero-length mbufs (nb_descs == 0 or pkt_len == 0).
  * Enforce packet size in [17, 9728] for non-TSO packets alongside
    the 10-descriptor limit (nb_descs <= GVE_TX_MAX_DATA_DESCS).
  * Reject split transport headers across mbuf segments
    (data_len < headlen).
  * Reject TSO packets missing an L4 header (l4_len == 0).
  * Reject TSO packets without payload (pkt_len <= headlen).
  * Enforce MSS limits [88, 9728] and header length limit <= 255.

Fixes: 403c671a46b6 ("net/gve: support TSO in DQO RDA")
Cc: stable@dpdk.org

Acked-by: Joshua Washington <joshwash@google.com>
Signed-off-by: Tathagat Priyadarshi <tathagat.dpdk@gmail.com>
---
 drivers/net/gve/base/gve_desc_dqo.h |  3 ++
 drivers/net/gve/gve_tx_dqo.c        | 44 ++++++++++++++++++++++++-----
 2 files changed, 40 insertions(+), 7 deletions(-)

diff --git a/drivers/net/gve/base/gve_desc_dqo.h b/drivers/net/gve/base/gve_desc_dqo.h
index c1534959c2..1740f69d73 100644
--- a/drivers/net/gve/base/gve_desc_dqo.h
+++ b/drivers/net/gve/base/gve_desc_dqo.h
@@ -12,6 +12,9 @@
 
 #define GVE_TX_MAX_HDR_SIZE_DQO 255
 #define GVE_TX_MIN_TSO_MSS_DQO 88
+#define GVE_TX_MAX_TSO_MSS_DQO 9728
+#define GVE_TX_MIN_PKT_SIZE_DQO 17
+#define GVE_TX_MAX_PKT_SIZE_DQO 9728
 
 /* Basic TX descriptor (DTYPE 0x0C) */
 struct gve_tx_pkt_desc_dqo {
diff --git a/drivers/net/gve/gve_tx_dqo.c b/drivers/net/gve/gve_tx_dqo.c
index 9210b02eb1..03ac3bd5dd 100644
--- a/drivers/net/gve/gve_tx_dqo.c
+++ b/drivers/net/gve/gve_tx_dqo.c
@@ -134,12 +134,43 @@ gve_tx_clean_descs_dqo(struct gve_tx_queue *txq, uint16_t nb_descs) {
 static inline bool
 gve_tx_validate_descs(struct rte_mbuf *tx_pkt, uint16_t nb_descs, bool is_tso)
 {
-	if (!is_tso)
+	int tso_segsz, seg_remain, mbuf_remain;
+	int num_descs, seg_offset, mbuf_len;
+	int headlen;
+
+	/* Reject zero-length packet */
+	if (unlikely(nb_descs == 0 || tx_pkt->pkt_len == 0))
+		return false;
+
+	if (!is_tso) {
+		/* Reject packets that violate hardware length constraints */
+		if (unlikely(tx_pkt->pkt_len < GVE_TX_MIN_PKT_SIZE_DQO ||
+			     tx_pkt->pkt_len > GVE_TX_MAX_PKT_SIZE_DQO))
+			return false;
 		return nb_descs <= GVE_TX_MAX_DATA_DESCS;
+	}
 
-	int tso_segsz = tx_pkt->tso_segsz;
-	int num_descs, seg_offset, mbuf_len;
-	int headlen = tx_pkt->l2_len + tx_pkt->l3_len + tx_pkt->l4_len;
+	headlen = tx_pkt->l2_len + tx_pkt->l3_len + tx_pkt->l4_len;
+
+	/* Reject split transport header */
+	if (unlikely(tx_pkt->data_len < headlen))
+		return false;
+
+	/* Reject missing TCP header on TSO */
+	if (unlikely(tx_pkt->l4_len == 0))
+		return false;
+
+	/* Reject TSO packet without payload */
+	if (unlikely(tx_pkt->pkt_len <= (uint32_t)headlen))
+		return false;
+
+	/* Enforce GVE hardware MSS and header limits */
+	if (unlikely(tx_pkt->tso_segsz < GVE_TX_MIN_TSO_MSS_DQO ||
+		     tx_pkt->tso_segsz > GVE_TX_MAX_TSO_MSS_DQO ||
+		     headlen > GVE_TX_MAX_HDR_SIZE_DQO))
+		return false;
+
+	tso_segsz = tx_pkt->tso_segsz;
 
 	/* Headers will be split into their own buffer. */
 	num_descs = 1;
@@ -150,7 +181,7 @@ gve_tx_validate_descs(struct rte_mbuf *tx_pkt, uint16_t nb_descs, bool is_tso)
 		if (!mbuf_len)
 			goto next_mbuf;
 
-		int seg_remain = tso_segsz - seg_offset;
+		seg_remain = tso_segsz - seg_offset;
 		if (num_descs == GVE_TX_MAX_DATA_DESCS && seg_remain)
 			return false;
 
@@ -162,7 +193,7 @@ gve_tx_validate_descs(struct rte_mbuf *tx_pkt, uint16_t nb_descs, bool is_tso)
 			 * is not at risk of defying this limitation.
 			 * Thus, such segments are ignored.
 			 */
-			int mbuf_remain = tx_pkt->data_len % GVE_TX_MAX_BUF_SIZE_DQO;
+			mbuf_remain = tx_pkt->data_len % GVE_TX_MAX_BUF_SIZE_DQO;
 
 			/* For each TSO segment, HW will prepend
 			 * headers. The remaining bytes of this mbuf
@@ -184,7 +215,6 @@ gve_tx_validate_descs(struct rte_mbuf *tx_pkt, uint16_t nb_descs, bool is_tso)
 			mbuf_len = tx_pkt->data_len;
 	}
 
-
 	return true;
 }
 
-- 
2.43.0

Re: [PATCH 2/4] net/gve: fix DQO TX queue stall on multi-seg and TSO

[Joshua Washington]

On Mon, Oct 12, 2026 at 3:50 AM Tathagat Priyadarshi
<tathagat.dpdk@gmail.com> wrote:
>
> The DPDK GVE driver previously tracked transmitted packet count (re_cnt)
> at the end of each burst rather than tracking descriptor intervals per
> packet against GVE_TX_MIN_RE_INTERVAL (32).
>
> In multi-segment and TSO transmissions, each packet consumes multiple
> descriptors. The descriptor ring could exhaust before 32 packets were
> sent, resulting in zero descriptors requesting report_event and
> preventing the hardware DMA engine from returning GVE_COMPL_TYPE_DQO_DESC
> completions. Without these completions, txq->nb_free was never
> replenished, permanently stalling the TX queue.
>
> Align with the Linux kernel driver (gve_tx_update_tail()) by tracking
> descriptor indices with last_re_idx:
> - Replace re_cnt with last_re_idx in struct gve_tx_queue.
> - Reset last_re_idx = 0 in gve_reset_tx_ring_state_dqo().
> - On each packet transmission, check descriptor interval
>   (last_desc_idx - last_re_idx) & mask >= GVE_TX_MIN_RE_INTERVAL.
>   When met, set report_event on the packet's last descriptor and
>   update last_re_idx.
>
> Fixes: 4022f9999f56 ("net/gve: support basic Tx data path for DQO")
> Fixes: 403c671a46b6 ("net/gve: support TSO in DQO RDA")
> Cc: stable@dpdk.org
>
> Acked-by: Joshua Washington <joshwash@google.com>
> Signed-off-by: Tathagat Priyadarshi <tathagat.dpdk@gmail.com>
> ---
>  drivers/net/gve/gve_ethdev.h |  2 +-
>  drivers/net/gve/gve_tx_dqo.c | 18 ++++++++++--------
>  2 files changed, 11 insertions(+), 9 deletions(-)
>
> diff --git a/drivers/net/gve/gve_ethdev.h b/drivers/net/gve/gve_ethdev.h
> index c9a176ff17..2033f49444 100644
> --- a/drivers/net/gve/gve_ethdev.h
> +++ b/drivers/net/gve/gve_ethdev.h
> @@ -193,8 +193,8 @@ struct gve_tx_queue {
>         uint16_t sw_size;
>         uint8_t cur_gen_bit;
>         uint32_t last_desc_cleaned;
> +       uint32_t last_re_idx;
>         void **txqs;
> -       uint16_t re_cnt;
>
>         /* Only valid for DQO_RDA queue format */
>         struct gve_tx_queue *complq;
> diff --git a/drivers/net/gve/gve_tx_dqo.c b/drivers/net/gve/gve_tx_dqo.c
> index c864d722d9..a8382ff3c4 100644
> --- a/drivers/net/gve/gve_tx_dqo.c
> +++ b/drivers/net/gve/gve_tx_dqo.c
> @@ -264,6 +264,8 @@ gve_tx_burst_dqo(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
>         uint16_t nb_descs;
>         bool csum, tso;
>         uint16_t tx_id;
> +       uint32_t last_desc_idx;
> +       uint32_t last_re_interval;

Minor nit: in this driver, the common practice is to follow reverse
Christmas tree declaration ordering where practical.

>
>         pkts = txq->pkt_ring_dqo;
>         txr = txq->tx_ring;
> @@ -331,20 +333,19 @@ gve_tx_burst_dqo(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
>                 txd = &txr[(tx_id - 1) & mask];
>                 txd->pkt.end_of_packet = 1;
>
> +               last_desc_idx = (tx_id - 1) & mask;
> +               last_re_interval = (last_desc_idx - txq->last_re_idx) & mask;
> +               if (unlikely(last_re_interval >= GVE_TX_MIN_RE_INTERVAL)) {
> +                       txd->pkt.report_event = true;
> +                       txq->last_re_idx = last_desc_idx;

Should we write the tail here? Allowing the device to start processing
packets right away instead of only at the end of TX might save some
failed bursts if dealing with a bursts of many-descriptor TSO packets.
The MMIO writes would also be amortized over RE_INTERVAL descriptors.

> +               }
> +
>                 txq->nb_free -= nb_descs;
>                 txq->nb_used += nb_descs;
>         }
>
>         /* update the tail pointer if any packets were processed */
>         if (nb_tx > 0) {
> -               /* Request a descriptor completion on the last descriptor */
> -               txq->re_cnt += nb_tx;
> -               if (txq->re_cnt >= GVE_TX_MIN_RE_INTERVAL) {
> -                       txd = &txr[(tx_id - 1) & mask];
> -                       txd->pkt.report_event = true;
> -                       txq->re_cnt = 0;
> -               }
> -
>                 rte_write32(tx_id, txq->qtx_tail);
>                 txq->tx_tail = tx_id;
>
> @@ -445,6 +446,7 @@ gve_reset_tx_ring_state_dqo(struct gve_tx_queue *txq)
>         txq->nb_used = 0;
>
>         txq->last_desc_cleaned = 0;
> +       txq->last_re_idx = 0;
>         txq->nb_free = txq->nb_tx_desc - 1;
>
>         txq->complq_tail = 0;
> --
> 2.43.0
>

Re: [PATCH 4/4] net/gve: add packet and TSO hardware constraints for DQO

[Joshua Washington]

On Mon, Oct 12, 2026 at 3:50 AM Tathagat Priyadarshi
<tathagat.dpdk@gmail.com> wrote:
>
> The GVNIC DQO hardware restricts each packet (or MTU segment in TSO)
> to at most 10 data descriptors (GVE_TX_MAX_DATA_DESCS), beyond which
> the hardware drops traffic. In addition to this 10-descriptor limit,
> the hardware imposes strict packet sizing and header constraints.
>
> Enforce these hardware packet and TSO validation constraints in DQO
> RDA mode within gve_tx_validate_descs() matching the upstream Linux
> kernel driver:
> - Define GVE_TX_MAX_TSO_MSS_DQO (9728 bytes) to avoid 14-bit MSS
>   field overflow in struct gve_tx_tso_context_desc_dqo.
> - Define GVE_TX_MIN_PKT_SIZE_DQO (17 bytes) and GVE_TX_MAX_PKT_SIZE_DQO
>   (9728 bytes) to reject runts (which cause hardware DMA parse errors)
>   and oversized non-TSO packets.
> - In gve_tx_validate_descs():
>   * Reject zero-length mbufs (nb_descs == 0 or pkt_len == 0).
>   * Enforce packet size in [17, 9728] for non-TSO packets alongside
>     the 10-descriptor limit (nb_descs <= GVE_TX_MAX_DATA_DESCS).
>   * Reject split transport headers across mbuf segments
>     (data_len < headlen).
>   * Reject TSO packets missing an L4 header (l4_len == 0).
>   * Reject TSO packets without payload (pkt_len <= headlen).
>   * Enforce MSS limits [88, 9728] and header length limit <= 255.
>
> Fixes: 403c671a46b6 ("net/gve: support TSO in DQO RDA")
> Cc: stable@dpdk.org
>
> Acked-by: Joshua Washington <joshwash@google.com>
> Signed-off-by: Tathagat Priyadarshi <tathagat.dpdk@gmail.com>
> ---
>  drivers/net/gve/base/gve_desc_dqo.h |  3 ++
>  drivers/net/gve/gve_tx_dqo.c        | 44 ++++++++++++++++++++++++-----
>  2 files changed, 40 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/net/gve/base/gve_desc_dqo.h b/drivers/net/gve/base/gve_desc_dqo.h
> index c1534959c2..1740f69d73 100644
> --- a/drivers/net/gve/base/gve_desc_dqo.h
> +++ b/drivers/net/gve/base/gve_desc_dqo.h
> @@ -12,6 +12,9 @@
>
>  #define GVE_TX_MAX_HDR_SIZE_DQO 255
>  #define GVE_TX_MIN_TSO_MSS_DQO 88
> +#define GVE_TX_MAX_TSO_MSS_DQO 9728
> +#define GVE_TX_MIN_PKT_SIZE_DQO 17
> +#define GVE_TX_MAX_PKT_SIZE_DQO 9728

GVE_TX_MAX_TSO_MSS_DQO and GVE_TX_MAX_PKT_SIZE_DQO have the same
semantic meaning: they represent the maximum wire packet allowed on
the NIC. Let's opt to use a single value here.

>
>  /* Basic TX descriptor (DTYPE 0x0C) */
>  struct gve_tx_pkt_desc_dqo {
> diff --git a/drivers/net/gve/gve_tx_dqo.c b/drivers/net/gve/gve_tx_dqo.c
> index 9210b02eb1..03ac3bd5dd 100644
> --- a/drivers/net/gve/gve_tx_dqo.c
> +++ b/drivers/net/gve/gve_tx_dqo.c
> @@ -134,12 +134,43 @@ gve_tx_clean_descs_dqo(struct gve_tx_queue *txq, uint16_t nb_descs) {
>  static inline bool
>  gve_tx_validate_descs(struct rte_mbuf *tx_pkt, uint16_t nb_descs, bool is_tso)
>  {
> -       if (!is_tso)
> +       int tso_segsz, seg_remain, mbuf_remain;
> +       int num_descs, seg_offset, mbuf_len;
> +       int headlen;
> +
> +       /* Reject zero-length packet */
> +       if (unlikely(nb_descs == 0 || tx_pkt->pkt_len == 0))
> +               return false;
> +
> +       if (!is_tso) {
> +               /* Reject packets that violate hardware length constraints */
> +               if (unlikely(tx_pkt->pkt_len < GVE_TX_MIN_PKT_SIZE_DQO ||
> +                            tx_pkt->pkt_len > GVE_TX_MAX_PKT_SIZE_DQO))
> +                       return false;
>                 return nb_descs <= GVE_TX_MAX_DATA_DESCS;
> +       }
>
> -       int tso_segsz = tx_pkt->tso_segsz;
> -       int num_descs, seg_offset, mbuf_len;
> -       int headlen = tx_pkt->l2_len + tx_pkt->l3_len + tx_pkt->l4_len;
> +       headlen = tx_pkt->l2_len + tx_pkt->l3_len + tx_pkt->l4_len;
> +
> +       /* Reject split transport header */
> +       if (unlikely(tx_pkt->data_len < headlen))
> +               return false;
> +
> +       /* Reject missing TCP header on TSO */
> +       if (unlikely(tx_pkt->l4_len == 0))
> +               return false;
> +
> +       /* Reject TSO packet without payload */
> +       if (unlikely(tx_pkt->pkt_len <= (uint32_t)headlen))
> +               return false;
> +
> +       /* Enforce GVE hardware MSS and header limits */
> +       if (unlikely(tx_pkt->tso_segsz < GVE_TX_MIN_TSO_MSS_DQO ||
> +                    tx_pkt->tso_segsz > GVE_TX_MAX_TSO_MSS_DQO ||
> +                    headlen > GVE_TX_MAX_HDR_SIZE_DQO))
> +               return false;
> +
> +       tso_segsz = tx_pkt->tso_segsz;
>
>         /* Headers will be split into their own buffer. */
>         num_descs = 1;
> @@ -150,7 +181,7 @@ gve_tx_validate_descs(struct rte_mbuf *tx_pkt, uint16_t nb_descs, bool is_tso)
>                 if (!mbuf_len)
>                         goto next_mbuf;
>
> -               int seg_remain = tso_segsz - seg_offset;
> +               seg_remain = tso_segsz - seg_offset;

Thanks for fixing the forward declarations but generally, declarations
should happen in the smallest scope possible. Since seg_remain and
mbuf_remain aren't used outside of this loop and branch, respectively,
they should be declared at those scopes. It might also be appropriate
to just declare both at the start of the loop.

>                 if (num_descs == GVE_TX_MAX_DATA_DESCS && seg_remain)
>                         return false;
>
> @@ -162,7 +193,7 @@ gve_tx_validate_descs(struct rte_mbuf *tx_pkt, uint16_t nb_descs, bool is_tso)
>                          * is not at risk of defying this limitation.
>                          * Thus, such segments are ignored.
>                          */
> -                       int mbuf_remain = tx_pkt->data_len % GVE_TX_MAX_BUF_SIZE_DQO;
> +                       mbuf_remain = tx_pkt->data_len % GVE_TX_MAX_BUF_SIZE_DQO;
>
>                         /* For each TSO segment, HW will prepend
>                          * headers. The remaining bytes of this mbuf
> @@ -184,7 +215,6 @@ gve_tx_validate_descs(struct rte_mbuf *tx_pkt, uint16_t nb_descs, bool is_tso)
>                         mbuf_len = tx_pkt->data_len;
>         }
>
> -
>         return true;
>  }
>
> --
> 2.43.0
>

Thanks for sending this!

Re: [PATCH 0/4] net/gve: fix TX queue stalls and sync DQO RDA with Linux driver

[Stephen Hemminger]

On Mon, 12 Oct 2026 10:50:19 +0000
Tathagat Priyadarshi <tathagat.dpdk@gmail.com> wrote:

> This patch series aligns the Google Virtual NIC (gve) DQO RDA transmit path
> with the upstream Linux kernel driver (drivers/net/ethernet/google/gve/) to
> resolve transmit queue stalls and hardware validation issues:
> 
> 1. net/gve: fix DQO descriptor cleaning loop bound
>    Fixes the off-by-one error in gve_tx_clean_descs_dqo()
>    ("while (--nb_descs)" -> "while (nb_descs--)") and avoids stalling
>    the completion queue on unrecognized descriptor types.
> 
> 2. net/gve: fix DQO TX queue stall on multi-seg and TSO
>    Tracks descriptor intervals with last_re_idx per packet against
>    GVE_TX_MIN_RE_INTERVAL (32) instead of counting packets at burst end,
>    preventing descriptor ring starvation and queue deadlocks under
>    multi-segment and TSO traffic.
> 
> 3. net/gve: add cacheline overlap margin to DQO TX ring
>    Enforces GVE_TX_MIN_DESC_PREVENT_CACHE_OVERLAP (4 descriptors) margin
>    in ring space checks, avoiding CPU and device DMA cacheline collisions.
> 
> 4. net/gve: add packet and TSO hardware constraints for DQO
>    Augments gve_tx_validate_descs() (which enforces the 10-descriptor
>    limit per packet or MTU segment) with the remaining hardware length
>    constraints: [17, 9728] bytes for non-TSO packets, maximum MSS limit
>    (9728 bytes) to avoid 14-bit field wrap-around, and validates header
>    contiguity and payload presence on TSO packets.
> 
> Tathagat Priyadarshi (4):
>   net/gve: fix DQO descriptor cleaning loop bound
>   net/gve: fix DQO TX queue stall on multi-seg and TSO
>   net/gve: add cacheline overlap margin to DQO TX ring
>   net/gve: add packet and TSO hardware constraints for DQO
> 
>  drivers/net/gve/base/gve_desc_dqo.h |  3 ++
>  drivers/net/gve/gve_ethdev.h        |  2 +-
>  drivers/net/gve/gve_tx_dqo.c        | 80 +++++++++++++++++++++--------
>  3 files changed, 62 insertions(+), 23 deletions(-)

The problem with the series is in the last patch.
The way tx_burst works the driver has to consume packets that cause
errors.

Subject: Re: [PATCH 0/4] net/gve: DQO Tx fixes

Applies cleanly to main; every commit builds with -Dwerror=true.
Patches 1-3 match the Linux gve driver (unknown completion types are
skipped, report_event spaced by descriptor index, 4 descriptor
cacheline margin).

Errors

[PATCH 4/4] net/gve: add packet and TSO hardware constraints for DQO

1. An invalid packet stalls the queue instead of being dropped.

       if (!gve_tx_validate_descs(tx_pkt, nb_descs, tso)) {
               txq->stats.too_many_descs++;
               break;
       }

   tx_burst returns short, so the application gets the bad mbuf back
   and retries it at the head of the next burst. It is rejected again,
   and nothing behind it is ever sent. The break was already there for
   the descriptor limit; this patch extends it to runts, oversize
   frames, bad MSS, split headers and zero length, all per-packet
   errors. A short return should mean only ring full. Linux frees the
   skb and returns NETDEV_TX_OK for the same checks.

   Validate before the ring space check, then free and continue:

       nb_descs = gve_tx_pkt_nb_data_descs(tx_pkt);
       nb_descs += tso;

       /* Drop packet if it doesn't adhere to hardware limits. */
       if (unlikely(!gve_tx_validate_descs(tx_pkt, nb_descs, tso))) {
               txq->stats.too_many_descs++;
               rte_pktmbuf_free(tx_pkt);
               nb_drop++;
               continue;
       }

       req_descs = nb_descs + GVE_TX_MIN_DESC_PREVENT_CACHE_OVERLAP;
       ...

   and use nb_tx - nb_drop for stats.packets. too_many_descs now counts
   more than descriptor overruns; renaming it, or adding a drop
   counter, would make the xstat less misleading. The rte_mbuf_check()
   failure just above has the same break and needs the same fix.

Info

[PATCH 4/4] net/gve: add packet and TSO hardware constraints for DQO

2. The non-TSO length check fixes the original DQO Tx path, not TSO
   support. Add:

   Fixes: 4022f9999f56 ("net/gve: support basic Tx data path for DQO")

[PATCH v2 0/4] net/gve: fix Tx queue stalls and sync DQO RDA with Linux driver

[Tathagat Priyadarshi]

This patch series aligns the Google Virtual NIC (gve) DQO RDA transmit path
with the upstream Linux kernel driver (drivers/net/ethernet/google/gve/) to
resolve transmit queue stalls and hardware validation issues:

1. net/gve: fix DQO descriptor cleaning loop bound
   Fixes the off-by-one error in gve_tx_clean_descs_dqo()
   ("while (--nb_descs)" -> "while (nb_descs--)") and avoids stalling
   the completion queue on unrecognized descriptor types.

2. net/gve: fix DQO Tx queue stall on multi-seg and TSO
   Tracks descriptor intervals with last_re_idx per packet against
   GVE_TX_MIN_RE_INTERVAL (32) instead of counting packets at burst end,
   preventing descriptor ring starvation and queue deadlocks under
   multi-segment and TSO traffic.

3. net/gve: add cacheline overlap margin to DQO Tx ring
   Enforces GVE_TX_MIN_DESC_PREVENT_CACHE_OVERLAP (4 descriptors) margin
   in ring space checks, avoiding CPU and device DMA cacheline collisions.

4. net/gve: add packet and TSO hardware constraints for DQO
   Augments gve_tx_validate_descs() (which enforces the 10-descriptor
   limit per packet or MTU segment) with the remaining hardware length
   constraints: [17, 9728] bytes for non-TSO packets, maximum MSS limit
   (9728 bytes), TSO total length limit (262143 bytes), and validates
   header contiguity and payload presence on TSO packets. Invalid packets
   are dropped instead of stalling the queue.

v2:
- Patch 1/4, 3/4: No changes.
- Patch 2/4:
  * Write the Tx tail when report_event is set so the device can start
    fetching descriptors before the end of the burst
    (per Joshua Washington).
  * Pop the completion tag before writing the TSO context descriptor.
  * Use reverse Christmas tree declaration ordering
    (per Joshua Washington).
- Patch 4/4:
  * Drop (free) invalid mbufs and continue the burst instead of
    returning short, which caused head-of-line blocking
    (per Stephen Hemminger).
  * Validate packets before the ring space check.
  * Exclude dropped packets from stats.packets.
  * Rename too_many_descs to dropped_pkts since it now counts all
    dropped packets (per Stephen Hemminger).
  * Use a single GVE_TX_MAX_PKT_SIZE_DQO for the max packet size and
    max MSS (per Joshua Washington).
  * Declare seg_remain and mbuf_remain in the smallest scope
    (per Joshua Washington).
  * Reject packets whose descriptor count can never fit in the Tx ring.
  * Enforce the TSO total length limit (262143 bytes).
  * Add Fixes tag for the original DQO Tx path (per Stephen Hemminger).

Tathagat Priyadarshi (4):
  net/gve: fix DQO descriptor cleaning loop bound
  net/gve: fix DQO Tx queue stall on multi-seg and TSO
  net/gve: add cacheline overlap margin to DQO Tx ring
  net/gve: add packet and TSO hardware constraints for DQO

 drivers/net/gve/base/gve_desc_dqo.h |   3 +
 drivers/net/gve/gve_ethdev.h        |   4 +-
 drivers/net/gve/gve_tx_dqo.c        | 152 ++++++++++++++++++++--------
 3 files changed, 115 insertions(+), 44 deletions(-)

-- 
2.43.0

[PATCH v2 1/4] net/gve: fix DQO descriptor cleaning loop bound

[Tathagat Priyadarshi]

In gve_tx_clean_descs_dqo(), the loop condition "while (--nb_descs)"
causes an off-by-one error:
- When nb_descs == 1, --nb_descs decrements to 0 on the first check,
  executing 0 iterations instead of 1.
- When nb_descs == 0, --nb_descs underflows uint16_t to 65535,
  executing 65535 iterations.

Change the condition to "while (nb_descs--)" so that exactly nb_descs
iterations are executed for any input value.

Additionally, in gve_tx_clean_dqo(), handle unrecognized completion
types by breaking from the switch rather than returning immediately,
allowing complq_tail to advance past unknown descriptors rather than
stalling the completion ring at the same entry.

Fixes: 4022f9999f56 ("net/gve: support basic Tx data path for DQO")
Cc: stable@dpdk.org

Signed-off-by: Tathagat Priyadarshi <tathagat.dpdk@gmail.com>
Acked-by: Joshua Washington <joshwash@google.com>
---
 drivers/net/gve/gve_tx_dqo.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/net/gve/gve_tx_dqo.c b/drivers/net/gve/gve_tx_dqo.c
index 5c03d3d1a2..c864d722d9 100644
--- a/drivers/net/gve/gve_tx_dqo.c
+++ b/drivers/net/gve/gve_tx_dqo.c
@@ -108,8 +108,8 @@ gve_tx_clean_dqo(struct gve_tx_queue *txq)
 		PMD_DRV_DP_LOG(DEBUG, "GVE_COMPL_TYPE_DQO_MISS ignored !!!");
 		break;
 	default:
-		PMD_DRV_DP_LOG(ERR, "unknown completion type.");
-		return;
+		PMD_DRV_DP_LOG(ERR, "unknown completion type %u", compl_desc->type);
+		break;
 	}
 
 	next++;
@@ -122,7 +122,7 @@ gve_tx_clean_dqo(struct gve_tx_queue *txq)
 
 static inline void
 gve_tx_clean_descs_dqo(struct gve_tx_queue *txq, uint16_t nb_descs) {
-	while (--nb_descs)
+	while (nb_descs--)
 		gve_tx_clean_dqo(txq);
 }
 
-- 
2.43.0

[PATCH v2 2/4] net/gve: fix DQO Tx queue stall on multi-seg and TSO

[Tathagat Priyadarshi]

The DPDK GVE driver previously tracked transmitted packet count (re_cnt)
at the end of each burst rather than tracking descriptor intervals per
packet against GVE_TX_MIN_RE_INTERVAL (32).

In multi-segment and TSO transmissions, each packet consumes multiple
descriptors. The descriptor ring could exhaust before 32 packets were
sent, resulting in zero descriptors requesting report_event and
preventing the hardware DMA engine from returning GVE_COMPL_TYPE_DQO_DESC
completions. Without these completions, txq->nb_free was never
replenished, permanently stalling the TX queue.

Align with the Linux kernel driver (gve_tx_update_tail()) by tracking
descriptor indices with last_re_idx:
- Replace re_cnt with last_re_idx in struct gve_tx_queue.
- Reset last_re_idx = 0 in gve_reset_tx_ring_state_dqo().
- On each packet transmission, check descriptor interval
  (last_desc_idx - last_re_idx) & mask >= GVE_TX_MIN_RE_INTERVAL.
  When met, set report_event on the packet's last descriptor and
  update last_re_idx.
- Ring the transmit doorbell immediately upon setting report_event to
  kick hardware processing early and reduce device fetch latency during
  bursts of multi-descriptor TSO packets, amortizing MMIO writes over
  RE_INTERVAL descriptors.
- Defensively pop completion tags before writing TSO context descriptors
  to avoid partial descriptor ring updates if completion tags are
  exhausted.

Fixes: 4022f9999f56 ("net/gve: support basic Tx data path for DQO")
Fixes: 403c671a46b6 ("net/gve: support TSO in DQO RDA")
Cc: stable@dpdk.org

Signed-off-by: Tathagat Priyadarshi <tathagat.dpdk@gmail.com>
---
 drivers/net/gve/gve_ethdev.h |  2 +-
 drivers/net/gve/gve_tx_dqo.c | 46 ++++++++++++++++++++----------------
 2 files changed, 27 insertions(+), 21 deletions(-)

diff --git a/drivers/net/gve/gve_ethdev.h b/drivers/net/gve/gve_ethdev.h
index c9a176ff17..2033f49444 100644
--- a/drivers/net/gve/gve_ethdev.h
+++ b/drivers/net/gve/gve_ethdev.h
@@ -193,8 +193,8 @@ struct gve_tx_queue {
 	uint16_t sw_size;
 	uint8_t cur_gen_bit;
 	uint32_t last_desc_cleaned;
+	uint32_t last_re_idx;
 	void **txqs;
-	uint16_t re_cnt;
 
 	/* Only valid for DQO_RDA queue format */
 	struct gve_tx_queue *complq;
diff --git a/drivers/net/gve/gve_tx_dqo.c b/drivers/net/gve/gve_tx_dqo.c
index c864d722d9..0839179d7f 100644
--- a/drivers/net/gve/gve_tx_dqo.c
+++ b/drivers/net/gve/gve_tx_dqo.c
@@ -250,21 +250,23 @@ gve_fill_tx_pkt_desc(struct gve_tx_queue *txq, uint16_t *tx_id,
 uint16_t
 gve_tx_burst_dqo(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 {
+	volatile union gve_tx_desc_dqo *txr, *txd;
 	struct gve_tx_queue *txq = tx_queue;
-	volatile union gve_tx_desc_dqo *txr;
-	volatile union gve_tx_desc_dqo *txd;
-	uint16_t mask = txq->nb_tx_desc - 1;
-	struct gve_tx_pkt *pkts;
+	uint32_t last_re_interval;
 	struct rte_mbuf *tx_pkt;
-	uint16_t compl_tag;
-	const char *reason;
+	struct gve_tx_pkt *pkts;
+	uint32_t last_desc_idx;
 	uint16_t nb_tx = 0;
 	uint64_t bytes = 0;
+	const char *reason;
+	uint16_t compl_tag;
 	uint64_t ol_flags;
 	uint16_t nb_descs;
-	bool csum, tso;
 	uint16_t tx_id;
+	bool csum, tso;
+	uint16_t mask;
 
+	mask = txq->nb_tx_desc - 1;
 	pkts = txq->pkt_ring_dqo;
 	txr = txq->tx_ring;
 
@@ -306,15 +308,15 @@ gve_tx_burst_dqo(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 			break;
 		}
 
+		if (!gve_free_compl_tags_pop(txq, &compl_tag))
+			break;
+
 		if (tso) {
 			txd = &txr[tx_id];
 			gve_tx_fill_seg_desc_dqo(txd, tx_pkt);
 			tx_id = (tx_id + 1) & mask;
 		}
 
-		if (!gve_free_compl_tags_pop(txq, &compl_tag))
-			break;
-
 		pkts[compl_tag].mbuf = tx_pkt;
 		while (tx_pkt) {
 			/* Skip writing descriptors if mbuf has no data. */
@@ -331,23 +333,26 @@ gve_tx_burst_dqo(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 		txd = &txr[(tx_id - 1) & mask];
 		txd->pkt.end_of_packet = 1;
 
+		last_desc_idx = (tx_id - 1) & mask;
+		last_re_interval = (last_desc_idx - txq->last_re_idx) & mask;
+		if (unlikely(last_re_interval >= GVE_TX_MIN_RE_INTERVAL)) {
+			txd->pkt.report_event = true;
+			txq->last_re_idx = last_desc_idx;
+			rte_write32(tx_id, txq->qtx_tail);
+			txq->tx_tail = tx_id;
+		}
+
 		txq->nb_free -= nb_descs;
 		txq->nb_used += nb_descs;
 	}
 
-	/* update the tail pointer if any packets were processed */
-	if (nb_tx > 0) {
-		/* Request a descriptor completion on the last descriptor */
-		txq->re_cnt += nb_tx;
-		if (txq->re_cnt >= GVE_TX_MIN_RE_INTERVAL) {
-			txd = &txr[(tx_id - 1) & mask];
-			txd->pkt.report_event = true;
-			txq->re_cnt = 0;
-		}
-
+	/* update the tail pointer if any uncommitted descriptors remain */
+	if (txq->tx_tail != tx_id) {
 		rte_write32(tx_id, txq->qtx_tail);
 		txq->tx_tail = tx_id;
+	}
 
+	if (nb_tx > 0) {
 		txq->stats.packets += nb_tx;
 		txq->stats.bytes += bytes;
 		txq->stats.errors += nb_pkts - nb_tx;
@@ -445,6 +450,7 @@ gve_reset_tx_ring_state_dqo(struct gve_tx_queue *txq)
 	txq->nb_used = 0;
 
 	txq->last_desc_cleaned = 0;
+	txq->last_re_idx = 0;
 	txq->nb_free = txq->nb_tx_desc - 1;
 
 	txq->complq_tail = 0;
-- 
2.43.0

[PATCH v2 3/4] net/gve: add cacheline overlap margin to DQO Tx ring

[Tathagat Priyadarshi]

The GVNIC DQO hardware DMA engine prefetches descriptors in cacheline
units (64 bytes = 4 descriptors of 16 bytes each). When producer tail
and consumer head indices share a cacheline, CPU and device DMA memory
collisions can occur.

Align with the Linux driver (gve_has_tx_slots_available()) by requiring
GVE_TX_MIN_DESC_PREVENT_CACHE_OVERLAP (4 descriptors) headroom when
checking descriptor availability before enqueueing packets.

Fixes: 4022f9999f56 ("net/gve: support basic Tx data path for DQO")
Cc: stable@dpdk.org

Signed-off-by: Tathagat Priyadarshi <tathagat.dpdk@gmail.com>
Acked-by: Joshua Washington <joshwash@google.com>
---
 drivers/net/gve/gve_tx_dqo.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/drivers/net/gve/gve_tx_dqo.c b/drivers/net/gve/gve_tx_dqo.c
index 0839179d7f..eee1f10df1 100644
--- a/drivers/net/gve/gve_tx_dqo.c
+++ b/drivers/net/gve/gve_tx_dqo.c
@@ -260,6 +260,7 @@ gve_tx_burst_dqo(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 	uint64_t bytes = 0;
 	const char *reason;
 	uint16_t compl_tag;
+	uint16_t req_descs;
 	uint64_t ol_flags;
 	uint16_t nb_descs;
 	uint16_t tx_id;
@@ -291,14 +292,17 @@ gve_tx_burst_dqo(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 
 		nb_descs = gve_tx_pkt_nb_data_descs(tx_pkt);
 		nb_descs += tso;
+		req_descs = nb_descs + GVE_TX_MIN_DESC_PREVENT_CACHE_OVERLAP;
 
-		/* Clean if there aren't enough descriptors to send the packet. */
-		if (unlikely(!gve_can_tx(txq, nb_descs, 1))) {
+		/* Clean if there aren't enough descriptors to send the packet
+		 * while maintaining the cacheline overlap margin.
+		 */
+		if (unlikely(!gve_can_tx(txq, req_descs, 1))) {
 			int nb_to_clean = RTE_MAX(DQO_TX_MULTIPLIER * txq->rs_thresh,
-						  nb_descs);
+						  (int)req_descs);
 
 			gve_tx_clean_descs_dqo(txq, nb_to_clean);
-			if (!gve_can_tx(txq, nb_descs, 1))
+			if (!gve_can_tx(txq, req_descs, 1))
 				break;
 		}
 
-- 
2.43.0

[PATCH v2 4/4] net/gve: add packet and TSO hardware constraints for DQO

[Tathagat Priyadarshi]

The GVNIC DQO hardware restricts each packet (or MTU segment in TSO)
to at most 10 data descriptors (GVE_TX_MAX_DATA_DESCS), beyond which
the hardware drops traffic. In addition to this 10-descriptor limit,
the hardware imposes strict packet sizing and header constraints.

Previously, failing the descriptor limit or rte_mbuf_check() broke
from the Tx burst loop without consuming the invalid mbuf. This caused
tx_burst to return a short count, leading the application to retry the
same rejected packet indefinitely and causing Head-of-Line (HOL)
queue stalls.

Enforce hardware validation and prevent HOL blocking in DPDK:
- Validate packet and hardware constraints prior to ring space checks.
- When packet validation or rte_mbuf_check() fails, drop and free the
  invalid mbuf with rte_pktmbuf_free(), update stats, and continue the
  burst to prevent Head-of-Line (HOL) blocking.
- Exclude dropped packets from txq->stats.packets while returning the
  total consumed count to the caller.
- Rename too_many_descs in struct gve_tx_stats to dropped_pkts to reflect
  all validation and malformed mbuf drop events.
- Reject packets whose descriptor count exceeds the maximum capacity of
  the ring (accounting for reserved slots, cacheline overlap margin, and
  unreported descriptors), preventing permanent queue stalls.
- Define GVE_TX_MIN_PKT_SIZE_DQO (17 bytes) and GVE_TX_MAX_PKT_SIZE_DQO
  (9728 bytes) to enforce wire packet boundaries for non-TSO packets
  and TSO MSS.
- Define GVE_TX_MAX_TSO_LEN_DQO (262143 bytes) to enforce the hardware
  L4 payload limit in the 24-bit tso_total_len field.
- In gve_tx_validate_descs():
  * Enforce packet size in [17, 9728] for non-TSO packets alongside
    the 10-descriptor limit (nb_descs <= GVE_TX_MAX_DATA_DESCS).
  * Reject TSO packets missing an L4 header (l4_len == 0).
  * Reject packet with headers (L2-L4) not contiguous in first segment.
  * Reject TSO packets without payload (pkt_len <= headlen).
  * Enforce TSO total payload length limit (pkt_len - headlen <= 262143).
  * Enforce MSS limits [88, 9728] and header length limit <= 255.

Fixes: 4022f9999f56 ("net/gve: support basic Tx data path for DQO")
Fixes: f4a70b449bfa ("net/gve: add DQO Tx descriptor limit")
Fixes: 403c671a46b6 ("net/gve: support TSO in DQO RDA")
Cc: stable@dpdk.org

Signed-off-by: Tathagat Priyadarshi <tathagat.dpdk@gmail.com>
---
 drivers/net/gve/base/gve_desc_dqo.h |  3 +
 drivers/net/gve/gve_ethdev.h        |  2 +-
 drivers/net/gve/gve_tx_dqo.c        | 92 +++++++++++++++++++++++------
 3 files changed, 79 insertions(+), 18 deletions(-)

diff --git a/drivers/net/gve/base/gve_desc_dqo.h b/drivers/net/gve/base/gve_desc_dqo.h
index c1534959c2..53e394f01c 100644
--- a/drivers/net/gve/base/gve_desc_dqo.h
+++ b/drivers/net/gve/base/gve_desc_dqo.h
@@ -12,6 +12,9 @@
 
 #define GVE_TX_MAX_HDR_SIZE_DQO 255
 #define GVE_TX_MIN_TSO_MSS_DQO 88
+#define GVE_TX_MIN_PKT_SIZE_DQO 17
+#define GVE_TX_MAX_PKT_SIZE_DQO 9728
+#define GVE_TX_MAX_TSO_LEN_DQO ((256 * 1024) - 1)
 
 /* Basic TX descriptor (DTYPE 0x0C) */
 struct gve_tx_pkt_desc_dqo {
diff --git a/drivers/net/gve/gve_ethdev.h b/drivers/net/gve/gve_ethdev.h
index 2033f49444..d6d71c31bb 100644
--- a/drivers/net/gve/gve_ethdev.h
+++ b/drivers/net/gve/gve_ethdev.h
@@ -113,7 +113,7 @@ struct gve_tx_stats {
 	uint64_t packets;
 	uint64_t bytes;
 	uint64_t errors;
-	uint64_t too_many_descs;
+	uint64_t dropped_pkts;
 };
 
 struct gve_rx_stats {
diff --git a/drivers/net/gve/gve_tx_dqo.c b/drivers/net/gve/gve_tx_dqo.c
index eee1f10df1..94ea426961 100644
--- a/drivers/net/gve/gve_tx_dqo.c
+++ b/drivers/net/gve/gve_tx_dqo.c
@@ -126,20 +126,67 @@ gve_tx_clean_descs_dqo(struct gve_tx_queue *txq, uint16_t nb_descs) {
 		gve_tx_clean_dqo(txq);
 }
 
+/* Max descriptors a single packet may use. A larger packet can never pass
+ * gve_can_tx(): one slot is always reserved, the cacheline overlap margin
+ * must stay free, and up to GVE_TX_MIN_RE_INTERVAL - 1 descriptors posted
+ * after the last report_event are not reclaimed until more are posted.
+ */
+static inline int
+gve_tx_max_pkt_descs(const struct gve_tx_queue *txq)
+{
+	return (int)txq->nb_tx_desc - 1 - GVE_TX_MIN_DESC_PREVENT_CACHE_OVERLAP -
+		(GVE_TX_MIN_RE_INTERVAL - 1);
+}
+
 /* GVE expects at most 10 data descriptors per mtu-sized segment. Beyond this,
  * the hardware will assume the driver is malicious and stop transmitting
  * packets altogether. Validate that a packet can be sent to avoid sending
  * posting descriptors for an invalid packet.
  */
 static inline bool
-gve_tx_validate_descs(struct rte_mbuf *tx_pkt, uint16_t nb_descs, bool is_tso)
+gve_tx_validate_descs(const struct gve_tx_queue *txq, struct rte_mbuf *tx_pkt,
+		      uint16_t nb_descs, bool is_tso)
 {
-	if (!is_tso)
+	int num_descs, seg_offset, mbuf_len;
+	int tso_segsz, headlen;
+
+	/* Reject packets that can never fit in the ring */
+	if (unlikely(nb_descs > gve_tx_max_pkt_descs(txq)))
+		return false;
+
+	if (!is_tso) {
+		/* Reject packets that violate hardware length constraints */
+		if (unlikely(tx_pkt->pkt_len < GVE_TX_MIN_PKT_SIZE_DQO ||
+			     tx_pkt->pkt_len > GVE_TX_MAX_PKT_SIZE_DQO))
+			return false;
 		return nb_descs <= GVE_TX_MAX_DATA_DESCS;
+	}
 
-	int tso_segsz = tx_pkt->tso_segsz;
-	int num_descs, seg_offset, mbuf_len;
-	int headlen = tx_pkt->l2_len + tx_pkt->l3_len + tx_pkt->l4_len;
+	headlen = tx_pkt->l2_len + tx_pkt->l3_len + tx_pkt->l4_len;
+
+	/* Reject missing TCP header on TSO */
+	if (unlikely(tx_pkt->l4_len == 0))
+		return false;
+
+	/* Reject packet with headers (L2-L4) not contiguous in first segment */
+	if (unlikely(tx_pkt->data_len < headlen))
+		return false;
+
+	/* Reject TSO packet without payload */
+	if (unlikely(tx_pkt->pkt_len <= (uint32_t)headlen))
+		return false;
+
+	/* Enforce GVE hardware TSO total length limit */
+	if (unlikely(tx_pkt->pkt_len - headlen > GVE_TX_MAX_TSO_LEN_DQO))
+		return false;
+
+	/* Enforce GVE hardware MSS and header limits */
+	if (unlikely(tx_pkt->tso_segsz < GVE_TX_MIN_TSO_MSS_DQO ||
+		     tx_pkt->tso_segsz > GVE_TX_MAX_PKT_SIZE_DQO ||
+		     headlen > GVE_TX_MAX_HDR_SIZE_DQO))
+		return false;
+
+	tso_segsz = tx_pkt->tso_segsz;
 
 	/* Headers will be split into their own buffer. */
 	num_descs = 1;
@@ -147,14 +194,18 @@ gve_tx_validate_descs(struct rte_mbuf *tx_pkt, uint16_t nb_descs, bool is_tso)
 	mbuf_len = tx_pkt->data_len - headlen;
 
 	while (tx_pkt) {
+		int seg_remain;
+
 		if (!mbuf_len)
 			goto next_mbuf;
 
-		int seg_remain = tso_segsz - seg_offset;
+		seg_remain = tso_segsz - seg_offset;
 		if (num_descs == GVE_TX_MAX_DATA_DESCS && seg_remain)
 			return false;
 
 		if (seg_remain < mbuf_len) {
+			int mbuf_remain = tx_pkt->data_len % GVE_TX_MAX_BUF_SIZE_DQO;
+
 			seg_offset = mbuf_len % tso_segsz;
 			/* The MSS is bound from above by 9728B, so a
 			 * single TSO segment in the middle of an mbuf
@@ -162,7 +213,6 @@ gve_tx_validate_descs(struct rte_mbuf *tx_pkt, uint16_t nb_descs, bool is_tso)
 			 * is not at risk of defying this limitation.
 			 * Thus, such segments are ignored.
 			 */
-			int mbuf_remain = tx_pkt->data_len % GVE_TX_MAX_BUF_SIZE_DQO;
 
 			/* For each TSO segment, HW will prepend
 			 * headers. The remaining bytes of this mbuf
@@ -184,7 +234,6 @@ gve_tx_validate_descs(struct rte_mbuf *tx_pkt, uint16_t nb_descs, bool is_tso)
 			mbuf_len = tx_pkt->data_len;
 	}
 
-
 	return true;
 }
 
@@ -256,6 +305,7 @@ gve_tx_burst_dqo(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 	struct rte_mbuf *tx_pkt;
 	struct gve_tx_pkt *pkts;
 	uint32_t last_desc_idx;
+	uint16_t nb_drop = 0;
 	uint16_t nb_tx = 0;
 	uint64_t bytes = 0;
 	const char *reason;
@@ -281,9 +331,13 @@ gve_tx_burst_dqo(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 					       txq->rs_thresh);
 
 
-		if (rte_mbuf_check(tx_pkt, true, &reason)) {
+		if (unlikely(rte_mbuf_check(tx_pkt, true, &reason))) {
 			PMD_DRV_DP_LOG(DEBUG, "Invalid mbuf: %s", reason);
-			break;
+			txq->stats.dropped_pkts++;
+			txq->stats.errors++;
+			rte_pktmbuf_free(tx_pkt);
+			nb_drop++;
+			continue;
 		}
 
 		ol_flags = tx_pkt->ol_flags;
@@ -292,6 +346,16 @@ gve_tx_burst_dqo(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 
 		nb_descs = gve_tx_pkt_nb_data_descs(tx_pkt);
 		nb_descs += tso;
+
+		/* Drop packet if it doesn't adhere to hardware limits. */
+		if (unlikely(!gve_tx_validate_descs(txq, tx_pkt, nb_descs, tso))) {
+			txq->stats.dropped_pkts++;
+			txq->stats.errors++;
+			rte_pktmbuf_free(tx_pkt);
+			nb_drop++;
+			continue;
+		}
+
 		req_descs = nb_descs + GVE_TX_MIN_DESC_PREVENT_CACHE_OVERLAP;
 
 		/* Clean if there aren't enough descriptors to send the packet
@@ -306,12 +370,6 @@ gve_tx_burst_dqo(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 				break;
 		}
 
-		/* Drop packet if it doesn't adhere to hardware limits. */
-		if (!gve_tx_validate_descs(tx_pkt, nb_descs, tso)) {
-			txq->stats.too_many_descs++;
-			break;
-		}
-
 		if (!gve_free_compl_tags_pop(txq, &compl_tag))
 			break;
 
@@ -357,7 +415,7 @@ gve_tx_burst_dqo(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 	}
 
 	if (nb_tx > 0) {
-		txq->stats.packets += nb_tx;
+		txq->stats.packets += (nb_tx - nb_drop);
 		txq->stats.bytes += bytes;
 		txq->stats.errors += nb_pkts - nb_tx;
 	}
-- 
2.43.0

Re: [PATCH v2 0/4] net/gve: fix Tx queue stalls and sync DQO RDA with Linux driver

[Stephen Hemminger]

On Tue, 13 Oct 2026 10:43:44 +0000
Tathagat Priyadarshi <tathagat.dpdk@gmail.com> wrote:

> This patch series aligns the Google Virtual NIC (gve) DQO RDA transmit path
> with the upstream Linux kernel driver (drivers/net/ethernet/google/gve/) to
> resolve transmit queue stalls and hardware validation issues:
> 
> 1. net/gve: fix DQO descriptor cleaning loop bound
>    Fixes the off-by-one error in gve_tx_clean_descs_dqo()
>    ("while (--nb_descs)" -> "while (nb_descs--)") and avoids stalling
>    the completion queue on unrecognized descriptor types.
> 
> 2. net/gve: fix DQO Tx queue stall on multi-seg and TSO
>    Tracks descriptor intervals with last_re_idx per packet against
>    GVE_TX_MIN_RE_INTERVAL (32) instead of counting packets at burst end,
>    preventing descriptor ring starvation and queue deadlocks under
>    multi-segment and TSO traffic.
> 
> 3. net/gve: add cacheline overlap margin to DQO Tx ring
>    Enforces GVE_TX_MIN_DESC_PREVENT_CACHE_OVERLAP (4 descriptors) margin
>    in ring space checks, avoiding CPU and device DMA cacheline collisions.
> 
> 4. net/gve: add packet and TSO hardware constraints for DQO
>    Augments gve_tx_validate_descs() (which enforces the 10-descriptor
>    limit per packet or MTU segment) with the remaining hardware length
>    constraints: [17, 9728] bytes for non-TSO packets, maximum MSS limit
>    (9728 bytes), TSO total length limit (262143 bytes), and validates
>    header contiguity and payload presence on TSO packets. Invalid packets
>    are dropped instead of stalling the queue.


Looks good you may want to address these comments from AI.

Subject: Re: [PATCH v2 0/4] net/gve: DQO Tx fixes

Applies cleanly to main; every commit builds with -Dwerror=true.
v2 drops invalid packets and continues instead of stalling the burst,
pops the completion tag before writing the TSO context descriptor,
and adds the Fixes tags. All three Fixes hashes resolve.

Info

[PATCH v2 2/4] net/gve: fix DQO Tx queue stall on multi-seg and TSO

1. Ringing the doorbell each time report_event is set is a
   performance change, not part of the stall fix. Descriptors posted
   earlier in the burst are reclaimed on the next burst either way,
   and Linux rings once per xmit batch. It is harmless, but for a
   stable backport it is better as a separate patch with numbers. At
   least drop "amortizing MMIO writes": it adds MMIO writes.

[PATCH v2 4/4] net/gve: add packet and TSO hardware constraints for DQO

2. tso_total_len is 24 bits wide; 262143 is the hardware limit (see
   the comment in the Linux gve_prep_tso()), not the field width.
   Reword: "Define GVE_TX_MAX_TSO_LEN_DQO (262143), the hardware limit
   on TSO payload length."

3. dropped_pkts is not in tx_xstats_name_offset[], so nothing reports
   it; too_many_descs had the same problem. Export it:

       { "dropped_pkts", TX_QUEUE_STATS_OFFSET(dropped_pkts) },

   or drop the counter and rely on errors.

4. The commit message lists the diff line by line. Cut it to the
   problem (invalid packets stalled the queue; hardware limits were
   not enforced) and the fix (drop and continue; the limits enforced).

Reviewed-by: Stephen Hemminger <stephen@networkplumber.org>