[PATCH 0/3] net/gve: TSO descriptor count fixes for GVE DQO

[PATCH 0/3] net/gve: TSO descriptor count fixes for GVE DQO

[Joshua Washington]

https://inbox.dpdk.org/dev/20250707231812.1937260-7-joshwash@google.com/
introduced descriptor limit checks for TSO packets in DPDK.
Unfortunately, a couple of bugs have been found in the implementation.

This patch series introduces fixes for those bugs while correcting the
overly restrictive behavior recently introduced mandating that all
headers be in the same buffer.

The final patch in the series introduces tx_pkt_prepare as an option to
ensure that there is a way to get a clear signal of when packets violate
hardware limitations. Before a recently introduced patch, the GVE
behavior was to stop transmitting at the first invalid packet, which
would allow the application to do any necessary linearization or packet
modification before attempting to transmit again. It turns out that that
behavior is considered incorrect, so offer tx_pkt_prepare as an
alternative.

Depends-on: series-39707 ("net/gve: fix DQO Tx queue stalls")

Joshua Washington (3):
  net/gve: fix segment TSO segment offset calculation
  net/gve: account for multi-mbuf headers when validating TSO
  net/gve: introduce Tx packet preparation for DQO format

 drivers/net/gve/base/gve_desc_dqo.h |   1 +
 drivers/net/gve/gve_ethdev.h        |   3 +
 drivers/net/gve/gve_tx_dqo.c        | 104 ++++++++++++++++++++++++----
 3 files changed, 96 insertions(+), 12 deletions(-)

-- 
2.56.0.307.g5fe7bd680c-goog

[PATCH 1/3] net/gve: fix segment TSO segment offset calculation

[Joshua Washington]

The descriptor limit validation for TSO packets was erroneously
computing the TSO segment offset at the end of an mbuf. The first bytes
of any mbuf will be included as part of the previous TSO segment which
has not yet been completed, as indicated by seg_remain. Subtract those
seg_remain bytes from the mbuf length before computing the segment
offset.

Visual representation:

Before:
------------------------------------------------------------------
|    tso_    |    tso_    |    tso_    |    tso_    |    tso_    |
|    segsz   |    segsz   |    segsz   |    segsz   |    segsz   |
------------------------------------------------------------------
|           16K               |            16K               |   |
------------------------------------------------------------------

After:
------------------------------------------------------------------
|rem |    tso_    |    tso_    |    tso_    |    tso_    | seg_  |
|ain |    segsz   |    segsz   |    segsz   |    segsz   |offset |
------------------------------------------------------------------
|           16K               |            16K               |   |
------------------------------------------------------------------
                                                               ^
                                                        mbuf_remain

In this example, the original code would have found that tso_segsz
evenly divides mbuf_len, and that there is no tail to be considered
split across two mbufs. So the number of descriptors it would have found
would be:

  1 (for the headers) +
  0 (no segment offset) +
  0 (mbuf_remain cannot be less than 0)

For a total of one descriptor counted in num_descs for the start of
the current TSO segment.

Meanwhile, the reality would be:

  1 (for the headers) +
  1 (seg_offset != 0) +
  1 (mbuf_remain < seg_offset)

For a total of 3 descriptors counted against the descriptor limit.

This undercount can cause TSO segments to incorrectly cross up to two
extra descriptors more than the hardware limit, leading to the datapath
being disabled.

Fixes: be8f0eb81f98 ("net/gve: fix DQO TSO descriptor limit")
Cc: stable@dpdk.org

Signed-off-by: Joshua Washington <joshwash@google.com>
Reviewed-by: Praveen Kaligineedi <pkaligineedi@google.com>
Reviewed-by: Ankit Garg <nktgrg@google.com>
---
Depends-on: series-39707 ("net/gve: fix DQO Tx queue stalls")
---
 drivers/net/gve/gve_tx_dqo.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/gve/gve_tx_dqo.c b/drivers/net/gve/gve_tx_dqo.c
index 52fcdb285a..3a20109a5d 100644
--- a/drivers/net/gve/gve_tx_dqo.c
+++ b/drivers/net/gve/gve_tx_dqo.c
@@ -191,7 +191,7 @@ gve_tx_validate_descs(const struct gve_tx_queue *txq, struct rte_mbuf *tx_pkt,
 			return false;
 
 		if (seg_remain < mbuf_len) {
-			seg_offset = mbuf_len % tso_segsz;
+			seg_offset = (mbuf_len - seg_remain) % tso_segsz;
 			/* The MSS is bound from above by 9728B, so a
 			 * single TSO segment in the middle of an mbuf
 			 * will be part of at most two descriptors, and
-- 
2.56.0.307.g5fe7bd680c-goog

[PATCH 2/3] net/gve: account for multi-mbuf headers when validating TSO

[Joshua Washington]

The DQ queue format allows TSO headers to span up to 3 descriptors.
Before this patch, the TSO descriptor limit check logic assumed that
headers would only be in a single descriptor. This can cause under
counting when processing descriptors, which can lead to hardware
failures.

As an example: suppose a header spans 3 mbufs. In that case, 3
descriptors will need to be reserved per TSO segment. Now imagine that
there is a TSO segment whose payload spans 8 buffers. The previous logic
would add one for the headers, making a total of 9, which would be
considered valid. In actuality, since the header spans 3 descriptors
themselves, there are actually 11 descriptors for the one wire segment,
which is invalid.

Compute the number of descriptors needed for the TSO header, reject
mbufs in which the headers span more than 3 buffers, and count the
number of descriptors needed against the descriptor budget per wire
packet/TSO segment.

Also, relax the strong restriction that all headers be in the same
buffer, as this is not a requirement by the hardware.

Fixes: be8f0eb81f98 ("net/gve: fix DQO TSO descriptor limit")
Fixes: 6b0452f76bf0 ("net/gve: enforce DQO Tx packet length limits")
Cc: stable@dpdk.org

Signed-off-by: Joshua Washington <joshwash@google.com>
Reviewed-by: Praveen Kaligineedi <pkaligineedi@google.com>
Reviewed-by: Ankit Garg <nktgrg@google.com>
---
Depends-on: series-39707 ("net/gve: fix DQO Tx queue stalls")
---
 drivers/net/gve/base/gve_desc_dqo.h |  1 +
 drivers/net/gve/gve_tx_dqo.c        | 38 ++++++++++++++++++++++-------
 2 files changed, 30 insertions(+), 9 deletions(-)

diff --git a/drivers/net/gve/base/gve_desc_dqo.h b/drivers/net/gve/base/gve_desc_dqo.h
index 53e394f01c..b7d8b27d68 100644
--- a/drivers/net/gve/base/gve_desc_dqo.h
+++ b/drivers/net/gve/base/gve_desc_dqo.h
@@ -45,6 +45,7 @@ GVE_CHECK_STRUCT_LEN(16, gve_tx_pkt_desc_dqo);
 
 /* Maximum number of data descriptors allowed per packet, or per-TSO segment. */
 #define GVE_TX_MAX_DATA_DESCS 10
+#define GVE_TX_MAX_HEADER_DESCS_TSO 3
 
 /* Min gap between tail and head to avoid cacheline overlap */
 #define GVE_TX_MIN_DESC_PREVENT_CACHE_OVERLAP 4
diff --git a/drivers/net/gve/gve_tx_dqo.c b/drivers/net/gve/gve_tx_dqo.c
index 3a20109a5d..00a4ea9d27 100644
--- a/drivers/net/gve/gve_tx_dqo.c
+++ b/drivers/net/gve/gve_tx_dqo.c
@@ -138,7 +138,7 @@ gve_tx_validate_pkt(const struct rte_mbuf *tx_pkt, bool is_tso)
 	 */
 	headlen = tx_pkt->l2_len + tx_pkt->l3_len + tx_pkt->l4_len;
 	if (tx_pkt->l4_len == 0 || headlen > GVE_TX_MAX_HDR_SIZE_DQO ||
-	    tx_pkt->data_len < headlen || tx_pkt->pkt_len <= headlen)
+	    tx_pkt->pkt_len <= headlen)
 		return false;
 
 	return tx_pkt->pkt_len - headlen <= GVE_TX_MAX_TSO_LEN_DQO &&
@@ -167,20 +167,41 @@ static inline bool
 gve_tx_validate_descs(const struct gve_tx_queue *txq, struct rte_mbuf *tx_pkt,
 		      uint16_t nb_descs, bool is_tso)
 {
+	int headlen = tx_pkt->l2_len + tx_pkt->l3_len + tx_pkt->l4_len;
+	int num_descs, seg_offset, mbuf_len;
+	int tso_segsz = tx_pkt->tso_segsz;
+	int header_remain = headlen;
+	uint16_t num_header_descs;
+
 	if (nb_descs > gve_tx_max_pkt_descs(txq))
 		return false;
 
 	if (!is_tso)
 		return nb_descs <= GVE_TX_MAX_DATA_DESCS;
 
-	int tso_segsz = tx_pkt->tso_segsz;
-	int num_descs, seg_offset, mbuf_len;
-	int headlen = tx_pkt->l2_len + tx_pkt->l3_len + tx_pkt->l4_len;
+	/* Process headers. Headers can span up to 3 descriptors. */
+	if (headlen == 0)
+		return false;
+
+	num_header_descs = 1;
+	while (tx_pkt != NULL) {
+		if (header_remain <= tx_pkt->data_len)
+			break;
+
+		if (likely(tx_pkt->data_len > 0)) {
+			header_remain -= tx_pkt->data_len;
+			num_header_descs++;
+		}
 
-	/* Headers will be split into their own buffer. */
-	num_descs = 1;
+		tx_pkt = tx_pkt->next;
+	}
+
+	if (tx_pkt == NULL || num_header_descs > GVE_TX_MAX_HEADER_DESCS_TSO)
+		return false;
+
+	num_descs = num_header_descs;
 	seg_offset = 0;
-	mbuf_len = tx_pkt->data_len - headlen;
+	mbuf_len = tx_pkt->data_len - header_remain;
 
 	while (tx_pkt) {
 		if (!mbuf_len)
@@ -207,7 +228,7 @@ gve_tx_validate_descs(const struct gve_tx_queue *txq, struct rte_mbuf *tx_pkt,
 			 * segment in this mbuf is divided between two
 			 * descriptors, both must be counted.
 			 */
-			num_descs = 1 + !!(seg_offset) +
+			num_descs = num_header_descs + !!(seg_offset) +
 				(mbuf_remain < seg_offset && mbuf_remain);
 		} else {
 			seg_offset += mbuf_len;
@@ -220,7 +241,6 @@ gve_tx_validate_descs(const struct gve_tx_queue *txq, struct rte_mbuf *tx_pkt,
 			mbuf_len = tx_pkt->data_len;
 	}
 
-
 	return true;
 }
 
-- 
2.56.0.307.g5fe7bd680c-goog

[PATCH 3/3] net/gve: introduce Tx packet preparation for DQO format

[Joshua Washington]

Before this patch, packets that are invalid in the DQO format with
respect to offloads, descriptor limits, or packet size limits, are
dropped by the driver without any form of recourse for the application,
potentially leading to a disruption in traffic that can be hard to
debug.

Introduce tx_pkt_prepare for the DQ queue format to avoid dropping
packets without first giving the application a way to know that they
are invalid.

Fixes: 84472d3edffa ("net/gve: fix Tx stall on invalid packets")
Cc: stable@dpdk.org

Signed-off-by: Joshua Washington <joshwash@google.com>
Reviewed-by: Praveen Kaligineedi <pkaligineedi@google.com>
Reviewed-by: Ankit Garg <nktgrg@google.com>
---
Depends-on: series-39707 ("net/gve: fix DQO Tx queue stalls")
---
 drivers/net/gve/gve_ethdev.h |  3 ++
 drivers/net/gve/gve_tx_dqo.c | 64 ++++++++++++++++++++++++++++++++++--
 2 files changed, 65 insertions(+), 2 deletions(-)

diff --git a/drivers/net/gve/gve_ethdev.h b/drivers/net/gve/gve_ethdev.h
index 26c6a304a6..ab5af33ca9 100644
--- a/drivers/net/gve/gve_ethdev.h
+++ b/drivers/net/gve/gve_ethdev.h
@@ -649,6 +649,9 @@ gve_rx_burst_dqo(void *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts);
 uint16_t
 gve_tx_burst_dqo(void *txq, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);
 
+uint16_t
+gve_tx_pkt_prepare_dqo(void *txq, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);
+
 void
 gve_set_rx_function_dqo(struct rte_eth_dev *dev);
 
diff --git a/drivers/net/gve/gve_tx_dqo.c b/drivers/net/gve/gve_tx_dqo.c
index 00a4ea9d27..67ad41d618 100644
--- a/drivers/net/gve/gve_tx_dqo.c
+++ b/drivers/net/gve/gve_tx_dqo.c
@@ -3,6 +3,8 @@
  * Copyright (c) 2022-2023 Intel Corporation
  */
 
+#include <rte_net.h>
+
 #include "gve_ethdev.h"
 #include "base/gve_adminq.h"
 #include "rte_malloc.h"
@@ -125,7 +127,7 @@ gve_tx_clean_descs_dqo(struct gve_tx_queue *txq, uint16_t nb_descs) {
 
 /* Validate packet lengths against the device limits. */
 static inline bool
-gve_tx_validate_pkt(const struct rte_mbuf *tx_pkt, bool is_tso)
+gve_tx_validate_pkt_limits(const struct rte_mbuf *tx_pkt, bool is_tso)
 {
 	uint32_t headlen;
 
@@ -303,6 +305,63 @@ gve_fill_tx_pkt_desc(struct gve_tx_queue *txq, uint16_t *tx_id,
 	}
 }
 
+uint16_t
+gve_tx_pkt_prepare_dqo(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
+{
+	struct gve_tx_queue *txq = tx_queue;
+	uint16_t i;
+
+	for (i = 0; i < nb_pkts; i++) {
+		struct rte_mbuf *tx_pkt;
+		uint16_t nb_descs;
+		bool tso;
+		int err;
+
+		tx_pkt = tx_pkts[i];
+
+#ifdef RTE_ETHDEV_DEBUG_TX
+		const char *reason;
+
+		if (rte_mbuf_check(tx_pkt, true, &reason)) {
+			rte_errno = EINVAL;
+			return i;
+		}
+
+		err = rte_validate_tx_offload(tx_pkt);
+		if (err != 0) {
+			rte_errno = -err;
+			return i;
+		}
+#endif
+
+		tso = !!(tx_pkt->ol_flags & RTE_MBUF_F_TX_TCP_SEG);
+		if (!gve_tx_validate_pkt_limits(tx_pkt, tso)) {
+			rte_errno = EINVAL;
+			return i;
+		}
+
+		/* Handle non-TSO packets with a direct check of nb_segs. We
+		 * already validated that the packet is no larger than 9728B, so
+		 * every segment is guaranteed to only require a single
+		 * descriptor.
+		 */
+		nb_descs = tso ? gve_tx_pkt_nb_data_descs(tx_pkt) + 1 :
+			tx_pkt->nb_segs;
+		if (!gve_tx_validate_descs(txq, tx_pkt, nb_descs, tso)) {
+			rte_errno = ENOTSUP;
+			return i;
+		}
+
+		err = rte_net_intel_cksum_prepare(tx_pkt);
+		if (err != 0) {
+			rte_errno = -err;
+			return i;
+		}
+	}
+
+	return nb_pkts;
+}
+
 uint16_t
 gve_tx_burst_dqo(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 {
@@ -349,7 +408,7 @@ gve_tx_burst_dqo(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 		tso = !!(ol_flags & RTE_MBUF_F_TX_TCP_SEG);
 		csum = !!(ol_flags & GVE_TX_CKSUM_OFFLOAD_MASK_DQO);
 
-		if (!gve_tx_validate_pkt(tx_pkt, tso)) {
+		if (!gve_tx_validate_pkt_limits(tx_pkt, tso)) {
 			txq->stats.invalid_pkts++;
 			rte_pktmbuf_free(tx_pkt);
 			continue;
@@ -703,4 +762,5 @@ void
 gve_set_tx_function_dqo(struct rte_eth_dev *dev)
 {
 	dev->tx_pkt_burst = gve_tx_burst_dqo;
+	dev->tx_pkt_prepare = gve_tx_pkt_prepare_dqo;
 }
-- 
2.56.0.307.g5fe7bd680c-goog

Re: [PATCH 0/3] net/gve: TSO descriptor count fixes for GVE DQO

[Stephen Hemminger]

On Mon, 26 Oct 2026 13:23:24 -0700
Joshua Washington <joshwash@google.com> wrote:

> https://inbox.dpdk.org/dev/20250707231812.1937260-7-joshwash@google.com/
> introduced descriptor limit checks for TSO packets in DPDK.
> Unfortunately, a couple of bugs have been found in the implementation.
> 
> This patch series introduces fixes for those bugs while correcting the
> overly restrictive behavior recently introduced mandating that all
> headers be in the same buffer.
> 
> The final patch in the series introduces tx_pkt_prepare as an option to
> ensure that there is a way to get a clear signal of when packets violate
> hardware limitations. Before a recently introduced patch, the GVE
> behavior was to stop transmitting at the first invalid packet, which
> would allow the application to do any necessary linearization or packet
> modification before attempting to transmit again. It turns out that that
> behavior is considered incorrect, so offer tx_pkt_prepare as an
> alternative.
> 
> Depends-on: series-39707 ("net/gve: fix DQO Tx queue stalls")

AI spotted issues here. 

Subject: Re: [PATCH 0/3] net/gve: DQO TSO descriptor validation fixes

Series summary

Patch 1 is correct. Patch 2 relaxes the requirement that TSO headers
be in the first mbuf, but patch 3 then calls
rte_net_intel_cksum_prepare(), which rejects exactly those packets.
The series needs to pick one behaviour.


Patch 2/3 + 3/3

Error: rte_eth_tx_prepare() rejects split-header TSO packets that
patch 2 now allows.

rte_net_intel_cksum_flags_prepare() in lib/net/rte_net.h has:

	if (unlikely(rte_pktmbuf_data_len(m) <
		     inner_l3_offset + m->l3_len + m->l4_len))
		return -ENOTSUP;

So after patch 3, any application that follows the documented
prepare-then-burst model (e.g. testpmd csum engine) still cannot send
a TSO packet whose headers span mbufs. Only applications that skip
tx_prepare get the relaxation from patch 2.

Does DQO actually need the TCP pseudo-header checksum seeded? If not,
drop the rte_net_intel_cksum_prepare() call. If it does, either keep
the data_len < headlen check for TSO in gve_tx_validate_pkt_limits()
and drop the relaxation from patch 2, or do the fixup across segments
with rte_pktmbuf_read() and copy the result back.


Patch 3/3

Warning: prepare and burst count descriptors differently.

prepare uses nb_segs for non-TSO packets, but gve_tx_pkt_nb_data_descs()
(and the burst fill loop) skip zero-length segments. A non-TSO packet
with empty segments can fail prepare with ENOTSUP while burst would
send it. Use the same count as burst:

	tso = !!(tx_pkt->ol_flags & RTE_MBUF_F_TX_TCP_SEG);
	...
	nb_descs = gve_tx_pkt_nb_data_descs(tx_pkt) + tso;

and drop the comment about nb_segs.

Info: adding a tx_pkt_prepare op is a behaviour change for stable
users; stable maintainers may not want it backported with patches 1
and 2.


Patch 2/3

Info: the header walk can stop as soon as the limit is exceeded
rather than walking the rest of a chain of empty mbufs:

	if (likely(tx_pkt->data_len > 0)) {
		header_remain -= tx_pkt->data_len;
		if (++num_header_descs > GVE_TX_MAX_HEADER_DESCS_TSO)
			return false;
	}


Patch 1/3

Info: subject has "segment" twice ("fix segment TSO segment offset").