FAILED: patch "[PATCH] gve: fix header buffer corruption with header-split and" failed to apply to 6.12-stable tree

FAILED: patch "[PATCH] gve: fix header buffer corruption with header-split and" failed to apply to 6.12-stable tree

[gregkh]

The patch below does not apply to the 6.12-stable tree.
If someone wants it applied there, or to any other stable or longterm
tree, then please email the backport, including the original git commit
id to <stable@vger.kernel.org>.

To reproduce the conflict and resubmit, you may use the following commands:

git fetch https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ linux-6.12.y
git checkout FETCH_HEAD
git cherry-pick -x d676c9a73bdcd8237425dbb826f2bd1a25c36e40
# <resolve conflicts, build, test, etc.>
git commit -s
git send-email --to '<stable@vger.kernel.org>' --in-reply-to '2026072027-feline-deserving-7fc7@gregkh' --subject-prefix 'PATCH 6.12.y' 'HEAD^..'

Possible dependencies:



thanks,

greg k-h

------------------ original commit in Linus's tree ------------------

From d676c9a73bdcd8237425dbb826f2bd1a25c36e40 Mon Sep 17 00:00:00 2001
From: Ankit Garg <nktgrg@google.com>
Date: Tue, 16 Jun 2026 18:32:08 -0700
Subject: [PATCH] gve: fix header buffer corruption with header-split and
 HW-GRO

The DQO RX datapath programs a per-buffer-queue-descriptor
header_buf_addr at post time and reads the split header back at
completion time. Both the post and the read currently index the
header buffer by queue position rather than by the buffer's identity:

  - post (gve_rx_post_buffers_dqo): header_buf_addr is computed from
    bufq->tail
  - read (gve_rx_dqo): the header is read from desc_idx (the completion
    queue head index)

This relies on the buffer-queue index and the completion-queue index
being equal for the start of every packet, i.e. on the device consuming
posted buffers and returning completions in the exact same order. That
assumption does not hold once HW-GRO is enabled with multiple
flows: coalesced segments are accepted and completed in an order that
may differ from the order buffers were posted, and segments from
different flows may interleave.

That results in two problems:

1. Wrong header slot on read. Because the read offset is derived from
   the completion index (desc_idx) while the device wrote the header to
   the address programmed for the buffer's buf_id, the driver can copy
   a header belonging to a different packet. This shows up as
   throughput drop (about 30% drop and large numbers of TCP
   retransmissions) with header-split and HW-GRO both enabled and many
   streams.

2. Header buffer reused while still owned by the device. The driver
   advances bufq->head by one per completion and re-posts buffers based
   on that. Arrival of N RX completions only guarantees that at least N
   RX buffer descriptors have been read by the device. It does not
   guarantee that the device has relinquished the ownership of all the
   buffers corresponding to those N descriptors. With out-of-order
   completions (e.g. the completion for a packet copied into buffer N
   arrives before the completion for a packet copied into buffer N-1),
   the driver can re-post and overwrite a header buffer that the device
   is still going to write into, corrupting the header of a packet
   whose completion has not yet been processed.

Fix both issues by indexing the header buffer by buf_id on both the post
and read paths. Reading from buf_id's slot is therefore always correct
regardless of completion ordering (fixes problem 1).

Indexing by buf_id also ties each header slot to the lifetime of its
buffer state. A buffer state is only returned to the free/recycle lists
when its own completion (buf_id) is processed, so its header slot can
only be re-posted after the device is done with it. This makes header
slot reuse safe under out-of-order completions (fixes problem 2).

Allocate (gve_rx_alloc_hdr_bufs) and free (gve_rx_free_hdr_bufs) the
header buffers based on num_buf_states to match the buf_id indexing.

Cc: stable@vger.kernel.org
Fixes: 5e37d8254e7f ("gve: Add header split data path")
Signed-off-by: Ankit Garg <nktgrg@google.com>
Reviewed-by: Praveen Kaligineedi <pkaligineedi@google.com>
Reviewed-by: Jordan Rhee <jordanrhee@google.com>
Reviewed-by: Harshitha Ramamurthy <hramamurthy@google.com>
Signed-off-by: Joshua Washington <joshwash@google.com>
Reviewed-by: Eric Dumazet <edumazet@google.com>
Link: https://patch.msgid.link/20260617013208.3781453-1-joshwash@google.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

diff --git a/drivers/net/ethernet/google/gve/gve_rx_dqo.c b/drivers/net/ethernet/google/gve/gve_rx_dqo.c
index 7924dce719e2..02cba280d81a 100644
--- a/drivers/net/ethernet/google/gve/gve_rx_dqo.c
+++ b/drivers/net/ethernet/google/gve/gve_rx_dqo.c
@@ -21,11 +21,13 @@
 static void gve_rx_free_hdr_bufs(struct gve_priv *priv, struct gve_rx_ring *rx)
 {
 	struct device *hdev = &priv->pdev->dev;
-	int buf_count = rx->dqo.bufq.mask + 1;
 
 	if (rx->dqo.hdr_bufs.data) {
-		dma_free_coherent(hdev, priv->header_buf_size * buf_count,
-				  rx->dqo.hdr_bufs.data, rx->dqo.hdr_bufs.addr);
+		size_t size =
+			(size_t)priv->header_buf_size * rx->dqo.num_buf_states;
+
+		dma_free_coherent(hdev, size, rx->dqo.hdr_bufs.data,
+				  rx->dqo.hdr_bufs.addr);
 		rx->dqo.hdr_bufs.data = NULL;
 	}
 }
@@ -254,7 +256,7 @@ int gve_rx_alloc_ring_dqo(struct gve_priv *priv,
 
 	/* Allocate header buffers for header-split */
 	if (cfg->enable_header_split)
-		if (gve_rx_alloc_hdr_bufs(priv, rx, buffer_queue_slots))
+		if (gve_rx_alloc_hdr_bufs(priv, rx, rx->dqo.num_buf_states))
 			goto err;
 
 	/* Allocate RX completion queue */
@@ -381,10 +383,13 @@ void gve_rx_post_buffers_dqo(struct gve_rx_ring *rx)
 			break;
 		}
 
-		if (rx->dqo.hdr_bufs.data)
+		if (rx->dqo.hdr_bufs.data) {
+			u16 buf_id = le16_to_cpu(desc->buf_id);
+
 			desc->header_buf_addr =
 				cpu_to_le64(rx->dqo.hdr_bufs.addr +
-					    priv->header_buf_size * bufq->tail);
+					(size_t)priv->header_buf_size * buf_id);
+		}
 
 		bufq->tail = (bufq->tail + 1) & bufq->mask;
 		complq->num_free_slots--;
@@ -826,10 +831,13 @@ static int gve_rx_dqo(struct napi_struct *napi, struct gve_rx_ring *rx,
 		int unsplit = 0;
 
 		if (hdr_len && !hbo) {
-			rx->ctx.skb_head = gve_rx_copy_data(priv->dev, napi,
-							    rx->dqo.hdr_bufs.data +
-							    desc_idx * priv->header_buf_size,
-							    hdr_len);
+			size_t offset =
+				(size_t)buffer_id * priv->header_buf_size;
+
+			rx->ctx.skb_head =
+				gve_rx_copy_data(priv->dev, napi,
+						 rx->dqo.hdr_bufs.data + offset,
+						 hdr_len);
 			if (unlikely(!rx->ctx.skb_head))
 				goto error;
 			rx->ctx.skb_tail = rx->ctx.skb_head;

[PATCH 6.12.y 1/2] gve: move DQO rx buffer management related code to a new file

[Sasha Levin]

From: Harshitha Ramamurthy <hramamurthy@google.com>

[ Upstream commit 93c68f1275f9e21ccfed9ee292aedb11c3f6241b ]

In preparation for the upcoming page pool adoption for DQO
raw addressing mode, move RX buffer management code to a new
file. In the follow on patches, page pool code will be added
to this file.

No functional change, just movement of code.

Reviewed-by: Praveen Kaligineedi <pkaligineedi@google.com>
Reviewed-by: Shailend Chand <shailend@google.com>
Reviewed-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: Harshitha Ramamurthy <hramamurthy@google.com>
Reviewed-by: Jacob Keller <jacob.e.keller@intel.com>
Link: https://patch.msgid.link/20241014202108.1051963-2-pkaligineedi@google.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Stable-dep-of: d676c9a73bdc ("gve: fix header buffer corruption with header-split and HW-GRO")
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 drivers/net/ethernet/google/gve/Makefile      |   3 +-
 drivers/net/ethernet/google/gve/gve.h         |  18 ++
 .../ethernet/google/gve/gve_buffer_mgmt_dqo.c | 230 ++++++++++++++++++
 drivers/net/ethernet/google/gve/gve_rx_dqo.c  | 225 -----------------
 4 files changed, 250 insertions(+), 226 deletions(-)
 create mode 100644 drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c

diff --git a/drivers/net/ethernet/google/gve/Makefile b/drivers/net/ethernet/google/gve/Makefile
index 9ed07080b38a25..4520f1c07a638f 100644
--- a/drivers/net/ethernet/google/gve/Makefile
+++ b/drivers/net/ethernet/google/gve/Makefile
@@ -1,4 +1,5 @@
 # Makefile for the Google virtual Ethernet (gve) driver
 
 obj-$(CONFIG_GVE) += gve.o
-gve-objs := gve_main.o gve_tx.o gve_tx_dqo.o gve_rx.o gve_rx_dqo.o gve_ethtool.o gve_adminq.o gve_utils.o gve_flow_rule.o
+gve-objs := gve_main.o gve_tx.o gve_tx_dqo.o gve_rx.o gve_rx_dqo.o gve_ethtool.o gve_adminq.o gve_utils.o gve_flow_rule.o \
+	    gve_buffer_mgmt_dqo.o
diff --git a/drivers/net/ethernet/google/gve/gve.h b/drivers/net/ethernet/google/gve/gve.h
index 8ddd366d9fde54..99fd933a1f30d4 100644
--- a/drivers/net/ethernet/google/gve/gve.h
+++ b/drivers/net/ethernet/google/gve/gve.h
@@ -1173,6 +1173,24 @@ void gve_rx_stop_ring_gqi(struct gve_priv *priv, int idx);
 u16 gve_get_pkt_buf_size(const struct gve_priv *priv, bool enable_hplit);
 bool gve_header_split_supported(const struct gve_priv *priv);
 int gve_set_hsplit_config(struct gve_priv *priv, u8 tcp_data_split);
+/* rx buffer handling */
+int gve_buf_ref_cnt(struct gve_rx_buf_state_dqo *bs);
+void gve_free_page_dqo(struct gve_priv *priv, struct gve_rx_buf_state_dqo *bs,
+		       bool free_page);
+struct gve_rx_buf_state_dqo *gve_alloc_buf_state(struct gve_rx_ring *rx);
+bool gve_buf_state_is_allocated(struct gve_rx_ring *rx,
+				struct gve_rx_buf_state_dqo *buf_state);
+void gve_free_buf_state(struct gve_rx_ring *rx,
+			struct gve_rx_buf_state_dqo *buf_state);
+struct gve_rx_buf_state_dqo *gve_dequeue_buf_state(struct gve_rx_ring *rx,
+						   struct gve_index_list *list);
+void gve_enqueue_buf_state(struct gve_rx_ring *rx, struct gve_index_list *list,
+			   struct gve_rx_buf_state_dqo *buf_state);
+struct gve_rx_buf_state_dqo *gve_get_recycled_buf_state(struct gve_rx_ring *rx);
+int gve_alloc_page_dqo(struct gve_rx_ring *rx,
+		       struct gve_rx_buf_state_dqo *buf_state);
+void gve_try_recycle_buf(struct gve_priv *priv, struct gve_rx_ring *rx,
+			 struct gve_rx_buf_state_dqo *buf_state);
 /* Reset */
 void gve_schedule_reset(struct gve_priv *priv);
 int gve_reset(struct gve_priv *priv, bool attempt_teardown);
diff --git a/drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c b/drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c
new file mode 100644
index 00000000000000..8e50f0e4bb2ecb
--- /dev/null
+++ b/drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c
@@ -0,0 +1,230 @@
+// SPDX-License-Identifier: (GPL-2.0 OR MIT)
+/* Google virtual Ethernet (gve) driver
+ *
+ * Copyright (C) 2015-2024 Google, Inc.
+ */
+
+#include "gve.h"
+#include "gve_utils.h"
+
+int gve_buf_ref_cnt(struct gve_rx_buf_state_dqo *bs)
+{
+	return page_count(bs->page_info.page) - bs->page_info.pagecnt_bias;
+}
+
+void gve_free_page_dqo(struct gve_priv *priv, struct gve_rx_buf_state_dqo *bs,
+		       bool free_page)
+{
+	page_ref_sub(bs->page_info.page, bs->page_info.pagecnt_bias - 1);
+	if (free_page)
+		gve_free_page(&priv->pdev->dev, bs->page_info.page, bs->addr,
+			      DMA_FROM_DEVICE);
+	bs->page_info.page = NULL;
+}
+
+struct gve_rx_buf_state_dqo *gve_alloc_buf_state(struct gve_rx_ring *rx)
+{
+	struct gve_rx_buf_state_dqo *buf_state;
+	s16 buffer_id;
+
+	buffer_id = rx->dqo.free_buf_states;
+	if (unlikely(buffer_id == -1))
+		return NULL;
+
+	buf_state = &rx->dqo.buf_states[buffer_id];
+
+	/* Remove buf_state from free list */
+	rx->dqo.free_buf_states = buf_state->next;
+
+	/* Point buf_state to itself to mark it as allocated */
+	buf_state->next = buffer_id;
+
+	return buf_state;
+}
+
+bool gve_buf_state_is_allocated(struct gve_rx_ring *rx,
+				struct gve_rx_buf_state_dqo *buf_state)
+{
+	s16 buffer_id = buf_state - rx->dqo.buf_states;
+
+	return buf_state->next == buffer_id;
+}
+
+void gve_free_buf_state(struct gve_rx_ring *rx,
+			struct gve_rx_buf_state_dqo *buf_state)
+{
+	s16 buffer_id = buf_state - rx->dqo.buf_states;
+
+	buf_state->next = rx->dqo.free_buf_states;
+	rx->dqo.free_buf_states = buffer_id;
+}
+
+struct gve_rx_buf_state_dqo *gve_dequeue_buf_state(struct gve_rx_ring *rx,
+						   struct gve_index_list *list)
+{
+	struct gve_rx_buf_state_dqo *buf_state;
+	s16 buffer_id;
+
+	buffer_id = list->head;
+	if (unlikely(buffer_id == -1))
+		return NULL;
+
+	buf_state = &rx->dqo.buf_states[buffer_id];
+
+	/* Remove buf_state from list */
+	list->head = buf_state->next;
+	if (buf_state->next == -1)
+		list->tail = -1;
+
+	/* Point buf_state to itself to mark it as allocated */
+	buf_state->next = buffer_id;
+
+	return buf_state;
+}
+
+void gve_enqueue_buf_state(struct gve_rx_ring *rx, struct gve_index_list *list,
+			   struct gve_rx_buf_state_dqo *buf_state)
+{
+	s16 buffer_id = buf_state - rx->dqo.buf_states;
+
+	buf_state->next = -1;
+
+	if (list->head == -1) {
+		list->head = buffer_id;
+		list->tail = buffer_id;
+	} else {
+		int tail = list->tail;
+
+		rx->dqo.buf_states[tail].next = buffer_id;
+		list->tail = buffer_id;
+	}
+}
+
+struct gve_rx_buf_state_dqo *gve_get_recycled_buf_state(struct gve_rx_ring *rx)
+{
+	struct gve_rx_buf_state_dqo *buf_state;
+	int i;
+
+	/* Recycled buf states are immediately usable. */
+	buf_state = gve_dequeue_buf_state(rx, &rx->dqo.recycled_buf_states);
+	if (likely(buf_state))
+		return buf_state;
+
+	if (unlikely(rx->dqo.used_buf_states.head == -1))
+		return NULL;
+
+	/* Used buf states are only usable when ref count reaches 0, which means
+	 * no SKBs refer to them.
+	 *
+	 * Search a limited number before giving up.
+	 */
+	for (i = 0; i < 5; i++) {
+		buf_state = gve_dequeue_buf_state(rx, &rx->dqo.used_buf_states);
+		if (gve_buf_ref_cnt(buf_state) == 0) {
+			rx->dqo.used_buf_states_cnt--;
+			return buf_state;
+		}
+
+		gve_enqueue_buf_state(rx, &rx->dqo.used_buf_states, buf_state);
+	}
+
+	/* For QPL, we cannot allocate any new buffers and must
+	 * wait for the existing ones to be available.
+	 */
+	if (rx->dqo.qpl)
+		return NULL;
+
+	/* If there are no free buf states discard an entry from
+	 * `used_buf_states` so it can be used.
+	 */
+	if (unlikely(rx->dqo.free_buf_states == -1)) {
+		buf_state = gve_dequeue_buf_state(rx, &rx->dqo.used_buf_states);
+		if (gve_buf_ref_cnt(buf_state) == 0)
+			return buf_state;
+
+		gve_free_page_dqo(rx->gve, buf_state, true);
+		gve_free_buf_state(rx, buf_state);
+	}
+
+	return NULL;
+}
+
+int gve_alloc_page_dqo(struct gve_rx_ring *rx,
+		       struct gve_rx_buf_state_dqo *buf_state)
+{
+	struct gve_priv *priv = rx->gve;
+	u32 idx;
+
+	if (!rx->dqo.qpl) {
+		int err;
+
+		err = gve_alloc_page(priv, &priv->pdev->dev,
+				     &buf_state->page_info.page,
+				     &buf_state->addr,
+				     DMA_FROM_DEVICE, GFP_ATOMIC);
+		if (err)
+			return err;
+	} else {
+		idx = rx->dqo.next_qpl_page_idx;
+		if (idx >= gve_get_rx_pages_per_qpl_dqo(priv->rx_desc_cnt)) {
+			net_err_ratelimited("%s: Out of QPL pages\n",
+					    priv->dev->name);
+			return -ENOMEM;
+		}
+		buf_state->page_info.page = rx->dqo.qpl->pages[idx];
+		buf_state->addr = rx->dqo.qpl->page_buses[idx];
+		rx->dqo.next_qpl_page_idx++;
+	}
+	buf_state->page_info.page_offset = 0;
+	buf_state->page_info.page_address =
+		page_address(buf_state->page_info.page);
+	buf_state->last_single_ref_offset = 0;
+
+	/* The page already has 1 ref. */
+	page_ref_add(buf_state->page_info.page, INT_MAX - 1);
+	buf_state->page_info.pagecnt_bias = INT_MAX;
+
+	return 0;
+}
+
+void gve_try_recycle_buf(struct gve_priv *priv, struct gve_rx_ring *rx,
+			 struct gve_rx_buf_state_dqo *buf_state)
+{
+	const u16 data_buffer_size = priv->data_buffer_size_dqo;
+	int pagecount;
+
+	/* Can't reuse if we only fit one buffer per page */
+	if (data_buffer_size * 2 > PAGE_SIZE)
+		goto mark_used;
+
+	pagecount = gve_buf_ref_cnt(buf_state);
+
+	/* Record the offset when we have a single remaining reference.
+	 *
+	 * When this happens, we know all of the other offsets of the page are
+	 * usable.
+	 */
+	if (pagecount == 1) {
+		buf_state->last_single_ref_offset =
+			buf_state->page_info.page_offset;
+	}
+
+	/* Use the next buffer sized chunk in the page. */
+	buf_state->page_info.page_offset += data_buffer_size;
+	buf_state->page_info.page_offset &= (PAGE_SIZE - 1);
+
+	/* If we wrap around to the same offset without ever dropping to 1
+	 * reference, then we don't know if this offset was ever freed.
+	 */
+	if (buf_state->page_info.page_offset ==
+	    buf_state->last_single_ref_offset) {
+		goto mark_used;
+	}
+
+	gve_enqueue_buf_state(rx, &rx->dqo.recycled_buf_states, buf_state);
+	return;
+
+mark_used:
+	gve_enqueue_buf_state(rx, &rx->dqo.used_buf_states, buf_state);
+	rx->dqo.used_buf_states_cnt++;
+}
diff --git a/drivers/net/ethernet/google/gve/gve_rx_dqo.c b/drivers/net/ethernet/google/gve/gve_rx_dqo.c
index 1154c1d8f66f05..b343be2fb118ad 100644
--- a/drivers/net/ethernet/google/gve/gve_rx_dqo.c
+++ b/drivers/net/ethernet/google/gve/gve_rx_dqo.c
@@ -16,189 +16,6 @@
 #include <net/ipv6.h>
 #include <net/tcp.h>
 
-static int gve_buf_ref_cnt(struct gve_rx_buf_state_dqo *bs)
-{
-	return page_count(bs->page_info.page) - bs->page_info.pagecnt_bias;
-}
-
-static void gve_free_page_dqo(struct gve_priv *priv,
-			      struct gve_rx_buf_state_dqo *bs,
-			      bool free_page)
-{
-	page_ref_sub(bs->page_info.page, bs->page_info.pagecnt_bias - 1);
-	if (free_page)
-		gve_free_page(&priv->pdev->dev, bs->page_info.page, bs->addr,
-			      DMA_FROM_DEVICE);
-	bs->page_info.page = NULL;
-}
-
-static struct gve_rx_buf_state_dqo *gve_alloc_buf_state(struct gve_rx_ring *rx)
-{
-	struct gve_rx_buf_state_dqo *buf_state;
-	s16 buffer_id;
-
-	buffer_id = rx->dqo.free_buf_states;
-	if (unlikely(buffer_id == -1))
-		return NULL;
-
-	buf_state = &rx->dqo.buf_states[buffer_id];
-
-	/* Remove buf_state from free list */
-	rx->dqo.free_buf_states = buf_state->next;
-
-	/* Point buf_state to itself to mark it as allocated */
-	buf_state->next = buffer_id;
-
-	return buf_state;
-}
-
-static bool gve_buf_state_is_allocated(struct gve_rx_ring *rx,
-				       struct gve_rx_buf_state_dqo *buf_state)
-{
-	s16 buffer_id = buf_state - rx->dqo.buf_states;
-
-	return buf_state->next == buffer_id;
-}
-
-static void gve_free_buf_state(struct gve_rx_ring *rx,
-			       struct gve_rx_buf_state_dqo *buf_state)
-{
-	s16 buffer_id = buf_state - rx->dqo.buf_states;
-
-	buf_state->next = rx->dqo.free_buf_states;
-	rx->dqo.free_buf_states = buffer_id;
-}
-
-static struct gve_rx_buf_state_dqo *
-gve_dequeue_buf_state(struct gve_rx_ring *rx, struct gve_index_list *list)
-{
-	struct gve_rx_buf_state_dqo *buf_state;
-	s16 buffer_id;
-
-	buffer_id = list->head;
-	if (unlikely(buffer_id == -1))
-		return NULL;
-
-	buf_state = &rx->dqo.buf_states[buffer_id];
-
-	/* Remove buf_state from list */
-	list->head = buf_state->next;
-	if (buf_state->next == -1)
-		list->tail = -1;
-
-	/* Point buf_state to itself to mark it as allocated */
-	buf_state->next = buffer_id;
-
-	return buf_state;
-}
-
-static void gve_enqueue_buf_state(struct gve_rx_ring *rx,
-				  struct gve_index_list *list,
-				  struct gve_rx_buf_state_dqo *buf_state)
-{
-	s16 buffer_id = buf_state - rx->dqo.buf_states;
-
-	buf_state->next = -1;
-
-	if (list->head == -1) {
-		list->head = buffer_id;
-		list->tail = buffer_id;
-	} else {
-		int tail = list->tail;
-
-		rx->dqo.buf_states[tail].next = buffer_id;
-		list->tail = buffer_id;
-	}
-}
-
-static struct gve_rx_buf_state_dqo *
-gve_get_recycled_buf_state(struct gve_rx_ring *rx)
-{
-	struct gve_rx_buf_state_dqo *buf_state;
-	int i;
-
-	/* Recycled buf states are immediately usable. */
-	buf_state = gve_dequeue_buf_state(rx, &rx->dqo.recycled_buf_states);
-	if (likely(buf_state))
-		return buf_state;
-
-	if (unlikely(rx->dqo.used_buf_states.head == -1))
-		return NULL;
-
-	/* Used buf states are only usable when ref count reaches 0, which means
-	 * no SKBs refer to them.
-	 *
-	 * Search a limited number before giving up.
-	 */
-	for (i = 0; i < 5; i++) {
-		buf_state = gve_dequeue_buf_state(rx, &rx->dqo.used_buf_states);
-		if (gve_buf_ref_cnt(buf_state) == 0) {
-			rx->dqo.used_buf_states_cnt--;
-			return buf_state;
-		}
-
-		gve_enqueue_buf_state(rx, &rx->dqo.used_buf_states, buf_state);
-	}
-
-	/* For QPL, we cannot allocate any new buffers and must
-	 * wait for the existing ones to be available.
-	 */
-	if (rx->dqo.qpl)
-		return NULL;
-
-	/* If there are no free buf states discard an entry from
-	 * `used_buf_states` so it can be used.
-	 */
-	if (unlikely(rx->dqo.free_buf_states == -1)) {
-		buf_state = gve_dequeue_buf_state(rx, &rx->dqo.used_buf_states);
-		if (gve_buf_ref_cnt(buf_state) == 0)
-			return buf_state;
-
-		gve_free_page_dqo(rx->gve, buf_state, true);
-		gve_free_buf_state(rx, buf_state);
-	}
-
-	return NULL;
-}
-
-static int gve_alloc_page_dqo(struct gve_rx_ring *rx,
-			      struct gve_rx_buf_state_dqo *buf_state)
-{
-	struct gve_priv *priv = rx->gve;
-	u32 idx;
-
-	if (!rx->dqo.qpl) {
-		int err;
-
-		err = gve_alloc_page(priv, &priv->pdev->dev,
-				     &buf_state->page_info.page,
-				     &buf_state->addr,
-				     DMA_FROM_DEVICE, GFP_ATOMIC);
-		if (err)
-			return err;
-	} else {
-		idx = rx->dqo.next_qpl_page_idx;
-		if (idx >= gve_get_rx_pages_per_qpl_dqo(priv->rx_desc_cnt)) {
-			net_err_ratelimited("%s: Out of QPL pages\n",
-					    priv->dev->name);
-			return -ENOMEM;
-		}
-		buf_state->page_info.page = rx->dqo.qpl->pages[idx];
-		buf_state->addr = rx->dqo.qpl->page_buses[idx];
-		rx->dqo.next_qpl_page_idx++;
-	}
-	buf_state->page_info.page_offset = 0;
-	buf_state->page_info.page_address =
-		page_address(buf_state->page_info.page);
-	buf_state->last_single_ref_offset = 0;
-
-	/* The page already has 1 ref. */
-	page_ref_add(buf_state->page_info.page, INT_MAX - 1);
-	buf_state->page_info.pagecnt_bias = INT_MAX;
-
-	return 0;
-}
-
 static void gve_rx_free_hdr_bufs(struct gve_priv *priv, struct gve_rx_ring *rx)
 {
 	struct device *hdev = &priv->pdev->dev;
@@ -557,48 +374,6 @@ void gve_rx_post_buffers_dqo(struct gve_rx_ring *rx)
 	rx->fill_cnt += num_posted;
 }
 
-static void gve_try_recycle_buf(struct gve_priv *priv, struct gve_rx_ring *rx,
-				struct gve_rx_buf_state_dqo *buf_state)
-{
-	const u16 data_buffer_size = priv->data_buffer_size_dqo;
-	int pagecount;
-
-	/* Can't reuse if we only fit one buffer per page */
-	if (data_buffer_size * 2 > PAGE_SIZE)
-		goto mark_used;
-
-	pagecount = gve_buf_ref_cnt(buf_state);
-
-	/* Record the offset when we have a single remaining reference.
-	 *
-	 * When this happens, we know all of the other offsets of the page are
-	 * usable.
-	 */
-	if (pagecount == 1) {
-		buf_state->last_single_ref_offset =
-			buf_state->page_info.page_offset;
-	}
-
-	/* Use the next buffer sized chunk in the page. */
-	buf_state->page_info.page_offset += data_buffer_size;
-	buf_state->page_info.page_offset &= (PAGE_SIZE - 1);
-
-	/* If we wrap around to the same offset without ever dropping to 1
-	 * reference, then we don't know if this offset was ever freed.
-	 */
-	if (buf_state->page_info.page_offset ==
-	    buf_state->last_single_ref_offset) {
-		goto mark_used;
-	}
-
-	gve_enqueue_buf_state(rx, &rx->dqo.recycled_buf_states, buf_state);
-	return;
-
-mark_used:
-	gve_enqueue_buf_state(rx, &rx->dqo.used_buf_states, buf_state);
-	rx->dqo.used_buf_states_cnt++;
-}
-
 static void gve_rx_skb_csum(struct sk_buff *skb,
 			    const struct gve_rx_compl_desc_dqo *desc,
 			    struct gve_ptype ptype)
-- 
2.53.0

[PATCH 6.12.y 2/2] gve: fix header buffer corruption with header-split and HW-GRO

[Sasha Levin]

From: Ankit Garg <nktgrg@google.com>

[ Upstream commit d676c9a73bdcd8237425dbb826f2bd1a25c36e40 ]

The DQO RX datapath programs a per-buffer-queue-descriptor
header_buf_addr at post time and reads the split header back at
completion time. Both the post and the read currently index the
header buffer by queue position rather than by the buffer's identity:

  - post (gve_rx_post_buffers_dqo): header_buf_addr is computed from
    bufq->tail
  - read (gve_rx_dqo): the header is read from desc_idx (the completion
    queue head index)

This relies on the buffer-queue index and the completion-queue index
being equal for the start of every packet, i.e. on the device consuming
posted buffers and returning completions in the exact same order. That
assumption does not hold once HW-GRO is enabled with multiple
flows: coalesced segments are accepted and completed in an order that
may differ from the order buffers were posted, and segments from
different flows may interleave.

That results in two problems:

1. Wrong header slot on read. Because the read offset is derived from
   the completion index (desc_idx) while the device wrote the header to
   the address programmed for the buffer's buf_id, the driver can copy
   a header belonging to a different packet. This shows up as
   throughput drop (about 30% drop and large numbers of TCP
   retransmissions) with header-split and HW-GRO both enabled and many
   streams.

2. Header buffer reused while still owned by the device. The driver
   advances bufq->head by one per completion and re-posts buffers based
   on that. Arrival of N RX completions only guarantees that at least N
   RX buffer descriptors have been read by the device. It does not
   guarantee that the device has relinquished the ownership of all the
   buffers corresponding to those N descriptors. With out-of-order
   completions (e.g. the completion for a packet copied into buffer N
   arrives before the completion for a packet copied into buffer N-1),
   the driver can re-post and overwrite a header buffer that the device
   is still going to write into, corrupting the header of a packet
   whose completion has not yet been processed.

Fix both issues by indexing the header buffer by buf_id on both the post
and read paths. Reading from buf_id's slot is therefore always correct
regardless of completion ordering (fixes problem 1).

Indexing by buf_id also ties each header slot to the lifetime of its
buffer state. A buffer state is only returned to the free/recycle lists
when its own completion (buf_id) is processed, so its header slot can
only be re-posted after the device is done with it. This makes header
slot reuse safe under out-of-order completions (fixes problem 2).

Allocate (gve_rx_alloc_hdr_bufs) and free (gve_rx_free_hdr_bufs) the
header buffers based on num_buf_states to match the buf_id indexing.

Cc: stable@vger.kernel.org
Fixes: 5e37d8254e7f ("gve: Add header split data path")
Signed-off-by: Ankit Garg <nktgrg@google.com>
Reviewed-by: Praveen Kaligineedi <pkaligineedi@google.com>
Reviewed-by: Jordan Rhee <jordanrhee@google.com>
Reviewed-by: Harshitha Ramamurthy <hramamurthy@google.com>
Signed-off-by: Joshua Washington <joshwash@google.com>
Reviewed-by: Eric Dumazet <edumazet@google.com>
Link: https://patch.msgid.link/20260617013208.3781453-1-joshwash@google.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 drivers/net/ethernet/google/gve/gve_rx_dqo.c | 28 +++++++++++++-------
 1 file changed, 18 insertions(+), 10 deletions(-)

diff --git a/drivers/net/ethernet/google/gve/gve_rx_dqo.c b/drivers/net/ethernet/google/gve/gve_rx_dqo.c
index b343be2fb118ad..dce90a2b4c29cf 100644
--- a/drivers/net/ethernet/google/gve/gve_rx_dqo.c
+++ b/drivers/net/ethernet/google/gve/gve_rx_dqo.c
@@ -19,11 +19,13 @@
 static void gve_rx_free_hdr_bufs(struct gve_priv *priv, struct gve_rx_ring *rx)
 {
 	struct device *hdev = &priv->pdev->dev;
-	int buf_count = rx->dqo.bufq.mask + 1;
 
 	if (rx->dqo.hdr_bufs.data) {
-		dma_free_coherent(hdev, priv->header_buf_size * buf_count,
-				  rx->dqo.hdr_bufs.data, rx->dqo.hdr_bufs.addr);
+		size_t size =
+			(size_t)priv->header_buf_size * rx->dqo.num_buf_states;
+
+		dma_free_coherent(hdev, size, rx->dqo.hdr_bufs.data,
+				  rx->dqo.hdr_bufs.addr);
 		rx->dqo.hdr_bufs.data = NULL;
 	}
 }
@@ -223,7 +225,7 @@ int gve_rx_alloc_ring_dqo(struct gve_priv *priv,
 
 	/* Allocate header buffers for header-split */
 	if (cfg->enable_header_split)
-		if (gve_rx_alloc_hdr_bufs(priv, rx, buffer_queue_slots))
+		if (gve_rx_alloc_hdr_bufs(priv, rx, rx->dqo.num_buf_states))
 			goto err;
 
 	/* Allocate RX completion queue */
@@ -358,10 +360,13 @@ void gve_rx_post_buffers_dqo(struct gve_rx_ring *rx)
 		desc->buf_id = cpu_to_le16(buf_state - rx->dqo.buf_states);
 		desc->buf_addr = cpu_to_le64(buf_state->addr +
 					     buf_state->page_info.page_offset);
-		if (rx->dqo.hdr_bufs.data)
+		if (rx->dqo.hdr_bufs.data) {
+			u16 buf_id = le16_to_cpu(desc->buf_id);
+
 			desc->header_buf_addr =
 				cpu_to_le64(rx->dqo.hdr_bufs.addr +
-					    priv->header_buf_size * bufq->tail);
+					(size_t)priv->header_buf_size * buf_id);
+		}
 
 		bufq->tail = (bufq->tail + 1) & bufq->mask;
 		complq->num_free_slots--;
@@ -566,10 +571,13 @@ static int gve_rx_dqo(struct napi_struct *napi, struct gve_rx_ring *rx,
 		int unsplit = 0;
 
 		if (hdr_len && !hbo) {
-			rx->ctx.skb_head = gve_rx_copy_data(priv->dev, napi,
-							    rx->dqo.hdr_bufs.data +
-							    desc_idx * priv->header_buf_size,
-							    hdr_len);
+			size_t offset =
+				(size_t)buffer_id * priv->header_buf_size;
+
+			rx->ctx.skb_head =
+				gve_rx_copy_data(priv->dev, napi,
+						 rx->dqo.hdr_bufs.data + offset,
+						 hdr_len);
 			if (unlikely(!rx->ctx.skb_head))
 				goto error;
 			rx->ctx.skb_tail = rx->ctx.skb_head;
-- 
2.53.0

Re: [PATCH 6.12.y 1/2] gve: move DQO rx buffer management related code to a new file

[Greg KH]

On Tue, Jul 28, 2026 at 09:20:03AM -0400, Sasha Levin wrote:
> From: Harshitha Ramamurthy <hramamurthy@google.com>
> 
> [ Upstream commit 93c68f1275f9e21ccfed9ee292aedb11c3f6241b ]
> 
> In preparation for the upcoming page pool adoption for DQO
> raw addressing mode, move RX buffer management code to a new
> file. In the follow on patches, page pool code will be added
> to this file.
> 
> No functional change, just movement of code.
> 
> Reviewed-by: Praveen Kaligineedi <pkaligineedi@google.com>
> Reviewed-by: Shailend Chand <shailend@google.com>
> Reviewed-by: Willem de Bruijn <willemb@google.com>
> Signed-off-by: Harshitha Ramamurthy <hramamurthy@google.com>
> Reviewed-by: Jacob Keller <jacob.e.keller@intel.com>
> Link: https://patch.msgid.link/20241014202108.1051963-2-pkaligineedi@google.com
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> Stable-dep-of: d676c9a73bdc ("gve: fix header buffer corruption with header-split and HW-GRO")
> Signed-off-by: Sasha Levin <sashal@kernel.org>
> ---
>  drivers/net/ethernet/google/gve/Makefile      |   3 +-
>  drivers/net/ethernet/google/gve/gve.h         |  18 ++
>  .../ethernet/google/gve/gve_buffer_mgmt_dqo.c | 230 ++++++++++++++++++
>  drivers/net/ethernet/google/gve/gve_rx_dqo.c  | 225 -----------------
>  4 files changed, 250 insertions(+), 226 deletions(-)
>  create mode 100644 drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c
> 
> diff --git a/drivers/net/ethernet/google/gve/Makefile b/drivers/net/ethernet/google/gve/Makefile
> index 9ed07080b38a25..4520f1c07a638f 100644
> --- a/drivers/net/ethernet/google/gve/Makefile
> +++ b/drivers/net/ethernet/google/gve/Makefile
> @@ -1,4 +1,5 @@
>  # Makefile for the Google virtual Ethernet (gve) driver
>  
>  obj-$(CONFIG_GVE) += gve.o
> -gve-objs := gve_main.o gve_tx.o gve_tx_dqo.o gve_rx.o gve_rx_dqo.o gve_ethtool.o gve_adminq.o gve_utils.o gve_flow_rule.o
> +gve-objs := gve_main.o gve_tx.o gve_tx_dqo.o gve_rx.o gve_rx_dqo.o gve_ethtool.o gve_adminq.o gve_utils.o gve_flow_rule.o \
> +	    gve_buffer_mgmt_dqo.o
> diff --git a/drivers/net/ethernet/google/gve/gve.h b/drivers/net/ethernet/google/gve/gve.h
> index 8ddd366d9fde54..99fd933a1f30d4 100644
> --- a/drivers/net/ethernet/google/gve/gve.h
> +++ b/drivers/net/ethernet/google/gve/gve.h
> @@ -1173,6 +1173,24 @@ void gve_rx_stop_ring_gqi(struct gve_priv *priv, int idx);
>  u16 gve_get_pkt_buf_size(const struct gve_priv *priv, bool enable_hplit);
>  bool gve_header_split_supported(const struct gve_priv *priv);
>  int gve_set_hsplit_config(struct gve_priv *priv, u8 tcp_data_split);
> +/* rx buffer handling */
> +int gve_buf_ref_cnt(struct gve_rx_buf_state_dqo *bs);
> +void gve_free_page_dqo(struct gve_priv *priv, struct gve_rx_buf_state_dqo *bs,
> +		       bool free_page);
> +struct gve_rx_buf_state_dqo *gve_alloc_buf_state(struct gve_rx_ring *rx);
> +bool gve_buf_state_is_allocated(struct gve_rx_ring *rx,
> +				struct gve_rx_buf_state_dqo *buf_state);
> +void gve_free_buf_state(struct gve_rx_ring *rx,
> +			struct gve_rx_buf_state_dqo *buf_state);
> +struct gve_rx_buf_state_dqo *gve_dequeue_buf_state(struct gve_rx_ring *rx,
> +						   struct gve_index_list *list);
> +void gve_enqueue_buf_state(struct gve_rx_ring *rx, struct gve_index_list *list,
> +			   struct gve_rx_buf_state_dqo *buf_state);
> +struct gve_rx_buf_state_dqo *gve_get_recycled_buf_state(struct gve_rx_ring *rx);
> +int gve_alloc_page_dqo(struct gve_rx_ring *rx,
> +		       struct gve_rx_buf_state_dqo *buf_state);
> +void gve_try_recycle_buf(struct gve_priv *priv, struct gve_rx_ring *rx,
> +			 struct gve_rx_buf_state_dqo *buf_state);
>  /* Reset */
>  void gve_schedule_reset(struct gve_priv *priv);
>  int gve_reset(struct gve_priv *priv, bool attempt_teardown);
> diff --git a/drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c b/drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c
> new file mode 100644
> index 00000000000000..8e50f0e4bb2ecb
> --- /dev/null
> +++ b/drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c
> @@ -0,0 +1,230 @@
> +// SPDX-License-Identifier: (GPL-2.0 OR MIT)
> +/* Google virtual Ethernet (gve) driver
> + *
> + * Copyright (C) 2015-2024 Google, Inc.
> + */
> +
> +#include "gve.h"
> +#include "gve_utils.h"
> +
> +int gve_buf_ref_cnt(struct gve_rx_buf_state_dqo *bs)
> +{
> +	return page_count(bs->page_info.page) - bs->page_info.pagecnt_bias;
> +}
> +
> +void gve_free_page_dqo(struct gve_priv *priv, struct gve_rx_buf_state_dqo *bs,
> +		       bool free_page)
> +{
> +	page_ref_sub(bs->page_info.page, bs->page_info.pagecnt_bias - 1);
> +	if (free_page)
> +		gve_free_page(&priv->pdev->dev, bs->page_info.page, bs->addr,
> +			      DMA_FROM_DEVICE);
> +	bs->page_info.page = NULL;
> +}
> +
> +struct gve_rx_buf_state_dqo *gve_alloc_buf_state(struct gve_rx_ring *rx)
> +{
> +	struct gve_rx_buf_state_dqo *buf_state;
> +	s16 buffer_id;
> +
> +	buffer_id = rx->dqo.free_buf_states;
> +	if (unlikely(buffer_id == -1))
> +		return NULL;
> +
> +	buf_state = &rx->dqo.buf_states[buffer_id];
> +
> +	/* Remove buf_state from free list */
> +	rx->dqo.free_buf_states = buf_state->next;
> +
> +	/* Point buf_state to itself to mark it as allocated */
> +	buf_state->next = buffer_id;
> +
> +	return buf_state;
> +}
> +
> +bool gve_buf_state_is_allocated(struct gve_rx_ring *rx,
> +				struct gve_rx_buf_state_dqo *buf_state)
> +{
> +	s16 buffer_id = buf_state - rx->dqo.buf_states;
> +
> +	return buf_state->next == buffer_id;
> +}
> +
> +void gve_free_buf_state(struct gve_rx_ring *rx,
> +			struct gve_rx_buf_state_dqo *buf_state)
> +{
> +	s16 buffer_id = buf_state - rx->dqo.buf_states;
> +
> +	buf_state->next = rx->dqo.free_buf_states;
> +	rx->dqo.free_buf_states = buffer_id;
> +}
> +
> +struct gve_rx_buf_state_dqo *gve_dequeue_buf_state(struct gve_rx_ring *rx,
> +						   struct gve_index_list *list)
> +{
> +	struct gve_rx_buf_state_dqo *buf_state;
> +	s16 buffer_id;
> +
> +	buffer_id = list->head;
> +	if (unlikely(buffer_id == -1))
> +		return NULL;
> +
> +	buf_state = &rx->dqo.buf_states[buffer_id];
> +
> +	/* Remove buf_state from list */
> +	list->head = buf_state->next;
> +	if (buf_state->next == -1)
> +		list->tail = -1;
> +
> +	/* Point buf_state to itself to mark it as allocated */
> +	buf_state->next = buffer_id;
> +
> +	return buf_state;
> +}
> +
> +void gve_enqueue_buf_state(struct gve_rx_ring *rx, struct gve_index_list *list,
> +			   struct gve_rx_buf_state_dqo *buf_state)
> +{
> +	s16 buffer_id = buf_state - rx->dqo.buf_states;
> +
> +	buf_state->next = -1;
> +
> +	if (list->head == -1) {
> +		list->head = buffer_id;
> +		list->tail = buffer_id;
> +	} else {
> +		int tail = list->tail;
> +
> +		rx->dqo.buf_states[tail].next = buffer_id;
> +		list->tail = buffer_id;
> +	}
> +}
> +
> +struct gve_rx_buf_state_dqo *gve_get_recycled_buf_state(struct gve_rx_ring *rx)
> +{
> +	struct gve_rx_buf_state_dqo *buf_state;
> +	int i;
> +
> +	/* Recycled buf states are immediately usable. */
> +	buf_state = gve_dequeue_buf_state(rx, &rx->dqo.recycled_buf_states);
> +	if (likely(buf_state))
> +		return buf_state;
> +
> +	if (unlikely(rx->dqo.used_buf_states.head == -1))
> +		return NULL;
> +
> +	/* Used buf states are only usable when ref count reaches 0, which means
> +	 * no SKBs refer to them.
> +	 *
> +	 * Search a limited number before giving up.
> +	 */
> +	for (i = 0; i < 5; i++) {
> +		buf_state = gve_dequeue_buf_state(rx, &rx->dqo.used_buf_states);
> +		if (gve_buf_ref_cnt(buf_state) == 0) {
> +			rx->dqo.used_buf_states_cnt--;
> +			return buf_state;
> +		}
> +
> +		gve_enqueue_buf_state(rx, &rx->dqo.used_buf_states, buf_state);
> +	}
> +
> +	/* For QPL, we cannot allocate any new buffers and must
> +	 * wait for the existing ones to be available.
> +	 */
> +	if (rx->dqo.qpl)
> +		return NULL;
> +
> +	/* If there are no free buf states discard an entry from
> +	 * `used_buf_states` so it can be used.
> +	 */
> +	if (unlikely(rx->dqo.free_buf_states == -1)) {
> +		buf_state = gve_dequeue_buf_state(rx, &rx->dqo.used_buf_states);
> +		if (gve_buf_ref_cnt(buf_state) == 0)
> +			return buf_state;
> +
> +		gve_free_page_dqo(rx->gve, buf_state, true);
> +		gve_free_buf_state(rx, buf_state);
> +	}
> +
> +	return NULL;
> +}
> +
> +int gve_alloc_page_dqo(struct gve_rx_ring *rx,
> +		       struct gve_rx_buf_state_dqo *buf_state)
> +{
> +	struct gve_priv *priv = rx->gve;
> +	u32 idx;
> +
> +	if (!rx->dqo.qpl) {
> +		int err;
> +
> +		err = gve_alloc_page(priv, &priv->pdev->dev,
> +				     &buf_state->page_info.page,
> +				     &buf_state->addr,
> +				     DMA_FROM_DEVICE, GFP_ATOMIC);
> +		if (err)
> +			return err;
> +	} else {
> +		idx = rx->dqo.next_qpl_page_idx;
> +		if (idx >= gve_get_rx_pages_per_qpl_dqo(priv->rx_desc_cnt)) {
> +			net_err_ratelimited("%s: Out of QPL pages\n",
> +					    priv->dev->name);
> +			return -ENOMEM;
> +		}
> +		buf_state->page_info.page = rx->dqo.qpl->pages[idx];
> +		buf_state->addr = rx->dqo.qpl->page_buses[idx];
> +		rx->dqo.next_qpl_page_idx++;
> +	}
> +	buf_state->page_info.page_offset = 0;
> +	buf_state->page_info.page_address =
> +		page_address(buf_state->page_info.page);
> +	buf_state->last_single_ref_offset = 0;
> +
> +	/* The page already has 1 ref. */
> +	page_ref_add(buf_state->page_info.page, INT_MAX - 1);
> +	buf_state->page_info.pagecnt_bias = INT_MAX;
> +
> +	return 0;
> +}
> +
> +void gve_try_recycle_buf(struct gve_priv *priv, struct gve_rx_ring *rx,
> +			 struct gve_rx_buf_state_dqo *buf_state)
> +{
> +	const u16 data_buffer_size = priv->data_buffer_size_dqo;
> +	int pagecount;
> +
> +	/* Can't reuse if we only fit one buffer per page */
> +	if (data_buffer_size * 2 > PAGE_SIZE)
> +		goto mark_used;
> +
> +	pagecount = gve_buf_ref_cnt(buf_state);
> +
> +	/* Record the offset when we have a single remaining reference.
> +	 *
> +	 * When this happens, we know all of the other offsets of the page are
> +	 * usable.
> +	 */
> +	if (pagecount == 1) {
> +		buf_state->last_single_ref_offset =
> +			buf_state->page_info.page_offset;
> +	}
> +
> +	/* Use the next buffer sized chunk in the page. */
> +	buf_state->page_info.page_offset += data_buffer_size;
> +	buf_state->page_info.page_offset &= (PAGE_SIZE - 1);
> +
> +	/* If we wrap around to the same offset without ever dropping to 1
> +	 * reference, then we don't know if this offset was ever freed.
> +	 */
> +	if (buf_state->page_info.page_offset ==
> +	    buf_state->last_single_ref_offset) {
> +		goto mark_used;
> +	}
> +
> +	gve_enqueue_buf_state(rx, &rx->dqo.recycled_buf_states, buf_state);
> +	return;
> +
> +mark_used:
> +	gve_enqueue_buf_state(rx, &rx->dqo.used_buf_states, buf_state);
> +	rx->dqo.used_buf_states_cnt++;
> +}
> diff --git a/drivers/net/ethernet/google/gve/gve_rx_dqo.c b/drivers/net/ethernet/google/gve/gve_rx_dqo.c
> index 1154c1d8f66f05..b343be2fb118ad 100644
> --- a/drivers/net/ethernet/google/gve/gve_rx_dqo.c
> +++ b/drivers/net/ethernet/google/gve/gve_rx_dqo.c
> @@ -16,189 +16,6 @@
>  #include <net/ipv6.h>
>  #include <net/tcp.h>
>  
> -static int gve_buf_ref_cnt(struct gve_rx_buf_state_dqo *bs)
> -{
> -	return page_count(bs->page_info.page) - bs->page_info.pagecnt_bias;
> -}
> -
> -static void gve_free_page_dqo(struct gve_priv *priv,
> -			      struct gve_rx_buf_state_dqo *bs,
> -			      bool free_page)
> -{
> -	page_ref_sub(bs->page_info.page, bs->page_info.pagecnt_bias - 1);
> -	if (free_page)
> -		gve_free_page(&priv->pdev->dev, bs->page_info.page, bs->addr,
> -			      DMA_FROM_DEVICE);
> -	bs->page_info.page = NULL;
> -}
> -
> -static struct gve_rx_buf_state_dqo *gve_alloc_buf_state(struct gve_rx_ring *rx)
> -{
> -	struct gve_rx_buf_state_dqo *buf_state;
> -	s16 buffer_id;
> -
> -	buffer_id = rx->dqo.free_buf_states;
> -	if (unlikely(buffer_id == -1))
> -		return NULL;
> -
> -	buf_state = &rx->dqo.buf_states[buffer_id];
> -
> -	/* Remove buf_state from free list */
> -	rx->dqo.free_buf_states = buf_state->next;
> -
> -	/* Point buf_state to itself to mark it as allocated */
> -	buf_state->next = buffer_id;
> -
> -	return buf_state;
> -}
> -
> -static bool gve_buf_state_is_allocated(struct gve_rx_ring *rx,
> -				       struct gve_rx_buf_state_dqo *buf_state)
> -{
> -	s16 buffer_id = buf_state - rx->dqo.buf_states;
> -
> -	return buf_state->next == buffer_id;
> -}
> -
> -static void gve_free_buf_state(struct gve_rx_ring *rx,
> -			       struct gve_rx_buf_state_dqo *buf_state)
> -{
> -	s16 buffer_id = buf_state - rx->dqo.buf_states;
> -
> -	buf_state->next = rx->dqo.free_buf_states;
> -	rx->dqo.free_buf_states = buffer_id;
> -}
> -
> -static struct gve_rx_buf_state_dqo *
> -gve_dequeue_buf_state(struct gve_rx_ring *rx, struct gve_index_list *list)
> -{
> -	struct gve_rx_buf_state_dqo *buf_state;
> -	s16 buffer_id;
> -
> -	buffer_id = list->head;
> -	if (unlikely(buffer_id == -1))
> -		return NULL;
> -
> -	buf_state = &rx->dqo.buf_states[buffer_id];
> -
> -	/* Remove buf_state from list */
> -	list->head = buf_state->next;
> -	if (buf_state->next == -1)
> -		list->tail = -1;
> -
> -	/* Point buf_state to itself to mark it as allocated */
> -	buf_state->next = buffer_id;
> -
> -	return buf_state;
> -}
> -
> -static void gve_enqueue_buf_state(struct gve_rx_ring *rx,
> -				  struct gve_index_list *list,
> -				  struct gve_rx_buf_state_dqo *buf_state)
> -{
> -	s16 buffer_id = buf_state - rx->dqo.buf_states;
> -
> -	buf_state->next = -1;
> -
> -	if (list->head == -1) {
> -		list->head = buffer_id;
> -		list->tail = buffer_id;
> -	} else {
> -		int tail = list->tail;
> -
> -		rx->dqo.buf_states[tail].next = buffer_id;
> -		list->tail = buffer_id;
> -	}
> -}
> -
> -static struct gve_rx_buf_state_dqo *
> -gve_get_recycled_buf_state(struct gve_rx_ring *rx)
> -{
> -	struct gve_rx_buf_state_dqo *buf_state;
> -	int i;
> -
> -	/* Recycled buf states are immediately usable. */
> -	buf_state = gve_dequeue_buf_state(rx, &rx->dqo.recycled_buf_states);
> -	if (likely(buf_state))
> -		return buf_state;
> -
> -	if (unlikely(rx->dqo.used_buf_states.head == -1))
> -		return NULL;
> -
> -	/* Used buf states are only usable when ref count reaches 0, which means
> -	 * no SKBs refer to them.
> -	 *
> -	 * Search a limited number before giving up.
> -	 */
> -	for (i = 0; i < 5; i++) {
> -		buf_state = gve_dequeue_buf_state(rx, &rx->dqo.used_buf_states);
> -		if (gve_buf_ref_cnt(buf_state) == 0) {
> -			rx->dqo.used_buf_states_cnt--;
> -			return buf_state;
> -		}
> -
> -		gve_enqueue_buf_state(rx, &rx->dqo.used_buf_states, buf_state);
> -	}
> -
> -	/* For QPL, we cannot allocate any new buffers and must
> -	 * wait for the existing ones to be available.
> -	 */
> -	if (rx->dqo.qpl)
> -		return NULL;
> -
> -	/* If there are no free buf states discard an entry from
> -	 * `used_buf_states` so it can be used.
> -	 */
> -	if (unlikely(rx->dqo.free_buf_states == -1)) {
> -		buf_state = gve_dequeue_buf_state(rx, &rx->dqo.used_buf_states);
> -		if (gve_buf_ref_cnt(buf_state) == 0)
> -			return buf_state;
> -
> -		gve_free_page_dqo(rx->gve, buf_state, true);
> -		gve_free_buf_state(rx, buf_state);
> -	}
> -
> -	return NULL;
> -}
> -
> -static int gve_alloc_page_dqo(struct gve_rx_ring *rx,
> -			      struct gve_rx_buf_state_dqo *buf_state)
> -{
> -	struct gve_priv *priv = rx->gve;
> -	u32 idx;
> -
> -	if (!rx->dqo.qpl) {
> -		int err;
> -
> -		err = gve_alloc_page(priv, &priv->pdev->dev,
> -				     &buf_state->page_info.page,
> -				     &buf_state->addr,
> -				     DMA_FROM_DEVICE, GFP_ATOMIC);
> -		if (err)
> -			return err;
> -	} else {
> -		idx = rx->dqo.next_qpl_page_idx;
> -		if (idx >= gve_get_rx_pages_per_qpl_dqo(priv->rx_desc_cnt)) {
> -			net_err_ratelimited("%s: Out of QPL pages\n",
> -					    priv->dev->name);
> -			return -ENOMEM;
> -		}
> -		buf_state->page_info.page = rx->dqo.qpl->pages[idx];
> -		buf_state->addr = rx->dqo.qpl->page_buses[idx];
> -		rx->dqo.next_qpl_page_idx++;
> -	}
> -	buf_state->page_info.page_offset = 0;
> -	buf_state->page_info.page_address =
> -		page_address(buf_state->page_info.page);
> -	buf_state->last_single_ref_offset = 0;
> -
> -	/* The page already has 1 ref. */
> -	page_ref_add(buf_state->page_info.page, INT_MAX - 1);
> -	buf_state->page_info.pagecnt_bias = INT_MAX;
> -
> -	return 0;
> -}
> -
>  static void gve_rx_free_hdr_bufs(struct gve_priv *priv, struct gve_rx_ring *rx)
>  {
>  	struct device *hdev = &priv->pdev->dev;
> @@ -557,48 +374,6 @@ void gve_rx_post_buffers_dqo(struct gve_rx_ring *rx)
>  	rx->fill_cnt += num_posted;
>  }
>  
> -static void gve_try_recycle_buf(struct gve_priv *priv, struct gve_rx_ring *rx,
> -				struct gve_rx_buf_state_dqo *buf_state)
> -{
> -	const u16 data_buffer_size = priv->data_buffer_size_dqo;
> -	int pagecount;
> -
> -	/* Can't reuse if we only fit one buffer per page */
> -	if (data_buffer_size * 2 > PAGE_SIZE)
> -		goto mark_used;
> -
> -	pagecount = gve_buf_ref_cnt(buf_state);
> -
> -	/* Record the offset when we have a single remaining reference.
> -	 *
> -	 * When this happens, we know all of the other offsets of the page are
> -	 * usable.
> -	 */
> -	if (pagecount == 1) {
> -		buf_state->last_single_ref_offset =
> -			buf_state->page_info.page_offset;
> -	}
> -
> -	/* Use the next buffer sized chunk in the page. */
> -	buf_state->page_info.page_offset += data_buffer_size;
> -	buf_state->page_info.page_offset &= (PAGE_SIZE - 1);
> -
> -	/* If we wrap around to the same offset without ever dropping to 1
> -	 * reference, then we don't know if this offset was ever freed.
> -	 */
> -	if (buf_state->page_info.page_offset ==
> -	    buf_state->last_single_ref_offset) {
> -		goto mark_used;
> -	}
> -
> -	gve_enqueue_buf_state(rx, &rx->dqo.recycled_buf_states, buf_state);
> -	return;
> -
> -mark_used:
> -	gve_enqueue_buf_state(rx, &rx->dqo.used_buf_states, buf_state);
> -	rx->dqo.used_buf_states_cnt++;
> -}
> -
>  static void gve_rx_skb_csum(struct sk_buff *skb,
>  			    const struct gve_rx_compl_desc_dqo *desc,
>  			    struct gve_ptype ptype)
> -- 
> 2.53.0
> 
> 

Did not apply, and patch 2/2 worked on it's own.