All of lore.kernel.org
 help / color / mirror / Atom feed
From: Stephen Hemminger <stephen@networkplumber.org>
To: Ryan Wolfe <rnwolfe@google.com>
Cc: dev@dpdk.org, Junfeng Guo <junfeng.guo@intel.com>,
	Joshua Washington <joshwash@google.com>,
	stable@dpdk.org, Harshitha Ramamurthy <hramamurthy@google.com>,
	Rushil Gupta <rushilg@google.com>
Subject: Re: [PATCH] net/gve: fix GQI QPL jumbo Tx FIFO reclaim and doorbell
Date: Tue, 17 Nov 2026 09:34:22 -0800	[thread overview]
Message-ID: <20261117093422.752cc7ad@phoenix.local> (raw)
In-Reply-To: <20261115140016.3830109-1-rnwolfe@google.com>

On Sun, 15 Nov 2026 09:00:16 -0500
Ryan Wolfe <rnwolfe@google.com> wrote:

> In GQI Queue Page List (GQI_QPL) mode each Tx queue copies outgoing
> frames into a fixed QPL bounce FIFO (512 pages = 2 MiB on GCE, ~236
> 8,832-byte cacheline-aligned jumbo frames) even when nb_tx_desc is
> configured to 2048. Under sustained jumbo bursts three bugs cause
> unnecessary packet drops and mbuf tracking hazards:
> 
> 1. Entry cleanup in gve_tx_burst_qpl() only ran when nb_free < free_thresh
>    (32) or fifo_avail == 0. With 2048 descriptors, 236 jumbo frames leave
>    nb_free = 1811 and non-zero remainder bytes in fifo_avail, so entry
>    cleanup never ran and every QPL wrap stalled mid-burst. Reclaim when
>    fifo_avail drops below half of fifo_size, and do not count normal
>    FIFO/ring-full backpressure as hardware stats.errors.
> 2. When is_fifo_avail() failed mid-burst after queuing packets, the driver
>    polled gve_tx_clean() once before writing qtx_tail, so the device had
>    not yet started draining the newly queued frames, and a single
>    GVE_TX_MAX_FREE_SZ batch after many small frames could stop short of
>    freeing a jumbo slot. Flush qtx_tail first and loop gve_tx_clean()
>    while next_to_clean advances.
> 3. sw_ring[sw_id] and the first TSO header slice were allocated/written
>    before verifying total FIFO space, gve_free_bulk_mbuf() did not clear
>    the first freed slot, and gve_tx_burst_ra() skipped mid-burst ring
>    cleanup and wrote pkt_len instead of data_len into the first segment
>    descriptor of multi-segment non-TSO packets.
> 
> Fixes: a46583cf43c8 ("net/gve: support Rx/Tx")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Ryan Wolfe <rnwolfe@google.com>
> ---

The AI review spotted some changes needed.

[PATCH] net/gve: fix GQI QPL jumbo Tx FIFO reclaim and doorbell

The FIFO fixes look right. Checking the TSO header and payload
together before allocating also fixes a real leak: the old code
allocated the header slice, then hit goto end_of_tx when the payload
check failed. iov_ring[tx_id] was overwritten on the next burst, so
that FIFO space was never returned to fifo_avail. The reclaim helper
rings the doorbell before polling, which is the right order.

Error

TSO packets with pkt_len <= hlen now goto end_of_tx. That is a
per-packet validation failure handled as backpressure. The app
retries the same mbuf forever and the queue stalls. Drop the packet
and move on:

	if (ol_flags & RTE_MBUF_F_TX_TCP_SEG) {
		uint32_t data_len, need;

		if (unlikely(tx_pkt->pkt_len <= hlen)) {
			txq->stats.errors++;
			rte_pktmbuf_free(tx_pkt);
			nb_drop++;
			continue;
		}

Leave nb_drop out of stats.packets. A better long-term fix is a
tx_pkt_prepare callback that rejects these before tx_burst, since
gve has none today.

Warning

The commit message does not match the code. Item 3 says
sw_ring[sw_id] is written before the FIFO check, but commit
1299806a ("net/gve: delay adding mbuf head to software ring")
already moved it. This patch does not touch that ordering. Drop the
claim.

This patch makes six separate changes to two Tx paths, under one
Fixes tag, and is marked for stable:
  - gve_free_bulk_mbuf() clears slot s
  - QPL FIFO reclaim, doorbell flush and wider is_fifo_avail()
  - QPL TSO check before allocating
  - RA: clean the ring mid-burst
  - RA: use data_len for the first descriptor
  - stats.errors removal in both paths
Split it into at least three patches: the free_bulk fix, the QPL
FIFO changes, and the RA fixes. Give each one a Fixes: tag pointing
at the commit that added that bug. Not all of them date back to
a46583cf43c8. Splitting lets stable maintainers backport each fix
on its own.

Say in the commit message why the free_bulk fix matters. In RA
mode a TSO packet uses two descriptors, but sw_ring[] is set only
for the first one. The second slot can still hold a stale pointer
from a previous wrap. The old code did not clear slot s, or earlier
slots whose refcount was still held, so gve_tx_clean() could free
that stale mbuf a second time. That is a double free, not a vague
"mbuf tracking hazard". The RA data_len fix is the same kind of
bug: the device was told to read pkt_len bytes from the first
segment, past the end of that segment. Both deserve their own
patch and a clear description.

The commit message is long and reads like generated text. Make it
short and factual, and say how this was tested: which instance
type, which queue mode, and the traffic used to reproduce.

Info

Use for (;;) rather than do { } while (true) in
gve_tx_reclaim_fifo_qpl(). Add a blank line between it and
gve_tx_alloc_from_fifo().

      reply	other threads:[~2026-11-17 17:34 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-15 14:00 [PATCH] net/gve: fix GQI QPL jumbo Tx FIFO reclaim and doorbell Ryan Wolfe
2026-11-17 17:34 ` Stephen Hemminger [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261117093422.752cc7ad@phoenix.local \
    --to=stephen@networkplumber.org \
    --cc=dev@dpdk.org \
    --cc=hramamurthy@google.com \
    --cc=joshwash@google.com \
    --cc=junfeng.guo@intel.com \
    --cc=rnwolfe@google.com \
    --cc=rushilg@google.com \
    --cc=stable@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.