[PATCH net-next v14 0/5] tun/tap & vhost-net: apply qdisc backpressure on full ptr_ring to reduce TX drops

[PATCH net-next v14 0/5] tun/tap & vhost-net: apply qdisc backpressure on full ptr_ring to reduce TX drops

[Simon Schippers]

This patch series deals with tun/tap & vhost-net which drop incoming
SKBs whenever their internal ptr_ring buffer is full. Instead, with this
patch series, the associated netdev queue is stopped, but only when the
new IFF_BACKPRESSURE flag is set and a qdisc is attached. Without the
flag, or if no qdisc is present, the existing behavior is preserved. The
XDP transmit path is not affected. This patch series touches tun/tap and
vhost-net, as they share common logic and must be updated together.
Modifying only one of them would break the other.

By applying proper backpressure, this change allows the connected qdisc to
operate correctly, as reported in [1], and significantly improves
performance in real-world scenarios, as demonstrated in our paper [2]. For
example, we observed a 36% TCP throughput improvement for an OpenVPN
connection between Germany and the USA.

The previous version of this work was applied and then reverted in 7.2,
because the backpressure was unconditional: it caused a significant
throughput drop in an IPv6 multicast testcase with multiple iperf3 TCP
threads sending on Brett Sheffield's librecast testbed [3]. This version
therefore makes the behavior opt-in via IFF_BACKPRESSURE from the very
first patch, so that a tun/tap device which does not set the flag behaves
exactly as before.

The series is ordered so that no patch changes how packets are handled
unless the flag is set:

- Patch 1 adds the flag only. It has no effect yet: TUNSETIFF silently
  masks it off, as it does for any flag outside TUN_FEATURES, until
  patch 5 adds it there.
- Patches 2 and 3 add the consumer side, which wakes a stopped netdev
  queue. __tun_wake_queue() returns early unless IFF_BACKPRESSURE is set,
  and no queue is stopped at this point anyway.
- Patch 4 is a pure ptr_ring refactor required by patch 5.
- Patch 5 adds the queue stopping, gated on IFF_BACKPRESSURE, together
  with the wake needed when the flag is cleared again, and only there is
  the flag added to TUN_FEATURES.

That way no intermediate commit changes the behavior of an existing
tun/tap user beyond the added checks, and bisecting inside the series can
not hit the regression that led to the revert.

Thanks!

[1] https://unix.stackexchange.com/questions/762935/traffic-shaping-ineffective-on-tun-device
[2] https://cni.etit.tu-dortmund.de/storages/cni-etit/r/Research/Publications/2025/Gebauer_2025_VTCFall/Gebauer_VTCFall2025_AuthorsVersion.pdf
[3] https://lore.kernel.org/netdev/akVnoOYQOrt8k-Gu@karahi.librecast.net/

---
Changelog:
v14:
Addressing the regression found by Sashiko:
- Move IFF_BACKPRESSURE from TUN_FEATURES in patch 1 to patch 5, so that
  TUNSETIFF only honours the flag once the implementation is complete.
- Patch 2: move the queue wake for the ntfile taking over the slot in
  __tun_detach() after synchronize_net() and tun_queue_purge(). Waking
  before the grace period could leave the subqueue stopped with no
  consumer left to wake it.
- Patch 2: only wake a subqueue if the tfile still owns its slot in
  tun->tfiles[], as a detached tfile keeps its queue_index.
- Patch 2: every place that wakes a queue now checks netif_running()
  under a ring lock that tun_net_close() takes, so none of them can undo
  its stop. tun_net_close() takes and releases both ring locks of every
  tfile before it stops the queues, so no wake can still be running.
- Patches 2 and 3: enforce the consumer_lock precondition with
  lockdep_assert_held() and document it on the declaration.
- Patch 4: document the return values of __ptr_ring_check_produce().
- Patch 5: use a separate ntfile in the tun_set_iff() wake loop.

v13: https://lore.kernel.org/netdev/20260730213639.726381-1-simon.schippers@tu-dortmund.de/
--> Apart from the three items below, the resulting code is identical to
    v12 plus the separate IFF_BACKPRESSURE patch [4]:
- tuntap.rst does not name a kernel version anymore.
- Add the IFF_BACKPRESSURE comment to tools/include/uapi/linux/if_tun.h
  as well (Sashiko).
- Add a comment explaining the __ptr_ring_empty() check in
  __tun_detach() (MST).

- Include the IFF_BACKPRESSURE opt-in into the series instead of adding it
  as a follow-up fix, so that the series can not be bisected into the
  regression that got v12 reverted.
--> New patch 1 adds the flag on its own.
- Patch 2 carries the IFF_BACKPRESSURE check in __tun_wake_queue() from
  the start and introduces tun_force_wake_queue(), which is used by
  tun_attach() and tun_queue_resize().
- Patch 5 gates the queue stopping on IFF_BACKPRESSURE and calls
  tun_force_wake_queue() for the attached tfiles in tun_set_iff(), so
  clearing the flag can not leave a queue stopped.
- Dropped the pktgen numbers from patches 2 and 3, as they only measured
  the consumer side without any queue stopping.

[4] https://lore.kernel.org/netdev/20260709095511.168235-1-simon.schippers@tu-dortmund.de/

v12: https://lore.kernel.org/netdev/20260510151529.43895-1-simon.schippers@tu-dortmund.de/
Patch 1:
- Revert tun_queue_purge() to plain ptr_ring_consume() and instead
  explicitly wake the queue in __tun_detach() for the ntfile taking
  over the queue slot (if its ring is empty).
- Inlined tun_reset_cons_cnt(), because only tun_attach() uses it.

- Patches 2-4 and cover letter unchanged.
- Compiled and short pktgen test.

v11:
- Renamed __ptr_ring_produce_peek() to __ptr_ring_check_produce()
  (Sashiko)
- Add return code -EINVAL to __ptr_ring_check_produce() which lets
  tun_net_xmit() stop the queue only on -ENOSPC. (MST)
- Resolve race on tfile->queue_index by locking tx_ring.consumer_lock
  in __tun_detach(). (Sashiko)
- Wake the queue in tun_queue_resize() to avoid possible stalls.
- Other minor adjustments & reran the benchmarks.

v10: https://lore.kernel.org/netdev/20260506141033.180450-1-simon.schippers@tu-dortmund.de/
- Changed the term "Transmitted" to "Received" in the benchmarks,
  as correctly pointed out by MST, and reran the benchmarks.

Addressed the Sashiko AI review:
- Avoid a data race on tfile->cons_cnt by always locking.
- Correctly count the number of consumed packets for vhost-net.
- Corrected a typo in the commit message of commit 3.
- Added a missing barrier on the consumer side.
--> The barriers now follow the "store buffering" principle.
- No longer return NETDEV_TX_BUSY at all, because it is unsafe.
--> Result: There are still a few drops with multiple senders, which
            would be avoided by disabling LLTX.

V9: https://lore.kernel.org/netdev/20260428123859.19578-1-simon.schippers@tu-dortmund.de/
- Addressed minor nit by MST in patches 1 and 2.
- Rebased patch 3 because of commit d748047
  ("ptr_ring: disable KCSAN warnings").
- Documented the pair of the smp_mb__after_atomic() in tun_net_xmit()
  with tun_ring_consume().
  --> It simply pairs with the test_and_clear_bit() inside of
      netif_wake_subqueue().
- Use 1 ptr_ring consumer spinlock instead of 2.
- Ran pktgen benchmarks with pg_set SHARED for 50 iterations on
  latest kernel
  --> No significant performance difference noticed

V8: https://lore.kernel.org/netdev/20260312130639.138988-1-simon.schippers@tu-dortmund.de/
- Drop code changes in drivers/net/tap.c; The code there deals with
  ipvtap/macvtap which are unrelated to the goal of this patch series
  and I did not realize that before
-> Greatly simplified logic, 4 instead of 9 commits
-> No more duplicated logics and distinction in vhost required
- Only wake after the queue stopped and half of the ring was consumed
  as suggested by MST
-> Performance improvements for TAP, but still slightly slower
- Better benchmarking with pinned threads, XDP drop program for
  tap+vhost-net and disabling CPU mitigations (and newer Ryzen 5 5600X
  processor) as suggested by Jason Wang

V7: https://lore.kernel.org/netdev/20260107210448.37851-1-simon.schippers@tu-dortmund.de/
- Switch to an approach similar to veth (excluding the recently fixed
variant), as suggested by MST, with minor adjustments discussed in V6
- Rename the cover-letter title
- Add multithreaded pktgen and iperf3 benchmarks, as suggested by Jason
Wang
- Rework __ptr_ring_consume_created_space() so it can also be used after
batched consume

...

Simon Schippers (5):
  tun/tap: add IFF_BACKPRESSURE flag
  tun/tap: add ptr_ring consume helper with netdev queue wakeup
  vhost-net: wake queue of tun/tap after ptr_ring consume
  ptr_ring: move free-space check into separate helper
  tun/tap & vhost-net: stop tail-drop when IFF_BACKPRESSURE is set

 Documentation/networking/tuntap.rst |  32 +++++
 drivers/net/tun.c                   | 182 ++++++++++++++++++++++++++--
 drivers/vhost/net.c                 |  21 +++-
 include/linux/if_tun.h              |   4 +
 include/linux/ptr_ring.h            |  26 +++-
 include/uapi/linux/if_tun.h         |   4 +
 tools/include/uapi/linux/if_tun.h   |   4 +
 7 files changed, 255 insertions(+), 18 deletions(-)


base-commit: 69963a0678a347d57c4ac8b16939dba216eb95ce
-- 
2.43.0

[PATCH net-next v14 1/5] tun/tap: add IFF_BACKPRESSURE flag

[Simon Schippers]

Add the IFF_BACKPRESSURE flag to the UAPI header and to its tools/ copy.
The flag has no effect yet, it is the opt-in switch for the qdisc
backpressure logic added by the following patches.

It is added to TUN_FEATURES only in the last patch of the series, once the
implementation is complete. Until then TUNSETIFF silently masks it off, as
it does for any flag outside TUN_FEATURES.

Keeping the flag and its users in separate patches would either leave a
window where backpressure is unconditional, or make the opt-in a later
add-on. Adding the flag first lets every following patch be a no-op
unless it is set.

Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
---
 include/uapi/linux/if_tun.h       | 4 ++++
 tools/include/uapi/linux/if_tun.h | 4 ++++
 2 files changed, 8 insertions(+)

diff --git a/include/uapi/linux/if_tun.h b/include/uapi/linux/if_tun.h
index 79d53c7a1ebd..a0ddc50a7534 100644
--- a/include/uapi/linux/if_tun.h
+++ b/include/uapi/linux/if_tun.h
@@ -69,6 +69,10 @@
 #define IFF_NAPI_FRAGS	0x0020
 /* Used in TUNSETIFF to bring up tun/tap without carrier */
 #define IFF_NO_CARRIER	0x0040
+/* Stop the queue instead of dropping when the internal ring is full, so an
+ * attached qdisc applies backpressure instead of being bypassed.
+ */
+#define IFF_BACKPRESSURE	0x0080
 #define IFF_NO_PI	0x1000
 /* This flag has no real effect */
 #define IFF_ONE_QUEUE	0x2000
diff --git a/tools/include/uapi/linux/if_tun.h b/tools/include/uapi/linux/if_tun.h
index 2ec07de1d73b..2c85525704c1 100644
--- a/tools/include/uapi/linux/if_tun.h
+++ b/tools/include/uapi/linux/if_tun.h
@@ -67,6 +67,10 @@
 #define IFF_TAP		0x0002
 #define IFF_NAPI	0x0010
 #define IFF_NAPI_FRAGS	0x0020
+/* Stop the queue instead of dropping when the internal ring is full, so an
+ * attached qdisc applies backpressure instead of being bypassed.
+ */
+#define IFF_BACKPRESSURE	0x0080
 #define IFF_NO_PI	0x1000
 /* This flag has no real effect */
 #define IFF_ONE_QUEUE	0x2000
-- 
2.43.0

[PATCH net-next v14 2/5] tun/tap: add ptr_ring consume helper with netdev queue wakeup

[Simon Schippers]

Introduce tun_ring_consume() that wraps ptr_ring_consume() and calls
__tun_wake_queue(). The latter wakes the stopped netdev subqueue once
half of the ring capacity has been consumed, tracked via the new
cons_cnt field in tun_file. As a safety net, the queue is also woken on
the last consumed entry if it leaves the ring empty. The point is to
allow the queue to be stopped when it gets full, which is required for
traffic shaping, implemented by the following "stop tail-drop when
IFF_BACKPRESSURE is set".

__tun_wake_queue() returns early unless IFF_BACKPRESSURE is set, so for a
tun/tap device that does not opt in only the added check on the consume
path remains.

Every site that clears __QUEUE_STATE_DRV_XOFF now checks netif_running()
under a ring lock that tun_net_close() takes, so that none of them undoes
its stop. The core sets it before it calls ndo_open() and clears it
before it calls ndo_stop(), so it is false for exactly as long as the
device is down. IFF_UP would not do, it is only cleared after ndo_stop()
returns.

Some implementation details:
- tun_ring_recv() replaces ptr_ring_consume() with tun_ring_consume()
  to properly wake the queue.
- __tun_wake_queue() returns early for a device that is not running, so a
  stop from tun_net_close() is not mistaken for backpressure, and it only
  wakes if the tfile still owns its slot in tun->tfiles[]. A detached
  tfile keeps its queue_index, which __tun_detach() may already have
  handed to the tfile that took over the slot.
- lockdep_assert_held() enforces the documented consumer_lock
  precondition of __tun_wake_queue().
- __tun_detach() locks the tx_ring.consumer_lock to avoid races with
  the consumer on the queue_index, and that of tfile across the hand-over
  of the slot, which makes the ownership check above exact.
- The ptr_ring_consume() call in tun_queue_purge() is not replaced with
  tun_ring_consume(). Instead __tun_detach() wakes the netdev queue for
  the ntfile taking it over, to avoid a possible stall. The queue is only
  woken if the ring of the ntfile is empty, as otherwise the consumer
  wakes it after consuming the remaining entries. This does not matter
  for tun_detach_all(), as it is called during device teardown and no
  tfile takes over any queue.
- That wake sits after synchronize_net() and tun_queue_purge(), so it can
  not be undone by a concurrent tun_net_xmit() or __tun_wake_queue().
- Ensure detached queues are woken on re-attach by calling the new
  tun_force_wake_queue() helper from tun_attach(), and reuse it across
  the existing wake paths. Unlike __tun_wake_queue() it ignores
  IFF_BACKPRESSURE, so a queue can not stay stopped after the flag is
  cleared. It does honour netif_running(), but it always clears cons_cnt,
  so no old count is left over when the queue is stopped again.
- tun_net_close() takes and releases both ring locks of every tfile
  before netif_tx_stop_all_queues(), so that its stop is the last write
  to __QUEUE_STATE_DRV_XOFF.
- The aforementioned upcoming patch explains the pairing of the smp_mb()
  of __tun_wake_queue().

Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
---
 drivers/net/tun.c | 118 ++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 114 insertions(+), 4 deletions(-)

diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index 51e80000bd0e..d49b6bfd104d 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -145,6 +145,8 @@ struct tun_file {
 	struct list_head next;
 	struct tun_struct *detached;
 	struct ptr_ring tx_ring;
+	/* Protected by tx_ring.consumer_lock */
+	int cons_cnt;
 	struct xdp_rxq_info xdp_rxq;
 };
 
@@ -585,11 +587,16 @@ static void __tun_detach(struct tun_file *tfile, bool clean)
 		u16 index = tfile->queue_index;
 		BUG_ON(index >= tun->numqueues);
 
+		spin_lock(&tfile->tx_ring.consumer_lock);
 		rcu_assign_pointer(tun->tfiles[index],
 				   tun->tfiles[tun->numqueues - 1]);
+		spin_unlock(&tfile->tx_ring.consumer_lock);
 		ntfile = rtnl_dereference(tun->tfiles[index]);
+		spin_lock(&ntfile->tx_ring.consumer_lock);
 		ntfile->queue_index = index;
 		ntfile->xdp_rxq.queue_index = index;
+		ntfile->cons_cnt = 0;
+		spin_unlock(&ntfile->tx_ring.consumer_lock);
 		rcu_assign_pointer(tun->tfiles[tun->numqueues - 1],
 				   NULL);
 
@@ -606,6 +613,14 @@ static void __tun_detach(struct tun_file *tfile, bool clean)
 		tun_flow_delete_by_queue(tun, tun->numqueues + 1);
 		/* Drop read queue */
 		tun_queue_purge(tfile);
+		spin_lock_bh(&ntfile->tx_ring.consumer_lock);
+		spin_lock(&ntfile->tx_ring.producer_lock);
+		ntfile->cons_cnt = 0;
+		if (netif_running(tun->dev) &&
+		    __ptr_ring_empty(&ntfile->tx_ring))
+			netif_wake_subqueue(tun->dev, index);
+		spin_unlock(&ntfile->tx_ring.producer_lock);
+		spin_unlock_bh(&ntfile->tx_ring.consumer_lock);
 		tun_set_real_num_queues(tun);
 	} else if (tfile->detached && clean) {
 		tun = tun_enable_queue(tfile);
@@ -687,6 +702,25 @@ static void tun_detach_all(struct net_device *dev)
 		module_put(THIS_MODULE);
 }
 
+static void tun_force_wake_queue(struct tun_struct *tun,
+				 struct tun_file *tfile)
+{
+	/* Ensure that the producer can not stop the
+	 * queue concurrently by taking locks.
+	 */
+	spin_lock_bh(&tfile->tx_ring.consumer_lock);
+	spin_lock(&tfile->tx_ring.producer_lock);
+	tfile->cons_cnt = 0;
+	/* Tested under the locks that tun_net_close() takes, so this can not
+	 * undo its stop. tun_net_open() wakes the queues of a device that
+	 * comes back up.
+	 */
+	if (netif_running(tun->dev))
+		netif_wake_subqueue(tun->dev, tfile->queue_index);
+	spin_unlock(&tfile->tx_ring.producer_lock);
+	spin_unlock_bh(&tfile->tx_ring.consumer_lock);
+}
+
 static int tun_attach(struct tun_struct *tun, struct file *file,
 		      bool skip_filter, bool napi, bool napi_frags,
 		      bool publish_tun)
@@ -730,8 +764,11 @@ static int tun_attach(struct tun_struct *tun, struct file *file,
 		goto out;
 	}
 
+	spin_lock(&tfile->tx_ring.consumer_lock);
 	tfile->queue_index = tun->numqueues;
+	spin_unlock(&tfile->tx_ring.consumer_lock);
 	tfile->socket.sk->sk_shutdown &= ~RCV_SHUTDOWN;
+	tun_force_wake_queue(tun, tfile);
 
 	if (tfile->detached) {
 		/* Re-attach detached tfile, updating XDP queue_index */
@@ -964,6 +1001,24 @@ static int tun_net_open(struct net_device *dev)
 /* Net device close. */
 static int tun_net_close(struct net_device *dev)
 {
+	struct tun_struct *tun = netdev_priv(dev);
+	struct tun_file *tfile;
+	int i;
+
+	/* netif_running() is already false: take both ring locks to keep the
+	 * wake sites out, so the stop below is the last write to
+	 * __QUEUE_STATE_DRV_XOFF.
+	 */
+	for (i = 0; i < tun->numqueues; i++) {
+		tfile = rtnl_dereference(tun->tfiles[i]);
+
+		spin_lock_bh(&tfile->tx_ring.consumer_lock);
+		spin_lock(&tfile->tx_ring.producer_lock);
+		tfile->cons_cnt = 0;
+		spin_unlock(&tfile->tx_ring.producer_lock);
+		spin_unlock_bh(&tfile->tx_ring.consumer_lock);
+	}
+
 	netif_tx_stop_all_queues(dev);
 	return 0;
 }
@@ -2116,13 +2171,61 @@ static ssize_t tun_put_user(struct tun_struct *tun,
 	return total;
 }
 
-static void *tun_ring_recv(struct tun_file *tfile, int noblock, int *err)
+/* Callers must hold ring.consumer_lock */
+static void __tun_wake_queue(struct tun_struct *tun,
+			     struct tun_file *tfile, int consumed)
+{
+	u16 queue_index = tfile->queue_index;
+	struct netdev_queue *txq;
+
+	lockdep_assert_held(&tfile->tx_ring.consumer_lock);
+
+	if (!(tun->flags & IFF_BACKPRESSURE))
+		return;
+
+	/* A stop from tun_net_close() is not backpressure, leave it alone. */
+	if (unlikely(!netif_running(tun->dev)))
+		return;
+
+	/* Only the current owner of the slot may wake its subqueue. */
+	if (unlikely(rcu_access_pointer(tun->tfiles[queue_index]) != tfile))
+		return;
+
+	txq = netdev_get_tx_queue(tun->dev, queue_index);
+
+	/* Paired with smp_mb__after_atomic() in tun_net_xmit() */
+	smp_mb();
+	if (netif_tx_queue_stopped(txq)) {
+		tfile->cons_cnt += consumed;
+		if (tfile->cons_cnt >= tfile->tx_ring.size / 2 ||
+		    __ptr_ring_empty(&tfile->tx_ring)) {
+			netif_tx_wake_queue(txq);
+			tfile->cons_cnt = 0;
+		}
+	}
+}
+
+static void *tun_ring_consume(struct tun_struct *tun, struct tun_file *tfile)
+{
+	void *ptr;
+
+	spin_lock(&tfile->tx_ring.consumer_lock);
+	ptr = __ptr_ring_consume(&tfile->tx_ring);
+	if (ptr)
+		__tun_wake_queue(tun, tfile, 1);
+
+	spin_unlock(&tfile->tx_ring.consumer_lock);
+	return ptr;
+}
+
+static void *tun_ring_recv(struct tun_struct *tun, struct tun_file *tfile,
+			   int noblock, int *err)
 {
 	DECLARE_WAITQUEUE(wait, current);
 	void *ptr = NULL;
 	int error = 0;
 
-	ptr = ptr_ring_consume(&tfile->tx_ring);
+	ptr = tun_ring_consume(tun, tfile);
 	if (ptr)
 		goto out;
 	if (noblock) {
@@ -2134,7 +2237,7 @@ static void *tun_ring_recv(struct tun_file *tfile, int noblock, int *err)
 
 	while (1) {
 		set_current_state(TASK_INTERRUPTIBLE);
-		ptr = ptr_ring_consume(&tfile->tx_ring);
+		ptr = tun_ring_consume(tun, tfile);
 		if (ptr)
 			break;
 		if (signal_pending(current)) {
@@ -2171,7 +2274,7 @@ static ssize_t tun_do_read(struct tun_struct *tun, struct tun_file *tfile,
 
 	if (!ptr) {
 		/* Read frames from ring */
-		ptr = tun_ring_recv(tfile, noblock, &err);
+		ptr = tun_ring_recv(tun, tfile, noblock, &err);
 		if (!ptr)
 			return err;
 	}
@@ -3630,6 +3733,13 @@ static int tun_queue_resize(struct tun_struct *tun)
 					  dev->tx_queue_len, GFP_KERNEL,
 					  tun_ptr_free);
 
+	if (!ret) {
+		for (i = 0; i < tun->numqueues; i++) {
+			tfile = rtnl_dereference(tun->tfiles[i]);
+			tun_force_wake_queue(tun, tfile);
+		}
+	}
+
 	kfree(rings);
 	return ret;
 }
-- 
2.43.0

[PATCH net-next v14 3/5] vhost-net: wake queue of tun/tap after ptr_ring consume

[Simon Schippers]

Add tun_wake_queue() to tun.c and export it for use by vhost-net. The
function validates that the file belongs to a device implemented by
drivers/net/tun.c, in IFF_TUN as well as in IFF_TAP mode, and that the
tfile exists, dereferences the tun_struct under RCU, and delegates to
__tun_wake_queue().

vhost_net_buf_produce() now calls tun_wake_queue() after a successful
batched consume of the ring to allow the netdev subqueue to be woken up.
The point is to allow the queue to be stopped when it gets full, which is
required for traffic shaping, implemented by the following
"stop tail-drop when IFF_BACKPRESSURE is set".
As __tun_wake_queue() returns early unless IFF_BACKPRESSURE is set, a
tun/tap device that does not opt in only pays for the added check.

macvtap and ipvtap rings, which get_tap_ptr_ring() accepts too, are
unaffected: their producer is the tap_handle_frame() rx_handler and not
ndo_start_xmit, so stopping a netdev TX queue would not hold it back.
drivers/net/tap.c has no netdev_ops of its own either. No
tap_wake_queue() is needed.

cons_cnt and the wake decision are best-effort and are not reverted by
ptr_ring_unconsume(), so vhost_net_buf_unproduce() can leave the subqueue
woken over a full ring. The producer re-stops it on the next packet, and
that path only runs from vhost_net_stop_vq() and vhost_net_set_backend(),
when the consumer is going away, so a stopped queue is the correct end
state rather than a stall.

Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
---
 drivers/net/tun.c      | 25 +++++++++++++++++++++++++
 drivers/vhost/net.c    | 21 +++++++++++++++------
 include/linux/if_tun.h |  4 ++++
 3 files changed, 44 insertions(+), 6 deletions(-)

diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index d49b6bfd104d..dc32566588d1 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -3848,6 +3848,31 @@ struct ptr_ring *tun_get_tx_ring(struct file *file)
 }
 EXPORT_SYMBOL_GPL(tun_get_tx_ring);
 
+/* Callers must hold ring.consumer_lock */
+void tun_wake_queue(struct file *file, int consumed)
+{
+	struct tun_file *tfile;
+	struct tun_struct *tun;
+
+	if (file->f_op != &tun_fops)
+		return;
+
+	tfile = file->private_data;
+	if (!tfile)
+		return;
+
+	lockdep_assert_held(&tfile->tx_ring.consumer_lock);
+
+	rcu_read_lock();
+
+	tun = rcu_dereference(tfile->tun);
+	if (tun)
+		__tun_wake_queue(tun, tfile, consumed);
+
+	rcu_read_unlock();
+}
+EXPORT_SYMBOL_GPL(tun_wake_queue);
+
 module_init(tun_init);
 module_exit(tun_cleanup);
 MODULE_DESCRIPTION(DRV_DESCRIPTION);
diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index 6949b704166d..3e72b9c6af0c 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -176,13 +176,21 @@ static void *vhost_net_buf_consume(struct vhost_net_buf *rxq)
 	return ret;
 }
 
-static int vhost_net_buf_produce(struct vhost_net_virtqueue *nvq)
+static int vhost_net_buf_produce(struct sock *sk,
+				 struct vhost_net_virtqueue *nvq)
 {
+	struct file *file = sk->sk_socket->file;
 	struct vhost_net_buf *rxq = &nvq->rxq;
 
 	rxq->head = 0;
-	rxq->tail = ptr_ring_consume_batched(nvq->rx_ring, rxq->queue,
-					      VHOST_NET_BATCH);
+	spin_lock(&nvq->rx_ring->consumer_lock);
+	rxq->tail = __ptr_ring_consume_batched(nvq->rx_ring, rxq->queue,
+					       VHOST_NET_BATCH);
+
+	if (rxq->tail)
+		tun_wake_queue(file, rxq->tail);
+
+	spin_unlock(&nvq->rx_ring->consumer_lock);
 	return rxq->tail;
 }
 
@@ -209,14 +217,15 @@ static int vhost_net_buf_peek_len(void *ptr)
 	return __skb_array_len_with_tag(ptr);
 }
 
-static int vhost_net_buf_peek(struct vhost_net_virtqueue *nvq)
+static int vhost_net_buf_peek(struct sock *sk,
+			      struct vhost_net_virtqueue *nvq)
 {
 	struct vhost_net_buf *rxq = &nvq->rxq;
 
 	if (!vhost_net_buf_is_empty(rxq))
 		goto out;
 
-	if (!vhost_net_buf_produce(nvq))
+	if (!vhost_net_buf_produce(sk, nvq))
 		return 0;
 
 out:
@@ -1004,7 +1013,7 @@ static int peek_head_len(struct vhost_net_virtqueue *rvq, struct sock *sk)
 	unsigned long flags;
 
 	if (rvq->rx_ring)
-		return vhost_net_buf_peek(rvq);
+		return vhost_net_buf_peek(sk, rvq);
 
 	spin_lock_irqsave(&sk->sk_receive_queue.lock, flags);
 	head = skb_peek(&sk->sk_receive_queue);
diff --git a/include/linux/if_tun.h b/include/linux/if_tun.h
index 80166eb62f41..eeb9ed3c5a23 100644
--- a/include/linux/if_tun.h
+++ b/include/linux/if_tun.h
@@ -22,6 +22,8 @@ struct tun_msg_ctl {
 #if defined(CONFIG_TUN) || defined(CONFIG_TUN_MODULE)
 struct socket *tun_get_socket(struct file *);
 struct ptr_ring *tun_get_tx_ring(struct file *file);
+/* Callers must hold the consumer_lock of the ring of file */
+void tun_wake_queue(struct file *file, int consumed);
 
 static inline bool tun_is_xdp_frame(void *ptr)
 {
@@ -55,6 +57,8 @@ static inline struct ptr_ring *tun_get_tx_ring(struct file *f)
 	return ERR_PTR(-EINVAL);
 }
 
+static inline void tun_wake_queue(struct file *f, int consumed) {}
+
 static inline bool tun_is_xdp_frame(void *ptr)
 {
 	return false;
-- 
2.43.0

[PATCH net-next v14 4/5] ptr_ring: move free-space check into separate helper

[Simon Schippers]

This patch moves the check for available free space for a new entry into
a separate function. Existing callers that only check for a non-zero
return value are unaffected. __ptr_ring_produce() now returns -EINVAL
for a zero-size ring and -ENOSPC when full, whereas before both cases
returned -ENOSPC. The new helper allows callers to determine in advance
whether a single subsequent __ptr_ring_produce() call will succeed. This
information can, for example, be used to temporarily stop producing until
__ptr_ring_check_produce() indicates that space is available again.

The return values are documented above the helper, as a caller that waits
for space must distinguish the transient -ENOSPC from the permanent
-EINVAL.

Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
---
 include/linux/ptr_ring.h | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)

diff --git a/include/linux/ptr_ring.h b/include/linux/ptr_ring.h
index d2c3629bbe45..631c43fde440 100644
--- a/include/linux/ptr_ring.h
+++ b/include/linux/ptr_ring.h
@@ -96,6 +96,26 @@ static inline bool ptr_ring_full_bh(struct ptr_ring *r)
 	return ret;
 }
 
+/* Report whether the next __ptr_ring_produce() has room for one entry:
+ * 0 means the single slot at r->queue[r->producer] is free, -ENOSPC means
+ * the ring is full, which is transient, and -EINVAL means r->size is 0,
+ * which is permanent. A caller that stops producing and waits for space
+ * must therefore do so only for -ENOSPC.
+ *
+ * Note: callers invoking this in a loop must use a compiler barrier,
+ * for example cpu_relax(). Callers must hold producer_lock.
+ */
+static inline int __ptr_ring_check_produce(struct ptr_ring *r)
+{
+	if (unlikely(!r->size))
+		return -EINVAL;
+
+	if (data_race(r->queue[r->producer]))
+		return -ENOSPC;
+
+	return 0;
+}
+
 /* Note: callers invoking this in a loop must use a compiler barrier,
  * for example cpu_relax(). Callers must hold producer_lock.
  * Callers are responsible for making sure pointer that is being queued
@@ -103,8 +123,10 @@ static inline bool ptr_ring_full_bh(struct ptr_ring *r)
  */
 static inline int __ptr_ring_produce(struct ptr_ring *r, void *ptr)
 {
-	if (unlikely(!r->size) || data_race(r->queue[r->producer]))
-		return -ENOSPC;
+	int ret = __ptr_ring_check_produce(r);
+
+	if (ret)
+		return ret;
 
 	/* Make sure the pointer we are storing points to a valid data. */
 	/* Pairs with the dependency ordering in __ptr_ring_consume. */
-- 
2.43.0

[PATCH net-next v14 5/5] tun/tap & vhost-net: stop tail-drop when IFF_BACKPRESSURE is set

[Simon Schippers]

This commit prevents tail-drop when IFF_BACKPRESSURE is set, a qdisc is
present and the ptr_ring becomes full. Once the ring reaches capacity
after a produce attempt, the netdev queue is stopped instead of dropping
subsequent packets. Without the flag, or if no qdisc is present, the
previous tail-drop behavior is preserved.

IFF_BACKPRESSURE is added to TUN_FEATURES here and not in the patch that
defines it, so that TUNSETIFF honours the flag only once the
implementation behind it is complete.

The unconditional version of this behavior was reverted because it caused
a significant throughput drop in an IPv6 multicast testcase on Brett
Sheffield's librecast testbed [1]: with 8 iperf3 TCP threads sending, the
throughput dropped from 13.5 Gbit/s to 9.13 Gbit/s. This is why the queue
stopping is now gated on IFF_BACKPRESSURE.

If producing an entry fails anyway due to a race, tun_net_xmit() drops
the packet. Such rare races are expected because LLTX is enabled and the
transmit path operates without the usual locking.

The queue state is only touched while the device is running. The stop
itself would be harmless during teardown, as tun_net_close() sets the
same bit, but the re-check below it wakes the queue again and must not
clear that stop.

A later TUNSETIFF can clear the flag again while the device has at most
one queue. Past that point tun_set_iff() returns before it writes
tun->flags, which is how it already treats every other TUN_FEATURES bit.
For the case where the flag does change, tun_set_iff() calls
tun_force_wake_queue() for the attached tfiles, so that no queue stays
stopped without a consumer that would wake it.

The __tun_wake_queue() function of the consumer races with the producer
for waking/stopping the netdev queue, which could result in a stalled
queue. Therefore, an smp_mb__after_atomic() is introduced that pairs
with the smp_mb() of the consumer. It follows the principle of store
buffering described in tools/memory-model/Documentation/recipes.txt:

- The producer in tun_net_xmit() first sets __QUEUE_STATE_DRV_XOFF,
  followed by an smp_mb__after_atomic() (= smp_mb()), and then reads the
  ring with __ptr_ring_check_produce().

- The consumer in __tun_wake_queue() first writes zero to the ring in
  __ptr_ring_consume(), followed by an smp_mb(), and then reads the queue
  status with netif_tx_queue_stopped().

=> Following the aforementioned principle, it is impossible for the
   producer to see a full ring (and therefore not wake the queue on the
   re-check) while the consumer simultaneously fails to see a stopped
   queue (and therefore also does not wake it).

tun_net_xmit() holds only the producer_lock and can not reset cons_cnt,
which the consumer_lock protects, so the wake on the re-check leaves
stale credit behind. That is accepted as best-effort, the re-check rarely
succeeds and the next drain corrects the count.

The documentation in tuntap.rst is updated accordingly.

Benchmarks:
My own benchmarks show a slight regression in raw transmission performance
when using two sending threads. Packet loss also occurs only in the
two-thread sending case; no packet loss was observed with a single
sending thread.

Test setup:
AMD Ryzen 5 5600X at 4.3 GHz, 3200 MHz RAM, isolated QEMU threads;
Average over 50 runs @ 100,000,000 packets. SRSO and spectre v2
mitigations disabled.

Note for tap+vhost-net:
XDP drop program active in VM -> ~2.5x faster; slower for tap due to
more syscalls (high utilization of entry_SYSRETQ_unsafe_stack in perf)

+--------------------------+--------------+----------------+----------+
| 1 thread                 | Stock        | Patched with   | diff     |
| sending                  |              | fq_codel qdisc |          |
+------------+-------------+--------------+----------------+----------+
| TAP        | Received    | 1.132 Mpps   | 1.123 Mpps     | -0.8%    |
|            +-------------+--------------+----------------+----------+
|            | Lost/s      | 3.765 Mpps   | 0 pps          |          |
+------------+-------------+--------------+----------------+----------+
| TAP        | Received    | 3.857 Mpps   | 3.901 Mpps     | +1.1%    |
|            +-------------+--------------+----------------+----------+
| +vhost-net | Lost/s      | 0.802 Mpps   | 0 pps          |          |
+------------+-------------+--------------+----------------+----------+

+--------------------------+--------------+----------------+----------+
| 2 threads                | Stock        | Patched with   | diff     |
| sending                  |              | fq_codel qdisc |          |
+------------+-------------+--------------+----------------+----------+
| TAP        | Received    | 1.115 Mpps   | 1.081 Mpps     | -3.0%    |
|            +-------------+--------------+----------------+----------+
|            | Lost/s      | 8.490 Mpps   | 391 pps        |          |
+------------+-------------+--------------+----------------+----------+
| TAP        | Received    | 3.664 Mpps   | 3.555 Mpps     | -3.0%    |
|            +-------------+--------------+----------------+----------+
| +vhost-net | Lost/s      | 5.330 Mpps   | 938 pps        |          |
+------------+-------------+--------------+----------------+----------+

[1] https://lore.kernel.org/netdev/akVnoOYQOrt8k-Gu@karahi.librecast.net/

Co-developed-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Tim Gebauer <tim.gebauer@tu-dortmund.de>
Signed-off-by: Simon Schippers <simon.schippers@tu-dortmund.de>
Link: https://lore.kernel.org/netdev/akVnoOYQOrt8k-Gu@karahi.librecast.net/
---
 Documentation/networking/tuntap.rst | 32 +++++++++++++++++++++++
 drivers/net/tun.c                   | 39 ++++++++++++++++++++++++-----
 2 files changed, 65 insertions(+), 6 deletions(-)

diff --git a/Documentation/networking/tuntap.rst b/Documentation/networking/tuntap.rst
index 4d7087f727be..56c9dc7b96af 100644
--- a/Documentation/networking/tuntap.rst
+++ b/Documentation/networking/tuntap.rst
@@ -206,6 +206,38 @@ enable is true we enable it, otherwise we disable it::
       return ioctl(fd, TUNSETQUEUE, (void *)&ifr);
   }
 
+3.4 qdisc backpressure
+----------------------
+
+IFF_BACKPRESSURE can be set to enable qdisc backpressure. Without it, TX
+drops occur when the internal ring buffer is full, so any attached qdisc
+is effectively bypassed and applications only learn about congestion
+through those drops.
+
+With it, the kernel stops the queue instead, letting the qdisc hold and
+schedule packets, so its AQM, shaping and fairness actually apply. This
+helps protocols like TCP, which cut throughput in reaction to packet
+drops. With IFF_BACKPRESSURE, drops then only occur as a rare race.
+Backpressure requires a qdisc to be attached and has no effect with
+noqueue.
+
+The flag is a property of the TUN/TAP device rather than of the file
+descriptor it was set on, so it applies to all queues of the device,
+regardless of which process opened which queue.
+
+The flag can only be changed while the device has at most one queue. On a
+multiqueue device that already has a second queue attached or detached, a
+later TUNSETIFF succeeds but leaves the flag as it is. All the other
+TUNSETIFF flags behave the same way.
+
+The txqueuelen can be reduced alongside this flag to further shift
+buffering into the qdisc and reduce bufferbloat, at a possible
+performance cost.
+
+When running multiple network streams in parallel through a single
+TUN/TAP queue, the flag may reduce performance due to the extra overhead
+of the backpressure mechanism.
+
 Universal TUN/TAP device driver Frequently Asked Question
 =========================================================
 
diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index dc32566588d1..ec90fef4a42f 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -98,7 +98,8 @@ static void tun_default_link_ksettings(struct net_device *dev,
 #define TUN_FASYNC	IFF_ATTACH_QUEUE
 
 #define TUN_FEATURES (IFF_NO_PI | IFF_ONE_QUEUE | IFF_VNET_HDR | \
-		      IFF_MULTI_QUEUE | IFF_NAPI | IFF_NAPI_FRAGS)
+		      IFF_MULTI_QUEUE | IFF_NAPI | IFF_NAPI_FRAGS | \
+		      IFF_BACKPRESSURE)
 
 #define GOODCOPY_LEN 128
 
@@ -1063,6 +1064,7 @@ static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev)
 	struct netdev_queue *queue;
 	struct tun_file *tfile;
 	int len = skb->len;
+	int ret;
 
 	rcu_read_lock();
 	tfile = rcu_dereference(tun->tfiles[txq]);
@@ -1117,13 +1119,35 @@ static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev)
 
 	nf_reset_ct(skb);
 
-	if (ptr_ring_produce(&tfile->tx_ring, skb)) {
+	queue = netdev_get_tx_queue(dev, txq);
+
+	spin_lock(&tfile->tx_ring.producer_lock);
+	ret = __ptr_ring_produce(&tfile->tx_ring, skb);
+	/* Do not touch the queue state of a device that is going down. */
+	if ((tun->flags & IFF_BACKPRESSURE) && netif_running(dev) &&
+	    !qdisc_txq_has_no_queue(queue) &&
+	    __ptr_ring_check_produce(&tfile->tx_ring) == -ENOSPC) {
+		netif_tx_stop_queue(queue);
+		/* Paired with smp_mb() in __tun_wake_queue() */
+		smp_mb__after_atomic();
+		if (!__ptr_ring_check_produce(&tfile->tx_ring))
+			netif_tx_wake_queue(queue);
+	}
+	spin_unlock(&tfile->tx_ring.producer_lock);
+
+	if (ret) {
+		/* This should be a rare case if IFF_BACKPRESSURE is enabled and
+		 * a qdisc is present, but can happen due to lltx.
+		 * Since skb_tx_timestamp(), skb_orphan(),
+		 * run_ebpf_filter() and pskb_trim() could have tinkered
+		 * with the SKB, returning NETDEV_TX_BUSY is unsafe and
+		 * we must drop instead.
+		 */
 		drop_reason = SKB_DROP_REASON_FULL_RING;
 		goto drop;
 	}
 
 	/* dev->lltx requires to do our own update of trans_start */
-	queue = netdev_get_tx_queue(dev, txq);
 	txq_trans_cond_update(queue);
 
 	/* Notify and wake up reader process */
@@ -2806,8 +2830,9 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
 {
 	struct tun_struct *tun;
 	struct tun_file *tfile = file->private_data;
+	struct tun_file *ntfile;
 	struct net_device *dev;
-	int err;
+	int err, i;
 
 	if (tfile->detached)
 		return -EINVAL;
@@ -2936,8 +2961,10 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr)
 	/* Make sure persistent devices do not get stuck in
 	 * xoff state.
 	 */
-	if (netif_running(tun->dev))
-		netif_tx_wake_all_queues(tun->dev);
+	for (i = 0; i < tun->numqueues; i++) {
+		ntfile = rtnl_dereference(tun->tfiles[i]);
+		tun_force_wake_queue(tun, ntfile);
+	}
 
 	strscpy(ifr->ifr_name, tun->dev->name);
 	return 0;
-- 
2.43.0

Re: [PATCH net-next v14 0/5] tun/tap & vhost-net: apply qdisc backpressure on full ptr_ring to reduce TX drops

[Simon Schippers]

On 8/3/26 20:36, Simon Schippers wrote:
> This patch series deals with tun/tap & vhost-net which drop incoming
> SKBs whenever their internal ptr_ring buffer is full. Instead, with this
> patch series, the associated netdev queue is stopped, but only when the
> new IFF_BACKPRESSURE flag is set and a qdisc is attached. Without the
> flag, or if no qdisc is present, the existing behavior is preserved. The
> XDP transmit path is not affected. This patch series touches tun/tap and
> vhost-net, as they share common logic and must be updated together.
> Modifying only one of them would break the other.
> 

Sashiko is happy now.

Gemini reports in patch 2:
"[Severity: Medium]
Will vhost-net bypass this wrapper and cause permanent TX queue stalls when
IFF_BACKPRESSURE is enabled?"
... but this is added in patch 3. It just cant cross-reference it.
So it is fine.