* [PATCH RFC net-next v2 0/4] xdp: reuse generic skb XDP handling for veth
@ 2026-11-10 10:19 Slawomir Stepien
2026-11-10 10:19 ` [PATCH RFC net-next v2 1/4] xdp: add mixed page_pool/page_shared memory type Slawomir Stepien
` (4 more replies)
0 siblings, 5 replies; 6+ messages in thread
From: Slawomir Stepien @ 2026-11-10 10:19 UTC (permalink / raw)
To: netdev
Cc: bpf, Maciej Fijalkowski, Jakub Kicinski, Paolo Abeni,
Eric Dumazet, David S . Miller, Andrew Lunn, Simon Horman,
Björn Töpel, Jesper Dangaard Brouer,
Toke Høiland-Jørgensen, Lorenzo Bianconi,
Magnus Karlsson, Stanislav Fomichev, Alexei Starovoitov,
Daniel Borkmann, John Fastabend, Andrii Nakryiko,
Eduard Zingerman, Kumar Kartikeya Dwivedi, Martin KaFai Lau,
Song Liu, Yonghong Song, Jiri Olsa, Emil Tsalapatis,
Ihor Solodrai, Vladimir Oltean, Tianyi Gao, Liang Chen,
Yunsheng Lin, huangjie . albert, Slawomir Stepien
Hi,
this is v2 of Maciej's RFC [1], which makes veth run XDP on skb-backed
packets through the generic skb XDP path instead of converting them
into xdp_frames for XDP_TX and XDP_REDIRECT. The patches are Maciej's;
I addressed the review of v1 and measured the performance, which was
the main open question there. Changes I made are noted in [sst: ...]
lines in each commit message.
The motivation is unchanged: veth pins the skb data with plain page
references (veth_xdp_get()) to build an xdp_frame from an skb, and the
rxq memory type it reports does not match memory that was copied into
a page_pool, which makes the AF_XDP selftests splat when a program
shrinks a multi-buffer packet. With this series, pages copied into
veth's page_pool are recycled (0 rx_pp_recycle_released_ref, was
~4.8M/s in the benchmark below).
Dependencies
============
The series applies to net-next with these two net series applied
first (prerequisite-patch-id below):
- Tianyi Gao, "net: fix mac_len of skbs copied by generic XDP" (1/2)
[2]. Without it, cpumap redirect of skbs that veth copied drops
every packet once veth uses the generic path.
- Vladimir Oltean, "net: pass the XDP action to generic_xdp_tx() for
tracing" and "net: validate xmit skbs in generic_xdp_tx()" [3].
Patch 2/4 changes the same function.
Changes in v2
=============
1/4: MEM_TYPE_PAGE_POOL_OR_SHARED renamed to MEM_TYPE_MIXED (Björn);
comment on what the type means.
2/4: Rebased on Vladimir's series. The redirect callers in devmap and
filter still return success: generic_xdp_tx() frees and counts a
failed skb itself, so propagating the error would free it twice
(Björn asked why the error is not propagated).
3/4: One skb_needs_xdp_cow() predicate for generic XDP and veth
(shared, cloned, nonlinear or headroom too small) instead of a
per-caller callback and INDIRECT_CALL_2 (Jakub). The
skb_head_is_locked() check is gone, it would copy every locally
generated skb. Copies go to the caller's page_pool with
skb_pp_cow_data() also for programs without frags support (found
by the xdp_metadata selftest).
4/4: Commit message describes how XDP_TX/XDP_REDIRECT change for
skb-backed packets (Björn's table). A failed skb XDP_TX is counted
once, in the device's tx_dropped (Sashiko). veth sets
skb->mac_len before running generic XDP, as
__netif_receive_skb_core() does; without it cpumap redirect of
veth skbs is dropped.
Performance
===========
QEMU/KVM, 8 vCPUs, minimal config without debug options. pktgen sends
skbs into veth v0, the XDP program runs on the peer v1 (skb path).
Packets/s delivered to the destination, median of 12 runs of 8 s over
4 alternating boots per kernel, 95% bootstrap confidence intervals
within about +-2%. Base: v7.4-rc1+ (f66469d9905b) with the
dependencies above.
64 B 8000 B (3 frags)
XDP_TX 4.49 -> 4.04 Mpps (-10%) 2.25 -> 1.87 (-17%)
REDIRECT devmap -> veth 4.30 -> 3.95 (-8%) 2.18 -> 1.81 (-17%)
REDIRECT cpumap 4.49 -> 4.47 (0%) 2.02 -> 2.27 (+12%)
XDP_DROP (native veth) 5.65 -> 5.42 (-4%) 2.82 -> 2.71 (-4%)
XDP_DROP (xdpgeneric) 5.63 -> 5.83 (+4%) 2.94 -> 3.00 (+2%)
(cpumap 8000 B base: base without Vladimir's series, which does not
touch that path; 3 of the 12 runs of the other base kernel stalled.)
cpumap avoids the skb -> xdp_frame -> skb round trip. XDP_TX and
devmap redirect to veth are slower for two reasons:
- no bulking: one ndo_start_xmit() per skb instead of ndo_xdp_xmit()
with up to 16 frames;
- at 8000 B the receiving veth copies the packet a second time.
Before, it got an xdp_frame; now it gets an skb with frags, and
nonlinear skbs are always copied (kprobe on skb_pp_cow_data(): 1
copy per packet before, 2 after).
Vladimir's validation itself costs about 3% at 64 B on these two
paths and nothing measurable elsewhere.
Questions
=========
- Is the XDP_TX/devmap regression acceptable for the simpler and
correct memory handling, or should it be fixed first? Options we
see: skb-list bulking in the generic TX path (Jesper measured +30%
with xmit_more), and not copying nonlinear skbs whose frags are
already page_pool pages. Raising veth's needed_headroom (Jesper)
only helps linear skbs from senders that honour it; pktgen skbs are
shared and always copied, so the benchmark above cannot show it.
We left all three for follow-ups.
- MEM_TYPE_MIXED vs. separate memory types per source (Jesper).
- The redirect semantics change in 4/4 (Björn).
Testing
=======
Each commit builds with W=1 without new warnings, checkpatch --strict is
clean. test_progs -t ns_xsk,xdp passes; test_xsk.sh shows the same DRV
XDP_ADJUST_TAIL_GROW failures (also with busy poll) as without the
series. Reproducers for shrinking frags in generic and veth XDP, XDP_TX
of multi-buffer packets, devmap egress programs and bpf_prog_test_run
after generic XDP on lo pass without kernel warnings.
The scripts, kernel configs and raw logs behind the numbers and tests
above are in [4], with instructions to rerun them.
AI disclosure: the review changes, the tests and the benchmark were
prepared with help from an AI coding assistant (Claude); I reviewed
and tested all changes.
[1] https://lore.kernel.org/all/20260509084858.773921-1-maciej.fijalkowski@intel.com/
[2] https://lore.kernel.org/all/20261103200036.65764-3-tianyi@cloudflare.com/
[3] https://lore.kernel.org/all/20261030114753.185281-1-vladimir.oltean@nxp.com/
[4] https://github.com/s-stepien/xdp-rfc
Thanks,
Slawomir
Maciej Fijalkowski (4):
xdp: add mixed page_pool/page_shared memory type
xdp: return status from generic_xdp_tx()
xdp: split generic XDP skb handling
veth: use generic skb XDP handling
drivers/net/veth.c | 191 +++++++++-----------------------------
include/linux/netdevice.h | 33 ++++++-
include/net/xdp.h | 1 +
net/core/dev.c | 122 ++++++++++++++++++------
net/core/filter.c | 7 ++
net/core/xdp.c | 54 ++++++++++-
6 files changed, 222 insertions(+), 186 deletions(-)
base-commit: fa5196392691051963726787944db872420dfa75
prerequisite-patch-id: 7db3107f14b28c509845871045e6af93b7e267bc
prerequisite-patch-id: 40350101232837c5a38ba1662825228d01734b39
prerequisite-patch-id: 0b57f8c2a409470b1ddf8df747d906d5d3dbd9f3
--
2.56.0
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH RFC net-next v2 1/4] xdp: add mixed page_pool/page_shared memory type
2026-11-10 10:19 [PATCH RFC net-next v2 0/4] xdp: reuse generic skb XDP handling for veth Slawomir Stepien
@ 2026-11-10 10:19 ` Slawomir Stepien
2026-11-10 10:19 ` [PATCH RFC net-next v2 2/4] xdp: return status from generic_xdp_tx() Slawomir Stepien
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Slawomir Stepien @ 2026-11-10 10:19 UTC (permalink / raw)
To: netdev
Cc: bpf, Maciej Fijalkowski, Jakub Kicinski, Paolo Abeni,
Eric Dumazet, David S . Miller, Andrew Lunn, Simon Horman,
Björn Töpel, Jesper Dangaard Brouer,
Toke Høiland-Jørgensen, Lorenzo Bianconi,
Magnus Karlsson, Stanislav Fomichev, Alexei Starovoitov,
Daniel Borkmann, John Fastabend, Andrii Nakryiko,
Eduard Zingerman, Kumar Kartikeya Dwivedi, Martin KaFai Lau,
Song Liu, Yonghong Song, Jiri Olsa, Emil Tsalapatis,
Ihor Solodrai, Vladimir Oltean, Tianyi Gao, Liang Chen,
Yunsheng Lin, huangjie . albert, Slawomir Stepien
From: Maciej Fijalkowski <maciej.fijalkowski@intel.com>
Generic XDP runs on skb-backed data. In that mode the skb head remains
owned by the skb, but XDP helpers may still release frags, for example
when a program trims a non-linear packet.
With the generic page_pool CoW path, the frags visible to XDP may be
backed by the generic system page_pool. In the fallback path, or for
other skb-backed memory, the same generic XDP rxq may still describe
page-frag based memory. Selecting MEM_TYPE_PAGE_POOL or
MEM_TYPE_PAGE_SHARED purely from the rxq therefore either lies about
page_pool ownership or misses recycling opportunities.
Add MEM_TYPE_MIXED for skb-backed generic XDP users. The
return path inspects the actual netmem: page_pool-backed netmems are
returned through their page_pool, and everything else falls back to
page_frag_free(). Transition netdev_rx_queue's xdp_rxq_info from
MEM_TYPE_PAGE_SHARED to MEM_TYPE_MIXED.
This keeps rxq identity stable for users which inspect xdp->rxq->dev and
xdp->rxq->queue_index, while avoiding per-packet rxq->mem mutation.
Respect new mem_type in __xdp_build_skb_from_frame() as veth could
redirect xdp_frame onto cpumap.
Signed-off-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com>
[sst: rename MEM_TYPE_PAGE_POOL_OR_SHARED to MEM_TYPE_MIXED (Björn asked
for a new name, MEM_TYPE_MIXED was Maciej's proposal), document the type]
Signed-off-by: Slawomir Stepien <sst@poczta.fm>
---
include/net/xdp.h | 1 +
net/core/dev.c | 7 ++++++
net/core/xdp.c | 54 ++++++++++++++++++++++++++++++++++++++++++-----
3 files changed, 57 insertions(+), 5 deletions(-)
diff --git a/include/net/xdp.h b/include/net/xdp.h
index 07231adfb5f8..ba055439e497 100644
--- a/include/net/xdp.h
+++ b/include/net/xdp.h
@@ -45,6 +45,7 @@ enum xdp_mem_type {
MEM_TYPE_PAGE_ORDER0, /* Orig XDP full page model */
MEM_TYPE_PAGE_POOL,
MEM_TYPE_XSK_BUFF_POOL,
+ MEM_TYPE_MIXED, /* skb memory: page_pool or page_frag, per page */
MEM_TYPE_MAX,
};
diff --git a/net/core/dev.c b/net/core/dev.c
index 9665f3b2f5cb..6e330fc415cb 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -11459,6 +11459,13 @@ static int netif_alloc_rx_queues(struct net_device *dev)
err = xdp_rxq_info_reg(&rx[i].xdp_rxq, dev, i, 0);
if (err < 0)
goto err_rxq_info;
+ err = xdp_rxq_info_reg_mem_model(&rx[i].xdp_rxq,
+ MEM_TYPE_MIXED,
+ NULL);
+ if (err < 0) {
+ xdp_rxq_info_unreg(&rx[i].xdp_rxq);
+ goto err_rxq_info;
+ }
}
return 0;
diff --git a/net/core/xdp.c b/net/core/xdp.c
index 1d679e8fd649..7077bc54d7d5 100644
--- a/net/core/xdp.c
+++ b/net/core/xdp.c
@@ -22,6 +22,7 @@
#include <net/xdp_priv.h> /* struct xdp_mem_allocator */
#include <trace/events/xdp.h>
#include <net/xdp_sock_drv.h>
+#include "netmem_priv.h"
#define REG_STATE_NEW 0x0
#define REG_STATE_REGISTERED 0x1
@@ -280,6 +281,12 @@ static struct xdp_mem_allocator *__xdp_reg_mem_model(struct xdp_mem_info *mem,
if (!__is_supported_mem_type(type))
return ERR_PTR(-EOPNOTSUPP);
+ /* MEM_TYPE_MIXED has no allocator of its own: __xdp_return()
+ * checks every page and returns page_pool pages to their pool.
+ */
+ if (type == MEM_TYPE_MIXED && allocator)
+ return ERR_PTR(-EINVAL);
+
mem->type = type;
if (!allocator) {
@@ -424,6 +431,23 @@ void xdp_rxq_info_attach_page_pool(struct xdp_rxq_info *xdp_rxq,
}
EXPORT_SYMBOL_GPL(xdp_rxq_info_attach_page_pool);
+static bool xdp_netmem_is_pp(netmem_ref netmem)
+{
+#if IS_ENABLED(CONFIG_PAGE_POOL)
+ return netmem_is_pp(netmem);
+#else
+ return false;
+#endif
+}
+
+static void __xdp_return_page_pool(netmem_ref netmem, bool napi_direct)
+{
+ if (napi_direct && xdp_return_frame_no_direct())
+ napi_direct = false;
+
+ page_pool_put_full_netmem(netmem_get_pp(netmem), netmem, napi_direct);
+}
+
/* XDP RX runs under NAPI protection, and in different delivery error
* scenarios (e.g. queue full), it is possible to return the xdp_frame
* while still leveraging this protection. The @napi_direct boolean
@@ -433,20 +457,26 @@ EXPORT_SYMBOL_GPL(xdp_rxq_info_attach_page_pool);
void __xdp_return(netmem_ref netmem, enum xdp_mem_type mem_type,
bool napi_direct, struct xdp_buff *xdp)
{
+ netmem_ref head;
+
switch (mem_type) {
case MEM_TYPE_PAGE_POOL:
netmem = netmem_compound_head(netmem);
- if (napi_direct && xdp_return_frame_no_direct())
- napi_direct = false;
/* No need to check netmem_is_pp() as mem->type knows this a
* page_pool page
*/
- page_pool_put_full_netmem(netmem_get_pp(netmem), netmem,
- napi_direct);
+ __xdp_return_page_pool(netmem, napi_direct);
break;
case MEM_TYPE_PAGE_SHARED:
page_frag_free(__netmem_address(netmem));
break;
+ case MEM_TYPE_MIXED:
+ head = netmem_compound_head(netmem);
+ if (xdp_netmem_is_pp(head))
+ __xdp_return_page_pool(head, napi_direct);
+ else
+ page_frag_free(__netmem_address(netmem));
+ break;
case MEM_TYPE_PAGE_ORDER0:
put_page(__netmem_to_page(netmem));
break;
@@ -791,6 +821,19 @@ struct sk_buff *xdp_build_skb_from_zc(struct xdp_buff *xdp)
}
EXPORT_SYMBOL_GPL(xdp_build_skb_from_zc);
+static bool xdp_mem_is_page_pool_backed(enum xdp_mem_type mem_type,
+ netmem_ref netmem)
+{
+ switch (mem_type) {
+ case MEM_TYPE_PAGE_POOL:
+ return true;
+ case MEM_TYPE_MIXED:
+ return xdp_netmem_is_pp(netmem_compound_head(netmem));
+ default:
+ return false;
+ }
+}
+
struct sk_buff *__xdp_build_skb_from_frame(struct xdp_frame *xdpf,
struct sk_buff *skb,
struct net_device *dev)
@@ -836,7 +879,8 @@ struct sk_buff *__xdp_build_skb_from_frame(struct xdp_frame *xdpf,
* - RX ring dev queue index (skb_record_rx_queue)
*/
- if (xdpf->mem_type == MEM_TYPE_PAGE_POOL)
+ if (xdp_mem_is_page_pool_backed(xdpf->mem_type,
+ virt_to_netmem(xdpf->data)))
skb_mark_for_recycle(skb);
/* Allow SKB to reuse area used by xdp_frame */
--
2.56.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH RFC net-next v2 2/4] xdp: return status from generic_xdp_tx()
2026-11-10 10:19 [PATCH RFC net-next v2 0/4] xdp: reuse generic skb XDP handling for veth Slawomir Stepien
2026-11-10 10:19 ` [PATCH RFC net-next v2 1/4] xdp: add mixed page_pool/page_shared memory type Slawomir Stepien
@ 2026-11-10 10:19 ` Slawomir Stepien
2026-11-10 10:19 ` [PATCH RFC net-next v2 3/4] xdp: split generic XDP skb handling Slawomir Stepien
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Slawomir Stepien @ 2026-11-10 10:19 UTC (permalink / raw)
To: netdev
Cc: bpf, Maciej Fijalkowski, Jakub Kicinski, Paolo Abeni,
Eric Dumazet, David S . Miller, Andrew Lunn, Simon Horman,
Björn Töpel, Jesper Dangaard Brouer,
Toke Høiland-Jørgensen, Lorenzo Bianconi,
Magnus Karlsson, Stanislav Fomichev, Alexei Starovoitov,
Daniel Borkmann, John Fastabend, Andrii Nakryiko,
Eduard Zingerman, Kumar Kartikeya Dwivedi, Martin KaFai Lau,
Song Liu, Yonghong Song, Jiri Olsa, Emil Tsalapatis,
Ihor Solodrai, Vladimir Oltean, Tianyi Gao, Liang Chen,
Yunsheng Lin, huangjie . albert, Slawomir Stepien
From: Maciej Fijalkowski <maciej.fijalkowski@intel.com>
Once veth will start to use __do_xdp_generic(), this will be helpful for
correct stats values being incremented - otherwise we would not have a
knowledge whether XDP_TX succeeded or not. With this being implemented we
will only bump xdp_tx stat when actual transmission happened.
The generic redirect callers in devmap and filter keep returning success.
generic_xdp_tx() frees and accounts a failed transmit itself, while the
caller of a failed redirect frees the skb again, and in devmap the
original skb may already be consumed after skb_copy(). As with native
XDP, a redirect succeeds once the packet reached the target device.
Signed-off-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com>
[sst: drop the (void) casts and explain why the redirect paths do not
propagate the transmit error (Björn);
rebased onto Vladimir Oltean's "net: pass the XDP action to
generic_xdp_tx() for tracing" and "net: validate xmit skbs in
generic_xdp_tx()": keep the act argument, and return 0 only if the
driver took the skb or every segment that validation made of it]
Signed-off-by: Slawomir Stepien <sst@poczta.fm>
---
include/linux/netdevice.h | 4 ++--
net/core/dev.c | 16 ++++++++++++----
net/core/filter.c | 7 +++++++
3 files changed, 21 insertions(+), 6 deletions(-)
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index f18bafb4efe2..b251c4b5c9ce 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -4357,8 +4357,8 @@ static inline void dev_consume_skb_any(struct sk_buff *skb)
u32 bpf_prog_run_generic_xdp(struct sk_buff *skb, struct xdp_buff *xdp,
const struct bpf_prog *xdp_prog);
-void generic_xdp_tx(struct sk_buff *skb, const struct bpf_prog *xdp_prog,
- u32 act);
+int generic_xdp_tx(struct sk_buff *skb, const struct bpf_prog *xdp_prog,
+ u32 act);
int do_xdp_generic(const struct bpf_prog *xdp_prog, struct sk_buff **pskb);
int netif_rx(struct sk_buff *skb);
int __netif_rx(struct sk_buff *skb);
diff --git a/net/core/dev.c b/net/core/dev.c
index 6e330fc415cb..12595f6b9eba 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -5854,13 +5854,17 @@ static u32 netif_receive_generic_xdp(struct sk_buff **pskb,
* that XDP packets are able to starve other packets going through a qdisc,
* and DDOS attacks will be more effective. In-driver-XDP use dedicated TX
* queues, so they do not have this starvation issue.
+ *
+ * Returns 0 if the driver took the skb, or all segments that validation made
+ * of it. Otherwise returns an error; the skb is consumed either way.
*/
-void generic_xdp_tx(struct sk_buff *skb, const struct bpf_prog *xdp_prog,
- u32 act)
+int generic_xdp_tx(struct sk_buff *skb, const struct bpf_prog *xdp_prog,
+ u32 act)
{
struct net_device *dev = skb->dev;
enum skb_drop_reason reason;
struct netdev_queue *txq;
+ int err = -ENETDOWN;
bool again = false;
int cpu, rc;
@@ -5869,14 +5873,14 @@ void generic_xdp_tx(struct sk_buff *skb, const struct bpf_prog *xdp_prog,
trace_xdp_exception(dev, xdp_prog, act);
dev_core_stats_tx_dropped_inc(dev);
kfree_skb_reason(skb, reason);
- return;
+ return -EINVAL;
}
txq = netdev_core_pick_tx(dev, skb, NULL);
skb = validate_xmit_skb_list(skb, dev, &again);
if (!skb) {
trace_xdp_exception(dev, xdp_prog, act);
- return;
+ return -EINVAL;
}
cpu = smp_processor_id();
@@ -5888,6 +5892,7 @@ void generic_xdp_tx(struct sk_buff *skb, const struct bpf_prog *xdp_prog,
rc = netdev_start_xmit(skb, dev, txq, false);
if (!dev_xmit_complete(rc)) {
skb->next = next;
+ err = -EBUSY;
break;
}
skb = next;
@@ -5897,7 +5902,10 @@ void generic_xdp_tx(struct sk_buff *skb, const struct bpf_prog *xdp_prog,
trace_xdp_exception(dev, xdp_prog, act);
dev_core_stats_tx_dropped_inc(dev);
kfree_skb_list(skb);
+ return err;
}
+
+ return 0;
}
static DEFINE_STATIC_KEY_FALSE(generic_xdp_needed_key);
diff --git a/net/core/filter.c b/net/core/filter.c
index aca8160b26a1..b7241cbfd8f1 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -4883,6 +4883,13 @@ int xdp_do_generic_redirect(struct net_device *dev, struct sk_buff *skb,
skb->dev = fwd;
_trace_xdp_redirect(dev, xdp_prog, ri->tgt_index);
+ /* generic_xdp_tx() consumes the skb: either the driver took it,
+ * or the transmit failed and generic_xdp_tx() traced the drop,
+ * counted it in the target's tx_dropped and freed the skb.
+ * Returning its error would make our caller free the skb again,
+ * so report success, as native XDP does once a frame has been
+ * handed to the target device.
+ */
generic_xdp_tx(skb, xdp_prog, XDP_REDIRECT);
return 0;
}
--
2.56.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH RFC net-next v2 3/4] xdp: split generic XDP skb handling
2026-11-10 10:19 [PATCH RFC net-next v2 0/4] xdp: reuse generic skb XDP handling for veth Slawomir Stepien
2026-11-10 10:19 ` [PATCH RFC net-next v2 1/4] xdp: add mixed page_pool/page_shared memory type Slawomir Stepien
2026-11-10 10:19 ` [PATCH RFC net-next v2 2/4] xdp: return status from generic_xdp_tx() Slawomir Stepien
@ 2026-11-10 10:19 ` Slawomir Stepien
2026-11-10 10:19 ` [PATCH RFC net-next v2 4/4] veth: use generic skb XDP handling Slawomir Stepien
2026-11-10 12:48 ` [PATCH RFC net-next v2 0/4] xdp: reuse generic skb XDP handling for veth Maciej Fijalkowski
4 siblings, 0 replies; 6+ messages in thread
From: Slawomir Stepien @ 2026-11-10 10:19 UTC (permalink / raw)
To: netdev
Cc: bpf, Maciej Fijalkowski, Jakub Kicinski, Paolo Abeni,
Eric Dumazet, David S . Miller, Andrew Lunn, Simon Horman,
Björn Töpel, Jesper Dangaard Brouer,
Toke Høiland-Jørgensen, Lorenzo Bianconi,
Magnus Karlsson, Stanislav Fomichev, Alexei Starovoitov,
Daniel Borkmann, John Fastabend, Andrii Nakryiko,
Eduard Zingerman, Kumar Kartikeya Dwivedi, Martin KaFai Lau,
Song Liu, Yonghong Song, Jiri Olsa, Emil Tsalapatis,
Ihor Solodrai, Vladimir Oltean, Tianyi Gao, Liang Chen,
Yunsheng Lin, huangjie . albert, Slawomir Stepien
From: Maciej Fijalkowski <maciej.fijalkowski@intel.com>
veth has its own page_pool and xdp_rxq_info and also embeds struct
xdp_buff into a larger context used by its metadata kfuncs. At the same
time, the skb-backed veth XDP path currently open-codes most of what
generic XDP already does and then converts skb-backed xdp_buffs into
xdp_frames for XDP_TX and XDP_REDIRECT.
Add a lower-level generic XDP helper, __do_xdp_generic(), that lets
callers provide a small context object. The context carries the
caller-provided xdp_buff storage, optional page_pool and optional
xdp_rxq_info, and returns the actual XDP action and redirect error to
the caller. A NULL page_pool keeps the existing behaviour and uses the
per-CPU system page_pool. A NULL xdp_rxq_info keeps deriving the rxq
from the skb device/rx queue.
This lets drivers such as veth preserve stats and redirect flush
decisions while using the generic skb XDP action handling.
Address also existing bpf_prog_run_generic_xdp() callsites
({cpu,dev}map) so they can keep on using netdev's xdp_rxq_info.
Both users decide with one predicate whether the program has to run on a
page_pool-backed copy of the skb: when it is shared or cloned, when it
has frags (XDP helpers must be able to release them), or when the XDP
headroom is too small. veth also used to require a page-frag head
(skb_head_is_locked()) because it turned the head into an xdp_frame;
skb-backed XDP does not do that, so this check is not needed. Generic
XDP now also copies shared skbs, which it previously ran on in place.
Signed-off-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com>
[sst: merge the generic and veth CoW predicates into one
skb_needs_xdp_cow() and drop the per-caller callback (Jakub); copy
into a caller-provided page_pool with skb_pp_cow_data() as veth did,
also for programs without frags support (fixes xdp_metadata)]
Signed-off-by: Slawomir Stepien <sst@poczta.fm>
---
include/linux/netdevice.h | 29 ++++++++++++
net/core/dev.c | 99 +++++++++++++++++++++++++++++----------
2 files changed, 102 insertions(+), 26 deletions(-)
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index b251c4b5c9ce..56a77c8c67ba 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -4355,10 +4355,39 @@ static inline void dev_consume_skb_any(struct sk_buff *skb)
dev_kfree_skb_any_reason(skb, SKB_CONSUMED);
}
+struct page_pool;
+struct xdp_rxq_info;
+
+/**
+ * struct xdp_generic_ctx - caller context for skb-backed generic XDP
+ * @xdp: caller-provided xdp_buff storage
+ * @page_pool: optional page_pool used when skb COW is needed; the caller
+ * must keep packets for programs without frags support within one page
+ * @xdp_rxq: optional rxq used to initialise @xdp
+ * @xdp_skb: optional pointer updated with the skb used for the XDP run
+ * @act: actual XDP action returned by the program
+ * @err: transmit or redirect error, valid when @act is XDP_TX or XDP_REDIRECT
+ *
+ * If @page_pool is NULL, the generic path uses the per-CPU system
+ * page_pool. If @xdp_rxq is NULL, the generic path derives the rxq
+ * from the skb device/rx-queue, preserving existing do_xdp_generic()
+ * behaviour.
+ */
+struct xdp_generic_ctx {
+ struct xdp_buff *xdp;
+ struct page_pool *page_pool;
+ struct xdp_rxq_info *xdp_rxq;
+ struct sk_buff **xdp_skb;
+ u32 act;
+ int err;
+};
+
u32 bpf_prog_run_generic_xdp(struct sk_buff *skb, struct xdp_buff *xdp,
const struct bpf_prog *xdp_prog);
int generic_xdp_tx(struct sk_buff *skb, const struct bpf_prog *xdp_prog,
u32 act);
+int __do_xdp_generic(const struct bpf_prog *xdp_prog, struct sk_buff **pskb,
+ struct xdp_generic_ctx *ctx);
int do_xdp_generic(const struct bpf_prog *xdp_prog, struct sk_buff **pskb);
int netif_rx(struct sk_buff *skb);
int __netif_rx(struct sk_buff *skb);
diff --git a/net/core/dev.c b/net/core/dev.c
index 12595f6b9eba..81848e638173 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -5668,11 +5668,11 @@ static struct netdev_rx_queue *netif_get_rxqueue(struct sk_buff *skb)
return rxqueue;
}
-u32 bpf_prog_run_generic_xdp(struct sk_buff *skb, struct xdp_buff *xdp,
- const struct bpf_prog *xdp_prog)
+static u32 __bpf_prog_run_generic_xdp(struct sk_buff *skb, struct xdp_buff *xdp,
+ const struct bpf_prog *xdp_prog,
+ struct xdp_rxq_info *xdp_rxq)
{
void *orig_data, *orig_data_end, *hard_start;
- struct netdev_rx_queue *rxqueue;
bool orig_bcast, orig_host;
u32 mac_len, frame_sz;
__be16 orig_eth_type;
@@ -5690,8 +5690,13 @@ u32 bpf_prog_run_generic_xdp(struct sk_buff *skb, struct xdp_buff *xdp,
frame_sz = (void *)skb_end_pointer(skb) - hard_start;
frame_sz += SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
- rxqueue = netif_get_rxqueue(skb);
- xdp_init_buff(xdp, frame_sz, &rxqueue->xdp_rxq);
+ if (!xdp_rxq) {
+ struct netdev_rx_queue *rxqueue;
+
+ rxqueue = netif_get_rxqueue(skb);
+ xdp_rxq = &rxqueue->xdp_rxq;
+ }
+ xdp_init_buff(xdp, frame_sz, xdp_rxq);
xdp_prepare_buff(xdp, hard_start, skb_headroom(skb) - mac_len,
skb_headlen(skb) + mac_len, true);
if (skb_is_nonlinear(skb)) {
@@ -5774,15 +5779,31 @@ u32 bpf_prog_run_generic_xdp(struct sk_buff *skb, struct xdp_buff *xdp,
return act;
}
+u32 bpf_prog_run_generic_xdp(struct sk_buff *skb, struct xdp_buff *xdp,
+ const struct bpf_prog *xdp_prog)
+{
+ return __bpf_prog_run_generic_xdp(skb, xdp, xdp_prog, NULL);
+}
+
static int
-netif_skb_check_for_xdp(struct sk_buff **pskb, const struct bpf_prog *prog)
+netif_skb_check_for_xdp(struct sk_buff **pskb, const struct bpf_prog *prog,
+ struct page_pool *page_pool)
{
struct sk_buff *skb = *pskb;
int err, hroom, troom;
- local_lock_nested_bh(&system_page_pool.bh_lock);
- err = skb_cow_data_for_xdp(this_cpu_read(system_page_pool.pool), pskb, prog);
- local_unlock_nested_bh(&system_page_pool.bh_lock);
+ if (page_pool) {
+ /* veth, the caller with its own page_pool, limits the MTU so
+ * that the copy is linear even for programs without frags
+ * support, and always copied into its page_pool.
+ */
+ err = skb_pp_cow_data(page_pool, pskb, XDP_PACKET_HEADROOM);
+ } else {
+ local_lock_nested_bh(&system_page_pool.bh_lock);
+ err = skb_cow_data_for_xdp(this_cpu_read(system_page_pool.pool),
+ pskb, prog);
+ local_unlock_nested_bh(&system_page_pool.bh_lock);
+ }
if (!err)
return 0;
@@ -5800,9 +5821,22 @@ netif_skb_check_for_xdp(struct sk_buff **pskb, const struct bpf_prog *prog)
return skb_linearize(skb);
}
+/* Run the program on a private, page_pool-backed copy when it could otherwise
+ * change data someone else sees (shared or cloned skb), when frags have to be
+ * page_pool-backed so that XDP helpers can release them, or when the XDP
+ * headroom contract is not met. The head does not have to be a page frag:
+ * skb-backed XDP never turns the head into an xdp_frame.
+ */
+static bool skb_needs_xdp_cow(const struct sk_buff *skb)
+{
+ return skb_shared(skb) || skb_cloned(skb) || skb_is_nonlinear(skb) ||
+ skb_headroom(skb) < XDP_PACKET_HEADROOM;
+}
+
static u32 netif_receive_generic_xdp(struct sk_buff **pskb,
struct xdp_buff *xdp,
- const struct bpf_prog *xdp_prog)
+ const struct bpf_prog *xdp_prog,
+ struct xdp_generic_ctx *ctx)
{
struct sk_buff *skb = *pskb;
u32 mac_len, act = XDP_DROP;
@@ -5820,15 +5854,17 @@ static u32 netif_receive_generic_xdp(struct sk_buff **pskb,
mac_len = skb->data - skb_mac_header(skb);
__skb_push(skb, mac_len);
- if (skb_cloned(skb) || skb_is_nonlinear(skb) ||
- skb_headroom(skb) < XDP_PACKET_HEADROOM) {
- if (netif_skb_check_for_xdp(pskb, xdp_prog))
+ if (skb_needs_xdp_cow(skb)) {
+ if (netif_skb_check_for_xdp(pskb, xdp_prog, ctx->page_pool))
goto do_drop;
}
__skb_pull(*pskb, mac_len);
- act = bpf_prog_run_generic_xdp(*pskb, xdp, xdp_prog);
+ if (ctx->xdp_skb)
+ *ctx->xdp_skb = *pskb;
+
+ act = __bpf_prog_run_generic_xdp(*pskb, xdp, xdp_prog, ctx->xdp_rxq);
switch (act) {
case XDP_REDIRECT:
case XDP_TX:
@@ -5910,27 +5946,27 @@ int generic_xdp_tx(struct sk_buff *skb, const struct bpf_prog *xdp_prog,
static DEFINE_STATIC_KEY_FALSE(generic_xdp_needed_key);
-int do_xdp_generic(const struct bpf_prog *xdp_prog, struct sk_buff **pskb)
+int __do_xdp_generic(const struct bpf_prog *xdp_prog, struct sk_buff **pskb,
+ struct xdp_generic_ctx *ctx)
{
struct bpf_net_context __bpf_net_ctx, *bpf_net_ctx;
- if (xdp_prog) {
- struct xdp_buff xdp;
- u32 act;
- int err;
+ ctx->act = XDP_PASS;
+ ctx->err = 0;
+ if (xdp_prog) {
bpf_net_ctx = bpf_net_ctx_set(&__bpf_net_ctx);
- act = netif_receive_generic_xdp(pskb, &xdp, xdp_prog);
- if (act != XDP_PASS) {
- switch (act) {
+ ctx->act = netif_receive_generic_xdp(pskb, ctx->xdp, xdp_prog, ctx);
+ if (ctx->act != XDP_PASS) {
+ switch (ctx->act) {
case XDP_REDIRECT:
- err = xdp_do_generic_redirect((*pskb)->dev, *pskb,
- &xdp, xdp_prog);
- if (err)
+ ctx->err = xdp_do_generic_redirect((*pskb)->dev, *pskb,
+ ctx->xdp, xdp_prog);
+ if (ctx->err)
goto out_redir;
break;
case XDP_TX:
- generic_xdp_tx(*pskb, xdp_prog, XDP_TX);
+ ctx->err = generic_xdp_tx(*pskb, xdp_prog, XDP_TX);
break;
}
bpf_net_ctx_clear(bpf_net_ctx);
@@ -5944,6 +5980,17 @@ int do_xdp_generic(const struct bpf_prog *xdp_prog, struct sk_buff **pskb)
kfree_skb_reason(*pskb, SKB_DROP_REASON_XDP);
return XDP_DROP;
}
+EXPORT_SYMBOL_GPL(__do_xdp_generic);
+
+int do_xdp_generic(const struct bpf_prog *xdp_prog, struct sk_buff **pskb)
+{
+ struct xdp_generic_ctx ctx = {};
+ struct xdp_buff xdp;
+
+ ctx.xdp = &xdp;
+
+ return __do_xdp_generic(xdp_prog, pskb, &ctx);
+}
EXPORT_SYMBOL_GPL(do_xdp_generic);
static int netif_rx_internal(struct sk_buff *skb)
--
2.56.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH RFC net-next v2 4/4] veth: use generic skb XDP handling
2026-11-10 10:19 [PATCH RFC net-next v2 0/4] xdp: reuse generic skb XDP handling for veth Slawomir Stepien
` (2 preceding siblings ...)
2026-11-10 10:19 ` [PATCH RFC net-next v2 3/4] xdp: split generic XDP skb handling Slawomir Stepien
@ 2026-11-10 10:19 ` Slawomir Stepien
2026-11-10 12:48 ` [PATCH RFC net-next v2 0/4] xdp: reuse generic skb XDP handling for veth Maciej Fijalkowski
4 siblings, 0 replies; 6+ messages in thread
From: Slawomir Stepien @ 2026-11-10 10:19 UTC (permalink / raw)
To: netdev
Cc: bpf, Maciej Fijalkowski, Jakub Kicinski, Paolo Abeni,
Eric Dumazet, David S . Miller, Andrew Lunn, Simon Horman,
Björn Töpel, Jesper Dangaard Brouer,
Toke Høiland-Jørgensen, Lorenzo Bianconi,
Magnus Karlsson, Stanislav Fomichev, Alexei Starovoitov,
Daniel Borkmann, John Fastabend, Andrii Nakryiko,
Eduard Zingerman, Kumar Kartikeya Dwivedi, Martin KaFai Lau,
Song Liu, Yonghong Song, Jiri Olsa, Emil Tsalapatis,
Ihor Solodrai, Vladimir Oltean, Tianyi Gao, Liang Chen,
Yunsheng Lin, huangjie . albert, Slawomir Stepien
From: Maciej Fijalkowski <maciej.fijalkowski@intel.com>
veth currently runs XDP for skb-backed packets by constructing an
xdp_buff and then, for XDP_TX and XDP_REDIRECT, converting that
skb-backed xdp_buff into an xdp_frame. The backing memory is still
skb-owned, so veth has to pin the skb data and frags manually before
consuming the skb.
Use the generic skb XDP helper for skb-backed packets instead. This
keeps skb-backed packets on the skb generic XDP path: XDP_REDIRECT uses
xdp_do_generic_redirect() and XDP_TX uses generic_xdp_tx(). Packets that
already arrive as xdp_frames keep using the existing veth xdp_frame path.
veth still provides its own page_pool and xdp_rxq_info through struct
xdp_generic_ctx. It also keeps using struct veth_xdp_buff storage so
metadata kfuncs that need the skb continue to work after a possible skb
COW.
This changes what XDP_TX and XDP_REDIRECT do for skb-backed packets in
veth (packets the peer sent with ndo_start_xmit()):
target before after
XDP_TX xdp_frame, bulked to peer skb via generic_xdp_tx()
NIC via devmap ndo_xdp_xmit(), bulked netdev_start_xmit()
cpumap xdp_frame on remote ring skb on remote backlog
AF_XDP socket copy, flush per poll copy, flush per packet
veth peer (devmap) xdp_frame in peer's ring skb in peer's ring
Packets that arrive as xdp_frames are not affected.
A failed skb XDP_TX is counted in the device's tx_dropped by
generic_xdp_tx(), so veth does not add it to xdp_tx_err as well.
Signed-off-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com>
[sst: describe the redirect semantics change (Björn), document the
XDP_TX error accounting and drop the unused xdp_tx_err update (Sashiko);
set skb->mac_len before generic XDP, needed by the generic cpumap redirect]
Signed-off-by: Slawomir Stepien <sst@poczta.fm>
---
drivers/net/veth.c | 191 ++++++++++-----------------------------------
1 file changed, 42 insertions(+), 149 deletions(-)
diff --git a/drivers/net/veth.c b/drivers/net/veth.c
index a61842b2585b..049c95dbb8be 100644
--- a/drivers/net/veth.c
+++ b/drivers/net/veth.c
@@ -49,6 +49,10 @@ struct veth_stats {
u64 xdp_drops;
u64 xdp_tx;
u64 xdp_tx_err;
+ /* skb-backed XDP_TX is handled immediately by generic_xdp_tx().
+ * xdp_frame XDP_TX is accounted from veth_xdp_flush_bq().
+ */
+ u64 xdp_skb_tx;
u64 peer_tq_xdp_xmit;
u64 peer_tq_xdp_xmit_err;
};
@@ -63,7 +67,6 @@ struct veth_rq {
struct napi_struct __rcu *napi; /* points to xdp_napi when the latter is initialized */
struct net_device *dev;
struct bpf_prog __rcu *xdp_prog;
- struct xdp_mem_info xdp_mem;
struct veth_rq_stats stats;
bool rx_notify_masked;
struct ptr_ring xdp_ring;
@@ -654,10 +657,15 @@ static struct xdp_frame *veth_xdp_rcv_one(struct veth_rq *rq,
if (likely(xdp_prog)) {
struct veth_xdp_buff vxbuf;
struct xdp_buff *xdp = &vxbuf.xdp;
+ struct xdp_rxq_info rxq;
u32 act;
xdp_convert_frame_to_buff(frame, xdp);
- xdp->rxq = &rq->xdp_rxq;
+
+ rxq = rq->xdp_rxq;
+ rxq.mem.type = frame->mem_type;
+ xdp->rxq = &rxq;
+
vxbuf.skb = NULL;
act = bpf_prog_run_xdp(xdp_prog, xdp);
@@ -669,19 +677,16 @@ static struct xdp_frame *veth_xdp_rcv_one(struct veth_rq *rq,
break;
case XDP_TX:
orig_frame = *frame;
- xdp->rxq->mem.type = frame->mem_type;
if (unlikely(veth_xdp_tx(rq, xdp, bq) < 0)) {
trace_xdp_exception(rq->dev, xdp_prog, act);
frame = &orig_frame;
stats->rx_drops++;
goto err_xdp;
}
- stats->xdp_tx++;
rcu_read_unlock();
goto xdp_xmit;
case XDP_REDIRECT:
orig_frame = *frame;
- xdp->rxq->mem.type = frame->mem_type;
if (xdp_do_redirect(rq->dev, xdp, xdp_prog)) {
frame = &orig_frame;
stats->rx_drops++;
@@ -741,71 +746,26 @@ static void veth_xdp_rcv_bulk_skb(struct veth_rq *rq, void **frames,
}
}
-static void veth_xdp_get(struct xdp_buff *xdp)
-{
- struct skb_shared_info *sinfo = xdp_get_shared_info_from_buff(xdp);
- int i;
-
- get_page(virt_to_page(xdp->data));
- if (likely(!xdp_buff_has_frags(xdp)))
- return;
-
- for (i = 0; i < sinfo->nr_frags; i++)
- netmem_inc_ref(skb_frag_netmem(&sinfo->frags[i]));
-}
-
-static int veth_convert_skb_to_xdp_buff(struct veth_rq *rq,
- struct xdp_buff *xdp,
- struct sk_buff **pskb)
-{
- struct sk_buff *skb = *pskb;
- u32 frame_sz;
-
- if (skb_shared(skb) || skb_head_is_locked(skb) ||
- skb_is_nonlinear(skb) ||
- skb_headroom(skb) < XDP_PACKET_HEADROOM) {
- if (skb_pp_cow_data(rq->page_pool, pskb, XDP_PACKET_HEADROOM))
- goto drop;
-
- skb = *pskb;
- }
-
- /* SKB "head" area always have tailroom for skb_shared_info */
- frame_sz = skb_end_pointer(skb) - skb->head;
- frame_sz += SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
- xdp_init_buff(xdp, frame_sz, &rq->xdp_rxq);
- xdp_prepare_buff(xdp, skb->head, skb_headroom(skb),
- skb_headlen(skb), true);
-
- if (skb_shinfo(skb)->nr_frags) {
- skb_shinfo(skb)->xdp_frags_size = skb->data_len;
- xdp_buff_set_frags_flag(xdp);
- } else {
- xdp_buff_clear_frags_flag(xdp);
- }
- *pskb = skb;
-
- return 0;
-drop:
- consume_skb(skb);
- *pskb = NULL;
-
- return -ENOMEM;
-}
-
static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq,
struct sk_buff *skb,
- struct veth_xdp_tx_bq *bq,
struct veth_stats *stats)
{
- void *orig_data, *orig_data_end;
struct bpf_prog *xdp_prog;
struct veth_xdp_buff vxbuf;
- struct xdp_buff *xdp = &vxbuf.xdp;
- u32 act, metalen;
- int off;
+ struct xdp_generic_ctx ctx = {
+ .page_pool = rq->page_pool,
+ .xdp_rxq = &rq->xdp_rxq,
+ .xdp_skb = &vxbuf.skb,
+ .xdp = &vxbuf.xdp,
+ };
+ int ret;
skb_prepare_for_gro(skb);
+ /* Generic XDP expects skb->mac_len to be set, as
+ * __netif_receive_skb_core() does before running it: the generic
+ * cpumap redirect pulls it before queueing the skb.
+ */
+ skb->mac_len = skb->data - skb_mac_header(skb);
rcu_read_lock();
xdp_prog = rcu_dereference(rq->xdp_prog);
@@ -814,100 +774,36 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq,
goto out;
}
- __skb_push(skb, skb->data - skb_mac_header(skb));
- if (veth_convert_skb_to_xdp_buff(rq, xdp, &skb))
- goto drop;
- vxbuf.skb = skb;
-
- orig_data = xdp->data;
- orig_data_end = xdp->data_end;
-
- act = bpf_prog_run_xdp(xdp_prog, xdp);
-
- switch (act) {
+ ret = __do_xdp_generic(xdp_prog, &skb, &ctx);
+ switch (ctx.act) {
case XDP_PASS:
break;
case XDP_TX:
- veth_xdp_get(xdp);
- consume_skb(skb);
- xdp->rxq->mem = rq->xdp_mem;
- if (unlikely(veth_xdp_tx(rq, xdp, bq) < 0)) {
- trace_xdp_exception(rq->dev, xdp_prog, act);
- stats->rx_drops++;
- goto err_xdp;
- }
- stats->xdp_tx++;
- rcu_read_unlock();
- goto xdp_xmit;
+ /* A failed transmit is counted in the device's tx_dropped
+ * by generic_xdp_tx(), so it is not added to xdp_tx_err.
+ */
+ if (!ctx.err)
+ stats->xdp_skb_tx++;
+ break;
case XDP_REDIRECT:
- veth_xdp_get(xdp);
- consume_skb(skb);
- xdp->rxq->mem = rq->xdp_mem;
- if (xdp_do_redirect(rq->dev, xdp, xdp_prog)) {
+ if (ctx.err)
stats->rx_drops++;
- goto err_xdp;
- }
- stats->xdp_redirect++;
- rcu_read_unlock();
- goto xdp_xmit;
+ else
+ stats->xdp_redirect++;
+ break;
default:
- bpf_warn_invalid_xdp_action(rq->dev, xdp_prog, act);
- fallthrough;
case XDP_ABORTED:
- trace_xdp_exception(rq->dev, xdp_prog, act);
- fallthrough;
case XDP_DROP:
stats->xdp_drops++;
- goto xdp_drop;
+ break;
}
rcu_read_unlock();
- /* check if bpf_xdp_adjust_head was used */
- off = orig_data - xdp->data;
- if (off > 0)
- __skb_push(skb, off);
- else if (off < 0)
- __skb_pull(skb, -off);
-
- skb_reset_mac_header(skb);
-
- /* XDP frag metadata (e.g. nr_frags) are updated in eBPF helpers
- * (e.g. bpf_xdp_adjust_tail). Remove the old fragment contribution
- * from skb->len before updating data_len, then add the new one back.
- */
- skb->len -= skb->data_len;
- if (xdp_buff_has_frags(xdp)) {
- skb->data_len = skb_shinfo(skb)->xdp_frags_size;
- skb->len += skb->data_len;
- } else {
- skb->data_len = 0;
- }
-
- /* Synchronize the skb tail with XDP's updated linear area. */
- off = xdp->data_end - orig_data_end;
- if (off != 0) {
- skb_set_tail_pointer(skb, xdp->data_end - xdp->data);
- skb->len += off; /* positive on grow, negative on shrink */
- }
-
- skb->protocol = eth_type_trans(skb, rq->dev);
-
- metalen = xdp->data - xdp->data_meta;
- if (metalen)
- skb_metadata_set(skb, metalen);
+ if (ret == XDP_PASS)
+ return skb;
+ return NULL;
out:
return skb;
-drop:
- stats->rx_drops++;
-xdp_drop:
- rcu_read_unlock();
- kfree_skb(skb);
- return NULL;
-err_xdp:
- rcu_read_unlock();
- xdp_return_buff(xdp);
-xdp_xmit:
- return NULL;
}
static int veth_xdp_rcv(struct veth_rq *rq, int budget,
@@ -943,7 +839,7 @@ static int veth_xdp_rcv(struct veth_rq *rq, int budget,
struct sk_buff *skb = ptr;
stats->xdp_bytes += skb->len;
- skb = veth_xdp_rcv_skb(rq, skb, bq, stats);
+ skb = veth_xdp_rcv_skb(rq, skb, stats);
if (skb) {
if (skb_shared(skb) || skb_unclone(skb, GFP_ATOMIC))
netif_receive_skb(skb);
@@ -959,6 +855,7 @@ static int veth_xdp_rcv(struct veth_rq *rq, int budget,
u64_stats_update_begin(&rq->stats.syncp);
rq->stats.vs.xdp_redirect += stats->xdp_redirect;
+ rq->stats.vs.xdp_tx += stats->xdp_skb_tx;
rq->stats.vs.xdp_bytes += stats->xdp_bytes;
rq->stats.vs.xdp_drops += stats->xdp_drops;
rq->stats.vs.rx_drops += stats->rx_drops;
@@ -992,7 +889,7 @@ static int veth_poll(struct napi_struct *napi, int budget)
if (stats.xdp_redirect > 0)
xdp_do_flush();
- if (stats.xdp_tx > 0)
+ if (bq.count > 0)
veth_xdp_flush(rq, &bq);
xdp_clear_return_frame_no_direct();
@@ -1138,13 +1035,10 @@ static int veth_enable_xdp_range(struct net_device *dev, int start, int end,
goto err_rxq_reg;
err = xdp_rxq_info_reg_mem_model(&rq->xdp_rxq,
- MEM_TYPE_PAGE_SHARED,
+ MEM_TYPE_MIXED,
NULL);
if (err < 0)
goto err_reg_mem;
-
- /* Save original mem info as it can be overwritten */
- rq->xdp_mem = rq->xdp_rxq.mem;
}
return 0;
@@ -1173,7 +1067,6 @@ static void veth_disable_xdp_range(struct net_device *dev, int start, int end,
for (i = start; i < end; i++) {
struct veth_rq *rq = &priv->rq[i];
- rq->xdp_rxq.mem = rq->xdp_mem;
xdp_rxq_info_unreg(&rq->xdp_rxq);
if (delete_napi)
--
2.56.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH RFC net-next v2 0/4] xdp: reuse generic skb XDP handling for veth
2026-11-10 10:19 [PATCH RFC net-next v2 0/4] xdp: reuse generic skb XDP handling for veth Slawomir Stepien
` (3 preceding siblings ...)
2026-11-10 10:19 ` [PATCH RFC net-next v2 4/4] veth: use generic skb XDP handling Slawomir Stepien
@ 2026-11-10 12:48 ` Maciej Fijalkowski
4 siblings, 0 replies; 6+ messages in thread
From: Maciej Fijalkowski @ 2026-11-10 12:48 UTC (permalink / raw)
To: Slawomir Stepien
Cc: netdev, bpf, Jakub Kicinski, Paolo Abeni, Eric Dumazet,
David S . Miller, Andrew Lunn, Simon Horman,
Björn Töpel, Jesper Dangaard Brouer,
Toke Høiland-Jørgensen, Lorenzo Bianconi,
Magnus Karlsson, Stanislav Fomichev, Alexei Starovoitov,
Daniel Borkmann, John Fastabend, Andrii Nakryiko,
Eduard Zingerman, Kumar Kartikeya Dwivedi, Martin KaFai Lau,
Song Liu, Yonghong Song, Jiri Olsa, Emil Tsalapatis,
Ihor Solodrai, Vladimir Oltean, Tianyi Gao, Liang Chen,
Yunsheng Lin, huangjie . albert
On Tue, Nov 10, 2026 at 11:19:37AM +0100, Slawomir Stepien wrote:
> Hi,
>
> this is v2 of Maciej's RFC [1], which makes veth run XDP on skb-backed
> packets through the generic skb XDP path instead of converting them
> into xdp_frames for XDP_TX and XDP_REDIRECT. The patches are Maciej's;
> I addressed the review of v1 and measured the performance, which was
> the main open question there. Changes I made are noted in [sst: ...]
> lines in each commit message.
>
> The motivation is unchanged: veth pins the skb data with plain page
> references (veth_xdp_get()) to build an xdp_frame from an skb, and the
> rxq memory type it reports does not match memory that was copied into
> a page_pool, which makes the AF_XDP selftests splat when a program
> shrinks a multi-buffer packet. With this series, pages copied into
> veth's page_pool are recycled (0 rx_pp_recycle_released_ref, was
> ~4.8M/s in the benchmark below).
>
> Dependencies
> ============
>
> The series applies to net-next with these two net series applied
> first (prerequisite-patch-id below):
>
> - Tianyi Gao, "net: fix mac_len of skbs copied by generic XDP" (1/2)
> [2]. Without it, cpumap redirect of skbs that veth copied drops
> every packet once veth uses the generic path.
> - Vladimir Oltean, "net: pass the XDP action to generic_xdp_tx() for
> tracing" and "net: validate xmit skbs in generic_xdp_tx()" [3].
> Patch 2/4 changes the same function.
>
> Changes in v2
> =============
>
> 1/4: MEM_TYPE_PAGE_POOL_OR_SHARED renamed to MEM_TYPE_MIXED (Björn);
> comment on what the type means.
> 2/4: Rebased on Vladimir's series. The redirect callers in devmap and
> filter still return success: generic_xdp_tx() frees and counts a
> failed skb itself, so propagating the error would free it twice
> (Björn asked why the error is not propagated).
> 3/4: One skb_needs_xdp_cow() predicate for generic XDP and veth
> (shared, cloned, nonlinear or headroom too small) instead of a
> per-caller callback and INDIRECT_CALL_2 (Jakub). The
> skb_head_is_locked() check is gone, it would copy every locally
> generated skb. Copies go to the caller's page_pool with
> skb_pp_cow_data() also for programs without frags support (found
> by the xdp_metadata selftest).
> 4/4: Commit message describes how XDP_TX/XDP_REDIRECT change for
> skb-backed packets (Björn's table). A failed skb XDP_TX is counted
> once, in the device's tx_dropped (Sashiko). veth sets
> skb->mac_len before running generic XDP, as
> __netif_receive_skb_core() does; without it cpumap redirect of
> veth skbs is dropped.
That looks sane.
>
> Performance
> ===========
>
> QEMU/KVM, 8 vCPUs, minimal config without debug options. pktgen sends
> skbs into veth v0, the XDP program runs on the peer v1 (skb path).
> Packets/s delivered to the destination, median of 12 runs of 8 s over
> 4 alternating boots per kernel, 95% bootstrap confidence intervals
> within about +-2%. Base: v7.4-rc1+ (f66469d9905b) with the
> dependencies above.
>
> 64 B 8000 B (3 frags)
> XDP_TX 4.49 -> 4.04 Mpps (-10%) 2.25 -> 1.87 (-17%)
> REDIRECT devmap -> veth 4.30 -> 3.95 (-8%) 2.18 -> 1.81 (-17%)
> REDIRECT cpumap 4.49 -> 4.47 (0%) 2.02 -> 2.27 (+12%)
> XDP_DROP (native veth) 5.65 -> 5.42 (-4%) 2.82 -> 2.71 (-4%)
> XDP_DROP (xdpgeneric) 5.63 -> 5.83 (+4%) 2.94 -> 3.00 (+2%)
>
> (cpumap 8000 B base: base without Vladimir's series, which does not
> touch that path; 3 of the 12 runs of the other base kernel stalled.)
>
> cpumap avoids the skb -> xdp_frame -> skb round trip. XDP_TX and
> devmap redirect to veth are slower for two reasons:
>
> - no bulking: one ndo_start_xmit() per skb instead of ndo_xdp_xmit()
> with up to 16 frames;
> - at 8000 B the receiving veth copies the packet a second time.
> Before, it got an xdp_frame; now it gets an skb with frags, and
> nonlinear skbs are always copied (kprobe on skb_pp_cow_data(): 1
> copy per packet before, 2 after).
>
> Vladimir's validation itself costs about 3% at 64 B on these two
> paths and nothing measurable elsewhere.
>
> Questions
> =========
>
> - Is the XDP_TX/devmap regression acceptable for the simpler and
> correct memory handling, or should it be fixed first? Options we
> see: skb-list bulking in the generic TX path (Jesper measured +30%
> with xmit_more), and not copying nonlinear skbs whose frags are
> already page_pool pages. Raising veth's needed_headroom (Jesper)
> only helps linear skbs from senders that honour it; pktgen skbs are
> shared and always copied, so the benchmark above cannot show it.
> We left all three for follow-ups.
Thanks for these tests, for some reason my local benchmarks were not
showing such differences and also xmit_more was not giving me much of a
bump.
The impact is noticeable I'd say. Could you see how xmit_more change would
affect the benchmarks?
> - MEM_TYPE_MIXED vs. separate memory types per source (Jesper).
> - The redirect semantics change in 4/4 (Björn).
>
> Testing
> =======
>
> Each commit builds with W=1 without new warnings, checkpatch --strict is
> clean. test_progs -t ns_xsk,xdp passes; test_xsk.sh shows the same DRV
> XDP_ADJUST_TAIL_GROW failures (also with busy poll) as without the
> series. Reproducers for shrinking frags in generic and veth XDP, XDP_TX
> of multi-buffer packets, devmap egress programs and bpf_prog_test_run
> after generic XDP on lo pass without kernel warnings.
>
> The scripts, kernel configs and raw logs behind the numbers and tests
> above are in [4], with instructions to rerun them.
>
> AI disclosure: the review changes, the tests and the benchmark were
> prepared with help from an AI coding assistant (Claude); I reviewed
> and tested all changes.
>
> [1] https://lore.kernel.org/all/20260509084858.773921-1-maciej.fijalkowski@intel.com/
> [2] https://lore.kernel.org/all/20261103200036.65764-3-tianyi@cloudflare.com/
> [3] https://lore.kernel.org/all/20261030114753.185281-1-vladimir.oltean@nxp.com/
> [4] https://github.com/s-stepien/xdp-rfc
>
> Thanks,
> Slawomir
>
> Maciej Fijalkowski (4):
> xdp: add mixed page_pool/page_shared memory type
> xdp: return status from generic_xdp_tx()
> xdp: split generic XDP skb handling
> veth: use generic skb XDP handling
>
> drivers/net/veth.c | 191 +++++++++-----------------------------
> include/linux/netdevice.h | 33 ++++++-
> include/net/xdp.h | 1 +
> net/core/dev.c | 122 ++++++++++++++++++------
> net/core/filter.c | 7 ++
> net/core/xdp.c | 54 ++++++++++-
> 6 files changed, 222 insertions(+), 186 deletions(-)
>
>
> base-commit: fa5196392691051963726787944db872420dfa75
> prerequisite-patch-id: 7db3107f14b28c509845871045e6af93b7e267bc
> prerequisite-patch-id: 40350101232837c5a38ba1662825228d01734b39
> prerequisite-patch-id: 0b57f8c2a409470b1ddf8df747d906d5d3dbd9f3
> --
> 2.56.0
>
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-11-10 12:49 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-10 10:19 [PATCH RFC net-next v2 0/4] xdp: reuse generic skb XDP handling for veth Slawomir Stepien
2026-11-10 10:19 ` [PATCH RFC net-next v2 1/4] xdp: add mixed page_pool/page_shared memory type Slawomir Stepien
2026-11-10 10:19 ` [PATCH RFC net-next v2 2/4] xdp: return status from generic_xdp_tx() Slawomir Stepien
2026-11-10 10:19 ` [PATCH RFC net-next v2 3/4] xdp: split generic XDP skb handling Slawomir Stepien
2026-11-10 10:19 ` [PATCH RFC net-next v2 4/4] veth: use generic skb XDP handling Slawomir Stepien
2026-11-10 12:48 ` [PATCH RFC net-next v2 0/4] xdp: reuse generic skb XDP handling for veth Maciej Fijalkowski
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox