[PATCH RFC] xdp: recycle page_pool pages correctly in __xdp_return()

[PATCH RFC] xdp: recycle page_pool pages correctly in __xdp_return()

[syzbot]

In generic XDP with multi-buffer support (BPF_F_XDP_HAS_FRAGS), incoming
non-linear skbs have their fragments copied into page_pool-backed pages
allocated from the per-CPU system_page_pool via skb_cow_data_for_xdp() ->
skb_pp_cow_data(). However, bpf_prog_run_generic_xdp() initializes the
xdp_buff using &rxqueue->xdp_rxq, which has no registered memory model and
defaults to rxq->mem.type == MEM_TYPE_PAGE_SHARED (0). When the XDP program
shrinks the packet and completely consumes one or more fragments (such as
via bpf_xdp_adjust_tail() with a negative offset or bpf_xdp_pull_data()),
bpf_xdp_shrink_data() passes xdp->rxq->mem.type (MEM_TYPE_PAGE_SHARED) to
__xdp_return(). Instead of returning the page to its page_pool via
page_pool_put_full_netmem(), __xdp_return() invokes page_frag_free(), which
decrements the refcount to zero and frees the page directly to the buddy
allocator. Because the page was allocated from a page_pool, page->pp_magic
still contains PP_SIGNATURE, causing free_page_is_bad() to detect a
"page_pool leak" and trigger a bad_page() BUG:

BUG: Bad page state in process ...  pfn:31a2c
page: refcount:0 mapcount:0 mapping:0000000000000000
index:0xffff888031a2cd20 pfn:0x31a2c
flags: 0xfff00000000000(node=0|zone=1|lastcpupid=0x7ff)
page dumped because: page_pool leak
Call Trace:
 <TASK>
 bad_page+0x17f/0x1c0 mm/page_alloc.c:647
 free_page_is_bad mm/page_alloc.c:1082 [inline]
 __free_pages_prepare mm/page_alloc.c:1409 [inline]
 __free_frozen_pages+0xcd7/0xd30 mm/page_alloc.c:2962
 bpf_xdp_shrink_data net/core/filter.c:4383 [inline]
 bpf_xdp_frags_shrink_tail+0x4ed/0x800 net/core/filter.c:4407
 ____bpf_xdp_adjust_tail net/core/filter.c:4429 [inline]
 bpf_xdp_adjust_tail+0x1ce/0x210 net/core/filter.c:4422
 bpf_prog_run_generic_xdp+0x612/0x1560 net/core/dev.c:5579
 netif_receive_generic_xdp net/core/dev.c:5699 [inline]
 do_xdp_generic+0xae1/0x12f0 net/core/dev.c:5761
 __netif_receive_skb_core+0x10e3/0x30c0 net/core/dev.c:6107
 ...
 </TASK>

A similar mismatch occurs in veth and devmap. In veth_xdp_rcv_one(),
xdp->rxq->mem.type was only assigned to frame->mem_type after executing the
XDP program on XDP_TX and XDP_REDIRECT, leaving xdp->rxq->mem.type
uninitialized during program execution. In addition,
veth_convert_skb_to_xdp_buff() did not reset rq->xdp_rxq.mem to rq->xdp_mem
prior to running the program, allowing modifications from previous frames
to leak. In devmap, dev_map_bpf_prog_run() constructed a local struct
xdp_rxq_info on the stack without setting rxq.mem.type to xdpf->mem_type,
leaving it as MEM_TYPE_PAGE_SHARED (0) while running the egress program.

Fix this by checking netmem_is_pp() under MEM_TYPE_PAGE_SHARED in
__xdp_return(). If the netmem belongs to a page_pool, fall through to
MEM_TYPE_PAGE_POOL and return it via page_pool_put_full_netmem();
otherwise, release it with page_frag_free(). Placing MEM_TYPE_PAGE_SHARED
immediately before MEM_TYPE_PAGE_POOL keeps the native MEM_TYPE_PAGE_POOL
fast path branch-free. In addition, set xdp->rxq->mem.type =
frame->mem_type in veth_xdp_rcv_one() before calling bpf_prog_run_xdp(),
restore rq->xdp_rxq.mem prior to program execution in
veth_convert_skb_to_xdp_buff(), and initialize rxq.mem.type =
xdpf->mem_type in dev_map_bpf_prog_run().

Fixes: e6d5dbdd20aa ("xdp: add multi-buff support for xdp running in generic mode")
Assisted-by: Gemini:gemini-4-argon Gemini:gemini-3.8-flash syzbot
Reported-by: syzbot+ff145014d6b0ce64a173@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=ff145014d6b0ce64a173
Link: https://syzkaller.appspot.com/ai_job?id=10925bb2-1509-4501-b65c-50070666ed06
To: "Andrew Lunn" <andrew+netdev@lunn.ch>
To: "Andrii Nakryiko" <andrii@kernel.org>
To: "Alexei Starovoitov" <ast@kernel.org>
To: <bpf@vger.kernel.org>
To: "Daniel Borkmann" <daniel@iogearbox.net>
To: "David S. Miller" <davem@davemloft.net>
To: "Eduard Zingerman" <eddyz87@gmail.com>
To: "Eric Dumazet" <edumazet@kernel.org>
To: "Jesper Dangaard Brouer" <hawk@kernel.org>
To: "John Fastabend" <john.fastabend@gmail.com>
To: "Jakub Kicinski" <kuba@kernel.org>
To: "Kumar Kartikeya Dwivedi" <memxor@gmail.com>
To: <netdev@vger.kernel.org>
To: "Paolo Abeni" <pabeni@redhat.com>
To: "Lorenzo Bianconi" <lorenzo@kernel.org>
Cc: "Emil Tsalapatis" <emil@etsalapatis.com>
Cc: "Simon Horman" <horms@kernel.org>
Cc: "Ihor Solodrai" <ihor.solodrai@linux.dev>
Cc: "Jiri Olsa" <jolsa@kernel.org>
Cc: <linux-kernel@vger.kernel.org>
Cc: "Martin KaFai Lau" <martin.lau@linux.dev>
Cc: "Stanislav Fomichev" <sdf@fomichev.me>
Cc: "Song Liu" <song@kernel.org>
Cc: "Yonghong Song" <yonghong.song@linux.dev>

---
diff --git a/drivers/net/veth.c b/drivers/net/veth.c
index 643b97dc5..745edf2c7 100644
--- a/drivers/net/veth.c
+++ b/drivers/net/veth.c
@@ -653,6 +653,7 @@ static struct xdp_frame *veth_xdp_rcv_one(struct veth_rq *rq,
 
 		xdp_convert_frame_to_buff(frame, xdp);
 		xdp->rxq = &rq->xdp_rxq;
+		xdp->rxq->mem.type = frame->mem_type;
 		vxbuf.skb = NULL;
 
 		act = bpf_prog_run_xdp(xdp_prog, xdp);
@@ -664,7 +665,6 @@ static struct xdp_frame *veth_xdp_rcv_one(struct veth_rq *rq,
 			break;
 		case XDP_TX:
 			orig_frame = *frame;
-			xdp->rxq->mem.type = frame->mem_type;
 			if (unlikely(veth_xdp_tx(rq, xdp, bq) < 0)) {
 				trace_xdp_exception(rq->dev, xdp_prog, act);
 				frame = &orig_frame;
@@ -676,7 +676,6 @@ static struct xdp_frame *veth_xdp_rcv_one(struct veth_rq *rq,
 			goto xdp_xmit;
 		case XDP_REDIRECT:
 			orig_frame = *frame;
-			xdp->rxq->mem.type = frame->mem_type;
 			if (xdp_do_redirect(rq->dev, xdp, xdp_prog)) {
 				frame = &orig_frame;
 				stats->rx_drops++;
@@ -768,6 +767,7 @@ static int veth_convert_skb_to_xdp_buff(struct veth_rq *rq,
 	/* SKB "head" area always have tailroom for skb_shared_info */
 	frame_sz = skb_end_pointer(skb) - skb->head;
 	frame_sz += SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
+	rq->xdp_rxq.mem = rq->xdp_mem;
 	xdp_init_buff(xdp, frame_sz, &rq->xdp_rxq);
 	xdp_prepare_buff(xdp, skb->head, skb_headroom(skb),
 			 skb_headlen(skb), true);
@@ -825,7 +825,6 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq,
 	case XDP_TX:
 		veth_xdp_get(xdp);
 		consume_skb(skb);
-		xdp->rxq->mem = rq->xdp_mem;
 		if (unlikely(veth_xdp_tx(rq, xdp, bq) < 0)) {
 			trace_xdp_exception(rq->dev, xdp_prog, act);
 			stats->rx_drops++;
@@ -837,7 +836,6 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq,
 	case XDP_REDIRECT:
 		veth_xdp_get(xdp);
 		consume_skb(skb);
-		xdp->rxq->mem = rq->xdp_mem;
 		if (xdp_do_redirect(rq->dev, xdp, xdp_prog)) {
 			stats->rx_drops++;
 			goto err_xdp;
diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c
index dc7b859e8..a138b9e8f 100644
--- a/kernel/bpf/devmap.c
+++ b/kernel/bpf/devmap.c
@@ -348,6 +348,7 @@ static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog,
 		u32 act;
 		int err;
 
+		rxq.mem.type = xdpf->mem_type;
 		xdp_convert_frame_to_buff(xdpf, &xdp);
 		xdp.txq = &txq;
 		xdp.rxq = &rxq;
diff --git a/net/core/xdp.c b/net/core/xdp.c
index 1d679e8fd..40360ff95 100644
--- a/net/core/xdp.c
+++ b/net/core/xdp.c
@@ -23,6 +23,8 @@
 #include <trace/events/xdp.h>
 #include <net/xdp_sock_drv.h>
 
+#include "netmem_priv.h"
+
 #define REG_STATE_NEW		0x0
 #define REG_STATE_REGISTERED	0x1
 #define REG_STATE_UNREGISTERED	0x2
@@ -434,6 +436,12 @@ void __xdp_return(netmem_ref netmem, enum xdp_mem_type mem_type,
 		  bool napi_direct, struct xdp_buff *xdp)
 {
 	switch (mem_type) {
+	case MEM_TYPE_PAGE_SHARED:
+		if (!netmem_is_pp(netmem_compound_head(netmem))) {
+			page_frag_free(__netmem_address(netmem));
+			break;
+		}
+		fallthrough;
 	case MEM_TYPE_PAGE_POOL:
 		netmem = netmem_compound_head(netmem);
 		if (napi_direct && xdp_return_frame_no_direct())
@@ -444,9 +452,6 @@ void __xdp_return(netmem_ref netmem, enum xdp_mem_type mem_type,
 		page_pool_put_full_netmem(netmem_get_pp(netmem), netmem,
 					  napi_direct);
 		break;
-	case MEM_TYPE_PAGE_SHARED:
-		page_frag_free(__netmem_address(netmem));
-		break;
 	case MEM_TYPE_PAGE_ORDER0:
 		put_page(__netmem_to_page(netmem));
 		break;


base-commit: c03c1a14507dfaf8b856c2608f9ac216021fc222
-- 
This is an AI-generated patch subject to moderation.
Reply with '#syz upstream' to Sign-off the patch as a human author
and send it to the upstream kernel mailing lists.
Reply with '#syz reject' to reject it ('#syz unreject' to undo).

See https://goo.gle/syzbot-ai-patches for information about AI-generated patches.
You can comment on the patch as usual, syzbot will try to address
the comments and send a new version of the patch if necessary.
syzbot engineers can be reached at syzkaller@googlegroups.com.