[PATCH net] page_pool: keep fragments cache-line aligned

[PATCH net] page_pool: keep fragments cache-line aligned

[Fikret Pirim]

page_pool_alloc_frag_netmem() rounds the requested fragment size up to
dma_get_cache_alignment(), which is 1 on architectures that do not define
ARCH_DMA_MINALIGN -- x86 among them, where the helper falls through to
"return 1" (include/linux/dma-mapping.h, include/linux/cache.h).

Two callers on the shared per-CPU system page_pool ask for arbitrary byte
lengths.  skb_pp_cow_data() in its fragment loop:

	size = min_t(u32, len, PAGE_SIZE);
	truesize = size;
	page = page_pool_dev_alloc(pool, &page_off, &truesize);

and xdp_copy_frags_from_zc() likewise (u32 truesize = len).  Once such a
fragment is handed out, pool->frag_offset sits at an arbitrary byte offset,
and the next page_pool_alloc_va() from the same pool returns a misaligned
buffer.  Both callers turn that buffer into an skb with napi_build_skb(),
and __build_skb_around() stores it verbatim:

	skb->head = data;
	skb_set_end_offset(skb, size);

so skb_shinfo(skb) == skb->head + skb->end inherits the misalignment.
atomic_set(&shinfo->dataref, 1) is a plain store and stays silent, but the
first skb_clone() does atomic_inc() on it.  dataref sits at offset 32 in
struct skb_shared_info, so whenever (head + end) % 64 lands in 61..63 the
locked dword straddles a cache line and x86 raises #AC.  On a kernel with
split-lock detection fatal for kernel code that is a panic:

	Oops: Split lock detected
	RIP: 0010:skb_clone+0x159/0x1e0
	 raw_local_deliver+0x1dd/0x290		/* or __udp4_lib_rcv+0x703 */
	 ip_protocol_deliver_rcu+0x51/0x1a0
	 ip_local_deliver_finish+0x85/0x100
	 __netif_receive_skb_one_core+0x85/0xa0
	 process_backlog+0x98/0x150
	 ...
	 __local_bh_enable_ip+0x62/0x70
	 __dev_queue_xmit+0x3cf/0x11a0
	 udp_sendmsg+0xb79/0xfe0

I hit this three times on an i9-13900K, on 6.18.48 and on 7.1.11, while a
generic-mode XDP program with frags support was attached to lo by a NetBird
VPN agent.  Locally generated traffic on lo always has headroom <
XDP_PACKET_HEADROOM, so netif_skb_check_for_xdp() -> skb_cow_data_for_xdp()
-> skb_pp_cow_data() runs for every packet, and any nonlinear one poisons
the pool for the packets that follow.  The three dumps agree with the
arithmetic: skb_shinfo % 64 was 30, 29 and 30, putting the 4-byte dataref
at 62..65, 61..64 and 62..65.  Cloning came from multicast fan-out and from
raw sockets.  On the same machine, with no special load, a read-only
bpftrace kretprobe on __napi_build_skb() counted 40 misaligned heads in 85
seconds against 2033 aligned ones.

Where split-lock detection is not available the misplaced dataref corrupts
whatever it lands on instead; I suspect this is also behind the syzbot
report "KASAN: use-after-free Read in skb_clone" (31 Aug 2026,
extid=22c4f9a7026c86bcc3b8), where the head is freed early.

Round fragments up to at least SMP_CACHE_BYTES.  Architectures with
ARCH_DMA_MINALIGN already get this or stricter rounding from the same line,
so this only restores on x86 the alignment that page_pool_alloc_va() users
implicitly rely on.

Fixes: FIXME_HASH ("page_pool: introduce page_pool_alloc() API")
Signed-off-by: Fikret Pirim <fpirim@gmail.com>
---
Tested on 6.18.48 in an 8 vCPU VM: an XDP_PASS program declared
SEC("xdp.frags") attached to lo in generic mode, 8000-byte UDP datagrams
streamed over lo to drive the fragment loop, and 86-byte multicast to eight
listeners to clone the next skb.  Misalignment is visible without
split-lock hardware by adding

	WARN(!IS_ALIGNED((unsigned long)data, 8), "unaligned head %px frag_size %u\n", ...)

at the top of __build_skb_around() and booting with panic_on_warn=1:

	RIP: 0010:__build_skb_around+0x28c/0x370
	R12: ffff88812a58bba6			/* head, 38 mod 64 */
	R13: 000000000000045a			/* frag_size */
	 __napi_build_skb+0x39/0x50
	 skb_pp_cow_data+0x233/0xfc0
	 do_xdp_generic+0x377/0xd90
	 __netif_receive_skb_core.constprop.0+0x533/0x3f10
	 __netif_receive_skb_one_core+0xe5/0x520
	 process_backlog+0x196/0x5a0
	 __napi_poll+0xb2/0x450
	 net_rx_action+0x418/0xdc0
	 handle_softirqs+0x19e/0x4b0
	 </IRQ>
	 __local_bh_enable_ip+0x62/0x70
	 __dev_queue_xmit+0xcaf/0x3510
	 ip_finish_output2+0x990/0x1f20
	 udp_send_skb+0x60f/0x1310
	 udp_sendmsg+0x1586/0x2230

  unpatched                            trap after ~25 s (twice)
  this patch                           375 s clean, 97 M + 146 M deliveries
  align the caller instead             378 s clean, 91 M + 138 M deliveries
  unpatched, XDP program without frags 272 s clean (pskb_expand_head path)
  unpatched, no XDP program            200 s clean

I also have the caller-side alternative that rounds up the length
skb_pp_cow_data() asks for; I went with this one because
xdp_copy_frags_from_zc() has the same pattern and any future
page_pool_alloc_va() user would share the exposure.  Happy to send that
version instead if you prefer to leave page_pool's rounding alone.  Either
way a DEBUG_NET_WARN_ON_ONCE() on the head alignment in
__build_skb_around() would have caught this immediately.

 net/core/page_pool.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

--- a/net/core/page_pool.c
+++ b/net/core/page_pool.c
@@ -1061,7 +1061,16 @@
 	if (WARN_ON(size > max_size))
 		return 0;
 
-	size = ALIGN(size, dma_get_cache_alignment());
+	/* Keep fragments cache-line aligned. dma_get_cache_alignment() is 1 on
+	 * architectures without ARCH_DMA_MINALIGN (x86 among them), and callers
+	 * such as skb_pp_cow_data() and xdp_copy_frags_from_zc() request
+	 * arbitrary byte lengths; an odd-sized fragment would leave frag_offset
+	 * misaligned and the next page_pool_alloc_va() buffer would be built into
+	 * an skb whose skb_shared_info is misaligned (split-lock #AC in the first
+	 * skb_clone() on x86, refcount corruption elsewhere).
+	 */
+	size = ALIGN(size, max_t(unsigned int, dma_get_cache_alignment(),
+				 SMP_CACHE_BYTES));
 	*offset = pool->frag_offset;
 
 	if (netmem && *offset + size > max_size) {