[PATCH v2 bpf-next 0/2] bpf: Populate mmap-able array maps lazily

[PATCH v2 bpf-next 0/2] bpf: Populate mmap-able array maps lazily

[Song Liu]

An mmap-able BPF array map (BPF_F_MMAPABLE) has its backing memory
vmalloc'ed up front at map creation time. array_map_mmap() then wires up
the whole mapping eagerly via remap_vmalloc_range(), which calls
vm_insert_page() for every page of the map. This makes every mmap() cost
O(number of pages): an 8MiB map inserts 2048 PTEs on each mmap() and
tears them all down again on munmap(), even when user space only touches
a few pages (or none at all).

This series makes the mapping populate lazily instead, the same way the
arena map already does, turning mmap()/munmap() of an mmap-able array
from O(map size) into O(1). The pages are inserted on demand by a new
map_mmap_fault() callback, dispatched from the shared
bpf_map_default_vmops so that the existing VMA open/close accounting
(VM_MAYWRITE write-active tracking, freeze handling) stays centralized.

Callers that want the pages populated up front can still request that
explicitly with MAP_POPULATE, matching regular file semantics.
Kernel-side access to the map (via the vmalloc address) is unaffected.

Patch 1 implements the lazy faulting. Patch 2 adds an "arraymap-mmap"
benchmark that mmap()s/munmap()s an mmap-able array map, useful to
observe how the cost scales with the mapping size.

Changes in v2:
- Compute the mapping offset with 64-bit arithmetic in array_map_mmap()
  and the fault handler so that a large vm_pgoff cannot overflow the
  bounds check on 32-bit architectures. remap_vmalloc_range(), which is
  no longer used, used to guard against this.
- Benchmark: use the standard -p/--producers option for the thread count
  instead of a custom --nr-threads option. The v1 approach set
  env.producer_cnt from validate(), which runs after the framework has
  already sized the producers array, causing an out-of-bounds write.
(Both issues were pointed out by the Sashiko AI review.)

v1: https://lore.kernel.org/bpf/20260722065308.4116186-1-song@kernel.org/

Benchmark results (single producer thread, ./bench -p1):

Before the patch
  nr_threads: 1, map_size: 1048576
arraymap-mmap: throughput: 0.034 +/- 0.000 M ops/s, latency: 29187.164 ns/op

  nr_threads: 1, map_size: 8388608
arraymap-mmap: throughput: 0.005 +/- 0.000 M ops/s, latency: 220910.150 ns/op

  nr_threads: 1, map_size: 67108864
arraymap-mmap: throughput: 0.001 +/- 0.000 M ops/s, latency: 1765083.440 ns/op

After the patch
  nr_threads: 1, map_size: 1048576
arraymap-mmap: throughput: 0.838 +/- 0.001 M ops/s, latency: 1193.553 ns/op

  nr_threads: 1, map_size: 8388608
arraymap-mmap: throughput: 0.811 +/- 0.002 M ops/s, latency: 1232.728 ns/op

  nr_threads: 1, map_size: 67108864
arraymap-mmap: throughput: 0.747 +/- 0.004 M ops/s, latency: 1339.051 ns/op

Song Liu (2):
  bpf: Populate mmap-able array map memory lazily
  selftests/bpf: Add mmap/munmap benchmark for array maps

 include/linux/bpf.h                           |   1 +
 kernel/bpf/arraymap.c                         |  43 +++++-
 kernel/bpf/syscall.c                          |  17 +++
 tools/testing/selftests/bpf/Makefile          |   1 +
 tools/testing/selftests/bpf/bench.c           |   4 +
 .../bpf/benchs/bench_arraymap_mmap.c          | 127 ++++++++++++++++++
 .../bpf/benchs/run_bench_arraymap_mmap.sh     |  15 +++
 7 files changed, 204 insertions(+), 4 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c
 create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_arraymap_mmap.sh

--
2.53.0-Meta

[PATCH v2 bpf-next 1/2] bpf: Populate mmap-able array map memory lazily

[Song Liu]

An mmap-able BPF array map (BPF_F_MMAPABLE) has its backing memory
vmalloc'ed up front at map creation time. array_map_mmap() then wired up
the whole mapping eagerly via remap_vmalloc_range(), which calls
vm_insert_page() for every page of the map. For large maps this makes
every mmap() O(number of pages): an 8MiB map inserts 2048 PTEs per
mmap() and tears them all down again on munmap(), even when user space
only touches a few pages (or none at all).

Populate the mapping lazily instead, the same way the arena map already
does. array_map_mmap() now only performs the bounds check and returns,
leaving the PTEs unpopulated; the pages are inserted on demand by a new
array_map_mmap_fault() handler. Because the memory is already resident,
the fault handler simply resolves the vmalloc page and hands it to the
fault path.

To keep the existing VMA open/close accounting (VM_MAYWRITE write-active
tracking, freeze handling) centralized, the fault handler is reached
through a new optional ->map_mmap_fault callback dispatched from the
shared bpf_map_default_vmops, rather than each map installing its own
vm_operations_struct.

remap_vmalloc_range() previously guarded against offset overflow; since
it is no longer used, array_map_mmap() and the fault handler compute the
mapping offset with 64-bit arithmetic so that a large vm_pgoff cannot
overflow the bounds check on 32-bit architectures.

This turns mmap()/munmap() of an mmap-able array from O(map size) into
O(1). Callers that want the pages populated up front can still request
that explicitly with MAP_POPULATE, matching regular file semantics.
Kernel-side access to the map (via the vmalloc address) is unaffected.

Signed-off-by: Song Liu <song@kernel.org>
Assisted-by: Claude:claude-opus-4-8
---
 include/linux/bpf.h   |  1 +
 kernel/bpf/arraymap.c | 43 +++++++++++++++++++++++++++++++++++++++----
 kernel/bpf/syscall.c  | 17 +++++++++++++++++
 3 files changed, 57 insertions(+), 4 deletions(-)

diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index e066f44a9c05..b536f3c9e103 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -145,6 +145,7 @@ struct bpf_map_ops {
 	int (*map_direct_value_meta)(const struct bpf_map *map,
 				     u64 imm, u32 *off);
 	int (*map_mmap)(struct bpf_map *map, struct vm_area_struct *vma);
+	vm_fault_t (*map_mmap_fault)(struct bpf_map *map, struct vm_fault *vmf);
 	__poll_t (*map_poll)(struct bpf_map *map, struct file *filp,
 			     struct poll_table_struct *pts);
 	unsigned long (*map_get_unmapped_area)(struct file *filep, unsigned long addr,
diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c
index 248b4818178c..59fda56062a9 100644
--- a/kernel/bpf/arraymap.c
+++ b/kernel/bpf/arraymap.c
@@ -576,17 +576,51 @@ static int array_map_check_btf(struct bpf_map *map,
 static int array_map_mmap(struct bpf_map *map, struct vm_area_struct *vma)
 {
 	struct bpf_array *array = container_of(map, struct bpf_array, map);
-	pgoff_t pgoff = PAGE_ALIGN(sizeof(*array)) >> PAGE_SHIFT;
 
 	if (!(map->map_flags & BPF_F_MMAPABLE))
 		return -EINVAL;
 
-	if (vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) >
+	/* use u64 math so the offset cannot overflow on 32-bit archs */
+	if ((u64)vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) >
 	    PAGE_ALIGN((u64)array->map.max_entries * array->elem_size))
 		return -EINVAL;
 
-	return remap_vmalloc_range(vma, array_map_vmalloc_addr(array),
-				   vma->vm_pgoff + pgoff);
+	/*
+	 * The backing memory is vmalloc'ed up front, so instead of wiring up
+	 * every PTE here we let array_map_mmap_fault() insert pages on demand.
+	 * remap_vmalloc_range_partial() used to set these flags for us; keep
+	 * them to preserve behavior (no VMA expansion, excluded from coredump).
+	 */
+	vm_flags_set(vma, VM_DONTEXPAND | VM_DONTDUMP);
+
+	return 0;
+}
+
+static vm_fault_t array_map_mmap_fault(struct bpf_map *map,
+				       struct vm_fault *vmf)
+{
+	struct bpf_array *array = container_of(map, struct bpf_array, map);
+	pgoff_t pgoff = PAGE_ALIGN(sizeof(*array)) >> PAGE_SHIFT;
+	struct page *page;
+	void *kaddr;
+
+	/*
+	 * vmf->pgoff is the page offset within the mapping (vma->vm_pgoff plus
+	 * the offset of the faulting address). The data pages start after the
+	 * bpf_array header, hence the extra pgoff. This mirrors the offset that
+	 * remap_vmalloc_range(vma, ..., vma->vm_pgoff + pgoff) used to apply.
+	 */
+	kaddr = array_map_vmalloc_addr(array) +
+		(((u64)vmf->pgoff + pgoff) << PAGE_SHIFT);
+
+	page = vmalloc_to_page(kaddr);
+	if (!page)
+		return VM_FAULT_SIGBUS;
+
+	get_page(page);
+	vmf->page = page;
+
+	return 0;
 }
 
 static bool array_map_meta_equal(const struct bpf_map *meta0,
@@ -812,6 +846,7 @@ const struct bpf_map_ops array_map_ops = {
 	.map_direct_value_addr = array_map_direct_value_addr,
 	.map_direct_value_meta = array_map_direct_value_meta,
 	.map_mmap = array_map_mmap,
+	.map_mmap_fault = array_map_mmap_fault,
 	.map_seq_show_elem = array_map_seq_show_elem,
 	.map_check_btf = array_map_check_btf,
 	.map_lookup_batch = generic_map_lookup_batch,
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 0ff9e3aa293d..fb0e1f55c487 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -1077,9 +1077,26 @@ static void bpf_map_mmap_close(struct vm_area_struct *vma)
 		bpf_map_write_active_dec(map);
 }
 
+/*
+ * Called for maps that populate their memory-mapped region lazily, i.e. that
+ * return without wiring up any PTEs from their ->map_mmap callback. Maps that
+ * populate the whole VMA eagerly (e.g. via remap_vmalloc_range()) never reach
+ * here.
+ */
+static vm_fault_t bpf_map_mmap_fault(struct vm_fault *vmf)
+{
+	struct bpf_map *map = vmf->vma->vm_private_data;
+
+	if (!map->ops->map_mmap_fault)
+		return VM_FAULT_SIGBUS;
+
+	return map->ops->map_mmap_fault(map, vmf);
+}
+
 static const struct vm_operations_struct bpf_map_default_vmops = {
 	.open		= bpf_map_mmap_open,
 	.close		= bpf_map_mmap_close,
+	.fault		= bpf_map_mmap_fault,
 };
 
 static int bpf_map_mmap(struct file *filp, struct vm_area_struct *vma)
-- 
2.53.0-Meta

[PATCH v2 bpf-next 2/2] selftests/bpf: Add mmap/munmap benchmark for array maps

[Song Liu]

Add a benchmark that repeatedly mmap()s and munmap()s an mmap-able BPF
array map (BPF_F_MMAPABLE). The number of worker threads is controlled
by the standard -p/--producers option, and the size of the map by
--map-size (default 8MiB).

The benchmark is useful to observe how the cost of mmap()/munmap()
scales with the mapping size, e.g. eager vs lazy page-table population.

Signed-off-by: Song Liu <song@kernel.org>
Assisted-by: Claude:claude-opus-4-8
---
 tools/testing/selftests/bpf/Makefile          |   1 +
 tools/testing/selftests/bpf/bench.c           |   4 +
 .../bpf/benchs/bench_arraymap_mmap.c          | 127 ++++++++++++++++++
 .../bpf/benchs/run_bench_arraymap_mmap.sh     |  15 +++
 4 files changed, 147 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c
 create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_arraymap_mmap.sh

diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index 55d394438705..b6593fb1135b 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -1003,6 +1003,7 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \
 		 $(OUTPUT)/bench_bpf_timing.o \
 		 $(OUTPUT)/bench_bpf_nop.o \
 		 $(OUTPUT)/bench_xdp_lb.o \
+		 $(OUTPUT)/bench_arraymap_mmap.o \
 		 $(OUTPUT)/usdt_1.o \
 		 $(OUTPUT)/usdt_2.o \
 		 #
diff --git a/tools/testing/selftests/bpf/bench.c b/tools/testing/selftests/bpf/bench.c
index 3d9d2cd7764b..192d5fcb7a74 100644
--- a/tools/testing/selftests/bpf/bench.c
+++ b/tools/testing/selftests/bpf/bench.c
@@ -287,6 +287,7 @@ extern struct argp bench_crypto_argp;
 extern struct argp bench_sockmap_argp;
 extern struct argp bench_lpm_trie_map_argp;
 extern struct argp bench_xdp_lb_argp;
+extern struct argp bench_arraymap_mmap_argp;
 
 static const struct argp_child bench_parsers[] = {
 	{ &bench_ringbufs_argp, 0, "Ring buffers benchmark", 0 },
@@ -304,6 +305,7 @@ static const struct argp_child bench_parsers[] = {
 	{ &bench_sockmap_argp, 0, "bpf sockmap benchmark", 0 },
 	{ &bench_lpm_trie_map_argp, 0, "LPM trie map benchmark", 0 },
 	{ &bench_xdp_lb_argp, 0, "XDP load-balancer benchmark", 0 },
+	{ &bench_arraymap_mmap_argp, 0, "arraymap-mmap benchmark", 0 },
 	{},
 };
 
@@ -582,6 +584,7 @@ extern const struct bench bench_lpm_trie_delete;
 extern const struct bench bench_lpm_trie_free;
 extern const struct bench bench_bpf_nop;
 extern const struct bench bench_xdp_lb;
+extern const struct bench bench_arraymap_mmap;
 
 static const struct bench *benchs[] = {
 	&bench_count_global,
@@ -665,6 +668,7 @@ static const struct bench *benchs[] = {
 	&bench_lpm_trie_free,
 	&bench_bpf_nop,
 	&bench_xdp_lb,
+	&bench_arraymap_mmap,
 };
 
 static void find_benchmark(void)
diff --git a/tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c b/tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c
new file mode 100644
index 000000000000..d7c7615e2183
--- /dev/null
+++ b/tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c
@@ -0,0 +1,127 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. */
+#include <argp.h>
+#include <sys/mman.h>
+#include "bench.h"
+
+/* Benchmark mmap()/munmap() of an mmap-able BPF array map from N threads. */
+
+static struct ctx {
+	int map_fd;
+	size_t mmap_sz;
+} ctx;
+
+static struct {
+	__u64 map_size;
+} args = {
+	.map_size = 8 * 1024 * 1024, /* 8 MiB */
+};
+
+enum {
+	ARG_MAP_SIZE = 5000,
+};
+
+static const struct argp_option opts[] = {
+	{ "map-size", ARG_MAP_SIZE, "BYTES", 0,
+		"Size of the BPF array map in bytes (default 8388608)" },
+	{},
+};
+
+static error_t parse_arg(int key, char *arg, struct argp_state *state)
+{
+	switch (key) {
+	case ARG_MAP_SIZE:
+		args.map_size = strtoull(arg, NULL, 0);
+		if (args.map_size == 0) {
+			fprintf(stderr, "Invalid map-size: %s\n", arg);
+			argp_usage(state);
+		}
+		break;
+	default:
+		return ARGP_ERR_UNKNOWN;
+	}
+
+	return 0;
+}
+
+/* exported into benchmark runner */
+const struct argp bench_arraymap_mmap_argp = {
+	.options = opts,
+	.parser = parse_arg,
+};
+
+static void validate(void)
+{
+	if (env.consumer_cnt != 0) {
+		fprintf(stderr, "benchmark doesn't support consumer!\n");
+		exit(1);
+	}
+}
+
+static long hits;
+
+static void *producer(void *input)
+{
+	while (true) {
+		void *addr;
+
+		addr = mmap(NULL, ctx.mmap_sz, PROT_READ | PROT_WRITE,
+			    MAP_SHARED, ctx.map_fd, 0);
+		if (addr == MAP_FAILED) {
+			fprintf(stderr, "mmap failed: %d\n", -errno);
+			exit(1);
+		}
+		if (munmap(addr, ctx.mmap_sz)) {
+			fprintf(stderr, "munmap failed: %d\n", -errno);
+			exit(1);
+		}
+		atomic_inc(&hits);
+	}
+
+	return NULL;
+}
+
+static void measure(struct bench_res *res)
+{
+	res->hits = atomic_swap(&hits, 0);
+}
+
+static void setup(void)
+{
+	LIBBPF_OPTS(bpf_map_create_opts, opts, .map_flags = BPF_F_MMAPABLE);
+	long page_sz = sysconf(_SC_PAGESIZE);
+	__u32 value_size = sizeof(__u64);
+	__u32 max_entries;
+
+	setup_libbpf();
+
+	/*
+	 * mmap-able array maps round the value size up to 8 bytes and expose
+	 * PAGE_ALIGN(max_entries * elem_size) bytes to user space.
+	 */
+	max_entries = (args.map_size + value_size - 1) / value_size;
+	ctx.mmap_sz = ((__u64)max_entries * value_size + page_sz - 1) &
+		      ~(page_sz - 1);
+
+	ctx.map_fd = bpf_map_create(BPF_MAP_TYPE_ARRAY, "mmap_array",
+				    sizeof(__u32), value_size, max_entries,
+				    &opts);
+	if (ctx.map_fd < 0) {
+		fprintf(stderr, "failed to create map: %d\n", -errno);
+		exit(1);
+	}
+
+	printf("array map: %u entries, mmap size %zu bytes, %d threads\n",
+	       max_entries, ctx.mmap_sz, env.producer_cnt);
+}
+
+const struct bench bench_arraymap_mmap = {
+	.name = "arraymap-mmap",
+	.argp = &bench_arraymap_mmap_argp,
+	.validate = validate,
+	.setup = setup,
+	.producer_thread = producer,
+	.measure = measure,
+	.report_progress = ops_report_progress,
+	.report_final = ops_report_final,
+};
diff --git a/tools/testing/selftests/bpf/benchs/run_bench_arraymap_mmap.sh b/tools/testing/selftests/bpf/benchs/run_bench_arraymap_mmap.sh
new file mode 100755
index 000000000000..22c569311137
--- /dev/null
+++ b/tools/testing/selftests/bpf/benchs/run_bench_arraymap_mmap.sh
@@ -0,0 +1,15 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+source ./benchs/run_common.sh
+
+set -eufo pipefail
+
+for t in 1 4 8 16; do
+for s in 1048576 8388608 67108864; do
+subtitle "nr_threads: $t, map_size: $s"
+	summarize_ops "arraymap-mmap: " \
+	    "$($RUN_BENCH -p $t --map-size $s arraymap-mmap)"
+	printf "\n"
+done
+done
-- 
2.53.0-Meta