[PATCH v6 bpf-next] bpf: Populate mmap-able array map memory lazily

[PATCH v6 bpf-next] bpf: Populate mmap-able array map memory lazily

[Song Liu]

An mmap-able BPF array map (BPF_F_MMAPABLE) has its backing memory
vmalloc'ed up front at map creation time. array_map_mmap() then wired up
the whole mapping eagerly via remap_vmalloc_range(), which calls
vm_insert_page() for every page of the map. For large maps this makes
every mmap() O(number of pages): an 8MiB map inserts 2048 PTEs per
mmap() and tears them all down again on munmap(), even when user space
only touches a few pages (or none at all).

Populate the mapping lazily instead, the same way the arena map already
does. array_map_mmap() now only performs the bounds check and returns,
leaving the PTEs unpopulated; pages are inserted on demand by a new
array_map_mmap_fault() handler. Because the memory is already resident,
the fault handler simply resolves the vmalloc page and hands it to the
fault path. This makes mmap() O(1), and munmap() proportional to the
number of pages that were actually faulted in rather than to the size of
the map.

The handler is reached through a new optional ->map_mmap_fault callback
dispatched from the shared bpf_map_default_vmops, so the existing VMA
open/close accounting (VM_MAYWRITE write-active tracking, freeze
handling) stays centralized rather than each map installing its own
vm_operations_struct.

Callers that want the pages populated up front can still request that
explicitly with MAP_POPULATE. Kernel-side access to the map (via the
vmalloc address) is unaffected.

Time for one mmap()+munmap() of an 8MiB mmap-able array map:

                                       before     after
  no MAP_POPULATE, no access            226us     1.1us
  no MAP_POPULATE, access all pages     236us    1341us
  MAP_POPULATE, no access               312us     493us
  MAP_POPULATE, access all pages        318us     519us

Mapping without touching the data, which is what this change targets,
gets ~160x cheaper. Faulting in the whole mapping one page at a time is
more expensive than the eager remap_vmalloc_range() loop, so users that
do touch every page should ask for MAP_POPULATE. Note that MAP_POPULATE
is not free before this change either: it adds ~85us (226us => 312us)
for no benefit, as the mapping is already fully populated.

Signed-off-by: Song Liu <song@kernel.org>
Assisted-by: Claude:claude-opus-4-8

---
Changes in v6:
- Drop the !CONFIG_MMU branch. nommu cannot mmap() a BPF map fd in the
  first place: the fd is an anon inode and bpf_map_fops has no
  ->mmap_capabilities, so validate_mmap_request() returns -EINVAL before
  ->mmap() runs. (BPF CI AI review)
- Scope the O(1) claim to mmap(); munmap() still tears down whatever was
  faulted in. (BPF CI AI review)
v5: https://lore.kernel.org/bpf/20260812230248.2452103-1-song@kernel.org/

Changes in v5:
- Drop the ->map_pages (fault-around) handler, it pulls in too much mm
  internal API for the gain. (Andrii)
- Drop the fault path overflow and bounds checks, and the verbose
  comments; VM_DONTEXPAND plus the mmap() time check already bound the
  faulting offset. (Andrii)
- No cover letter for a single patch. (Andrii)
v4: https://lore.kernel.org/bpf/20260729192419.41331-1-song@kernel.org/

Changes in v4:
- Flush the D-cache before exposing a page at a new user address, as the
  eager vm_insert_page() path did. (Sashiko AI review)
- Fix the build on !CONFIG_MMU: keep populating the mapping eagerly
  there, as there are no page faults. (kernel test robot)
v3: https://lore.kernel.org/bpf/20260729001033.3433328-1-song@kernel.org/

Changes in v3:
- Add a ->map_pages (fault-around) handler so mmap(MAP_POPULATE) and
  linear access populate PTEs in batches instead of one fault per page.
- Harden the fault path with check_shl_overflow() and explicit bounds
  checks instead of a plain (u64) cast. (Andrii)
- Drop selftests (2/2 in v2). (Andrii)
v2: https://lore.kernel.org/bpf/20260722205032.1245094-1-song@kernel.org/

Changes in v2:
- Use 64-bit arithmetic for the mmap offset and bounds check to avoid a
  potential overflow on 32-bit architectures.
v1: https://lore.kernel.org/bpf/20260722065308.4116186-1-song@kernel.org/
---
 include/linux/bpf.h   |  1 +
 kernel/bpf/arraymap.c | 29 +++++++++++++++++++++++++----
 kernel/bpf/syscall.c  | 12 ++++++++++++
 3 files changed, 38 insertions(+), 4 deletions(-)

diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index f4e8d372253a..04cadd987169 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -145,6 +145,7 @@ struct bpf_map_ops {
 	int (*map_direct_value_meta)(const struct bpf_map *map,
 				     u64 imm, u32 *off);
 	int (*map_mmap)(struct bpf_map *map, struct vm_area_struct *vma);
+	vm_fault_t (*map_mmap_fault)(struct bpf_map *map, struct vm_fault *vmf);
 	__poll_t (*map_poll)(struct bpf_map *map, struct file *filp,
 			     struct poll_table_struct *pts);
 	unsigned long (*map_get_unmapped_area)(struct file *filep, unsigned long addr,
diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c
index 34865701f7f7..a6e44428a6c5 100644
--- a/kernel/bpf/arraymap.c
+++ b/kernel/bpf/arraymap.c
@@ -608,17 +608,37 @@ static int array_map_check_btf(struct bpf_map *map,
 static int array_map_mmap(struct bpf_map *map, struct vm_area_struct *vma)
 {
 	struct bpf_array *array = container_of(map, struct bpf_array, map);
-	pgoff_t pgoff = PAGE_ALIGN(sizeof(*array)) >> PAGE_SHIFT;
 
 	if (!(map->map_flags & BPF_F_MMAPABLE))
 		return -EINVAL;
 
-	if (vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) >
+	/* use u64 math so the offset cannot overflow on 32-bit archs */
+	if ((u64)vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) >
 	    PAGE_ALIGN((u64)array->map.max_entries * array->elem_size))
 		return -EINVAL;
 
-	return remap_vmalloc_range(vma, array_map_vmalloc_addr(array),
-				   vma->vm_pgoff + pgoff);
+	/* pages are faulted in on demand by array_map_mmap_fault() */
+	vm_flags_set(vma, VM_DONTEXPAND | VM_DONTDUMP);
+
+	return 0;
+}
+
+static vm_fault_t array_map_mmap_fault(struct bpf_map *map,
+				       struct vm_fault *vmf)
+{
+	struct bpf_array *array = container_of(map, struct bpf_array, map);
+	struct page *page;
+
+	page = vmalloc_to_page(array->value + ((u64)vmf->pgoff << PAGE_SHIFT));
+	if (!page)
+		return VM_FAULT_SIGBUS;
+
+	/* the eager remap_vmalloc_range() flushed via vm_insert_page() */
+	flush_dcache_folio(page_folio(page));
+	get_page(page);
+	vmf->page = page;
+
+	return 0;
 }
 
 static bool array_map_meta_equal(const struct bpf_map *meta0,
@@ -844,6 +864,7 @@ const struct bpf_map_ops array_map_ops = {
 	.map_direct_value_addr = array_map_direct_value_addr,
 	.map_direct_value_meta = array_map_direct_value_meta,
 	.map_mmap = array_map_mmap,
+	.map_mmap_fault = array_map_mmap_fault,
 	.map_seq_show_elem = array_map_seq_show_elem,
 	.map_check_btf = array_map_check_btf,
 	.map_lookup_batch = generic_map_lookup_batch,
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 7d8c3e8e6d62..9e5e65e73dfa 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -1076,9 +1076,21 @@ static void bpf_map_mmap_close(struct vm_area_struct *vma)
 		bpf_map_write_active_dec(map);
 }
 
+/* Called for maps that populate their memory-mapped region lazily. */
+static vm_fault_t bpf_map_mmap_fault(struct vm_fault *vmf)
+{
+	struct bpf_map *map = vmf->vma->vm_private_data;
+
+	if (!map->ops->map_mmap_fault)
+		return VM_FAULT_SIGBUS;
+
+	return map->ops->map_mmap_fault(map, vmf);
+}
+
 static const struct vm_operations_struct bpf_map_default_vmops = {
 	.open		= bpf_map_mmap_open,
 	.close		= bpf_map_mmap_close,
+	.fault		= bpf_map_mmap_fault,
 };
 
 static int bpf_map_mmap(struct file *filp, struct vm_area_struct *vma)

base-commit: 259d60f5bfa41056fe01cbf2ba3f6f0331865a16
-- 
2.53.0-Meta

Re: [PATCH v6 bpf-next] bpf: Populate mmap-able array map memory lazily

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 5718 bytes --]

> diff --git include/linux/bpf.h include/linux/bpf.h
> index f4e8d372253ac..04cadd9871696 100644
> --- include/linux/bpf.h
> +++ include/linux/bpf.h
> @@ -145,6 +145,7 @@ struct bpf_map_ops {
>  	int (*map_direct_value_meta)(const struct bpf_map *map,
>  				     u64 imm, u32 *off);
>  	int (*map_mmap)(struct bpf_map *map, struct vm_area_struct *vma);
> +	vm_fault_t (*map_mmap_fault)(struct bpf_map *map, struct vm_fault *vmf);
>  	__poll_t (*map_poll)(struct bpf_map *map, struct file *filp,
>  			     struct poll_table_struct *pts);
>  	unsigned long (*map_get_unmapped_area)(struct file *filep, unsigned long addr,
> diff --git kernel/bpf/arraymap.c kernel/bpf/arraymap.c
> index 34865701f7f75..a6e44428a6c52 100644
> --- kernel/bpf/arraymap.c
> +++ kernel/bpf/arraymap.c
> @@ -608,17 +608,37 @@ static int array_map_mmap(struct bpf_map *map, struct vm_area_struct *vma)
>  {
>  	struct bpf_array *array = container_of(map, struct bpf_array, map);
> -	pgoff_t pgoff = PAGE_ALIGN(sizeof(*array)) >> PAGE_SHIFT;
>
>  	if (!(map->map_flags & BPF_F_MMAPABLE))
>  		return -EINVAL;
>
> -	if (vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) >
> +	/* use u64 math so the offset cannot overflow on 32-bit archs */
> +	if ((u64)vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) >
>  	    PAGE_ALIGN((u64)array->map.max_entries * array->elem_size))
>  		return -EINVAL;
>
> -	return remap_vmalloc_range(vma, array_map_vmalloc_addr(array),
> -				   vma->vm_pgoff + pgoff);
> +	/* pages are faulted in on demand by array_map_mmap_fault() */
> +	vm_flags_set(vma, VM_DONTEXPAND | VM_DONTDUMP);

Should VM_MIXEDMAP be included here? The old path set this flag through
remap_vmalloc_range()->vm_insert_page():

mm/memory.c:vm_insert_page() {
    ...
    if (!(vma->vm_flags & VM_MIXEDMAP)) {
            BUG_ON(mmap_read_trylock(vma->vm_mm));
            BUG_ON(vma->vm_flags & VM_PFNMAP);
            vm_flags_set(vma, VM_MIXEDMAP);
    }
}

Without this flag, task_numa_work() no longer skips the VMA:

kernel/sched/fair.c:task_numa_work() {
    ...
    if (!vma_migratable(vma) || !vma_policy_mof(vma) ||
            is_vm_hugetlb_page(vma) || (vma->vm_flags & VM_MIXEDMAP)) {
            trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_UNSUITABLE);
            continue;
    }
}

The VMA now passes all the checks when numa_balancing is enabled, so the
region gets periodically PROT_NONE'd by change_prot_numa() and re-faulted
through do_numa_page(). The migration attempt always fails because
folio_isolate_lru() returns -EAGAIN on vmalloc pages, but the scanning
overhead (extra minor faults, TLB flushes, and cpupid writes to kernel
vmalloc page flags) is still paid. Is this intentional?

> +
> +	return 0;
> +}
> +
> +static vm_fault_t array_map_mmap_fault(struct bpf_map *map,
> +				       struct vm_fault *vmf)
> +{
> +	struct bpf_array *array = container_of(map, struct bpf_array, map);
> +	struct page *page;
> +
> +	page = vmalloc_to_page(array->value + ((u64)vmf->pgoff << PAGE_SHIFT));
> +	if (!page)
> +		return VM_FAULT_SIGBUS;
> +
> +	/* the eager remap_vmalloc_range() flushed via vm_insert_page() */
> +	flush_dcache_folio(page_folio(page));
> +	get_page(page);
> +	vmf->page = page;
> +
> +	return 0;
>  }

[ ... ]

> diff --git kernel/bpf/syscall.c kernel/bpf/syscall.c
> index 7d8c3e8e6d62e..9e5e65e73dfa4 100644
> --- kernel/bpf/syscall.c
> +++ kernel/bpf/syscall.c
> @@ -1076,9 +1076,21 @@ static void bpf_map_mmap_close(struct vm_area_struct *vma)
>  		bpf_map_write_active_dec(map);
>  }
>
> +/* Called for maps that populate their memory-mapped region lazily. */
> +static vm_fault_t bpf_map_mmap_fault(struct vm_fault *vmf)
> +{
> +	struct bpf_map *map = vmf->vma->vm_private_data;
> +
> +	if (!map->ops->map_mmap_fault)
> +		return VM_FAULT_SIGBUS;
> +
> +	return map->ops->map_mmap_fault(map, vmf);
> +}
> +
>  static const struct vm_operations_struct bpf_map_default_vmops = {
>  	.open		= bpf_map_mmap_open,
>  	.close		= bpf_map_mmap_close,
> +	.fault		= bpf_map_mmap_fault,
>  };

Can this affect ringbuf maps? Both ringbuf_map_ops and user_ringbuf_map_ops
provide .map_mmap but not .map_mmap_fault, and neither overrides vm_ops, so
they now use bpf_map_default_vmops.fault. Before this patch a fault on
their VMAs had no .fault handler, which let do_fault() take the path that
re-checks the PTE under the PTL:

mm/memory.c:do_fault() {
    if (!vma->vm_ops->fault) {
            vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd,
                                           vmf->address, &vmf->ptl);
            ...
            if (unlikely(pte_none(ptep_get(vmf->pte))))
                    ret = VM_FAULT_SIGBUS;
            else
                    ret = VM_FAULT_NOPAGE;
    ...
}

That re-check exists so a fault racing the zero-PTE window of a
read/modify/write update (like change_pte_range() for mprotect() or
clear_soft_dirty() for /proc/pid/clear_refs) sees the updated entry and
returns VM_FAULT_NOPAGE instead of SIGBUS.

After this patch bpf_map_mmap_fault() returns VM_FAULT_SIGBUS immediately
when map_mmap_fault is NULL, so the PTL re-check never happens. Could this
send SIGBUS to a userspace thread reading a ring buffer while another
thread calls mprotect() on the same mapping?

The comment says the handler is "Called for maps that populate their
memory-mapped region lazily," but it is installed on the shared vm_ops and
actually called for all mmap-able maps, including the eagerly-populated
ringbufs.

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/31752145119

Re: [PATCH v6 bpf-next] bpf: Populate mmap-able array map memory lazily

[Song Liu]

On Thu, Aug 13, 2026 at 5:06 PM <bot+bpf-ci@kernel.org> wrote:
[...]
>
> > diff --git kernel/bpf/syscall.c kernel/bpf/syscall.c
> > index 7d8c3e8e6d62e..9e5e65e73dfa4 100644
> > --- kernel/bpf/syscall.c
> > +++ kernel/bpf/syscall.c
> > @@ -1076,9 +1076,21 @@ static void bpf_map_mmap_close(struct vm_area_struct *vma)
> >               bpf_map_write_active_dec(map);
> >  }
> >
> > +/* Called for maps that populate their memory-mapped region lazily. */
> > +static vm_fault_t bpf_map_mmap_fault(struct vm_fault *vmf)
> > +{
> > +     struct bpf_map *map = vmf->vma->vm_private_data;
> > +
> > +     if (!map->ops->map_mmap_fault)
> > +             return VM_FAULT_SIGBUS;
> > +
> > +     return map->ops->map_mmap_fault(map, vmf);
> > +}
> > +
> >  static const struct vm_operations_struct bpf_map_default_vmops = {
> >       .open           = bpf_map_mmap_open,
> >       .close          = bpf_map_mmap_close,
> > +     .fault          = bpf_map_mmap_fault,
> >  };
>
> Can this affect ringbuf maps? Both ringbuf_map_ops and user_ringbuf_map_ops
> provide .map_mmap but not .map_mmap_fault, and neither overrides vm_ops, so
> they now use bpf_map_default_vmops.fault. Before this patch a fault on
> their VMAs had no .fault handler, which let do_fault() take the path that
> re-checks the PTE under the PTL:
>
> mm/memory.c:do_fault() {
>     if (!vma->vm_ops->fault) {
>             vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd,
>                                            vmf->address, &vmf->ptl);
>             ...
>             if (unlikely(pte_none(ptep_get(vmf->pte))))
>                     ret = VM_FAULT_SIGBUS;
>             else
>                     ret = VM_FAULT_NOPAGE;
>     ...
> }
>
> That re-check exists so a fault racing the zero-PTE window of a
> read/modify/write update (like change_pte_range() for mprotect() or
> clear_soft_dirty() for /proc/pid/clear_refs) sees the updated entry and
> returns VM_FAULT_NOPAGE instead of SIGBUS.
>
> After this patch bpf_map_mmap_fault() returns VM_FAULT_SIGBUS immediately
> when map_mmap_fault is NULL, so the PTL re-check never happens. Could this
> send SIGBUS to a userspace thread reading a ring buffer while another
> thread calls mprotect() on the same mapping?
>
> The comment says the handler is "Called for maps that populate their
> memory-mapped region lazily," but it is installed on the shared vm_ops and
> actually called for all mmap-able maps, including the eagerly-populated
> ringbufs.

Both comments appears to be valid. I incorporated the feedbacks in v7.

Thanks,
Song

Re: [PATCH v6 bpf-next] bpf: Populate mmap-able array map memory lazily

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] Integer overflow in `array_map_mmap` bounds check allows mapping arbitrary kernel memory, leading to arbitrary kernel code execution.
--

--- Patch [1]: [PATCH v6 bpf-next] bpf: Populate mmap-able array map memory lazily ---
commit ef1e2903b40b525ff2c8448b624a44742a4bc1a4
Author: Song Liu <song@kernel.org>

bpf: Populate mmap-able array map memory lazily

This patch replaces the eager remap_vmalloc_range() in array_map_mmap()
with lazy, on-demand page faulting via array_map_mmap_fault(). This makes
mmap() O(1) by only setting up the VMA bounds and inserting pages during
actual accesses.

> diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c
> index 34865701f7f75..a6e44428a6c52 100644
> --- a/kernel/bpf/arraymap.c
> +++ b/kernel/bpf/arraymap.c
> @@ -608,17 +608,37 @@ static int array_map_check_btf(struct bpf_map *map,
>  static int array_map_mmap(struct bpf_map *map, struct vm_area_struct *vma)
>  {
>  	struct bpf_array *array = container_of(map, struct bpf_array, map);
> -	pgoff_t pgoff = PAGE_ALIGN(sizeof(*array)) >> PAGE_SHIFT;
>  
>  	if (!(map->map_flags & BPF_F_MMAPABLE))
>  		return -EINVAL;
>  
> -	if (vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) >
> +	/* use u64 math so the offset cannot overflow on 32-bit archs */
> +	if ((u64)vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) >
>  	    PAGE_ALIGN((u64)array->map.max_entries * array->elem_size))
>  		return -EINVAL;

[Severity: Critical]
Can this arithmetic wrap around 64 bits to a small value, bypassing the safety
check in array_map_mmap()?

If the user-provided vma->vm_pgoff is carefully crafted, the 64-bit addition
could wrap around to a small value. This overflow was previously harmless
because the eagerly mapped remap_vmalloc_range() strictly guarded against
shift overflows.

> -	return remap_vmalloc_range(vma, array_map_vmalloc_addr(array),
> -				   vma->vm_pgoff + pgoff);
> +	/* pages are faulted in on demand by array_map_mmap_fault() */
> +	vm_flags_set(vma, VM_DONTEXPAND | VM_DONTDUMP);
> +
> +	return 0;
> +}
> +
> +static vm_fault_t array_map_mmap_fault(struct bpf_map *map,
> +				       struct vm_fault *vmf)
> +{
> +	struct bpf_array *array = container_of(map, struct bpf_array, map);
> +	struct page *page;
> +
> +	page = vmalloc_to_page(array->value + ((u64)vmf->pgoff << PAGE_SHIFT));

[Severity: Critical]
Does this shift lack overflow checking, allowing backwards pointer arithmetic
in array_map_mmap_fault()?

If a wrapped pgoff bypasses the bounds check in array_map_mmap(), shifting the
user-controlled pgoff here without validation can produce a large negative
offset. Since array->value is explicitly aligned to start on the second page
of the map's vmalloc allocation, a negative offset could walk backward to the
first page, inadvertently exposing the struct bpf_array metadata to userspace.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260813224853.1436734-1-song@kernel.org?part=1