From: "Lorenzo Stoakes (ARM)" <ljs@kernel.org>
To: Puranjay Mohan <puranjay@kernel.org>
Cc: bpf@vger.kernel.org, Puranjay Mohan <puranjay12@gmail.com>,
Alexei Starovoitov <ast@kernel.org>,
Andrii Nakryiko <andrii@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Martin KaFai Lau <martin.lau@kernel.org>,
Eduard Zingerman <eddyz87@gmail.com>,
Kumar Kartikeya Dwivedi <memxor@gmail.com>,
Mykyta Yatsenko <mykyta.yatsenko5@gmail.com>,
kernel-team@meta.com, "Liam R. Howlett" <liam@infradead.org>,
Suren Baghdasaryan <surenb@google.com>,
Vlastimil Babka <vbabka@kernel.org>,
Shakeel Butt <shakeel.butt@linux.dev>,
Pedro Falcato <pfalcato@suse.de>,
David Hildenbrand <david@kernel.org>,
Jann Horn <jannh@google.com>,
linux-mm@kvack.org, Chris Mason <mason@kernel.org>
Subject: Re: [PATCH bpf v6 2/3] bpf: switch task_vma iterator from mmap_lock to per-VMA locks
Date: Sat, 21 Nov 2026 13:40:54 +0000 [thread overview]
Message-ID: <awGZWsCp6v63H_py@gremlin> (raw)
In-Reply-To: <20260408154539.3832150-3-puranjay@kernel.org>
+cc VMA team, Chris [whose AI fix this relates to]
On Wed, Apr 08, 2026 at 08:45:36AM -0700, Puranjay Mohan wrote:
> The open-coded task_vma iterator holds mmap_lock for the entire duration
> of iteration, increasing contention on this highly contended lock.
>
> Switch to per-VMA locking. Find the next VMA via an RCU-protected maple
> tree walk and lock it with lock_vma_under_rcu(). lock_next_vma() is not
> used because its fallback takes mmap_read_lock(), and the iterator must
> work in non-sleepable contexts.
Hmm :) In future if an API doesn't give you what you need please do ask us
in mm to offer one that does give you what you need, we don't bite :)
>
> lock_vma_under_rcu() is a point lookup (mas_walk) that finds the VMA
> containing a given address but cannot iterate across gaps. An
> RCU-protected vma_next() walk (mas_find) first locates the next VMA's
> vm_start to pass to lock_vma_under_rcu().
>
> Between the RCU walk and the lock, the VMA may be removed, shrunk, or
> write-locked. On failure, advance past it using vm_end from the RCU
> walk. Because the VMA slab is SLAB_TYPESAFE_BY_RCU, vm_end may be
> stale; fall back to PAGE_SIZE advancement when it does not make forward
> progress. Concurrent VMA insertions at addresses already passed by the
> iterator are not detected.
>
> CONFIG_PER_VMA_LOCK is required; return -EOPNOTSUPP without it.
>
> Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Hi guys,
This is buggy and can result in deadlock. You need to revert it or restrict
which contexts it can be called in.
I'm a little surprised you didn't cc- or even ping mm or the VMA team here, we
are always willing to help and it's clearly sub-optimal to write duplicative,
open-coded VMA traversal in BPF.
Let's be collaborative in future please :)
Anyway, unfortunately it seems that you've rather got yourselves in rather a
pickle here, as vma_start_read() can call mmdrop() and in x86 that can take a
pgd_lock in an invalid context.
Consider:
mmdrop()
-> __mmdrop()
-> mm_free_pgd()
-> pgd_free()
-> pgd_dtor()
And looking at pgd_dtor():
static void pgd_dtor(pgd_t *pgd)
{
spin_lock(&pgd_lock);
pgd_list_del(pgd);
spin_unlock(&pgd_lock);
}
It's just a plain spin_[un]lock() which is invalid in softirq context, which you
are potentially calling from here.
You can get there from softirq via something like:
pgd_alloc()
-> spin_lock(&pgd_lock)
<softirq>
bpf_iter_task_vma_next()
-> vma_start_read()
... etc...
< maybe deadlock >
It's not really sensible to alter core mm to accommodate this at this stage, so
probably the best course is to disallow this code path for softirq context, or
you could look into patching the x86 code.
Thanks!
> ---
> kernel/bpf/task_iter.c | 91 +++++++++++++++++++++++++++++++++---------
> 1 file changed, 73 insertions(+), 18 deletions(-)
>
> diff --git a/kernel/bpf/task_iter.c b/kernel/bpf/task_iter.c
> index c1f5fbe9dc2f..87e87f18913d 100644
> --- a/kernel/bpf/task_iter.c
> +++ b/kernel/bpf/task_iter.c
> @@ -9,6 +9,7 @@
> #include <linux/bpf_mem_alloc.h>
> #include <linux/btf_ids.h>
> #include <linux/mm_types.h>
> +#include <linux/mmap_lock.h>
> #include <linux/sched/mm.h>
> #include "mmap_unlock_work.h"
>
> @@ -807,8 +808,8 @@ static inline void bpf_iter_mmput_async(struct mm_struct *mm)
> struct bpf_iter_task_vma_kern_data {
> struct task_struct *task;
> struct mm_struct *mm;
> - struct mmap_unlock_irq_work *work;
> - struct vma_iterator vmi;
> + struct vm_area_struct *locked_vma;
> + u64 next_addr;
> };
>
> struct bpf_iter_task_vma {
> @@ -829,21 +830,19 @@ __bpf_kfunc int bpf_iter_task_vma_new(struct bpf_iter_task_vma *it,
> struct task_struct *task, u64 addr)
> {
> struct bpf_iter_task_vma_kern *kit = (void *)it;
> - bool irq_work_busy = false;
> int err;
>
> BUILD_BUG_ON(sizeof(struct bpf_iter_task_vma_kern) != sizeof(struct bpf_iter_task_vma));
> BUILD_BUG_ON(__alignof__(struct bpf_iter_task_vma_kern) != __alignof__(struct bpf_iter_task_vma));
>
> - /* bpf_iter_mmput_async() needs mmput_async() which requires CONFIG_MMU */
> - if (!IS_ENABLED(CONFIG_MMU)) {
> + if (!IS_ENABLED(CONFIG_PER_VMA_LOCK)) {
> kit->data = NULL;
> return -EOPNOTSUPP;
> }
>
> /*
> * Reject irqs-disabled contexts including NMI. Operations used
> - * by _next() and _destroy() (mmap_read_unlock, bpf_iter_mmput_async)
> + * by _next() and _destroy() (vma_end_read, bpf_iter_mmput_async)
> * can take spinlocks with IRQs disabled (pi_lock, pool->lock).
> * Running from NMI or from a tracepoint that fires with those
> * locks held could deadlock.
> @@ -886,18 +885,10 @@ __bpf_kfunc int bpf_iter_task_vma_new(struct bpf_iter_task_vma *it,
> goto err_cleanup_iter;
> }
>
> - /* kit->data->work == NULL is valid after bpf_mmap_unlock_get_irq_work */
> - irq_work_busy = bpf_mmap_unlock_get_irq_work(&kit->data->work);
> - if (irq_work_busy || !mmap_read_trylock(kit->data->mm)) {
> - err = -EBUSY;
> - goto err_cleanup_mmget;
> - }
> -
> - vma_iter_init(&kit->data->vmi, kit->data->mm, addr);
> + kit->data->locked_vma = NULL;
> + kit->data->next_addr = addr;
> return 0;
>
> -err_cleanup_mmget:
> - bpf_iter_mmput_async(kit->data->mm);
> err_cleanup_iter:
> put_task_struct(kit->data->task);
> bpf_mem_free(&bpf_global_ma, kit->data);
> @@ -906,13 +897,76 @@ __bpf_kfunc int bpf_iter_task_vma_new(struct bpf_iter_task_vma *it,
> return err;
> }
>
> +/*
> + * Find and lock the next VMA at or after data->next_addr.
> + *
> + * lock_vma_under_rcu() is a point lookup (mas_walk): it finds the VMA
> + * containing a given address but cannot iterate. An RCU-protected
> + * maple tree walk with vma_next() (mas_find) is needed first to locate
> + * the next VMA's vm_start across any gap.
> + *
> + * Between the RCU walk and the lock, the VMA may be removed, shrunk,
> + * or write-locked. On failure, advance past it using vm_end from the
> + * RCU walk. SLAB_TYPESAFE_BY_RCU can make vm_end stale, so fall back
> + * to PAGE_SIZE advancement to guarantee forward progress.
> + */
> +static struct vm_area_struct *
> +bpf_iter_task_vma_find_next(struct bpf_iter_task_vma_kern_data *data)
> +{
> + struct vm_area_struct *vma;
> + struct vma_iterator vmi;
> + unsigned long start, end;
> +
> +retry:
> + rcu_read_lock();
> + vma_iter_init(&vmi, data->mm, data->next_addr);
> + vma = vma_next(&vmi);
> + if (!vma) {
> + rcu_read_unlock();
> + return NULL;
> + }
> + start = vma->vm_start;
> + end = vma->vm_end;
> + rcu_read_unlock();
> +
> + vma = lock_vma_under_rcu(data->mm, start);
> + if (!vma) {
> + if (end <= data->next_addr)
> + data->next_addr += PAGE_SIZE;
> + else
> + data->next_addr = end;
> + goto retry;
> + }
> +
> + if (unlikely(vma->vm_end <= data->next_addr)) {
> + data->next_addr += PAGE_SIZE;
> + vma_end_read(vma);
> + goto retry;
> + }
> +
> + return vma;
> +}
> +
> __bpf_kfunc struct vm_area_struct *bpf_iter_task_vma_next(struct bpf_iter_task_vma *it)
> {
> struct bpf_iter_task_vma_kern *kit = (void *)it;
> + struct vm_area_struct *vma;
>
> if (!kit->data) /* bpf_iter_task_vma_new failed */
> return NULL;
> - return vma_next(&kit->data->vmi);
> +
> + if (kit->data->locked_vma) {
> + vma_end_read(kit->data->locked_vma);
> + kit->data->locked_vma = NULL;
> + }
> +
> + vma = bpf_iter_task_vma_find_next(kit->data);
> + if (!vma)
> + return NULL;
> +
> + kit->data->locked_vma = vma;
> + kit->data->next_addr = vma->vm_end;
> + return vma;
> }
>
> __bpf_kfunc void bpf_iter_task_vma_destroy(struct bpf_iter_task_vma *it)
> @@ -920,7 +974,8 @@ __bpf_kfunc void bpf_iter_task_vma_destroy(struct bpf_iter_task_vma *it)
> struct bpf_iter_task_vma_kern *kit = (void *)it;
>
> if (kit->data) {
> - bpf_mmap_unlock_mm(kit->data->work, kit->data->mm);
> + if (kit->data->locked_vma)
> + vma_end_read(kit->data->locked_vma);
> put_task_struct(kit->data->task);
> bpf_iter_mmput_async(kit->data->mm);
> bpf_mem_free(&bpf_global_ma, kit->data);
> --
> 2.52.0
>
>
--
Cheers, Lorenzo
next prev parent reply other threads:[~2026-11-21 13:41 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-04-08 15:45 [PATCH bpf v6 0/3] bpf: fix and improve open-coded task_vma iterator Puranjay Mohan
2026-04-08 15:45 ` [PATCH bpf v6 1/3] bpf: fix mm lifecycle in " Puranjay Mohan
2026-04-08 15:45 ` [PATCH bpf v6 2/3] bpf: switch task_vma iterator from mmap_lock to per-VMA locks Puranjay Mohan
2026-11-21 13:40 ` Lorenzo Stoakes (ARM) [this message]
2026-04-08 15:45 ` [PATCH bpf v6 3/3] bpf: return VMA snapshot from task_vma iterator Puranjay Mohan
2026-04-09 13:15 ` Mykyta Yatsenko
2026-04-10 19:10 ` [PATCH bpf v6 0/3] bpf: fix and improve open-coded " patchwork-bot+netdevbpf
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=awGZWsCp6v63H_py@gremlin \
--to=ljs@kernel.org \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=david@kernel.org \
--cc=eddyz87@gmail.com \
--cc=jannh@google.com \
--cc=kernel-team@meta.com \
--cc=liam@infradead.org \
--cc=linux-mm@kvack.org \
--cc=martin.lau@kernel.org \
--cc=mason@kernel.org \
--cc=memxor@gmail.com \
--cc=mykyta.yatsenko5@gmail.com \
--cc=pfalcato@suse.de \
--cc=puranjay12@gmail.com \
--cc=puranjay@kernel.org \
--cc=shakeel.butt@linux.dev \
--cc=surenb@google.com \
--cc=vbabka@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox