[PATCH bpf] bpf: Don't hold pack_mutex over arm64 IPIs

[PATCH bpf] bpf: Don't hold pack_mutex over arm64 IPIs

[Alexei Starovoitov]

From: Alexei Starovoitov <ast@kernel.org>

bpf_prog_pack_free() calls bpf_arch_text_invalidate() with pack_mutex
held. On arm64 it interrupts all other CPUs and waits for them:

  bpf_prog_free_deferred
    bpf_jit_free
      bpf_prog_pack_free
        bpf_arch_text_invalidate
          aarch64_insn_set
            flush_icache_range
              kick_all_cpus_sync

When many progs are freed at once, every prog load waits in
bpf_prog_pack_alloc() for all of these broadcasts. On arm64 hosts with
many CPUs we saw hung task reports with sk_attach_filter() blocked on
pack_mutex for minutes.

The bits of the range are still set in the bitmap during the
invalidation, so nobody can allocate it and the pack cannot be freed.
Drop pack_mutex around bpf_arch_text_invalidate() and take it again to
clear the bits.

Do it on arm64 only. x86, riscv, powerpc and loongarch take text_mutex in
bpf_arch_text_invalidate(). The frees would wait there instead, and the
load would wait for them in bpf_arch_text_copy().

Tested in arm64 KVM guests with 16 and 96 cpus on a 96 cpu host. cpus/2
processes attach a cBPF filter to 256 sockets and close them in a loop
for 20 sec. One more process attaches a filter every msec. Median of
5 runs.

The loops get more done. Sockets attached and closed in 20 sec, in
millions (higher is better):

             before  after
  16 cpus       1.6    4.7
  96 cpus      0.38    4.2

setsockopt(SO_ATTACH_FILTER) of the process that attaches every msec
takes less time, in msec (lower is better):

                16 cpus          96 cpus
             before  after    before  after
  avg          0.97   0.18        11   0.54
  p99            20    2.8       421    4.0
  max           204     90      1738    255

Fixes: 1dad391daef1 ("bpf, arm64: use bpf_prog_pack for memory management")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
 kernel/bpf/core.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 3030beb531d9..0fef87cfa152 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -1069,8 +1069,17 @@ void bpf_prog_pack_free(void *ptr, u32 size)
 	nbits = BPF_PROG_SIZE_TO_NBITS(size);
 	pos = ((unsigned long)ptr - (unsigned long)pack->ptr) >> BPF_PROG_CHUNK_SHIFT;
 
+	/*
+	 * arm64 sends IPI to all CPUs in bpf_arch_text_invalidate().
+	 * Don't hold pack_mutex across it. The bits are still set,
+	 * so the range cannot be reused and the pack cannot be freed.
+	 */
+	if (IS_ENABLED(CONFIG_ARM64))
+		mutex_unlock(&pack_mutex);
 	WARN_ONCE(bpf_arch_text_invalidate(ptr, size),
 		  "bpf_prog_pack bug: missing bpf_arch_text_invalidate?\n");
+	if (IS_ENABLED(CONFIG_ARM64))
+		mutex_lock(&pack_mutex);
 
 	bitmap_clear(pack->bitmap, pos, nbits);
 

base-commit: b116ac6087541d3d98031e786adbe9c733cd6df3
-- 
2.55.0

Re: [PATCH bpf] bpf: Don't hold pack_mutex over arm64 IPIs

[Puranjay Mohan]

Alexei Starovoitov <alexei.starovoitov@gmail.com> writes:

> From: Alexei Starovoitov <ast@kernel.org>
>
> bpf_prog_pack_free() calls bpf_arch_text_invalidate() with pack_mutex
> held. On arm64 it interrupts all other CPUs and waits for them:
>
>   bpf_prog_free_deferred
>     bpf_jit_free
>       bpf_prog_pack_free
>         bpf_arch_text_invalidate
>           aarch64_insn_set
>             flush_icache_range
>               kick_all_cpus_sync
>
> When many progs are freed at once, every prog load waits in
> bpf_prog_pack_alloc() for all of these broadcasts. On arm64 hosts with
> many CPUs we saw hung task reports with sk_attach_filter() blocked on
> pack_mutex for minutes.
>
> The bits of the range are still set in the bitmap during the
> invalidation, so nobody can allocate it and the pack cannot be freed.
> Drop pack_mutex around bpf_arch_text_invalidate() and take it again to
> clear the bits.
>
> Do it on arm64 only. x86, riscv, powerpc and loongarch take text_mutex in
> bpf_arch_text_invalidate(). The frees would wait there instead, and the
> load would wait for them in bpf_arch_text_copy().
>
> Tested in arm64 KVM guests with 16 and 96 cpus on a 96 cpu host. cpus/2
> processes attach a cBPF filter to 256 sockets and close them in a loop
> for 20 sec. One more process attaches a filter every msec. Median of
> 5 runs.
>
> The loops get more done. Sockets attached and closed in 20 sec, in
> millions (higher is better):
>
>              before  after
>   16 cpus       1.6    4.7
>   96 cpus      0.38    4.2
>
> setsockopt(SO_ATTACH_FILTER) of the process that attaches every msec
> takes less time, in msec (lower is better):
>
>                 16 cpus          96 cpus
>              before  after    before  after
>   avg          0.97   0.18        11   0.54
>   p99            20    2.8       421    4.0
>   max           204     90      1738    255
>
> Fixes: 1dad391daef1 ("bpf, arm64: use bpf_prog_pack for memory management")
> Signed-off-by: Alexei Starovoitov <ast@kernel.org>
> ---
>  kernel/bpf/core.c | 9 +++++++++
>  1 file changed, 9 insertions(+)
>
> diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
> index 3030beb531d9..0fef87cfa152 100644
> --- a/kernel/bpf/core.c
> +++ b/kernel/bpf/core.c
> @@ -1069,8 +1069,17 @@ void bpf_prog_pack_free(void *ptr, u32 size)
>  	nbits = BPF_PROG_SIZE_TO_NBITS(size);
>  	pos = ((unsigned long)ptr - (unsigned long)pack->ptr) >> BPF_PROG_CHUNK_SHIFT;
>  
> +	/*
> +	 * arm64 sends IPI to all CPUs in bpf_arch_text_invalidate().
> +	 * Don't hold pack_mutex across it. The bits are still set,
> +	 * so the range cannot be reused and the pack cannot be freed.
> +	 */
> +	if (IS_ENABLED(CONFIG_ARM64))
> +		mutex_unlock(&pack_mutex);
>  	WARN_ONCE(bpf_arch_text_invalidate(ptr, size),
>  		  "bpf_prog_pack bug: missing bpf_arch_text_invalidate?\n");
> +	if (IS_ENABLED(CONFIG_ARM64))

Can't we do the bpf_arch_text_invalidate() before taking the pack_mutex?
then it works on all architectures and there is no lock nesting.

Because the caller owns [ptr, ptr+size) until bitmap_clear() so it
should work.

I can commandeer it if you want.

Thanks,
Puranjay

Re: [PATCH bpf] bpf: Don't hold pack_mutex over arm64 IPIs

[Alexei Starovoitov]

On Tue, Oct 27, 2026 at 12:59 PM Puranjay Mohan <puranjay@kernel.org> wrote:
> Can't we do the bpf_arch_text_invalidate() before taking the pack_mutex?
> then it works on all architectures and there is no lock nesting.
>
> Because the caller owns [ptr, ptr+size) until bitmap_clear() so it
> should work.

Interesting idea. I guess that's simpler.
bpf_arch_text_copy() already writes into a pack without pack_mutex.

but other archs gain nothing. fine.

> I can commandeer it if you want.

Sure. pls go ahead.

pw-bot:cr

first time for my own patches. next level unlocked :)