[PATCH bpf 2/2] bpf: Invalidate freed prog pack memory without holding pack_mutex

[Puranjay Mohan <puranjay@kernel.org>]

bpf_prog_pack_free() calls bpf_arch_text_invalidate() with pack_mutex
held. On arm64 it interrupts all other CPUs and waits for them:

  bpf_prog_free_deferred
    bpf_prog_free_images
      bpf_jit_free
        bpf_jit_binary_pack_free
          bpf_prog_pack_free
            bpf_arch_text_invalidate
              aarch64_insn_set
                __text_poke
                  flush_icache_range
                    kick_all_cpus_sync

When many progs are freed at once, every prog load waits in
bpf_prog_pack_alloc() for all of these broadcasts. On arm64 hosts with
many CPUs this showed up as hung task reports with sk_attach_filter()
blocked on pack_mutex for minutes.

pack_mutex protects pack_list, the bitmaps and arch_flush_needed, not
the text. Until bitmap_clear(), the bits of the range are set, so it
cannot be allocated again and the pack cannot be freed. Invalidate it
before taking pack_mutex. A ptr that is in no pack, which only a caller
bug can pass, is now written to before the lookup warns about it.

Tested in arm64 KVM guests with 16 and 64 cpus on a 72 cpu host. cpus/2
processes attach a cBPF filter to 256 sockets and close them in a loop
for 20 sec. One more process attaches a filter every msec. Median of 5
runs.

Sockets attached and closed in 20 sec, in millions (higher is better):

             before  after
  16 cpus      1.22   3.79
  64 cpus      0.42   3.38

setsockopt(SO_ATTACH_FILTER) of the process that attaches every msec,
in msec (lower is better):

                16 cpus          64 cpus
             before  after    before  after
  avg          1.32   0.27      7.21   0.56
  p99          5.39   5.52      79.7   9.89
  max           447    159      1598    253

Fixes: 1dad391daef1 ("bpf, arm64: use bpf_prog_pack for memory management")
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
---
 kernel/bpf/core.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 4dac33a4208aa..51f1b128f0d63 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -1073,6 +1073,9 @@ void bpf_prog_pack_free(void *ptr, u32 size)
 		return;
 	}
 
+	WARN_ONCE(bpf_arch_text_invalidate(ptr, size),
+		  "bpf_prog_pack bug: missing bpf_arch_text_invalidate?\n");
+
 	mutex_lock(&pack_mutex);
 	list_for_each_entry(tmp, &pack_list, list) {
 		if (ptr >= tmp->ptr && (tmp->ptr + BPF_PROG_PACK_SIZE) > ptr) {
@@ -1087,9 +1090,6 @@ void bpf_prog_pack_free(void *ptr, u32 size)
 	nbits = BPF_PROG_SIZE_TO_NBITS(size);
 	pos = ((unsigned long)ptr - (unsigned long)pack->ptr) >> BPF_PROG_CHUNK_SHIFT;
 
-	WARN_ONCE(bpf_arch_text_invalidate(ptr, size),
-		  "bpf_prog_pack bug: missing bpf_arch_text_invalidate?\n");
-
 	bitmap_clear(pack->bitmap, pos, nbits);
 
 	if (static_branch_unlikely(&bpf_pred_flush_enabled))
-- 
2.53.0-Meta