* [PATCH bpf 0/2] bpf: Avoid workqueue deadlock when freeing programs
@ 2026-11-02 7:34 Emil Tsalapatis
2026-11-02 7:34 ` [PATCH bpf 1/2] bpf: Use dedicated workqueue for freeing BPF programs Emil Tsalapatis
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Emil Tsalapatis @ 2026-11-02 7:34 UTC (permalink / raw)
To: bpf; +Cc: ast, andrii, eddyz87, memxor, daniel, Emil Tsalapatis
BPF program freeing currently runs from the system workqueue. With BPF
program pack allocations, freeing the last program in a pack can take
pack_mutex and call into execmem/vmalloc code to change page permissions.
That path may queue a vmalloc purge helper to the same CPU and wait for it
to finish.
If enough other BPF program free workers are already active on that CPU,
they can all block on pack_mutex and consume the system workqueue active
slots. The worker holding pack_mutex then waits for the vmalloc purge work,
but that purge work cannot run because the queue is full of pack_mutex
waiters:
[ 24.646667] INFO: task kworker/0:4:1291 blocked for more than 10 seconds.
[ 24.646669] Not tainted 7.3.0-00001-gdeaa120a9b25-dirty #11
[ 24.646670] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 24.646671] task:kworker/0:4 state:D stack:0 pid:1291 tgid:1291 ppid:2 task_flags:0x4208060 flags:0x00080000
[ 24.646675] Workqueue: events bpf_prog_free_deferred
[ 24.646678] Call Trace:
[ 24.646680] <TASK>
[ 24.646686] __schedule+0x5bc/0x1430
[ 24.646702] schedule+0x3f/0x100
[ 24.646706] schedule_timeout+0x106/0x110
[ 24.646711] ? srso_alias_return_thunk+0x5/0xfbef5
[ 24.646713] ? mark_held_locks+0x49/0x80
[ 24.646720] ? srso_alias_return_thunk+0x5/0xfbef5
[ 24.646723] ? lockdep_hardirqs_on_prepare+0xdc/0x190
[ 24.646726] ? srso_alias_return_thunk+0x5/0xfbef5
[ 24.646732] __wait_for_common+0xc1/0x1d0
[ 24.646735] ? __pfx_schedule_timeout+0x10/0x10
[ 24.646748] ? __flush_work+0x1f8/0x550
[ 24.646751] __flush_work+0x309/0x550
[ 24.646759] ? __pfx_wq_barrier_func+0x10/0x10
[ 24.646783] __purge_vmap_area_lazy+0x3e4/0x440
[ 24.646796] _vm_unmap_aliases+0x2d5/0x330
[ 24.646810] change_page_attr_set_clr+0xb9/0x580
[ 24.646817] ? srso_alias_return_thunk+0x5/0xfbef5
[ 24.646833] set_memory_nx+0x3b/0x50
[ 24.646838] __execmem_cache_free+0x35/0xb0
[ 24.646848] execmem_free+0x98/0x160
[ 24.646864] bpf_prog_pack_free+0x152/0x190
[ 24.646871] bpf_jit_binary_pack_free+0x17/0x90
[ 24.646876] bpf_jit_free+0x5e/0x130
[ 24.646882] process_one_work+0x289/0x790
[ 24.646886] ? srso_alias_return_thunk+0x5/0xfbef5
[ 24.646888] ? srso_alias_return_thunk+0x5/0xfbef5
[ 24.646902] worker_thread+0x1b3/0x370
[ 24.646908] ? __pfx_worker_thread+0x10/0x10
[ 24.646911] kthread+0xf7/0x130
[ 24.646914] ? __pfx_kthread+0x10/0x10
[ 24.646920] ret_from_fork+0x293/0x300
[ 24.646923] ? __pfx_kthread+0x10/0x10
[ 24.646927] ret_from_fork_asm+0x1a/0x30
[ 24.646947] </TASK>
Move deferred BPF program freeing to a dedicated workqueue to avoid
this.
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
Emil Tsalapatis (2):
bpf: Use dedicated workqueue for freeing BPF programs
selftests/bpf: Add a selftest for program allocation/freeing
kernel/bpf/core.c | 19 +-
tools/testing/selftests/bpf/.gitignore | 1 +
tools/testing/selftests/bpf/Makefile | 1 +
.../selftests/bpf/bpf_prog_free_stress.c | 184 ++++++++++++++++++
4 files changed, 204 insertions(+), 1 deletion(-)
create mode 100644 tools/testing/selftests/bpf/bpf_prog_free_stress.c
--
2.52.0
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH bpf 1/2] bpf: Use dedicated workqueue for freeing BPF programs
2026-11-02 7:34 [PATCH bpf 0/2] bpf: Avoid workqueue deadlock when freeing programs Emil Tsalapatis
@ 2026-11-02 7:34 ` Emil Tsalapatis
2026-11-02 17:58 ` Alexei Starovoitov
2026-11-02 7:34 ` [PATCH bpf 2/2] selftests/bpf: Add a selftest for program allocation/freeing Emil Tsalapatis
2026-11-02 17:55 ` [PATCH bpf 0/2] bpf: Avoid workqueue deadlock when freeing programs Andrii Nakryiko
2 siblings, 1 reply; 6+ messages in thread
From: Emil Tsalapatis @ 2026-11-02 7:34 UTC (permalink / raw)
To: bpf; +Cc: ast, andrii, eddyz87, memxor, daniel, Emil Tsalapatis
Move deferred BPF program frees to a dedicated workqueue.
BPF free workers can exhaust a CPU's system workqueue active slots while
waiting for pack_mutex. The worker holding pack_mutex then waits for a
vmalloc purge helper on that same queue, which cannot run until a slot
is freed.
Fixes: 6b88aaec81db ("x86/bpf: Enable EXECMEM_ROX_CACHE for BPF allocations")
Reported-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
---
kernel/bpf/core.c | 19 ++++++++++++++++++-
1 file changed, 18 insertions(+), 1 deletion(-)
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 7f11555a5070..a4994438a8af 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -23,6 +23,7 @@
#include <linux/skbuff.h>
#include <linux/static_call.h>
#include <linux/vmalloc.h>
+#include <linux/workqueue.h>
#include <linux/prandom.h>
#include <linux/bpf.h>
#include <linux/btf.h>
@@ -3073,6 +3074,17 @@ static void bpf_free_used_btfs(struct bpf_prog_aux *aux)
kfree(aux->used_btfs);
}
+static struct workqueue_struct *bpf_prog_free_wq __ro_after_init;
+
+static int __init bpf_prog_free_wq_init(void)
+{
+ bpf_prog_free_wq = alloc_workqueue("bpf_prog_free", WQ_UNBOUND, 0);
+ if (!bpf_prog_free_wq)
+ panic("Failed to allocate BPF program free workqueue\n");
+ return 0;
+}
+early_initcall(bpf_prog_free_wq_init);
+
static void bpf_prog_free_deferred(struct work_struct *work)
{
struct bpf_prog_aux *aux;
@@ -3122,7 +3134,12 @@ void bpf_prog_free(struct bpf_prog *fp)
bpf_prog_put(aux->dst_prog);
bpf_token_put(aux->token);
INIT_WORK(&aux->work, bpf_prog_free_deferred);
- schedule_work(&aux->work);
+ /*
+ * A pack_mutex holder may wait for vmalloc work on system_percpu_wq.
+ * Keep program frees off that queue so mutex waiters cannot exhaust
+ * the active slots needed by the vmalloc work.
+ */
+ queue_work(bpf_prog_free_wq, &aux->work);
}
EXPORT_SYMBOL_GPL(bpf_prog_free);
--
2.52.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH bpf 2/2] selftests/bpf: Add a selftest for program allocation/freeing
2026-11-02 7:34 [PATCH bpf 0/2] bpf: Avoid workqueue deadlock when freeing programs Emil Tsalapatis
2026-11-02 7:34 ` [PATCH bpf 1/2] bpf: Use dedicated workqueue for freeing BPF programs Emil Tsalapatis
@ 2026-11-02 7:34 ` Emil Tsalapatis
2026-11-02 7:47 ` sashiko-bot
2026-11-02 17:55 ` [PATCH bpf 0/2] bpf: Avoid workqueue deadlock when freeing programs Andrii Nakryiko
2 siblings, 1 reply; 6+ messages in thread
From: Emil Tsalapatis @ 2026-11-02 7:34 UTC (permalink / raw)
To: bpf; +Cc: ast, andrii, eddyz87, memxor, daniel, Emil Tsalapatis
Add a selftest that stresses BPF program allocation/free
paths.
Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
---
tools/testing/selftests/bpf/.gitignore | 1 +
tools/testing/selftests/bpf/Makefile | 1 +
.../selftests/bpf/bpf_prog_free_stress.c | 184 ++++++++++++++++++
3 files changed, 186 insertions(+)
create mode 100644 tools/testing/selftests/bpf/bpf_prog_free_stress.c
diff --git a/tools/testing/selftests/bpf/.gitignore b/tools/testing/selftests/bpf/.gitignore
index 986a6389186b..ba72f8ac1f72 100644
--- a/tools/testing/selftests/bpf/.gitignore
+++ b/tools/testing/selftests/bpf/.gitignore
@@ -2,6 +2,7 @@
bpftool
bpf-helpers*
bpf-syscall*
+/bpf_prog_free_stress
test_verifier
test_maps
test_lru_map
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index 93c707116fad..e59e07c4bd12 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -137,6 +137,7 @@ TEST_KMOD_TARGETS = $(addprefix $(OUTPUT)/,$(TEST_KMODS))
# Compile but not part of 'make run_tests'
TEST_GEN_PROGS_EXTENDED = \
+ bpf_prog_free_stress \
bench \
flow_dissector_load \
test_cpp \
diff --git a/tools/testing/selftests/bpf/bpf_prog_free_stress.c b/tools/testing/selftests/bpf/bpf_prog_free_stress.c
new file mode 100644
index 000000000000..8c761845c703
--- /dev/null
+++ b/tools/testing/selftests/bpf/bpf_prog_free_stress.c
@@ -0,0 +1,184 @@
+// SPDX-License-Identifier: GPL-2.0
+#define _GNU_SOURCE
+#include <errno.h>
+#include <err.h>
+#include <linux/bpf.h>
+#include <pthread.h>
+#include <sched.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/resource.h>
+#include <sys/syscall.h>
+#include <time.h>
+#include <unistd.h>
+
+#include "../../../include/linux/filter.h"
+
+/*
+ * Spread allocations over vmap nodes, then queue frees from CPU0, whose purge
+ * helper runs asynchronously. Tiny programs fill packs densely enough to build
+ * a backlog exceeding the system workqueue's active-work limit.
+ */
+#define LOAD_CPUS 64
+#define PROGRAMS 65536
+#define WAITERS 1024
+#define DEFAULT_DURATION_SEC 5
+
+static int fds[PROGRAMS];
+static const struct bpf_insn insns[] = {
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+};
+
+static void pin_cpu(int cpu)
+{
+ cpu_set_t mask;
+
+ CPU_ZERO(&mask);
+ CPU_SET(cpu, &mask);
+ if (sched_setaffinity(0, sizeof(mask), &mask))
+ err(EXIT_FAILURE, "sched_setaffinity CPU%d", cpu);
+}
+
+static int load_prog(void)
+{
+ union bpf_attr attr = {
+ .prog_type = BPF_PROG_TYPE_SOCKET_FILTER,
+ .insn_cnt = sizeof(insns) / sizeof(insns[0]),
+ .insns = (uintptr_t)insns,
+ .license = (uintptr_t)"GPL",
+ };
+ int fd;
+
+ fd = syscall(__NR_bpf, BPF_PROG_LOAD, &attr, sizeof(attr));
+ if (fd < 0)
+ err(EXIT_FAILURE, "BPF_PROG_LOAD");
+ return fd;
+}
+
+static void *load_worker(void *arg)
+{
+ int cpu = (uintptr_t)arg;
+ int first = cpu * (PROGRAMS / LOAD_CPUS);
+ int i;
+
+ pin_cpu(cpu);
+ for (i = first; i < first + PROGRAMS / LOAD_CPUS; i++)
+ fds[i] = load_prog();
+ return NULL;
+}
+
+static void *load_waiter(void *arg)
+{
+ (void)arg;
+
+ pin_cpu(0);
+ if (close(load_prog()))
+ err(EXIT_FAILURE, "close waiter program");
+ return NULL;
+}
+
+static void check_pthread(int error, const char *operation)
+{
+ if (error)
+ errx(EXIT_FAILURE, "%s: %s", operation, strerror(error));
+}
+
+static void run_threads(int count, void *(*worker)(void *))
+{
+ pthread_t threads[WAITERS];
+ pthread_attr_t attr;
+ int i;
+
+ check_pthread(pthread_attr_init(&attr), "pthread_attr_init");
+ check_pthread(pthread_attr_setstacksize(&attr, 64 * 1024),
+ "pthread_attr_setstacksize");
+ for (i = 0; i < count; i++)
+ check_pthread(pthread_create(&threads[i], &attr, worker,
+ (void *)(uintptr_t)i),
+ "pthread_create");
+ for (i = 0; i < count; i++)
+ check_pthread(pthread_join(threads[i], NULL), "pthread_join");
+ check_pthread(pthread_attr_destroy(&attr), "pthread_attr_destroy");
+}
+
+static long monotonic_sec(void)
+{
+ struct timespec now;
+
+ if (clock_gettime(CLOCK_MONOTONIC, &now))
+ err(EXIT_FAILURE, "clock_gettime");
+ return now.tv_sec;
+}
+
+static long parse_duration(int argc, char **argv)
+{
+ char *end;
+ long duration;
+
+ if (argc == 1)
+ return DEFAULT_DURATION_SEC;
+ if (argc != 2)
+ errx(EXIT_FAILURE, "usage: %s [duration_sec]", argv[0]);
+
+ errno = 0;
+ duration = strtol(argv[1], &end, 10);
+ if (errno || *end || duration <= 0)
+ errx(EXIT_FAILURE, "usage: %s [duration_sec]", argv[0]);
+ return duration;
+}
+
+static void run_burst(int round)
+{
+ struct sched_param priority = { .sched_priority = 80 };
+ int i;
+
+ printf("Round %d: loading %d programs across %d CPUs\n", round,
+ PROGRAMS, LOAD_CPUS);
+ run_threads(LOAD_CPUS, load_worker);
+ printf("Round %d: loaded %d programs; closing from CPU0\n", round,
+ PROGRAMS);
+
+ if (sched_setscheduler(0, SCHED_FIFO, &priority))
+ err(EXIT_FAILURE, "sched_setscheduler SCHED_FIFO");
+ for (i = 0; i < PROGRAMS; i++)
+ if (close(fds[i]))
+ err(EXIT_FAILURE, "close program %d", i);
+ priority.sched_priority = 0;
+ if (sched_setscheduler(0, SCHED_OTHER, &priority))
+ err(EXIT_FAILURE, "sched_setscheduler SCHED_OTHER");
+
+ printf("Closed %d programs; starting %d allocation waiters\n",
+ PROGRAMS, WAITERS);
+ run_threads(WAITERS, load_waiter);
+}
+
+int main(int argc, char **argv)
+{
+ struct rlimit limit = { PROGRAMS + 1024, PROGRAMS + 1024 };
+ long duration = parse_duration(argc, argv);
+ long start;
+ int round = 0;
+
+ setvbuf(stdout, NULL, _IOLBF, 0);
+ if (sysconf(_SC_NPROCESSORS_ONLN) < LOAD_CPUS)
+ errx(EXIT_FAILURE, "need at least %d online CPUs", LOAD_CPUS);
+
+ if (setrlimit(RLIMIT_NOFILE, &limit))
+ err(EXIT_FAILURE, "setrlimit RLIMIT_NOFILE");
+ limit.rlim_cur = limit.rlim_max = RLIM_INFINITY;
+ if (setrlimit(RLIMIT_MEMLOCK, &limit))
+ err(EXIT_FAILURE, "setrlimit RLIMIT_MEMLOCK");
+ pin_cpu(0);
+
+ printf("Running for %ld seconds\n", duration);
+ start = monotonic_sec();
+ do {
+ run_burst(++round);
+ } while (monotonic_sec() - start < duration);
+
+ printf("Completed %d rounds\n", round);
+ return EXIT_SUCCESS;
+}
--
2.52.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH bpf 2/2] selftests/bpf: Add a selftest for program allocation/freeing
2026-11-02 7:34 ` [PATCH bpf 2/2] selftests/bpf: Add a selftest for program allocation/freeing Emil Tsalapatis
@ 2026-11-02 7:47 ` sashiko-bot
0 siblings, 0 replies; 6+ messages in thread
From: sashiko-bot @ 2026-11-02 7:47 UTC (permalink / raw)
To: Emil Tsalapatis, bpf
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [Medium] selftests/bpf: standalone stress binary bypasses test_progs and kselftest
- [Medium] selftests/bpf: hardcoded 64-CPU check and contiguous CPU IDs in pin_cpu()
- [Medium] selftests/bpf: 64 KiB stack in run_threads() below PTHREAD_STACK_MIN
- [Low] selftests/bpf: commit message omits deadlock details and test design
--
commit f26b450586e3e82488cb0fdfca492b1cd1920f9b
Author: Emil Tsalapatis <emil@etsalapatis.com>
selftests/bpf: Add a selftest for program allocation/freeing
This commit adds a standalone stress test under tools/testing/selftests/bpf/
that repeatedly loads 65,536 socket filter programs across 64 CPUs, closes
them from CPU 0 under SCHED_FIFO, and spawns 1,024 waiter threads.
> selftests/bpf: Add a selftest for program allocation/freeing
>
> Add a selftest that stresses BPF program allocation/free
> paths.
>
> Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
[Severity: Low]
This isn't a bug, but the commit message only states that a selftest is
added without explaining the underlying kernel bug mechanism or why the
workload in run_burst() is structured this way.
Could the commit message be expanded to describe the deadlock scenario being
tested (such as deferred __bpf_prog_put_noref() work items saturating CPU 0's
system_percpu_wq max_active limit while blocked on pack_mutex and starving
the vmalloc purge_vmap_node work item), the rationale for the multi-phase
workload parameters in run_burst(), and why the binary is placed in
TEST_GEN_PROGS_EXTENDED?
> diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
> index 93c707116fad0..e59e07c4bd121 100644
> --- a/tools/testing/selftests/bpf/Makefile
> +++ b/tools/testing/selftests/bpf/Makefile
> @@ -137,6 +137,7 @@ TEST_KMOD_TARGETS = $(addprefix $(OUTPUT)/,$(TEST_KMODS))
>
> # Compile but not part of 'make run_tests'
> TEST_GEN_PROGS_EXTENDED = \
> + bpf_prog_free_stress \
> bench \
> flow_dissector_load \
> test_cpp \
[Severity: Medium]
By adding bpf_prog_free_stress only to TEST_GEN_PROGS_EXTENDED without a
corresponding runner in TEST_PROGS or prog_tests/, neither make run_tests
nor BPF CI (test_progs) will ever execute this test.
In addition, tools/testing/selftests/bpf/bpf_prog_free_stress.c defines a
standalone main() that reports status via printf() and err()/errx(), and
reimplements program loading in load_prog() via raw syscall(__NR_bpf,
BPF_PROG_LOAD, ...) instead of using libbpf's bpf_prog_load() and either
test_progs ASSERT_*() macros (like existing stress tests in prog_tests/ such
as task_work_stress.c, fexit_stress.c, and bpf_mod_race.c) or kselftest.h
ksft_* reporting helpers.
Should this test be integrated as a test_progs subtest under prog_tests/ (or
use libbpf's bpf_prog_load() and kselftest.h reporting if kept standalone)?
> diff --git a/tools/testing/selftests/bpf/bpf_prog_free_stress.c b/tools/testing/selftests/bpf/bpf_prog_free_stress.c
> new file mode 100644
> index 0000000000000..8c761845c7032
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/bpf_prog_free_stress.c
> @@ -0,0 +1,184 @@
[ ... ]
> +#define LOAD_CPUS 64
> +#define PROGRAMS 65536
> +#define WAITERS 1024
> +#define DEFAULT_DURATION_SEC 5
> +
> +static int fds[PROGRAMS];
> +static const struct bpf_insn insns[] = {
> + BPF_MOV64_IMM(BPF_REG_0, 0),
> + BPF_EXIT_INSN(),
> +};
> +
> +static void pin_cpu(int cpu)
> +{
> + cpu_set_t mask;
> +
> + CPU_ZERO(&mask);
> + CPU_SET(cpu, &mask);
> + if (sched_setaffinity(0, sizeof(mask), &mask))
> + err(EXIT_FAILURE, "sched_setaffinity CPU%d", cpu);
> +}
[ ... ]
> +static void *load_worker(void *arg)
> +{
> + int cpu = (uintptr_t)arg;
> + int first = cpu * (PROGRAMS / LOAD_CPUS);
> + int i;
> +
> + pin_cpu(cpu);
> + for (i = first; i < first + PROGRAMS / LOAD_CPUS; i++)
> + fds[i] = load_prog();
> + return NULL;
> +}
[ ... ]
> +static void run_threads(int count, void *(*worker)(void *))
> +{
> + pthread_t threads[WAITERS];
> + pthread_attr_t attr;
> + int i;
> +
> + check_pthread(pthread_attr_init(&attr), "pthread_attr_init");
> + check_pthread(pthread_attr_setstacksize(&attr, 64 * 1024),
> + "pthread_attr_setstacksize");
[Severity: Medium]
In run_threads(), does setting a hardcoded 64 KiB stack size via
pthread_attr_setstacksize(&attr, 64 * 1024) fail with EINVAL on architectures
where PTHREAD_STACK_MIN is larger than 64 KiB?
For example, on powerpc64le (where PTHREAD_STACK_MIN is 131072) or aarch64
with 64 KiB pages, pthread_attr_setstacksize() returns EINVAL when the
requested size is smaller than PTHREAD_STACK_MIN, causing check_pthread() to
abort the test with EXIT_FAILURE on the first burst.
Could this clamp the stack size to at least PTHREAD_STACK_MIN?
[ ... ]
> +int main(int argc, char **argv)
> +{
> + struct rlimit limit = { PROGRAMS + 1024, PROGRAMS + 1024 };
> + long duration = parse_duration(argc, argv);
> + long start;
> + int round = 0;
> +
> + setvbuf(stdout, NULL, _IOLBF, 0);
> + if (sysconf(_SC_NPROCESSORS_ONLN) < LOAD_CPUS)
> + errx(EXIT_FAILURE, "need at least %d online CPUs", LOAD_CPUS);
[Severity: Medium]
What happens in main() and load_worker() when this test runs on systems with
fewer than 64 online CPUs, or on systems with at least 64 online CPUs where
CPU IDs 0 through 63 are not all online and permitted by the process's CPU
affinity mask?
First, main() exits with EXIT_FAILURE when sysconf(_SC_NPROCESSORS_ONLN) <
LOAD_CPUS (64), or when unprivileged calls to setrlimit(),
sched_setscheduler(), or BPF_PROG_LOAD fail. Per kselftest policy, missing
environment or hardware prerequisites should skip rather than fail. Moreover,
since __purge_vmap_area_lazy() in mm/vmalloc.c queues asynchronous purge work
onto system_percpu_wq whenever nr_purge_nodes >= 2, could the test scale to
the number of available CPUs in the affinity mask and only skip when fewer
than 2 CPUs are available?
Second, load_worker() passes the worker index 0..LOAD_CPUS-1 directly to
pin_cpu(cpu), while main() and load_waiter() hardcode pin_cpu(0). Because
sysconf(_SC_NPROCESSORS_ONLN) only checks the total count of online CPUs,
sched_setaffinity() in pin_cpu() will fail with EINVAL if any CPU in 0..63 is
offline or excluded by the current cpuset or taskset affinity mask. Could
main() query sched_getaffinity() at startup and map workers onto the set of
usable CPU IDs?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261102073439.6822-1-emil@etsalapatis.com?part=2
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH bpf 0/2] bpf: Avoid workqueue deadlock when freeing programs
2026-11-02 7:34 [PATCH bpf 0/2] bpf: Avoid workqueue deadlock when freeing programs Emil Tsalapatis
2026-11-02 7:34 ` [PATCH bpf 1/2] bpf: Use dedicated workqueue for freeing BPF programs Emil Tsalapatis
2026-11-02 7:34 ` [PATCH bpf 2/2] selftests/bpf: Add a selftest for program allocation/freeing Emil Tsalapatis
@ 2026-11-02 17:55 ` Andrii Nakryiko
2 siblings, 0 replies; 6+ messages in thread
From: Andrii Nakryiko @ 2026-11-02 17:55 UTC (permalink / raw)
To: Emil Tsalapatis; +Cc: bpf, ast, andrii, eddyz87, memxor, daniel
On Sun, Nov 1, 2026 at 11:34 PM Emil Tsalapatis <emil@etsalapatis.com> wrote:
>
> BPF program freeing currently runs from the system workqueue. With BPF
> program pack allocations, freeing the last program in a pack can take
> pack_mutex and call into execmem/vmalloc code to change page permissions.
> That path may queue a vmalloc purge helper to the same CPU and wait for it
> to finish.
>
> If enough other BPF program free workers are already active on that CPU,
> they can all block on pack_mutex and consume the system workqueue active
> slots. The worker holding pack_mutex then waits for the vmalloc purge work,
> but that purge work cannot run because the queue is full of pack_mutex
> waiters:
>
> [ 24.646667] INFO: task kworker/0:4:1291 blocked for more than 10 seconds.
> [ 24.646669] Not tainted 7.3.0-00001-gdeaa120a9b25-dirty #11
> [ 24.646670] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 24.646671] task:kworker/0:4 state:D stack:0 pid:1291 tgid:1291 ppid:2 task_flags:0x4208060 flags:0x00080000
> [ 24.646675] Workqueue: events bpf_prog_free_deferred
> [ 24.646678] Call Trace:
> [ 24.646680] <TASK>
> [ 24.646686] __schedule+0x5bc/0x1430
> [ 24.646702] schedule+0x3f/0x100
> [ 24.646706] schedule_timeout+0x106/0x110
> [ 24.646711] ? srso_alias_return_thunk+0x5/0xfbef5
> [ 24.646713] ? mark_held_locks+0x49/0x80
> [ 24.646720] ? srso_alias_return_thunk+0x5/0xfbef5
> [ 24.646723] ? lockdep_hardirqs_on_prepare+0xdc/0x190
> [ 24.646726] ? srso_alias_return_thunk+0x5/0xfbef5
> [ 24.646732] __wait_for_common+0xc1/0x1d0
> [ 24.646735] ? __pfx_schedule_timeout+0x10/0x10
> [ 24.646748] ? __flush_work+0x1f8/0x550
> [ 24.646751] __flush_work+0x309/0x550
> [ 24.646759] ? __pfx_wq_barrier_func+0x10/0x10
> [ 24.646783] __purge_vmap_area_lazy+0x3e4/0x440
> [ 24.646796] _vm_unmap_aliases+0x2d5/0x330
> [ 24.646810] change_page_attr_set_clr+0xb9/0x580
> [ 24.646817] ? srso_alias_return_thunk+0x5/0xfbef5
> [ 24.646833] set_memory_nx+0x3b/0x50
> [ 24.646838] __execmem_cache_free+0x35/0xb0
> [ 24.646848] execmem_free+0x98/0x160
> [ 24.646864] bpf_prog_pack_free+0x152/0x190
> [ 24.646871] bpf_jit_binary_pack_free+0x17/0x90
> [ 24.646876] bpf_jit_free+0x5e/0x130
> [ 24.646882] process_one_work+0x289/0x790
> [ 24.646886] ? srso_alias_return_thunk+0x5/0xfbef5
> [ 24.646888] ? srso_alias_return_thunk+0x5/0xfbef5
> [ 24.646902] worker_thread+0x1b3/0x370
> [ 24.646908] ? __pfx_worker_thread+0x10/0x10
> [ 24.646911] kthread+0xf7/0x130
> [ 24.646914] ? __pfx_kthread+0x10/0x10
> [ 24.646920] ret_from_fork+0x293/0x300
> [ 24.646923] ? __pfx_kthread+0x10/0x10
> [ 24.646927] ret_from_fork_asm+0x1a/0x30
> [ 24.646947] </TASK>
>
> Move deferred BPF program freeing to a dedicated workqueue to avoid
> this.
>
Hey Emil,
My AI says that:
a) we might have similar problems due to trampoline freeing, so maybe
we should address both in the same fix?
b) there apparently was fix by Josef touching this code, so now your
fix doesn't apply cleanly, please double check
and me saying
c) do we really need to have an extra binary (bpf_prog_free_stress)
that we won't be really testing regularly? it's good as reproducer and
to confirm the fix, but I wouldn't check it into selftests
pw-bot: cr
> Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
>
> Emil Tsalapatis (2):
> bpf: Use dedicated workqueue for freeing BPF programs
> selftests/bpf: Add a selftest for program allocation/freeing
>
> kernel/bpf/core.c | 19 +-
> tools/testing/selftests/bpf/.gitignore | 1 +
> tools/testing/selftests/bpf/Makefile | 1 +
> .../selftests/bpf/bpf_prog_free_stress.c | 184 ++++++++++++++++++
> 4 files changed, 204 insertions(+), 1 deletion(-)
> create mode 100644 tools/testing/selftests/bpf/bpf_prog_free_stress.c
>
> --
> 2.52.0
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH bpf 1/2] bpf: Use dedicated workqueue for freeing BPF programs
2026-11-02 7:34 ` [PATCH bpf 1/2] bpf: Use dedicated workqueue for freeing BPF programs Emil Tsalapatis
@ 2026-11-02 17:58 ` Alexei Starovoitov
0 siblings, 0 replies; 6+ messages in thread
From: Alexei Starovoitov @ 2026-11-02 17:58 UTC (permalink / raw)
To: Emil Tsalapatis, bpf; +Cc: andrii, eddyz87, memxor, daniel
On Sun, Nov 01, 2026 at 11:34 PM Emil Tsalapatis <emil@etsalapatis.com> wrote:
> BPF free workers can exhaust a CPU's system workqueue active slots while
> waiting for pack_mutex. The worker holding pack_mutex then waits for a
> vmalloc purge helper on that same queue, which cannot run until a slot
> is freed.
This doesn't apply to bpf tree.
Commit 4eaf4d71b91a ("bpf: Keep the symbols of a program until it has
ended") moved bpf_jit_free() into bpf_prog_free_images().
bpf_prog_free_deferred() queues it with
queue_rcu_work(system_percpu_wq, &aux->rwork).
So with this patch the pack_mutex waiters are still on system_percpu_wq.
__bpf_tramp_image_put_deferred() has the same problem.
It runs from schedule_work(), and arch_free_bpf_trampoline() calls
bpf_prog_pack_free().
Let's move all three to the new wq in one patch?
> Fixes: 6b88aaec81db ("x86/bpf: Enable EXECMEM_ROX_CACHE for BPF allocations")
The ROX cache is not the cause.
Without it bpf_jit_free_exec() is vfree() of a VM_FLUSH_RESET_PERMS area.
vm_reset_perms() calls _vm_unmap_aliases() and gets to the same
flush_work(), also under pack_mutex.
That's every arch with bpf_prog_pack since
commit 72210662c5a2 ("mm: vmalloc: offload free_vmap_area_lock lock").
> + bpf_prog_free_wq = alloc_workqueue("bpf_prog_free", WQ_UNBOUND, 0);
0 means 1024. That's up to 1024 kworkers that sleep on pack_mutex.
max_active = 1 like cgroup_bpf_destroy_wq ?
pw-bot: cr
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-11-02 17:58 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-02 7:34 [PATCH bpf 0/2] bpf: Avoid workqueue deadlock when freeing programs Emil Tsalapatis
2026-11-02 7:34 ` [PATCH bpf 1/2] bpf: Use dedicated workqueue for freeing BPF programs Emil Tsalapatis
2026-11-02 17:58 ` Alexei Starovoitov
2026-11-02 7:34 ` [PATCH bpf 2/2] selftests/bpf: Add a selftest for program allocation/freeing Emil Tsalapatis
2026-11-02 7:47 ` sashiko-bot
2026-11-02 17:55 ` [PATCH bpf 0/2] bpf: Avoid workqueue deadlock when freeing programs Andrii Nakryiko
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.