* [PATCH bpf-next 0/4] bpf: BPF-driven proactive memcg reclaim
@ 2026-08-07 7:01 Hui Zhu
2026-08-07 7:01 ` [PATCH bpf-next 1/4] mm/bpf: Add bpf_try_to_free_mem_cgroup_pages kfunc Hui Zhu
` (3 more replies)
0 siblings, 4 replies; 5+ messages in thread
From: Hui Zhu @ 2026-08-07 7:01 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, John Fastabend,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Song Liu, Yonghong Song, Jiri Olsa,
Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, JP Kobryn, Andrew Morton, Shuah Khan, davem,
Jakub Kicinski, Jesper Dangaard Brouer, Stanislav Fomichev,
KP Singh, Tao Chen, Mykyta Yatsenko, Leon Hwang, Anton Protopopov,
Amery Hung, Tobias Klauser, Eyal Birger, Rong Tao, Hao Luo,
Peter Zijlstra, Miguel Ojeda, Nathan Chancellor, Kees Cook,
Tejun Heo, Jeff Xu, mkoutny, Jan Hendrik Farr, Christian Brauner,
Randy Dunlap, Brian Gerst, Masahiro Yamada, Willem de Bruijn,
Jason Xing, Paul Chaignon, Lance Yang, Jiayuan Chen,
Emil Tsalapatis, Ihor Solodrai, Barry Song, Geliang Tang,
linux-kernel, bpf, cgroups, linux-mm, netdev, linux-kselftest
Cc: Hui Zhu
From: Hui Zhu <zhuhui@kylinos.cn>
This series lets a BPF program decide when to trigger memcg reclaim
and how aggressively to do it, based on whatever runtime signal it
chooses to observe -- rather than reclaim only being triggered once a
cgroup's usage crosses a fixed threshold. The core idea is a new kfunc,
bpf_try_to_free_mem_cgroup_pages(), which gives BPF direct access to
the reclaim path so this decision can be made in BPF policy rather than
hard-coded threshold logic.
This was originally part of a larger series posted here [1].
That series also adds a memcg BPF struct_ops (memcg_charged,
memcg_uncharged, below_low, below_min) for synchronous, in-line memory
protection decisions. That mechanism and this one solve different
problems -- struct_ops hooks run inline on the charge/reclaim path,
while the kfunc here is for asynchronous, out-of-band reclaim decided
independently by a BPF program -- so I think they're better reviewed
as separate series rather than bundled together. This series carries
only the async reclaim piece: the bpf_try_to_free_mem_cgroup_pages
kfunc (patch 1), plus a new bpf_thread_wq mechanism (patch 2) that
grew out of discussion here [2].
Patch 1 adds bpf_try_to_free_mem_cgroup_pages(), a sleepable kfunc
wrapping try_to_free_mem_cgroup_pages(). With it, a BPF program can
reclaim from a given cgroup on its own terms -- any condition it can
observe at runtime -- instead of being limited to "usage hit
threshold X".
Patch 2 adds bpf_thread_wq, a bpf_wq-like map field backed by a
dedicated kthread_worker rather than the system workqueue, which can
be attached to a specific cgroup at init time. The motivation is
accounting: reclaim work triggered from BPF still costs CPU (and can
itself touch memory), and running it via a plain irq_work/system
workqueue callback would charge that cost to whatever context happens
to run it, not to the cgroup the policy cares about. bpf_thread_wq
lets that cost be attributed to a chosen cgroup instead -- e.g. the
low-priority cgroup being reclaimed from.
Patch 3 is a selftest that exercises bpf_thread_wq's cgroup attachment
in isolation: verifying the callback observes the target cgroup when
one is given, and does not when it isn't.
Patch 4 (selftests/bpf: add memcg async reclaim test for
bpf_wq/bpf_thread_wq) ties patches 1 and 2 together as a worked
example: it watches the WORKINGSET_REFAULT_FILE counter of a
high-priority cgroup as a proxy for memory-pressure impact, and once
it starts climbing, proactively reclaims pages from a low-priority
cgroup via bpf_try_to_free_mem_cgroup_pages, running that reclaim
inside a bpf_thread_wq attached to the low-priority cgroup so the
reclaim cost lands on it rather than leaking into an unrelated
context. This demonstrates the end-to-end use case: BPF observes
pressure on the cgroup it wants to protect, and reclaims from the
cgroup it wants to charge, in one self-contained mechanism.
[1] https://sashiko.dev/#/message/cover.1779760876.git.zhuhui%40kylinos.cn
[2] https://sashiko.dev/#/message/1b58d56976202f26818d31dbd0da2ecb2e2460f5%40linux.dev
Hui Zhu (4):
mm/bpf: Add bpf_try_to_free_mem_cgroup_pages kfunc
bpf: add bpf_thread_wq kthread-backed workqueue with cgroup placement
selftests/bpf: add thread_wq cgroup test
selftests/bpf: add memcg async reclaim test for bpf_wq/bpf_thread_wq
include/linux/bpf.h | 15 +-
include/linux/cgroup.h | 2 +
include/uapi/linux/bpf.h | 4 +
kernel/bpf/btf.c | 7 +
kernel/bpf/helpers.c | 418 +++++++++++++++
kernel/bpf/syscall.c | 15 +-
kernel/bpf/verifier.c | 44 +-
kernel/cgroup/cgroup.c | 13 +
mm/bpf_memcontrol.c | 58 +++
.../testing/selftests/bpf/bpf_experimental.h | 7 +
.../bpf/prog_tests/memcg_async_reclaim.c | 479 ++++++++++++++++++
.../bpf/prog_tests/thread_wq_cgroup.c | 87 ++++
.../selftests/bpf/progs/memcg_async_reclaim.c | 255 ++++++++++
.../selftests/bpf/progs/thread_wq_cgroup.c | 56 ++
14 files changed, 1455 insertions(+), 5 deletions(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/memcg_async_reclaim.c
create mode 100644 tools/testing/selftests/bpf/prog_tests/thread_wq_cgroup.c
create mode 100644 tools/testing/selftests/bpf/progs/memcg_async_reclaim.c
create mode 100644 tools/testing/selftests/bpf/progs/thread_wq_cgroup.c
--
2.53.0
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH bpf-next 1/4] mm/bpf: Add bpf_try_to_free_mem_cgroup_pages kfunc
2026-08-07 7:01 [PATCH bpf-next 0/4] bpf: BPF-driven proactive memcg reclaim Hui Zhu
@ 2026-08-07 7:01 ` Hui Zhu
2026-08-07 7:01 ` [PATCH bpf-next 2/4] bpf: add bpf_thread_wq kthread-backed workqueue with cgroup placement Hui Zhu
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Hui Zhu @ 2026-08-07 7:01 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, John Fastabend,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Song Liu, Yonghong Song, Jiri Olsa,
Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, JP Kobryn, Andrew Morton, Shuah Khan, davem,
Jakub Kicinski, Jesper Dangaard Brouer, Stanislav Fomichev,
KP Singh, Tao Chen, Mykyta Yatsenko, Leon Hwang, Anton Protopopov,
Amery Hung, Tobias Klauser, Eyal Birger, Rong Tao, Hao Luo,
Peter Zijlstra, Miguel Ojeda, Nathan Chancellor, Kees Cook,
Tejun Heo, Jeff Xu, mkoutny, Jan Hendrik Farr, Christian Brauner,
Randy Dunlap, Brian Gerst, Masahiro Yamada, Willem de Bruijn,
Jason Xing, Paul Chaignon, Lance Yang, Jiayuan Chen,
Emil Tsalapatis, Ihor Solodrai, Barry Song, Geliang Tang,
linux-kernel, bpf, cgroups, linux-mm, netdev, linux-kselftest
Cc: Hui Zhu
From: Hui Zhu <zhuhui@kylinos.cn>
Expose the memory cgroup reclaim interface to BPF programs by adding
the bpf_try_to_free_mem_cgroup_pages kfunc. This allows BPF to
trigger memory reclamation for a specific cgroup.
The kfunc wraps try_to_free_mem_cgroup_pages and introduces a
swappiness parameter with the following semantics:
Values in [MIN_SWAPPINESS, SWAPPINESS_ANON_ONLY] are passed through
as an explicit swappiness override.
Values below MIN_SWAPPINESS indicate the use of the system default
(passed as NULL to the core reclaim path).
Values above SWAPPINESS_ANON_ONLY result in 0.
Note that the swappiness override is only respected by the core
reclaim path if the MEMCG_RECLAIM_PROACTIVE flag is set in
reclaim_options.
Swap usage during reclaim is gated on reclaim_options: swap is
considered only when MEMCG_RECLAIM_MAY_SWAP is set. Without this
flag, reclaim is restricted to file-backed pages regardless of the
swappiness value or the cgroup's swappiness setting.
Also include <linux/swap.h> for the swappiness macro definitions and
register the function with the KF_SLEEPABLE flag.
Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
---
mm/bpf_memcontrol.c | 58 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 58 insertions(+)
diff --git a/mm/bpf_memcontrol.c b/mm/bpf_memcontrol.c
index 716df49d7647..3f7a5c97e135 100644
--- a/mm/bpf_memcontrol.c
+++ b/mm/bpf_memcontrol.c
@@ -6,6 +6,7 @@
*/
#include <linux/memcontrol.h>
+#include <linux/swap.h>
#include <linux/bpf.h>
__bpf_kfunc_start_defs();
@@ -159,6 +160,61 @@ __bpf_kfunc void bpf_mem_cgroup_flush_stats(struct mem_cgroup *memcg)
mem_cgroup_flush_stats(memcg);
}
+/**
+ * bpf_try_to_free_mem_cgroup_pages - attempt to reclaim pages from
+ * a memory cgroup
+ * @memcg: the target memory cgroup to reclaim from
+ * @nr_pages: the number of pages to reclaim
+ * @gfp_mask: GFP flags controlling the reclaim behavior
+ * @reclaim_options: bitmask of MEMCG_RECLAIM_* flags to tune
+ * reclaim strategy
+ * @swappiness: swappiness override value, or a sentinel to use
+ * the default
+ *
+ * BPF-facing wrapper around try_to_free_mem_cgroup_pages() that
+ * validates and translates the @swappiness argument before
+ * delegating to the core reclaim path.
+ *
+ * The @swappiness parameter follows these semantics:
+ * - Values in [MIN_SWAPPINESS, SWAPPINESS_ANON_ONLY] are passed
+ * through as an explicit swappiness override.
+ * - Values below MIN_SWAPPINESS are treated as "use the system
+ * default"; the override pointer is set to NULL and the cgroup's
+ * own swappiness setting takes effect.
+ * - Values above SWAPPINESS_ANON_ONLY are rejected as invalid.
+ * - If @reclaim_options does not include MEMCG_RECLAIM_PROACTIVE,
+ * the @swappiness override is ignored entirely by the core
+ * reclaim path and the system default is used regardless.
+ *
+ * Swap usage during reclaim is gated on @reclaim_options: swap is
+ * considered only when MEMCG_RECLAIM_MAY_SWAP is set. Without this
+ * flag, reclaim is restricted to file-backed pages regardless of the
+ * @swappiness value or the cgroup's swappiness setting.
+ *
+ * Return:
+ * The number of pages actually reclaimed on success, or 0
+ * if @swappiness exceeds SWAPPINESS_ANON_ONLY.
+ */
+__bpf_kfunc unsigned long
+bpf_try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,
+ unsigned long nr_pages,
+ gfp_t gfp_mask,
+ unsigned int reclaim_options,
+ int swappiness)
+{
+ int *swapiness_ptr;
+
+ if (swappiness > SWAPPINESS_ANON_ONLY)
+ return 0;
+ else if (swappiness < MIN_SWAPPINESS)
+ swapiness_ptr = NULL;
+ else
+ swapiness_ptr = &swappiness;
+
+ return try_to_free_mem_cgroup_pages(memcg, nr_pages, gfp_mask,
+ reclaim_options, swapiness_ptr);
+}
+
__bpf_kfunc_end_defs();
BTF_KFUNCS_START(bpf_memcontrol_kfuncs)
@@ -172,6 +228,8 @@ BTF_ID_FLAGS(func, bpf_mem_cgroup_usage)
BTF_ID_FLAGS(func, bpf_mem_cgroup_page_state)
BTF_ID_FLAGS(func, bpf_mem_cgroup_flush_stats, KF_SLEEPABLE)
+BTF_ID_FLAGS(func, bpf_try_to_free_mem_cgroup_pages, KF_SLEEPABLE)
+
BTF_KFUNCS_END(bpf_memcontrol_kfuncs)
static const struct btf_kfunc_id_set bpf_memcontrol_kfunc_set = {
--
2.53.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH bpf-next 2/4] bpf: add bpf_thread_wq kthread-backed workqueue with cgroup placement
2026-08-07 7:01 [PATCH bpf-next 0/4] bpf: BPF-driven proactive memcg reclaim Hui Zhu
2026-08-07 7:01 ` [PATCH bpf-next 1/4] mm/bpf: Add bpf_try_to_free_mem_cgroup_pages kfunc Hui Zhu
@ 2026-08-07 7:01 ` Hui Zhu
2026-08-07 7:04 ` [PATCH bpf-next 3/4] selftests/bpf: add thread_wq cgroup test Hui Zhu
2026-08-07 7:04 ` [PATCH bpf-next 4/4] selftests/bpf: add memcg async reclaim test for bpf_wq/bpf_thread_wq Hui Zhu
3 siblings, 0 replies; 5+ messages in thread
From: Hui Zhu @ 2026-08-07 7:01 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, John Fastabend,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Song Liu, Yonghong Song, Jiri Olsa,
Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, JP Kobryn, Andrew Morton, Shuah Khan, davem,
Jakub Kicinski, Jesper Dangaard Brouer, Stanislav Fomichev,
KP Singh, Tao Chen, Mykyta Yatsenko, Leon Hwang, Anton Protopopov,
Amery Hung, Tobias Klauser, Eyal Birger, Rong Tao, Hao Luo,
Peter Zijlstra, Miguel Ojeda, Nathan Chancellor, Kees Cook,
Tejun Heo, Jeff Xu, mkoutny, Jan Hendrik Farr, Christian Brauner,
Randy Dunlap, Brian Gerst, Masahiro Yamada, Willem de Bruijn,
Jason Xing, Paul Chaignon, Lance Yang, Jiayuan Chen,
Emil Tsalapatis, Ihor Solodrai, Barry Song, Geliang Tang,
linux-kernel, bpf, cgroups, linux-mm, netdev, linux-kselftest
Cc: Hui Zhu
From: Hui Zhu <zhuhui@kylinos.cn>
Introduce bpf_thread_wq, a new BPF embedded map field similar to
bpf_wq but backed by a dedicated kthread_worker instead of a system
workqueue. The worker kthread can be attached to a specific cgroup at
init time so BPF-deferred callbacks run under the resource limits of
the target cgroup.
Three kfuncs are exposed:
bpf_thread_wq_init(twq, map, cgroup_id, flags) [KF_SLEEPABLE]
bpf_thread_wq_set_callback(twq, cb, flags, aux)
bpf_thread_wq_start(twq, flags)
bpf_thread_wq_init() is registered only for BPF_PROG_TYPE_SYSCALL
programs. It creates a kthread worker and may attach it to a cgroup;
those paths can sleep and acquire kthread and cgroup locks. Restricting
init to syscall programs prevents it from running in BPF contexts that
may already hold locks which could deadlock with those paths.
bpf_thread_wq intentionally avoids the bpf_async infrastructure used by
bpf_timer and bpf_wq. That infrastructure drives cleanup from irq_work
in hardirq context, while bpf_thread_wq cancellation and final teardown
may need to sleep through kthread_cancel_work_sync(),
kthread_destroy_worker() and a final cgroup_put().
bpf_thread_wq_cancel_and_free() therefore cancels work synchronously and
drops the context reference; the last put waits for tasks-trace RCU
readers and then schedules process-context work to run bpf_prog_put(),
cgroup_put(), kthread_destroy_worker() and kfree().
Add BTF/map support for bpf_thread_wq fields, map teardown hooks,
verifier handling for the callback kfunc, and cgroup_kthread_attach() to
move the worker into the requested cgroup.
Supported map types are BPF_MAP_TYPE_HASH, BPF_MAP_TYPE_LRU_HASH, and
BPF_MAP_TYPE_ARRAY, consistent with bpf_wq and bpf_task_work.
Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
---
include/linux/bpf.h | 15 +-
include/linux/cgroup.h | 2 +
include/uapi/linux/bpf.h | 4 +
kernel/bpf/btf.c | 7 +
kernel/bpf/helpers.c | 418 ++++++++++++++++++
kernel/bpf/syscall.c | 15 +-
kernel/bpf/verifier.c | 44 +-
kernel/cgroup/cgroup.c | 13 +
.../testing/selftests/bpf/bpf_experimental.h | 7 +
9 files changed, 520 insertions(+), 5 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 73bacfc6444d..d63ce8319869 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -213,6 +213,7 @@ enum btf_field_type {
BPF_UPTR = (1 << 11),
BPF_RES_SPIN_LOCK = (1 << 12),
BPF_TASK_WORK = (1 << 13),
+ BPF_THREAD_WQ = (1 << 14),
};
enum bpf_cgroup_storage_type {
@@ -267,6 +268,7 @@ struct btf_record {
int wq_off;
int refcount_off;
int task_work_off;
+ int thread_wq_off;
struct btf_field fields[];
};
@@ -372,6 +374,8 @@ static inline const char *btf_field_type_name(enum btf_field_type type)
return "bpf_refcount";
case BPF_TASK_WORK:
return "bpf_task_work";
+ case BPF_THREAD_WQ:
+ return "bpf_thread_wq";
default:
WARN_ON_ONCE(1);
return "unknown";
@@ -412,6 +416,8 @@ static inline u32 btf_field_type_size(enum btf_field_type type)
return sizeof(struct bpf_refcount);
case BPF_TASK_WORK:
return sizeof(struct bpf_task_work);
+ case BPF_THREAD_WQ:
+ return sizeof(struct bpf_thread_wq);
default:
WARN_ON_ONCE(1);
return 0;
@@ -446,6 +452,8 @@ static inline u32 btf_field_type_align(enum btf_field_type type)
return __alignof__(struct bpf_refcount);
case BPF_TASK_WORK:
return __alignof__(struct bpf_task_work);
+ case BPF_THREAD_WQ:
+ return __alignof__(struct bpf_thread_wq);
default:
WARN_ON_ONCE(1);
return 0;
@@ -478,6 +486,7 @@ static inline void bpf_obj_init_field(const struct btf_field *field, void *addr)
case BPF_KPTR_PERCPU:
case BPF_UPTR:
case BPF_TASK_WORK:
+ case BPF_THREAD_WQ:
break;
default:
WARN_ON_ONCE(1);
@@ -502,6 +511,7 @@ static inline bool btf_field_is_nmi_safe(enum btf_field_type type)
case BPF_TASK_WORK:
case BPF_KPTR_UNREF:
case BPF_REFCOUNT:
+ case BPF_THREAD_WQ:
return true;
default:
return false;
@@ -644,6 +654,7 @@ void copy_map_value_locked(struct bpf_map *map, void *dst, void *src,
void bpf_timer_cancel_and_free(void *timer);
void bpf_wq_cancel_and_free(void *timer);
void bpf_task_work_cancel_and_free(void *timer);
+void bpf_thread_wq_cancel_and_free(void *val);
void bpf_list_head_free(const struct btf_field *field, void *list_head,
struct bpf_spin_lock *spin_lock);
void bpf_rb_root_free(const struct btf_field *field, void *rb_root,
@@ -701,7 +712,8 @@ bool bpf_map_meta_equal(const struct bpf_map *meta0,
static inline bool bpf_map_has_internal_structs(struct bpf_map *map)
{
- return btf_record_has_field(map->record, BPF_TIMER | BPF_WORKQUEUE | BPF_TASK_WORK);
+ return btf_record_has_field(map->record, BPF_TIMER | BPF_WORKQUEUE |
+ BPF_TASK_WORK | BPF_THREAD_WQ);
}
void bpf_map_free_internal_structs(struct bpf_map *map, void *obj);
@@ -2725,6 +2737,7 @@ bool btf_record_equal(const struct btf_record *rec_a, const struct btf_record *r
void bpf_obj_free_timer(const struct btf_record *rec, void *obj);
void bpf_obj_free_workqueue(const struct btf_record *rec, void *obj);
void bpf_obj_free_task_work(const struct btf_record *rec, void *obj);
+void bpf_obj_free_thread_wq(const struct btf_record *rec, void *obj);
void bpf_obj_cancel_fields(struct bpf_map *map, void *obj);
void bpf_obj_free_fields(const struct btf_record *rec, void *obj);
void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu);
diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h
index f2aa46a4f871..9b4a8dc748ac 100644
--- a/include/linux/cgroup.h
+++ b/include/linux/cgroup.h
@@ -923,4 +923,6 @@ struct cgroup *task_get_cgroup1(struct task_struct *tsk, int hierarchy_id);
struct cgroup_of_peak *of_peak(struct kernfs_open_file *of);
+int cgroup_kthread_attach(struct cgroup *cgrp, struct task_struct *task);
+
#endif /* _LINUX_CGROUP_H */
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index ffd96e8b920b..0558520f67fe 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -7574,6 +7574,10 @@ struct bpf_wq {
__u64 __opaque[2];
} __attribute__((aligned(8)));
+struct bpf_thread_wq {
+ __u64 __opaque[2];
+} __attribute__((aligned(8)));
+
struct bpf_dynptr {
__u64 __opaque[2];
} __attribute__((aligned(8)));
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 42414633cf26..bf6fb5f51d21 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -3665,6 +3665,7 @@ static int btf_get_field_type(const struct btf *btf, const struct btf_type *var_
{ BPF_TIMER, "bpf_timer", true },
{ BPF_WORKQUEUE, "bpf_wq", true },
{ BPF_TASK_WORK, "bpf_task_work", true },
+ { BPF_THREAD_WQ, "bpf_thread_wq", true },
{ BPF_LIST_HEAD, "bpf_list_head", false },
{ BPF_LIST_NODE, "bpf_list_node", false },
{ BPF_RB_ROOT, "bpf_rb_root", false },
@@ -3850,6 +3851,7 @@ static int btf_find_field_one(const struct btf *btf,
case BPF_RB_NODE:
case BPF_REFCOUNT:
case BPF_TASK_WORK:
+ case BPF_THREAD_WQ:
ret = btf_find_struct(btf, var_type, off, sz, field_type,
info_cnt ? &info[0] : &tmp);
if (ret < 0)
@@ -4145,6 +4147,7 @@ struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type
rec->wq_off = -EINVAL;
rec->refcount_off = -EINVAL;
rec->task_work_off = -EINVAL;
+ rec->thread_wq_off = -EINVAL;
for (i = 0; i < cnt; i++) {
field_type_size = btf_field_type_size(info_arr[i].type);
if (info_arr[i].off + field_type_size > value_size) {
@@ -4188,6 +4191,10 @@ struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type
WARN_ON_ONCE(rec->task_work_off >= 0);
rec->task_work_off = rec->fields[i].offset;
break;
+ case BPF_THREAD_WQ:
+ WARN_ON_ONCE(rec->thread_wq_off >= 0);
+ rec->thread_wq_off = rec->fields[i].offset;
+ break;
case BPF_REFCOUNT:
WARN_ON_ONCE(rec->refcount_off >= 0);
/* Cache offset for faster lookup at runtime */
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 4709a5ad0474..68e7456f0649 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -29,6 +29,8 @@
#include <linux/task_work.h>
#include <linux/irq_work.h>
#include <linux/buildid.h>
+#include <linux/kthread.h>
+#include <linux/jhash.h>
#include "../../lib/kstrtox.h"
@@ -1110,6 +1112,17 @@ static void *map_key_from_value(struct bpf_map *map, void *value, u32 *arr_idx)
return (void *)value - round_up(map->key_size, 8);
}
+static u32 bpf_map_elem_id_from_value(struct bpf_map *map, void *value)
+{
+ u32 arr_idx;
+ void *key;
+
+ key = map_key_from_value(map, value, &arr_idx);
+ if (map->map_type == BPF_MAP_TYPE_ARRAY)
+ return arr_idx;
+ return jhash(key, map->key_size, 0);
+}
+
enum bpf_async_type {
BPF_ASYNC_TYPE_TIMER = 0,
BPF_ASYNC_TYPE_WQ,
@@ -4769,6 +4782,397 @@ __bpf_kfunc int bpf_timer_cancel_async(struct bpf_timer *timer)
}
}
+/*
+ * BPF thread workqueue (kthread_worker based) implementation
+ *
+ * Why bpf_thread_wq does NOT use the bpf_async infrastructure:
+ *
+ * bpf_timer and bpf_wq share a common cleanup path via bpf_async:
+ *
+ * bpf_async_cancel_and_free()
+ * -> bpf_async_schedule_op()
+ * -> irq_work_queue() // schedule from any context
+ * -> bpf_async_process_op() // runs in hardirq context
+ * -> bpf_wq_work() / timer cb // atomic, non-sleepable
+ *
+ * This works for timer and workqueue because their callbacks and
+ * cancellation (hrtimer_cancel / bpf_wq_cancel_and_free) complete
+ * synchronously and need not sleep. The hardirq context is sufficient.
+ *
+ * bpf_thread_wq is different: cancellation and final cleanup may need to
+ * sleep:
+ *
+ * kthread_cancel_work_sync() - waits for the work to finish, can
+ * schedule out if the work is running.
+ * kthread_destroy_worker() - stops the kthread, internally
+ * synchronizes with kthread exit.
+ * cgroup_put() (final put) - may acquire cgroup_mutex and other
+ * sleeping locks during offline.
+ *
+ * None of these can be called from hardirq context (irq_work). Doing so
+ * would trigger might_sleep() warnings or deadlock.
+ *
+ * Therefore bpf_thread_wq implements its own RCU-based cleanup:
+ *
+ * bpf_thread_wq_cancel_and_free() // sleepable (called from
+ * // map free / elem delete)
+ * -> xchg(ctx, NULL) // detach atomically
+ * -> kthread_cancel_work_sync() // ok to sleep
+ * -> bpf_thread_wq_ctx_put()
+ * -> call_rcu_tasks_trace() // wait for BPF callbacks
+ * -> schedule_work() // switch to sleepable context
+ * -> bpf_thread_wq_destroy_work_fn()
+ * -> bpf_prog_put()
+ * -> cgroup_put()
+ * -> kthread_destroy_worker()
+ * -> kfree(ctx)
+ *
+ * The same design choice was made for bpf_task_work, which also avoids
+ * bpf_async because task_work cancellation synchronizes with the task
+ * and may sleep.
+ */
+
+struct bpf_thread_wq_ctx {
+ struct kthread_worker *worker;
+ struct kthread_work work;
+ struct bpf_prog *prog;
+ bpf_callback_t callback_fn;
+ struct bpf_map *map;
+ void *value;
+ struct cgroup *cgrp;
+ refcount_t refcnt;
+ struct rcu_head rcu;
+ struct work_struct destroy_work;
+};
+
+/* Kernel-internal representation that fits in struct bpf_thread_wq */
+struct bpf_thread_wq_kern {
+ struct bpf_thread_wq_ctx *ctx;
+} __aligned(8);
+
+static void bpf_thread_wq_destroy_work_fn(struct work_struct *work)
+{
+ struct bpf_thread_wq_ctx *ctx = container_of(work,
+ struct bpf_thread_wq_ctx,
+ destroy_work);
+
+ if (ctx->prog)
+ bpf_prog_put(ctx->prog);
+ if (ctx->cgrp)
+ cgroup_put(ctx->cgrp);
+ if (ctx->worker)
+ kthread_destroy_worker(ctx->worker);
+ kfree(ctx);
+}
+
+static void bpf_thread_wq_ctx_free_rcu(struct rcu_head *rcu)
+{
+ struct bpf_thread_wq_ctx *ctx = container_of(rcu,
+ struct bpf_thread_wq_ctx,
+ rcu);
+
+ INIT_WORK(&ctx->destroy_work, bpf_thread_wq_destroy_work_fn);
+ schedule_work(&ctx->destroy_work);
+}
+
+static void bpf_thread_wq_ctx_put(struct bpf_thread_wq_ctx *ctx)
+{
+ if (!refcount_dec_and_test(&ctx->refcnt))
+ return;
+ call_rcu_tasks_trace(&ctx->rcu, bpf_thread_wq_ctx_free_rcu);
+}
+
+static void bpf_thread_wq_work_fn(struct kthread_work *work)
+{
+ struct bpf_thread_wq_ctx *ctx = container_of(work,
+ struct bpf_thread_wq_ctx,
+ work);
+ bpf_callback_t callback_fn;
+ void *value = ctx->value;
+ struct bpf_map *map = ctx->map;
+ void *key;
+ u32 idx;
+
+ BTF_TYPE_EMIT(struct bpf_thread_wq);
+
+ callback_fn = READ_ONCE(ctx->callback_fn);
+ if (!callback_fn)
+ goto out;
+ key = map_key_from_value(map, value, &idx);
+
+ rcu_read_lock_trace();
+ migrate_disable();
+
+ callback_fn = READ_ONCE(ctx->callback_fn);
+ if (callback_fn)
+ callback_fn((u64)(long)map, (u64)(long)key, (u64)(long)value,
+ 0, 0);
+
+ migrate_enable();
+ rcu_read_unlock_trace();
+
+out:
+ bpf_thread_wq_ctx_put(ctx);
+}
+
+/*
+ * bpf_thread_wq_init() creates a kthread worker and may attach it to a cgroup.
+ * The helpers used here can sleep and acquire several locks through kthread
+ * creation/destruction, cgroup lookup and cgroup kthread attachment. Keep this
+ * kfunc available only to BPF_PROG_TYPE_SYSCALL programs so it is not invoked
+ * from BPF program contexts that already hold locks which could deadlock with
+ * those paths.
+ */
+__bpf_kfunc int bpf_thread_wq_init(struct bpf_thread_wq *twq, void *p__map,
+ u64 cgroup_id, unsigned int flags)
+{
+ struct bpf_thread_wq_kern *twk = (struct bpf_thread_wq_kern *)twq;
+ struct bpf_map *map = p__map;
+ struct bpf_thread_wq_ctx *ctx, *old_ctx;
+ struct kthread_worker *worker;
+ struct cgroup *cgrp = NULL;
+ void *value;
+ u32 elem_id;
+ int err;
+
+ BUILD_BUG_ON(sizeof(struct bpf_thread_wq_kern)
+ > sizeof(struct bpf_thread_wq));
+ BUILD_BUG_ON(__alignof__(struct bpf_thread_wq_kern)
+ != __alignof__(struct bpf_thread_wq));
+
+ if (flags)
+ return -EINVAL;
+
+ old_ctx = READ_ONCE(twk->ctx);
+ if (old_ctx)
+ return -EBUSY;
+
+ value = (void *)twq - map->record->thread_wq_off;
+ elem_id = bpf_map_elem_id_from_value(map, value);
+ worker = kthread_run_worker(0, "bpf_twq/%d/%x", map->id, elem_id);
+ if (IS_ERR(worker))
+ return PTR_ERR(worker);
+
+ /* Setup ctx. */
+ ctx = bpf_map_kmalloc_nolock(map, sizeof(*ctx), GFP_KERNEL,
+ map->numa_node);
+ if (!ctx) {
+ err = -ENOMEM;
+ goto destroy_worker;
+ }
+ memset(ctx, 0, sizeof(*ctx));
+ ctx->worker = worker;
+ ctx->map = map;
+ ctx->value = value;
+ refcount_set(&ctx->refcnt, 1);
+ kthread_init_work(&ctx->work, bpf_thread_wq_work_fn);
+
+ if (cgroup_id) {
+#ifdef CONFIG_CGROUPS
+ cgrp = cgroup_get_from_id(cgroup_id);
+ if (IS_ERR(cgrp)) {
+ err = PTR_ERR(cgrp);
+ goto kfree_ctx;
+ }
+ ctx->cgrp = cgrp;
+
+ /*
+ * kthread_run_worker() wakes the kthread, but it may not have
+ * executed cgroup_kthread_ready() yet, which clears
+ * no_cgroup_migration.
+ * Do a queue work and flush to wait the kthread run.
+ */
+ refcount_inc(&ctx->refcnt);
+ if (!kthread_queue_work(ctx->worker, &ctx->work)) {
+ refcount_dec(&ctx->refcnt);
+ err = -EBUSY;
+ goto cgroup_put;
+ }
+ kthread_flush_work(&ctx->work);
+
+ if (worker->task->no_cgroup_migration) {
+ err = -EAGAIN;
+ goto cgroup_put;
+ }
+
+ err = cgroup_kthread_attach(cgrp, worker->task);
+ if (err)
+ goto cgroup_put;
+#else
+ err = -EOPNOTSUPP;
+ goto kfree_ctx;
+#endif
+ }
+
+ old_ctx = cmpxchg(&twk->ctx, NULL, ctx);
+ if (old_ctx) {
+ err = -EBUSY;
+ goto cgroup_put;
+ }
+
+ /*
+ * Paired with the map destruction path. Ensures that ctx is globally
+ * visible before we check map->usercnt.
+ * If usercnt has dropped to zero, the destruction path will either see
+ * the ctx (and cancel it) or we see usercnt == 0 here and cancel
+ * ourselves.
+ * Without this barrier, a CPU could reorder the load of usercnt before
+ * the cmpxchg store becomes visible, breaking the mutual exclusion
+ * guarantee.
+ */
+ smp_mb();
+
+ if (!atomic64_read(&map->usercnt)) {
+ bpf_thread_wq_cancel_and_free(twq);
+ return -EPERM;
+ }
+
+ return 0;
+
+cgroup_put:
+#ifdef CONFIG_CGROUPS
+ if (cgrp)
+ cgroup_put(cgrp);
+#endif
+kfree_ctx:
+ /*
+ * Not use bpf_thread_wq_ctx_put because ctx has not yet entered
+ * the running state.
+ */
+ kfree(ctx);
+destroy_worker:
+ kthread_destroy_worker(worker);
+ return err;
+}
+
+__bpf_kfunc int bpf_thread_wq_set_callback(struct bpf_thread_wq *twq,
+ int (callback_fn)(void *map,
+ int *key,
+ void *value),
+ unsigned int flags,
+ struct bpf_prog_aux *aux)
+{
+ struct bpf_thread_wq_kern *twk = (struct bpf_thread_wq_kern *)twq;
+ struct bpf_thread_wq_ctx *ctx;
+ struct bpf_prog *prog;
+
+ if (flags)
+ return -EINVAL;
+
+ ctx = READ_ONCE(twk->ctx);
+ if (!ctx)
+ return -EINVAL;
+
+ prog = bpf_prog_inc_not_zero(aux->prog);
+ if (IS_ERR(prog))
+ return PTR_ERR(prog);
+
+ /*
+ * Allow set_callback only once to prevent UAF: a concurrent
+ * set_callback could bpf_prog_put() the prog while the worker
+ * kthread is still executing its callback.
+ */
+ if (cmpxchg(&ctx->prog, NULL, prog) != NULL) {
+ bpf_prog_put(prog);
+ return -EBUSY;
+ }
+ /*
+ * Safe to set callback_fn after prog: bpf_thread_wq_start() and
+ * bpf_thread_wq_work_fn() both check callback_fn with READ_ONCE()
+ * and bail out if it is still NULL.
+ */
+ WRITE_ONCE(ctx->callback_fn, (void *)callback_fn);
+
+ return 0;
+}
+
+__bpf_kfunc int
+bpf_thread_wq_start(struct bpf_thread_wq *twq, unsigned int flags)
+{
+ struct bpf_thread_wq_kern *twk = (struct bpf_thread_wq_kern *)twq;
+ struct bpf_thread_wq_ctx *ctx;
+ int err;
+
+ if (flags)
+ return -EINVAL;
+
+ rcu_read_lock_trace();
+
+ err = 0;
+
+ ctx = READ_ONCE(twk->ctx);
+ if (!ctx || !READ_ONCE(ctx->callback_fn)) {
+ err = -EINVAL;
+ goto unlock;
+ }
+
+ if (!refcount_inc_not_zero(&ctx->refcnt))
+ err = -ENOENT;
+
+unlock:
+ rcu_read_unlock_trace();
+ if (err)
+ return err;
+
+ if (!kthread_queue_work(ctx->worker, &ctx->work)) {
+ bpf_thread_wq_ctx_put(ctx);
+ return -EBUSY;
+ }
+
+ return 0;
+}
+
+void bpf_thread_wq_cancel_and_free(void *val)
+{
+ struct bpf_thread_wq_kern *twk = val;
+ struct bpf_thread_wq_ctx *ctx;
+
+ ctx = xchg(&twk->ctx, NULL);
+ if (!ctx)
+ return;
+
+ might_sleep();
+
+ /*
+ * Prevent future callbacks from running and wait for any
+ * in-progress execution to finish.
+ */
+ WRITE_ONCE(ctx->callback_fn, NULL);
+ /*
+ * kthread_cancel_work_sync() returns true when it dequeues a pending
+ * work item from the work_list without executing it. Each successful
+ * bpf_thread_wq_start() call increments ctx->refcnt and relies on the
+ * subsequent bpf_thread_wq_work_fn() execution to release that
+ * reference via bpf_thread_wq_ctx_put(). If the work was pending and
+ * got cancelled here, work_fn will never run for that queued instance,
+ * so we must drop the reference ourselves to avoid a permanent refcount
+ * leak.
+ *
+ * This covers two scenarios uniformly:
+ * 1. The work is purely pending (not currently executing) - e.g. a
+ * normal bpf_thread_wq_start() call queued it but the worker
+ * thread hasn't picked it up yet.
+ * 2. The work is currently in-flight AND was self-rescheduled from
+ * within the callback - kthread_cancel_work_sync() dequeues the
+ * re-queued pending node and then waits for the in-flight
+ * execution to complete.
+ * In both cases the return value is true, indicating one orphaned
+ * reference that needs to be released here.
+ */
+ if (kthread_cancel_work_sync(&ctx->work))
+ bpf_thread_wq_ctx_put(ctx);
+
+ /*
+ * Drop our own reference. If the work was still in-flight above,
+ * the refcount won't hit zero here - it will reach zero when the
+ * work path calls bpf_thread_wq_ctx_put() upon completion. Either
+ * way, final cleanup (worker destruction, prog put, cgroup put,
+ * kfree) happens exclusively in the RCU callback to keep the
+ * teardown path single-threaded.
+ */
+ bpf_thread_wq_ctx_put(ctx);
+}
+
__bpf_kfunc_end_defs();
static void bpf_task_work_cancel_scheduled(struct irq_work *irq_work)
@@ -4915,6 +5319,8 @@ BTF_ID_FLAGS(func, bpf_modify_return_test_tp)
BTF_ID_FLAGS(func, bpf_wq_init)
BTF_ID_FLAGS(func, bpf_wq_set_callback, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_wq_start)
+BTF_ID_FLAGS(func, bpf_thread_wq_set_callback, KF_IMPLICIT_ARGS)
+BTF_ID_FLAGS(func, bpf_thread_wq_start)
BTF_ID_FLAGS(func, bpf_preempt_disable)
BTF_ID_FLAGS(func, bpf_preempt_enable)
BTF_ID_FLAGS(func, bpf_iter_bits_new, KF_ITER_NEW)
@@ -4976,6 +5382,15 @@ static const struct btf_kfunc_id_set common_kfunc_set = {
.set = &common_btf_ids,
};
+BTF_KFUNCS_START(syscall_btf_ids)
+BTF_ID_FLAGS(func, bpf_thread_wq_init, KF_SLEEPABLE)
+BTF_KFUNCS_END(syscall_btf_ids)
+
+static const struct btf_kfunc_id_set syscall_kfunc_set = {
+ .owner = THIS_MODULE,
+ .set = &syscall_btf_ids,
+};
+
static int __init kfunc_init(void)
{
int ret;
@@ -4998,6 +5413,7 @@ static int __init kfunc_init(void)
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_CGROUP_SKB, &generic_kfunc_set);
+ ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &syscall_kfunc_set);
ret = ret ?: register_btf_id_dtor_kfuncs(generic_dtors,
ARRAY_SIZE(generic_dtors),
THIS_MODULE);
@@ -5035,4 +5451,6 @@ void bpf_map_free_internal_structs(struct bpf_map *map, void *val)
bpf_obj_free_workqueue(map->record, val);
if (btf_record_has_field(map->record, BPF_TASK_WORK))
bpf_obj_free_task_work(map->record, val);
+ if (btf_record_has_field(map->record, BPF_THREAD_WQ))
+ bpf_obj_free_thread_wq(map->record, val);
}
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 8d111da88655..dea14823bacd 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -688,6 +688,7 @@ void btf_record_free(struct btf_record *rec)
case BPF_REFCOUNT:
case BPF_WORKQUEUE:
case BPF_TASK_WORK:
+ case BPF_THREAD_WQ:
/* Nothing to release */
break;
default:
@@ -742,6 +743,7 @@ struct btf_record *btf_record_dup(const struct btf_record *rec)
case BPF_REFCOUNT:
case BPF_WORKQUEUE:
case BPF_TASK_WORK:
+ case BPF_THREAD_WQ:
/* Nothing to acquire */
break;
default:
@@ -807,6 +809,13 @@ void bpf_obj_free_task_work(const struct btf_record *rec, void *obj)
bpf_task_work_cancel_and_free(obj + rec->task_work_off);
}
+void bpf_obj_free_thread_wq(const struct btf_record *rec, void *obj)
+{
+ if (WARN_ON_ONCE(!btf_record_has_field(rec, BPF_THREAD_WQ)))
+ return;
+ bpf_thread_wq_cancel_and_free(obj + rec->thread_wq_off);
+}
+
void bpf_obj_cancel_fields(struct bpf_map *map, void *obj)
{
bpf_map_free_internal_structs(map, obj);
@@ -839,6 +848,9 @@ void bpf_obj_free_fields(const struct btf_record *rec, void *obj)
case BPF_TASK_WORK:
bpf_task_work_cancel_and_free(field_ptr);
break;
+ case BPF_THREAD_WQ:
+ bpf_thread_wq_cancel_and_free(field_ptr);
+ break;
case BPF_KPTR_UNREF:
WRITE_ONCE(*(u64 *)field_ptr, 0);
break;
@@ -1265,7 +1277,7 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token,
map->record = btf_parse_fields(btf, value_type,
BPF_SPIN_LOCK | BPF_RES_SPIN_LOCK | BPF_TIMER | BPF_KPTR | BPF_LIST_HEAD |
BPF_RB_ROOT | BPF_REFCOUNT | BPF_WORKQUEUE | BPF_UPTR |
- BPF_TASK_WORK,
+ BPF_TASK_WORK | BPF_THREAD_WQ,
map->value_size);
if (!IS_ERR_OR_NULL(map->record)) {
int i;
@@ -1299,6 +1311,7 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token,
case BPF_TIMER:
case BPF_WORKQUEUE:
case BPF_TASK_WORK:
+ case BPF_THREAD_WQ:
if (map->map_type != BPF_MAP_TYPE_HASH &&
map->map_type != BPF_MAP_TYPE_RHASH &&
map->map_type != BPF_MAP_TYPE_LRU_HASH &&
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 7439afdc851a..4d713bb767e0 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -477,6 +477,7 @@ static bool is_async_callback_calling_kfunc(u32 btf_id);
static bool is_callback_calling_kfunc(u32 btf_id);
static bool is_bpf_wq_set_callback_kfunc(u32 btf_id);
+static bool is_bpf_thread_wq_set_callback_kfunc(u32 btf_id);
static bool is_task_work_add_kfunc(u32 func_id);
static bool is_sync_callback_calling_function(enum bpf_func_id func_id)
@@ -516,9 +517,11 @@ static bool is_async_cb_sleepable(struct bpf_verifier_env *env, struct bpf_insn
if (bpf_helper_call(insn) && insn->imm == BPF_FUNC_timer_set_callback)
return false;
- /* bpf_wq and bpf_task_work callbacks are always sleepable. */
+ /* bpf_wq, bpf_thread_wq and bpf_task_work callbacks are always sleepable. */
if (bpf_pseudo_kfunc_call(insn) && insn->off == 0 &&
- (is_bpf_wq_set_callback_kfunc(insn->imm) || is_task_work_add_kfunc(insn->imm)))
+ (is_bpf_wq_set_callback_kfunc(insn->imm) ||
+ is_bpf_thread_wq_set_callback_kfunc(insn->imm) ||
+ is_task_work_add_kfunc(insn->imm)))
return true;
verifier_bug(env, "unhandled async callback in is_async_cb_sleepable");
@@ -1871,7 +1874,10 @@ static void refine_map_lookup_value(struct bpf_reg_state *reg)
* as UID of the inner map.
*/
if (btf_record_has_field(map->inner_map_meta->record,
- BPF_TIMER | BPF_WORKQUEUE | BPF_TASK_WORK))
+ BPF_TIMER |
+ BPF_WORKQUEUE |
+ BPF_TASK_WORK |
+ BPF_THREAD_WQ))
reg->map_uid = reg->id;
} else if (map->map_type == BPF_MAP_TYPE_XSKMAP) {
reg->type = PTR_TO_XDP_SOCK | maybe_null;
@@ -7243,6 +7249,9 @@ static int check_map_field_pointer(struct bpf_verifier_env *env, struct bpf_reg_
case BPF_WORKQUEUE:
field_off = map->record->wq_off;
break;
+ case BPF_THREAD_WQ:
+ field_off = map->record->thread_wq_off;
+ break;
default:
verifier_bug(env, "unsupported BTF field type: %s\n", struct_name);
return -EINVAL;
@@ -10940,6 +10949,7 @@ enum {
KF_ARG_WORKQUEUE_ID,
KF_ARG_RES_SPIN_LOCK_ID,
KF_ARG_TASK_WORK_ID,
+ KF_ARG_THREAD_WQ_ID,
KF_ARG_PROG_AUX_ID,
KF_ARG_TIMER_ID
};
@@ -10953,6 +10963,7 @@ BTF_ID(struct, bpf_rb_node)
BTF_ID(struct, bpf_wq)
BTF_ID(struct, bpf_res_spin_lock)
BTF_ID(struct, bpf_task_work)
+BTF_ID(struct, bpf_thread_wq)
BTF_ID(struct, bpf_prog_aux)
BTF_ID(struct, bpf_timer)
@@ -11013,6 +11024,11 @@ static bool is_kfunc_arg_task_work(const struct btf *btf, const struct btf_param
return __is_kfunc_ptr_arg_type(btf, arg, KF_ARG_TASK_WORK_ID);
}
+static bool is_kfunc_arg_thread_wq(const struct btf *btf, const struct btf_param *arg)
+{
+ return __is_kfunc_ptr_arg_type(btf, arg, KF_ARG_THREAD_WQ_ID);
+}
+
static bool is_kfunc_arg_res_spin_lock(const struct btf *btf, const struct btf_param *arg)
{
return __is_kfunc_ptr_arg_type(btf, arg, KF_ARG_RES_SPIN_LOCK_ID);
@@ -11132,6 +11148,7 @@ enum kfunc_ptr_arg_type {
KF_ARG_PTR_TO_IRQ_FLAG,
KF_ARG_PTR_TO_RES_SPIN_LOCK,
KF_ARG_PTR_TO_TASK_WORK,
+ KF_ARG_PTR_TO_THREAD_WQ,
};
enum special_kfunc_type {
@@ -11178,6 +11195,7 @@ enum special_kfunc_type {
KF_bpf_percpu_obj_drop,
KF_bpf_throw,
KF_bpf_wq_set_callback,
+ KF_bpf_thread_wq_set_callback,
KF_bpf_preempt_disable,
KF_bpf_preempt_enable,
KF_bpf_iter_css_task_new,
@@ -11258,6 +11276,7 @@ BTF_ID(func, bpf_percpu_obj_drop_impl)
BTF_ID(func, bpf_percpu_obj_drop)
BTF_ID(func, bpf_throw)
BTF_ID(func, bpf_wq_set_callback)
+BTF_ID(func, bpf_thread_wq_set_callback)
BTF_ID(func, bpf_preempt_disable)
BTF_ID(func, bpf_preempt_enable)
#ifdef CONFIG_CGROUPS
@@ -11460,6 +11479,8 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
arg_type = KF_ARG_PTR_TO_TIMER;
else if (is_kfunc_arg_task_work(meta->btf, &args[arg]))
arg_type = KF_ARG_PTR_TO_TASK_WORK;
+ else if (is_kfunc_arg_thread_wq(meta->btf, &args[arg]))
+ arg_type = KF_ARG_PTR_TO_THREAD_WQ;
else if (is_kfunc_arg_irq_flag(meta->btf, &args[arg]))
arg_type = KF_ARG_PTR_TO_IRQ_FLAG;
else if (is_kfunc_arg_res_spin_lock(meta->btf, &args[arg]))
@@ -11852,6 +11873,7 @@ static bool is_sync_callback_calling_kfunc(u32 btf_id)
static bool is_async_callback_calling_kfunc(u32 btf_id)
{
return is_bpf_wq_set_callback_kfunc(btf_id) ||
+ is_bpf_thread_wq_set_callback_kfunc(btf_id) ||
is_task_work_add_kfunc(btf_id);
}
@@ -11866,6 +11888,11 @@ static bool is_bpf_wq_set_callback_kfunc(u32 btf_id)
return btf_id == special_kfunc_list[KF_bpf_wq_set_callback];
}
+static bool is_bpf_thread_wq_set_callback_kfunc(u32 btf_id)
+{
+ return btf_id == special_kfunc_list[KF_bpf_thread_wq_set_callback];
+}
+
static bool is_callback_calling_kfunc(u32 btf_id)
{
return is_sync_callback_calling_kfunc(btf_id) ||
@@ -12211,6 +12238,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me
case KF_ARG_PTR_TO_WORKQUEUE:
case KF_ARG_PTR_TO_TIMER:
case KF_ARG_PTR_TO_TASK_WORK:
+ case KF_ARG_PTR_TO_THREAD_WQ:
case KF_ARG_PTR_TO_IRQ_FLAG:
case KF_ARG_PTR_TO_RES_SPIN_LOCK:
break;
@@ -13158,6 +13186,16 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
}
}
+ if (is_bpf_thread_wq_set_callback_kfunc(meta.func_id)) {
+ err = push_callback_call(env, insn, insn_idx, meta.subprogno,
+ set_timer_callback_state);
+ if (err) {
+ verbose(env, "kfunc %s#%d failed callback verification\n",
+ func_name, meta.func_id);
+ return err;
+ }
+ }
+
if (is_task_work_add_kfunc(meta.func_id)) {
err = push_callback_call(env, insn, insn_idx, meta.subprogno,
set_task_work_schedule_callback_state);
diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c
index 38f8d9df8fbc..164e069c7576 100644
--- a/kernel/cgroup/cgroup.c
+++ b/kernel/cgroup/cgroup.c
@@ -3042,6 +3042,19 @@ int cgroup_attach_task(struct cgroup *dst_cgrp, struct task_struct *leader,
return ret;
}
+int cgroup_kthread_attach(struct cgroup *cgrp, struct task_struct *task)
+{
+ int ret;
+
+ cgroup_lock();
+ cgroup_attach_lock(CGRP_ATTACH_LOCK_GLOBAL, NULL);
+ ret = cgroup_attach_task(cgrp, task, false);
+ cgroup_attach_unlock(CGRP_ATTACH_LOCK_GLOBAL, NULL);
+ cgroup_unlock();
+
+ return ret;
+}
+
struct task_struct *cgroup_procs_write_start(char *buf, bool threadgroup,
enum cgroup_attach_lock_mode *lock_mode)
{
diff --git a/tools/testing/selftests/bpf/bpf_experimental.h b/tools/testing/selftests/bpf/bpf_experimental.h
index ff37ae5a113d..ea905ea5603c 100644
--- a/tools/testing/selftests/bpf/bpf_experimental.h
+++ b/tools/testing/selftests/bpf/bpf_experimental.h
@@ -351,6 +351,13 @@ extern void bpf_iter_css_destroy(struct bpf_iter_css *it) __weak __ksym;
extern int bpf_wq_init(struct bpf_wq *wq, void *p__map, unsigned int flags) __weak __ksym;
extern int bpf_wq_start(struct bpf_wq *wq, unsigned int flags) __weak __ksym;
+struct bpf_thread_wq;
+extern int bpf_thread_wq_init(struct bpf_thread_wq *twq, void *p__map,
+ __u64 cgroup_id,
+ unsigned int flags) __weak __ksym;
+extern int bpf_thread_wq_start(struct bpf_thread_wq *twq,
+ unsigned int flags) __weak __ksym;
+
struct bpf_iter_kmem_cache;
extern int bpf_iter_kmem_cache_new(struct bpf_iter_kmem_cache *it) __weak __ksym;
extern struct kmem_cache *bpf_iter_kmem_cache_next(struct bpf_iter_kmem_cache *it) __weak __ksym;
--
2.53.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH bpf-next 3/4] selftests/bpf: add thread_wq cgroup test
2026-08-07 7:01 [PATCH bpf-next 0/4] bpf: BPF-driven proactive memcg reclaim Hui Zhu
2026-08-07 7:01 ` [PATCH bpf-next 1/4] mm/bpf: Add bpf_try_to_free_mem_cgroup_pages kfunc Hui Zhu
2026-08-07 7:01 ` [PATCH bpf-next 2/4] bpf: add bpf_thread_wq kthread-backed workqueue with cgroup placement Hui Zhu
@ 2026-08-07 7:04 ` Hui Zhu
2026-08-07 7:04 ` [PATCH bpf-next 4/4] selftests/bpf: add memcg async reclaim test for bpf_wq/bpf_thread_wq Hui Zhu
3 siblings, 0 replies; 5+ messages in thread
From: Hui Zhu @ 2026-08-07 7:04 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, John Fastabend,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Song Liu, Yonghong Song, Jiri Olsa,
Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, JP Kobryn, Andrew Morton, Shuah Khan, davem,
Jakub Kicinski, Jesper Dangaard Brouer, Stanislav Fomichev,
KP Singh, Tao Chen, Mykyta Yatsenko, Leon Hwang, Anton Protopopov,
Amery Hung, Tobias Klauser, Eyal Birger, Rong Tao, Hao Luo,
Peter Zijlstra, Miguel Ojeda, Nathan Chancellor, Kees Cook,
Tejun Heo, Jeff Xu, mkoutny, Jan Hendrik Farr, Christian Brauner,
Randy Dunlap, Brian Gerst, Masahiro Yamada, Willem de Bruijn,
Jason Xing, Paul Chaignon, Lance Yang, Jiayuan Chen,
Emil Tsalapatis, Ihor Solodrai, Barry Song, Geliang Tang,
linux-kernel, bpf, cgroups, linux-mm, netdev, linux-kselftest
Cc: Hui Zhu
From: Hui Zhu <zhuhui@kylinos.cn>
Add test cases for bpf_thread_wq with cgroup attachment:
- Test thread_wq execution in a specified cgroup and verify
callback runs in the target cgroup
- Test thread_wq execution without cgroup attachment and verify
callback runs in a different cgroup
This validates that bpf_thread_wq properly attaches to and
executes callbacks within the specified cgroup context.
Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
---
.../bpf/prog_tests/thread_wq_cgroup.c | 87 +++++++++++++++++++
.../selftests/bpf/progs/thread_wq_cgroup.c | 56 ++++++++++++
2 files changed, 143 insertions(+)
create mode 100644 tools/testing/selftests/bpf/prog_tests/thread_wq_cgroup.c
create mode 100644 tools/testing/selftests/bpf/progs/thread_wq_cgroup.c
diff --git a/tools/testing/selftests/bpf/prog_tests/thread_wq_cgroup.c b/tools/testing/selftests/bpf/prog_tests/thread_wq_cgroup.c
new file mode 100644
index 000000000000..7537b03f17e2
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/thread_wq_cgroup.c
@@ -0,0 +1,87 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include <unistd.h>
+#include "cgroup_helpers.h"
+#include "thread_wq_cgroup.skel.h"
+
+#define TEST_CGROUP "/thread_wq_test"
+#define WAIT_TIMEOUT_SECS 30
+
+void test_thread_wq_cgroup(void)
+{
+ struct thread_wq_cgroup *skel = NULL;
+ int err, prog_fd, cg_fd = -1;
+ unsigned long long cg_id;
+ int waited_secs;
+
+ LIBBPF_OPTS(bpf_test_run_opts, topts);
+
+ err = setup_cgroup_environment();
+ if (!ASSERT_OK(err, "setup_cgroup_environment"))
+ return;
+ cg_fd = create_and_get_cgroup(TEST_CGROUP);
+ if (!ASSERT_GE(cg_fd, 0, "create_and_get_cgroup"))
+ goto cleanup;
+ cg_id = get_cgroup_id(TEST_CGROUP);
+ if (!ASSERT_GT(cg_id, 0ULL, "get_cgroup_id"))
+ goto cleanup;
+
+ skel = thread_wq_cgroup__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "open_and_load"))
+ goto cleanup;
+
+ prog_fd = bpf_program__fd(skel->progs.start_thread_wq);
+
+ /* Run bpf_thread_wq in the specified cgroup. */
+ skel->bss->test_key = 0;
+ skel->bss->target_cgroup_id = cg_id;
+ skel->bss->callback_cgroup_id = 0;
+ skel->bss->twq_done = 0;
+ if (!ASSERT_OK(bpf_prog_test_run_opts(prog_fd, &topts),
+ "bpf_prog_test_run_opts in cgroup"))
+ goto cleanup;
+ if (!ASSERT_OK(topts.retval, "retval in cgroup"))
+ goto cleanup;
+ for (waited_secs = 0; waited_secs < WAIT_TIMEOUT_SECS; waited_secs++) {
+ if (skel->bss->twq_done)
+ break;
+ sleep(1);
+ }
+ if (!ASSERT_TRUE(skel->bss->twq_done, "twq_done in cgroup"))
+ goto cleanup;
+ if (!ASSERT_EQ(skel->bss->callback_cgroup_id, cg_id,
+ "callback_cgroup_id in cgroup"))
+ goto cleanup;
+
+ /* Run bpf_thread_wq without cgroup attachment (cgroup_id = 0). */
+ LIBBPF_OPTS_RESET(topts);
+ skel->bss->test_key = 1;
+ skel->bss->target_cgroup_id = 0;
+ skel->bss->callback_cgroup_id = 0;
+ skel->bss->twq_done = 0;
+ if (!ASSERT_OK(bpf_prog_test_run_opts(prog_fd, &topts),
+ "bpf_prog_test_run_opts without cgroup"))
+ goto cleanup;
+ if (!ASSERT_OK(topts.retval, "retval without cgroup"))
+ goto cleanup;
+ for (waited_secs = 0; waited_secs < WAIT_TIMEOUT_SECS; waited_secs++) {
+ if (skel->bss->twq_done)
+ break;
+ sleep(1);
+ }
+ if (!ASSERT_TRUE(skel->bss->twq_done, "twq_done without cgroup"))
+ goto cleanup;
+ if (!ASSERT_NEQ(skel->bss->callback_cgroup_id, cg_id,
+ "callback_cgroup_id without cgroup"))
+ goto cleanup;
+
+cleanup:
+ if (skel) {
+ thread_wq_cgroup__destroy(skel);
+ /* Wait thread_wq kthread quit. */
+ sleep(2);
+ }
+ if (cg_fd >= 0)
+ close(cg_fd);
+ cleanup_cgroup_environment();
+}
diff --git a/tools/testing/selftests/bpf/progs/thread_wq_cgroup.c b/tools/testing/selftests/bpf/progs/thread_wq_cgroup.c
new file mode 100644
index 000000000000..c70a37f55397
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/thread_wq_cgroup.c
@@ -0,0 +1,56 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 */
+
+#include "bpf_experimental.h"
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+char _license[] SEC("license") = "GPL";
+
+struct elem {
+ struct bpf_thread_wq twq;
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __uint(max_entries, 2);
+ __type(key, int);
+ __type(value, struct elem);
+} map_arr SEC(".maps");
+
+__u64 target_cgroup_id;
+__u64 callback_cgroup_id;
+int twq_done;
+int test_key;
+
+static int twq_callback(void *map, int *key, void *value)
+{
+ callback_cgroup_id = bpf_get_current_cgroup_id();
+ twq_done = 1;
+ return 0;
+}
+
+SEC("syscall")
+int start_thread_wq(void *ctx)
+{
+ struct elem *val;
+ int key = test_key;
+ int ret;
+
+ val = bpf_map_lookup_elem(&map_arr, &key);
+ if (!val)
+ return -1;
+
+ ret = bpf_thread_wq_init(&val->twq, &map_arr, target_cgroup_id, 0);
+ if (ret)
+ goto out;
+
+ ret = bpf_thread_wq_set_callback(&val->twq, twq_callback, 0);
+ if (ret)
+ goto out;
+
+ ret = bpf_thread_wq_start(&val->twq, 0);
+
+out:
+ return ret;
+}
--
2.53.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH bpf-next 4/4] selftests/bpf: add memcg async reclaim test for bpf_wq/bpf_thread_wq
2026-08-07 7:01 [PATCH bpf-next 0/4] bpf: BPF-driven proactive memcg reclaim Hui Zhu
` (2 preceding siblings ...)
2026-08-07 7:04 ` [PATCH bpf-next 3/4] selftests/bpf: add thread_wq cgroup test Hui Zhu
@ 2026-08-07 7:04 ` Hui Zhu
3 siblings, 0 replies; 5+ messages in thread
From: Hui Zhu @ 2026-08-07 7:04 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, John Fastabend,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Song Liu, Yonghong Song, Jiri Olsa,
Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, JP Kobryn, Andrew Morton, Shuah Khan, davem,
Jakub Kicinski, Jesper Dangaard Brouer, Stanislav Fomichev,
KP Singh, Tao Chen, Mykyta Yatsenko, Leon Hwang, Anton Protopopov,
Amery Hung, Tobias Klauser, Eyal Birger, Rong Tao, Hao Luo,
Peter Zijlstra, Miguel Ojeda, Nathan Chancellor, Kees Cook,
Tejun Heo, Jeff Xu, mkoutny, Jan Hendrik Farr, Christian Brauner,
Randy Dunlap, Brian Gerst, Masahiro Yamada, Willem de Bruijn,
Jason Xing, Paul Chaignon, Lance Yang, Jiayuan Chen,
Emil Tsalapatis, Ihor Solodrai, Barry Song, Geliang Tang,
linux-kernel, bpf, cgroups, linux-mm, netdev, linux-kselftest
Cc: Hui Zhu
From: Hui Zhu <zhuhui@kylinos.cn>
Add memcg_async_reclaim selftest that verifies BPF-driven async
proactive reclaim can mitigate refault-induced slowdown under memory
pressure.
The test creates a parent cgroup with a fixed memory.max, and two
child cgroups (high/low) under it. Both children concurrently write
and repeatedly read-fault a file larger than the shared limit. A BPF
program monitors the "high" cgroup's WORKINGSET_REFAULT_FILE stat via
a periodic timer, and when it detects refault growth beyond a
threshold, triggers async reclaim on the "low" cgroup using
bpf_try_to_free_mem_cgroup_pages(), expecting the "high" cgroup's
workload to finish faster than without such reclaim.
Two variants are covered:
- test_memcg_wq_async_reclaim: async work driven by bpf_wq.
- test_memcg_thread_wq_async_reclaim: async work driven by the new
bpf_thread_wq, which pins the reclaim work to the "low" cgroup's
resource context via bpf_thread_wq_init()'s cgroup_id argument.
Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
---
.../bpf/prog_tests/memcg_async_reclaim.c | 479 ++++++++++++++++++
.../selftests/bpf/progs/memcg_async_reclaim.c | 255 ++++++++++
2 files changed, 734 insertions(+)
create mode 100644 tools/testing/selftests/bpf/prog_tests/memcg_async_reclaim.c
create mode 100644 tools/testing/selftests/bpf/progs/memcg_async_reclaim.c
diff --git a/tools/testing/selftests/bpf/prog_tests/memcg_async_reclaim.c b/tools/testing/selftests/bpf/prog_tests/memcg_async_reclaim.c
new file mode 100644
index 000000000000..40a5fe62cb38
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/memcg_async_reclaim.c
@@ -0,0 +1,479 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Memory controller eBPF async reclaim test
+ */
+
+#include <test_progs.h>
+#include <sys/mman.h>
+#include <sys/stat.h>
+#include <sys/time.h>
+#include <sys/vfs.h>
+#include <sys/wait.h>
+#include <fcntl.h>
+#include <linux/magic.h>
+#include <unistd.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "cgroup_helpers.h"
+
+struct bpf_args_s {
+ u64 high_cgroup_id;
+ u64 low_cgroup_id;
+ u64 event_delta_threshold;
+ u64 check_ns;
+};
+
+#include "memcg_async_reclaim.skel.h"
+
+#define FILE_SIZE (32 * 1024 * 1024ul)
+#define BUFFER_SIZE (4096)
+#define CG_LIMIT (32 * 1024 * 1024ul)
+#define READ_TIMES 16
+
+#define CG_DIR "/memcg_async_reclaim"
+#define CG_HIGH_DIR CG_DIR "/high"
+#define CG_LOW_DIR CG_DIR "/low"
+
+#define CHECK_PERIOD_NS (2 * 1000 * 1000ull)
+#define EVENT_DELTA_THRESHOLD 1
+
+/*
+ * Test files must reside on a filesystem that supports page reclaim without
+ * swap (e.g. ext4). If /tmp is on tmpfs, the file pages are shmem-backed
+ * and can only be reclaimed through swap. But the test disables swap
+ * (memory.swap.max=0), making reclaim impossible and causing OOM.
+ *
+ * Pick a directory on a non-tmpfs filesystem: try $TMPDIR first, then /tmp,
+ * and fall back to the current directory if the chosen path is on tmpfs.
+ */
+static int get_test_dir(char *buf, size_t size)
+{
+ static const char * const candidates[] = { "/tmp", "." };
+ const char *tmpdir = getenv("TMPDIR");
+ struct statfs sfs;
+ size_t i;
+
+ if (tmpdir && tmpdir[0] && statfs(tmpdir, &sfs) == 0 &&
+ sfs.f_type != TMPFS_MAGIC) {
+ snprintf(buf, size, "%s", tmpdir);
+ return 0;
+ }
+
+ for (i = 0; i < ARRAY_SIZE(candidates); i++) {
+ if (statfs(candidates[i], &sfs) == 0 &&
+ sfs.f_type != TMPFS_MAGIC) {
+ snprintf(buf, size, "%s", candidates[i]);
+ return 0;
+ }
+ }
+
+ return -1;
+}
+
+static int setup_high_low_cgroups(u64 *high_cgroup_id, u64 *low_cgroup_id)
+{
+ int ret;
+ char limit_buf[20];
+
+ ret = setup_cgroup_environment();
+ if (!ASSERT_OK(ret, "setup_cgroup_environment"))
+ goto cleanup;
+
+ ret = create_and_get_cgroup(CG_DIR);
+ if (!ASSERT_GE(ret, 0, "create_and_get_cgroup " CG_DIR))
+ goto cleanup;
+ close(ret);
+
+ ret = enable_controllers(CG_DIR, "memory");
+ if (!ASSERT_OK(ret, "enable_controllers"))
+ goto cleanup;
+
+ snprintf(limit_buf, sizeof(limit_buf), "%lu", CG_LIMIT);
+ ret = write_cgroup_file(CG_DIR, "memory.max", limit_buf);
+ if (!ASSERT_OK(ret, "write_cgroup_file memory.max"))
+ goto cleanup;
+
+ ret = write_cgroup_file(CG_DIR, "memory.swap.max", "0");
+ if (!ASSERT_OK(ret, "write_cgroup_file memory.swap.max"))
+ goto cleanup;
+
+ ret = create_and_get_cgroup(CG_HIGH_DIR);
+ if (!ASSERT_GE(ret, 0, "create_and_get_cgroup " CG_HIGH_DIR))
+ goto cleanup;
+ close(ret);
+
+ *high_cgroup_id = get_cgroup_id(CG_HIGH_DIR);
+ if (!ASSERT_GT(*high_cgroup_id, 0, "get_cgroup_id"))
+ goto cleanup;
+
+ ret = create_and_get_cgroup(CG_LOW_DIR);
+ if (!ASSERT_GE(ret, 0, "create_and_get_cgroup " CG_LOW_DIR))
+ goto cleanup;
+ close(ret);
+
+ *low_cgroup_id = get_cgroup_id(CG_LOW_DIR);
+ if (!ASSERT_GT(*high_cgroup_id, 0, "get_cgroup_id"))
+ goto cleanup;
+
+ return 0;
+
+cleanup:
+ cleanup_cgroup_environment();
+ return -1;
+}
+
+static int write_file(const char *filename)
+{
+ int ret = -1;
+ size_t written = 0;
+ char *buffer;
+ FILE *fp;
+
+ fp = fopen(filename, "wb");
+ if (!fp)
+ goto out;
+
+ buffer = malloc(BUFFER_SIZE);
+ if (!buffer)
+ goto cleanup_fp;
+
+ memset(buffer, 'A', BUFFER_SIZE);
+
+ while (written < FILE_SIZE) {
+ size_t to_write = FILE_SIZE - written < BUFFER_SIZE ?
+ FILE_SIZE - written : BUFFER_SIZE;
+
+ if (fwrite(buffer, 1, to_write, fp) != to_write)
+ goto cleanup;
+ written += to_write;
+ }
+
+ ret = 0;
+cleanup:
+ free(buffer);
+cleanup_fp:
+ fclose(fp);
+out:
+ return ret;
+}
+
+static int read_file(const char *filename, int iterations)
+{
+ int ret = -1;
+ long page_size = sysconf(_SC_PAGESIZE);
+ char *map;
+ size_t i;
+ int fd;
+ struct stat sb;
+
+ fd = open(filename, O_RDONLY);
+ if (fd == -1)
+ goto out;
+
+ if (fstat(fd, &sb) == -1)
+ goto cleanup_fd;
+
+ if (sb.st_size != FILE_SIZE) {
+ fprintf(stderr, "File size mismatch: expected %lu, got %lu\n",
+ (unsigned long)FILE_SIZE, (unsigned long)sb.st_size);
+ goto cleanup_fd;
+ }
+
+ map = mmap(NULL, FILE_SIZE, PROT_READ, MAP_PRIVATE, fd, 0);
+ if (map == MAP_FAILED)
+ goto cleanup_fd;
+
+ for (int iter = 0; iter < iterations; iter++) {
+ for (i = 0; i < FILE_SIZE; i += page_size) {
+ /* access a byte to trigger page fault */
+ asm volatile("" :: "r"(map[i]) : "memory");
+ }
+ }
+
+ if (munmap(map, FILE_SIZE) == -1)
+ goto cleanup_fd;
+
+ ret = 0;
+
+cleanup_fd:
+ close(fd);
+out:
+ return ret;
+}
+
+static int real_test_child_work(const char *cgroup_path, char *data_filename,
+ char *time_filename, int read_times)
+{
+ struct timeval start, end;
+ double elapsed;
+ FILE *fp;
+
+ if (!ASSERT_OK(join_parent_cgroup(cgroup_path), "join_parent_cgroup"))
+ return -1;
+
+ gettimeofday(&start, NULL);
+
+ if (!ASSERT_OK(write_file(data_filename), "write_file"))
+ return -1;
+
+ if (!ASSERT_OK(read_file(data_filename, read_times), "read_file"))
+ return -1;
+
+ gettimeofday(&end, NULL);
+
+ if (!time_filename)
+ return 0;
+
+ elapsed = (end.tv_sec - start.tv_sec) +
+ (end.tv_usec - start.tv_usec) / 1000000.0;
+ printf("%.6f\n", elapsed);
+
+ fp = fopen(time_filename, "w");
+ if (!ASSERT_OK_PTR(fp, "fopen"))
+ return -1;
+ fprintf(fp, "%.6f", elapsed);
+ fclose(fp);
+
+ return 0;
+}
+
+static int get_time(char *time_filename, double *time)
+{
+ int ret = -1;
+ FILE *fp;
+ char buf[64];
+
+ fp = fopen(time_filename, "r");
+ if (!ASSERT_OK_PTR(fp, "fopen"))
+ goto out;
+
+ if (!ASSERT_OK_PTR(fgets(buf, sizeof(buf), fp), "fgets"))
+ goto cleanup;
+
+ if (sscanf(buf, "%lf", time) != 1) {
+ PRINT_FAIL("sscanf %s", buf);
+ goto cleanup;
+ }
+
+ ret = 0;
+cleanup:
+ fclose(fp);
+out:
+ return ret;
+}
+
+static int
+run_high_low_workload(double *high_elapsed, double *low_elapsed, int read_times)
+{
+ char test_dir[PATH_MAX], high_data_file[PATH_MAX], low_data_file[PATH_MAX];
+ char high_time_file[PATH_MAX], low_time_file[PATH_MAX];
+ int ret, fd;
+ pid_t high_pid, low_pid;
+ int status;
+
+ ret = get_test_dir(test_dir, sizeof(test_dir));
+ if (!ASSERT_OK(ret, "get_test_dir: no non-tmpfs directory found"))
+ return -1;
+
+ fd = snprintf(high_data_file, sizeof(high_data_file),
+ "%s/memcg_async_high_data_XXXXXX", test_dir);
+ if (!ASSERT_LT(fd, sizeof(high_data_file), "high_data_file path"))
+ return -1;
+
+ fd = snprintf(low_data_file, sizeof(low_data_file),
+ "%s/memcg_async_low_data_XXXXXX", test_dir);
+ if (!ASSERT_LT(fd, sizeof(low_data_file), "low_data_file path"))
+ return -1;
+
+ fd = snprintf(high_time_file, sizeof(high_time_file),
+ "%s/memcg_async_high_time_XXXXXX", test_dir);
+ if (!ASSERT_LT(fd, sizeof(high_time_file), "high_time_file path"))
+ return -1;
+
+ fd = snprintf(low_time_file, sizeof(low_time_file),
+ "%s/memcg_async_low_time_XXXXXX", test_dir);
+ if (!ASSERT_LT(fd, sizeof(low_time_file), "low_time_file path"))
+ return -1;
+
+ fd = mkstemp(high_data_file);
+ if (!ASSERT_GE(fd, 0, "mkstemp"))
+ return -1;
+ close(fd);
+
+ fd = mkstemp(low_data_file);
+ if (!ASSERT_GE(fd, 0, "mkstemp"))
+ goto cleanup_high_data;
+ close(fd);
+
+ fd = mkstemp(high_time_file);
+ if (!ASSERT_GE(fd, 0, "mkstemp"))
+ goto cleanup_low_data;
+ close(fd);
+
+ fd = mkstemp(low_time_file);
+ if (!ASSERT_GE(fd, 0, "mkstemp"))
+ goto cleanup_high_time;
+ close(fd);
+
+ low_pid = fork();
+ if (!ASSERT_GE(low_pid, 0, "fork low"))
+ goto cleanup_low_time;
+ if (low_pid == 0)
+ exit(real_test_child_work(CG_LOW_DIR, low_data_file,
+ low_time_file, read_times));
+
+ high_pid = fork();
+ if (!ASSERT_GE(high_pid, 0, "fork high")) {
+ (void)waitpid(low_pid, NULL, 0);
+ goto cleanup_low_time;
+ }
+ if (high_pid == 0)
+ exit(real_test_child_work(CG_HIGH_DIR, high_data_file,
+ high_time_file, read_times));
+
+ ret = waitpid(low_pid, &status, 0);
+ if (!ASSERT_GT(ret, 0, "low waitpid"))
+ goto cleanup_low_time;
+ if (!ASSERT_TRUE(WIFEXITED(status), "low exited"))
+ goto cleanup_low_time;
+ if (!ASSERT_EQ(WEXITSTATUS(status), 0, "low exit status"))
+ goto cleanup_low_time;
+
+ ret = waitpid(high_pid, &status, 0);
+ if (!ASSERT_GT(ret, 0, "high waitpid"))
+ goto cleanup_low_time;
+ if (!ASSERT_TRUE(WIFEXITED(status), "high exited"))
+ goto cleanup_low_time;
+ if (!ASSERT_EQ(WEXITSTATUS(status), 0, "high exit status"))
+ goto cleanup_low_time;
+
+ if (get_time(high_time_file, high_elapsed))
+ goto cleanup_low_time;
+ if (get_time(low_time_file, low_elapsed))
+ goto cleanup_low_time;
+
+ ret = 0;
+
+cleanup_low_time:
+ unlink(low_time_file);
+cleanup_high_time:
+ unlink(high_time_file);
+cleanup_low_data:
+ unlink(low_data_file);
+cleanup_high_data:
+ unlink(high_data_file);
+ return ret;
+}
+
+static int
+setup_bpf(u64 high_cgroup_id, u64 low_cgroup_id,
+ struct memcg_async_reclaim **skel_ptr, bool use_thread_wq)
+{
+ struct memcg_async_reclaim *skel;
+ struct bpf_args_s bpf_args = {
+ .high_cgroup_id = high_cgroup_id,
+ .low_cgroup_id = low_cgroup_id,
+ .event_delta_threshold = EVENT_DELTA_THRESHOLD,
+ .check_ns = CHECK_PERIOD_NS,
+ };
+ LIBBPF_OPTS(bpf_test_run_opts, run_opts,
+ .ctx_in = &bpf_args,
+ .ctx_size_in = sizeof(bpf_args));
+ int prog_init_fd, err;
+
+ skel = memcg_async_reclaim__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "memcg_async_reclaim__open_and_load"))
+ return -1;
+
+ if (use_thread_wq)
+ prog_init_fd = bpf_program__fd(skel->progs.thread_wq_prog_init);
+ else
+ prog_init_fd = bpf_program__fd(skel->progs.wq_prog_init);
+ if (!ASSERT_GE(prog_init_fd, 0, "bpf_program__fd"))
+ goto error_out;
+
+ err = bpf_prog_test_run_opts(prog_init_fd, &run_opts);
+ if (!ASSERT_OK(err, "bpf_prog_test_run_opts"))
+ goto error_out;
+ if (!ASSERT_EQ(run_opts.retval, 0, "prog_init retval"))
+ goto error_out;
+
+ *skel_ptr = skel;
+ return 0;
+
+error_out:
+ memcg_async_reclaim__destroy(skel);
+ return -1;
+}
+
+void test_memcg_wq_async_reclaim(void)
+{
+ u64 high_cgroup_id, low_cgroup_id;
+ int err;
+ double high_time = 0.0, low_time = 0.0;
+ struct memcg_async_reclaim *skel = NULL;
+
+ err = setup_high_low_cgroups(&high_cgroup_id, &low_cgroup_id);
+ if (!ASSERT_OK(err, "setup_high_low_cgroups reclaim"))
+ return;
+
+ err = setup_bpf(high_cgroup_id, low_cgroup_id, &skel, false);
+ if (!ASSERT_OK(err, "setup_bpf"))
+ goto out;
+
+ err = run_high_low_workload(&high_time, &low_time, READ_TIMES);
+ if (!ASSERT_OK(err, "run_high_low_workload reclaim"))
+ goto out;
+
+ if (high_time >= low_time) {
+ PRINT_FAIL("high cgroup not improved with async reclaim: high_time=%f low_time=%f",
+ high_time, low_time);
+ }
+
+out:
+ if (skel)
+ memcg_async_reclaim__destroy(skel);
+ /*
+ * Wait for bpf_wq to release the reference to cgroup
+ * to ensure the successful deletion of cgroup.
+ */
+ sleep(1);
+ cleanup_cgroup_environment();
+}
+
+void test_memcg_thread_wq_async_reclaim(void)
+{
+ u64 high_cgroup_id, low_cgroup_id;
+ int err;
+ double high_time = 0.0, low_time = 0.0;
+ struct memcg_async_reclaim *skel = NULL;
+
+ err = setup_high_low_cgroups(&high_cgroup_id, &low_cgroup_id);
+ if (!ASSERT_OK(err, "setup_high_low_cgroups reclaim"))
+ return;
+
+ err = setup_bpf(high_cgroup_id, low_cgroup_id, &skel, true);
+ if (!ASSERT_OK(err, "setup_bpf"))
+ goto out;
+
+ err = run_high_low_workload(&high_time, &low_time, READ_TIMES);
+ if (!ASSERT_OK(err, "run_high_low_workload reclaim"))
+ goto out;
+
+ if (high_time >= low_time) {
+ PRINT_FAIL("high cgroup not improved with async reclaim: high_time=%f low_time=%f",
+ high_time, low_time);
+ }
+
+out:
+ if (skel)
+ memcg_async_reclaim__destroy(skel);
+ /*
+ * Wait for bpf_thread_wq to release the reference to cgroup
+ * to ensure the successful deletion of cgroup.
+ */
+ sleep(1);
+ cleanup_cgroup_environment();
+}
diff --git a/tools/testing/selftests/bpf/progs/memcg_async_reclaim.c b/tools/testing/selftests/bpf/progs/memcg_async_reclaim.c
new file mode 100644
index 000000000000..eaccc8a37388
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/memcg_async_reclaim.c
@@ -0,0 +1,255 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include "vmlinux.h"
+#include "bpf_experimental.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+#define BIT(nr) (1UL << (nr))
+
+#define ___GFP_IO BIT(___GFP_IO_BIT)
+#define ___GFP_FS BIT(___GFP_FS_BIT)
+#define ___GFP_DIRECT_RECLAIM BIT(___GFP_DIRECT_RECLAIM_BIT)
+#define ___GFP_KSWAPD_RECLAIM BIT(___GFP_KSWAPD_RECLAIM_BIT)
+
+#define __GFP_IO ((gfp_t)___GFP_IO)
+#define __GFP_FS ((gfp_t)___GFP_FS)
+#define __GFP_DIRECT_RECLAIM ((gfp_t)___GFP_DIRECT_RECLAIM)
+#define __GFP_KSWAPD_RECLAIM ((gfp_t)___GFP_KSWAPD_RECLAIM)
+#define __GFP_RECLAIM ((gfp_t)(___GFP_DIRECT_RECLAIM | ___GFP_KSWAPD_RECLAIM))
+
+#define GFP_KERNEL (__GFP_RECLAIM | __GFP_IO | __GFP_FS)
+#define CLOCK_MONOTONIC_ID 1
+#define RECLAIM_PAGES 32
+#define RECLAIM_MAX_ITER 32
+
+struct bpf_args_s {
+ u64 high_cgroup_id;
+ u64 low_cgroup_id;
+ u64 event_delta_threshold;
+ u64 check_ns;
+};
+
+struct cgroup_memcg {
+ struct cgroup *cgrp;
+ struct mem_cgroup *memcg;
+};
+
+static u64 wq_high_cgroup_id;
+static u64 wq_low_cgroup_id;
+
+static int get_cgroup_memcg_from_id(u64 cgroup_id, struct cgroup_memcg *cm)
+{
+ cm->cgrp = bpf_cgroup_from_id(cgroup_id);
+ if (!cm->cgrp)
+ return -1;
+
+ cm->memcg = bpf_get_mem_cgroup(&cm->cgrp->self);
+ if (!cm->memcg) {
+ bpf_cgroup_release(cm->cgrp);
+ return -1;
+ }
+
+ return 0;
+}
+
+static void put_cgroup_memcg(struct cgroup_memcg *cm)
+{
+ bpf_put_mem_cgroup(cm->memcg);
+ bpf_cgroup_release(cm->cgrp);
+}
+
+static int get_cgroup_event(u64 cgroup_id, u64 *val)
+{
+ struct cgroup_memcg cm;
+
+ if (get_cgroup_memcg_from_id(cgroup_id, &cm))
+ return -1;
+ bpf_mem_cgroup_flush_stats(cm.memcg);
+ *val = bpf_mem_cgroup_page_state(cm.memcg, WORKINGSET_REFAULT_FILE);
+ put_cgroup_memcg(&cm);
+
+ return 0;
+}
+
+static bool
+should_reclaim_cgroup(u64 cgroup_id, u64 *prev_event, u64 event_delta_threshold)
+{
+ u64 cur, delta;
+
+ if (get_cgroup_event(cgroup_id, &cur))
+ return false;
+
+ delta = cur - *prev_event;
+ *prev_event = cur;
+
+ return delta >= event_delta_threshold;
+}
+
+static int reclaim_cgroup(u64 cgroup_id)
+{
+ struct cgroup_memcg cm;
+ int i;
+
+ if (get_cgroup_memcg_from_id(cgroup_id, &cm))
+ return 0;
+
+ for (i = 0; i < RECLAIM_MAX_ITER; i++) {
+ if (!bpf_try_to_free_mem_cgroup_pages(cm.memcg, RECLAIM_PAGES,
+ GFP_KERNEL, 0, -1))
+ break;
+ }
+
+ put_cgroup_memcg(&cm);
+
+ return 0;
+}
+
+struct wq_elem {
+ struct bpf_timer timer;
+ struct bpf_wq work;
+ u64 prev_event;
+ u64 event_delta_threshold;
+ u64 check_ns;
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, __u32);
+ __type(value, struct wq_elem);
+} wq_map SEC(".maps");
+
+static int async_free(void *map, int *key, void *value)
+{
+ struct wq_elem *elem = value;
+
+ if (should_reclaim_cgroup(wq_high_cgroup_id, &elem->prev_event,
+ elem->event_delta_threshold)) {
+ reclaim_cgroup(wq_low_cgroup_id);
+ bpf_wq_start(&elem->work, 0);
+ }
+
+ return 0;
+}
+
+static int wq_timer_cb(void *map, int *key, struct wq_elem *elem)
+{
+ bpf_wq_start(&elem->work, 0);
+ bpf_timer_start(&elem->timer, elem->check_ns, 0);
+
+ return 0;
+}
+
+SEC("syscall")
+int wq_prog_init(struct bpf_args_s *ctx)
+{
+ struct wq_elem *elem;
+ __u32 key = 0;
+ int ret;
+
+ elem = bpf_map_lookup_elem(&wq_map, &key);
+ if (!elem)
+ return -1;
+
+ ret = bpf_wq_init(&elem->work, &wq_map, 0);
+ if (ret)
+ return ret;
+
+ ret = bpf_wq_set_callback(&elem->work, async_free, 0);
+ if (ret)
+ return ret;
+
+ ret = bpf_timer_init(&elem->timer, &wq_map, CLOCK_MONOTONIC_ID);
+ if (ret)
+ return ret;
+
+ ret = bpf_timer_set_callback(&elem->timer, wq_timer_cb);
+ if (ret)
+ return ret;
+
+ elem->prev_event = 0;
+ elem->event_delta_threshold = ctx->event_delta_threshold;
+ elem->check_ns = ctx->check_ns;
+
+ wq_high_cgroup_id = ctx->high_cgroup_id;
+ wq_low_cgroup_id = ctx->low_cgroup_id;
+
+ return bpf_timer_start(&elem->timer, elem->check_ns, 0);
+}
+
+struct thread_wq_elem {
+ struct bpf_timer timer;
+ struct bpf_thread_wq work;
+ u64 prev_event;
+ u64 event_delta_threshold;
+ u64 check_ns;
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, __u32);
+ __type(value, struct thread_wq_elem);
+} thread_wq_map SEC(".maps");
+
+static int thread_async_free(void *map, int *key, void *value)
+{
+ struct thread_wq_elem *elem = value;
+
+ if (should_reclaim_cgroup(wq_high_cgroup_id, &elem->prev_event,
+ elem->event_delta_threshold)) {
+ reclaim_cgroup(wq_low_cgroup_id);
+ bpf_thread_wq_start(&elem->work, 0);
+ }
+
+ return 0;
+}
+
+static int thread_wq_timer_cb(void *map, int *key, struct thread_wq_elem *elem)
+{
+ bpf_thread_wq_start(&elem->work, 0);
+ bpf_timer_start(&elem->timer, elem->check_ns, 0);
+
+ return 0;
+}
+
+SEC("syscall")
+int thread_wq_prog_init(struct bpf_args_s *ctx)
+{
+ struct thread_wq_elem *elem;
+ __u32 key = 0;
+ int ret;
+
+ elem = bpf_map_lookup_elem(&thread_wq_map, &key);
+ if (!elem)
+ return -1;
+
+ ret = bpf_thread_wq_init(&elem->work, &thread_wq_map,
+ ctx->low_cgroup_id, 0);
+ if (ret)
+ return ret;
+
+ ret = bpf_thread_wq_set_callback(&elem->work, thread_async_free, 0);
+ if (ret)
+ return ret;
+
+ ret = bpf_timer_init(&elem->timer, &thread_wq_map, CLOCK_MONOTONIC_ID);
+ if (ret)
+ return ret;
+
+ ret = bpf_timer_set_callback(&elem->timer, thread_wq_timer_cb);
+ if (ret)
+ return ret;
+
+ elem->prev_event = 0;
+ elem->event_delta_threshold = ctx->event_delta_threshold;
+ elem->check_ns = ctx->check_ns;
+
+ wq_high_cgroup_id = ctx->high_cgroup_id;
+ wq_low_cgroup_id = ctx->low_cgroup_id;
+
+ return bpf_timer_start(&elem->timer, elem->check_ns, 0);
+}
+
+char LICENSE[] SEC("license") = "GPL";
--
2.53.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-08-07 7:04 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-08-07 7:01 [PATCH bpf-next 0/4] bpf: BPF-driven proactive memcg reclaim Hui Zhu
2026-08-07 7:01 ` [PATCH bpf-next 1/4] mm/bpf: Add bpf_try_to_free_mem_cgroup_pages kfunc Hui Zhu
2026-08-07 7:01 ` [PATCH bpf-next 2/4] bpf: add bpf_thread_wq kthread-backed workqueue with cgroup placement Hui Zhu
2026-08-07 7:04 ` [PATCH bpf-next 3/4] selftests/bpf: add thread_wq cgroup test Hui Zhu
2026-08-07 7:04 ` [PATCH bpf-next 4/4] selftests/bpf: add memcg async reclaim test for bpf_wq/bpf_thread_wq Hui Zhu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox