[PATCH bpf-next v4 0/2] bpf: BPF-driven proactive memcg reclaim

[PATCH bpf-next v4 0/2] bpf: BPF-driven proactive memcg reclaim

[Hui Zhu]

From: Hui Zhu <zhuhui@kylinos.cn>

This series lets a BPF program decide when to trigger memcg reclaim
and how aggressively to do it, based on whatever runtime signal it
chooses to observe -- rather than reclaim only being triggered once a
cgroup's usage crosses a fixed threshold. The core idea is a pair of
new kfuncs, bpf_proactive_reclaim() and
bpf_proactive_reclaim_swappiness(), which give BPF direct access to
the proactive reclaim path so this decision can be made in BPF policy
rather than hard-coded threshold logic.

This was originally part of a larger series posted here [1].
That series also adds a memcg BPF struct_ops (memcg_charged,
memcg_uncharged, below_low, below_min) for synchronous, in-line memory
protection decisions. That mechanism and this one solve different
problems -- struct_ops hooks run inline on the charge/reclaim path,
while the kfuncs here are for asynchronous, out-of-band reclaim
decided independently by a BPF program -- so they are reviewed as
separate series. This series carries only the async reclaim piece.

Compared to v1, the kfunc interface has been reworked based on review
feedback: instead of a thin wrapper around
try_to_free_mem_cgroup_pages() exposing raw gfp/reclaim-option knobs,
the series now provides use-case-driven kfuncs that perform one
proactive reclaim pass with the same parameters memory.reclaim uses.
The bpf_thread_wq patches from v1 (old patches 2-3) are dropped from
this series: following the discussion in [2], the cgroup-aware
workqueue is being superseded by a disaggregated set of async
primitives (bpf_kthread/bpf_waitq) that will be developed separately
(discussion in [3]), and the selftest now queues its reclaim work
through bpf_wq.

Patch 1 adds bpf_proactive_reclaim() and
bpf_proactive_reclaim_swappiness(), sleepable kfuncs that perform one
reclaim pass on a target memcg, like a write to memory.reclaim: swap
is allowed, and the anon/file balance follows the cgroup's swappiness
or an explicit override in [MIN_SWAPPINESS, MAX_SWAPPINESS] plus
SWAPPINESS_ANON_ONLY. Both go through a shared helper,
bpf_proactive_reclaim_pages(), which guards against reclaim recursion
and calls try_to_free_mem_cgroup_pages() with GFP_KERNEL and
MEMCG_RECLAIM_MAY_SWAP | MEMCG_RECLAIM_PROACTIVE, the same parameters
user_proactive_reclaim() uses, and unlike memory.reclaim they do not
retry until the requested size is reached. Both refuse to run when
the caller already holds PF_MEMALLOC or has a non-NULL
current->reclaim_state, since a nested try_to_free_mem_cgroup_pages()
would clobber the outer reclaim's current->reclaim_state (e.g. MGLRU
dereferences current->reclaim_state->mm_walk); the reclaim_state
check also closes the window where try_to_free_mem_cgroup_pages() has
installed it but not yet set PF_MEMALLOC, reachable by a sleepable
program attaching fentry to the generated trace iterator function.
The size argument and the return value are both in bytes, matching
the byte-based unit of bpf_mem_cgroup_usage() and
bpf_mem_cgroup_page_state() so callers can mix them without manual
page/byte conversions. An out-of-range swappiness is reported with
(unsigned long)-1 rather than 0, since 0 cannot be told apart from a
pass that reclaimed nothing.

Patch 2 (selftests/bpf: add memcg async reclaim test) ties the kfuncs
into a worked example: it watches the WORKINGSET_REFAULT_FILE counter
of a high-priority cgroup as a proxy for memory-pressure impact, and
once it starts climbing, proactively reclaims memory from a
low-priority cgroup via bpf_proactive_reclaim(), with the reclaim
work queued asynchronously through bpf_wq. The test asserts that the
monitored cgroup's workload finishes faster once async reclaim kicks
in, and -- as timing alone cannot distinguish a working reclaim from
a no-op one -- that the BPF program actually made reclaim calls and
reclaimed bytes, via counters it exports through its .bss. This
demonstrates the end-to-end use case: BPF observes pressure on the
cgroup it wants to protect, and reclaims from the cgroup it wants to
reclaim from, in one self-contained mechanism. Note that, without
bpf_thread_wq, the CPU cost of the reclaim work is not yet attributed
to a chosen cgroup; that part waits for the async primitives work
mentioned above.

Changelog:
v4:
According to the comments of bot+bpf-ci and sashiko, also check
current->reclaim_state to close the fentry-on-trace-iter recursion
window in bpf_in_reclaim_context.
Return bytes instead of pages ( nr * PAGE_SIZE ) in
bpf_proactive_reclaim_pages and bpf_proactive_reclaim_swappiness.
Return (unsigned long)-1 on out-of-range swappiness (was 0).
Kdoc of both kfuncs: updated Return descriptions; added FS-lock deadlock
warning to bpf_proactive_reclaim.
Fix potential child process leak in selftests.
Use _exit() instead of exit() in forked children in selftests.
Rename reclaimed_pages to reclaimed_bytes in selftests.
Fix comments issues in selftests.
v3:
According to the comments of bot+bpf-ci, add a shared helper
bpf_proactive_reclaim_pages() that is called by bpf_proactive_reclaim
and bpf_proactive_reclaim_swappiness.
According to the comments of sashiko and bot+bpf-ci, fix the issues of
selftests.
v2:
According to the comments of Shakeel Butt, replace
bpf_try_to_free_mem_cgroup_pages() with
bpf_proactive_reclaim(memcg, size) and
bpf_proactive_reclaim_swappiness(memcg, size, swappiness).
According to the comments of Kumar Kartikeya Dwivedi, drop patch 2
and patch 3.
Remove bpf_thread_wq code in patch 4.
According to the comments of sashiko-bot, fix the issues of selftests.

[1] https://sashiko.dev/#/message/cover.1779760876.git.zhuhui%40kylinos.cn
[2] https://sashiko.dev/#/message/1b58d56976202f26818d31dbd0da2ecb2e2460f5%40linux.dev
[3] https://sashiko.dev/#/message/DKNHV09PBQZP.IRQL20BY574I%40gmail.com

Hui Zhu (2):
  mm/bpf: Add bpf_proactive_reclaim kfuncs
  selftests/bpf: add memcg async reclaim test

 mm/bpf_memcontrol.c                           | 118 +++++
 .../bpf/prog_tests/memcg_async_reclaim.c      | 480 ++++++++++++++++++
 .../selftests/bpf/progs/memcg_async_reclaim.c | 181 +++++++
 3 files changed, 779 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/memcg_async_reclaim.c
 create mode 100644 tools/testing/selftests/bpf/progs/memcg_async_reclaim.c

-- 
2.53.0

[PATCH bpf-next v4 1/2] mm/bpf: Add bpf_proactive_reclaim kfuncs

[Hui Zhu]

From: Hui Zhu <zhuhui@kylinos.cn>

Expose memcg proactive reclaim to sleepable BPF programs:
unsigned long bpf_proactive_reclaim(memcg, size);
unsigned long bpf_proactive_reclaim_swappiness(memcg, size, swappiness);

They perform one reclaim pass on @memcg, like a write to memory.reclaim:
swap is allowed, and the anon/file balance follows the cgroup's
swappiness or an explicit override in [MIN_SWAPPINESS, MAX_SWAPPINESS]
plus SWAPPINESS_ANON_ONLY. Both go through a shared helper,
bpf_proactive_reclaim_pages(), which guards against reclaim recursion
and calls try_to_free_mem_cgroup_pages() with GFP_KERNEL and
MEMCG_RECLAIM_MAY_SWAP | MEMCG_RECLAIM_PROACTIVE, the same parameters
user_proactive_reclaim() uses, and unlike memory.reclaim they do not
retry until @size is reached.

Reclaim must not recurse: try_to_free_mem_cgroup_pages() overwrites
current->reclaim_state on entry and NULLs it on exit, so a nested call
from an in-flight reclaim would corrupt the outer reclaim state (e.g.
MGLRU dereferences current->reclaim_state->mm_walk). Both kfuncs
therefore refuse to reclaim when PF_MEMALLOC is set or
current->reclaim_state is non-NULL. The latter check also closes the
window in try_to_free_mem_cgroup_pages() where reclaim_state is already
installed but PF_MEMALLOC is not: only a tracepoint call sits in
between, and while a sleepable BPF program cannot attach to the
tracepoint itself, it can attach to the generated trace iterator
function (__traceiter_mm_vmscan_memcg_reclaim_begin) via fentry.

The kfuncs take @size in bytes; the return value is normalized to bytes
as well, matching the byte-based unit of bpf_mem_cgroup_usage() and
bpf_mem_cgroup_page_state(), so callers can mix them without manual
page/byte conversions.

An out-of-range @swappiness is reported with (unsigned long)-1 instead
of 0, following the convention of bpf_mem_cgroup_vm_events() and
bpf_mem_cgroup_page_state(), as 0 cannot be told apart from a
successful pass that reclaimed nothing.

Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
---
 mm/bpf_memcontrol.c | 118 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 118 insertions(+)

diff --git a/mm/bpf_memcontrol.c b/mm/bpf_memcontrol.c
index 716df49d7647..dc51868b3acf 100644
--- a/mm/bpf_memcontrol.c
+++ b/mm/bpf_memcontrol.c
@@ -6,6 +6,7 @@
  */
 
 #include <linux/memcontrol.h>
+#include <linux/swap.h>
 #include <linux/bpf.h>
 
 __bpf_kfunc_start_defs();
@@ -159,6 +160,120 @@ __bpf_kfunc void bpf_mem_cgroup_flush_stats(struct mem_cgroup *memcg)
 	mem_cgroup_flush_stats(memcg);
 }
 
+/*
+ * Reclaim must not recurse. try_to_free_mem_cgroup_pages() unconditionally
+ * overwrites current->reclaim_state on entry and resets it to NULL on exit.
+ * So invoking it from an in-flight reclaim would clobber the outer reclaim
+ * state and corrupt its accounting.
+ *
+ * The guards are PF_MEMALLOC and current->reclaim_state. Every reclaim
+ * entry point marks the current task with PF_MEMALLOC for the whole
+ * reclaim window: try_to_free_mem_cgroup_pages() and __perform_reclaim()
+ * do so via memalloc_noreclaim_save(), and kswapd keeps it set for its
+ * entire lifetime. A hook inside the reclaim path (shrink_node,
+ * shrink_slab, ...) executes in the context of the reclaiming task, where
+ * current->flags already carries the flag. The page allocator, the memcg
+ * charging path and node_reclaim() rely on the same flag to avoid reclaim
+ * recursion.
+ *
+ * reclaim_state is checked in addition because it is set slightly before
+ * PF_MEMALLOC in try_to_free_mem_cgroup_pages(), with only a tracepoint
+ * call in between. A sleepable BPF program cannot attach to the tracepoint
+ * itself, but it can attach to the generated trace iterator function
+ * (__traceiter_mm_vmscan_memcg_reclaim_begin) via fentry, so PF_MEMALLOC
+ * alone would leave that window open.
+ *
+ * Also, PF_MEMALLOC is set in some non-reclaim contexts (e.g. direct compaction
+ * and vmalloc), where the kfunc conservatively refuses to reclaim as well.
+ */
+static bool bpf_in_reclaim_context(void)
+{
+	return (current->flags & PF_MEMALLOC) || current->reclaim_state;
+}
+
+/*
+ * Shared implementation of the proactive reclaim kfuncs: performs one
+ * reclaim pass on @memcg with @nr_pages as the goal, allowing swap, and
+ * @swappiness as the anon/file balance override (NULL to follow the
+ * cgroup's own swappiness setting). Returns the reclaimed amount in
+ * bytes, keeping the byte-based unit of the kfuncs' @size argument.
+ */
+static unsigned long
+bpf_proactive_reclaim_pages(struct mem_cgroup *memcg, unsigned long nr_pages,
+			    int *swappiness)
+{
+	unsigned long nr_reclaimed;
+
+	if (!nr_pages || unlikely(bpf_in_reclaim_context()))
+		return 0;
+
+	nr_reclaimed = try_to_free_mem_cgroup_pages(memcg, nr_pages, GFP_KERNEL,
+						    MEMCG_RECLAIM_MAY_SWAP |
+						    MEMCG_RECLAIM_PROACTIVE,
+						    swappiness);
+
+	return nr_reclaimed * PAGE_SIZE;
+}
+
+/**
+ * bpf_proactive_reclaim - proactively reclaim memory from a memory
+ *                         cgroup
+ * @memcg: the target memory cgroup to reclaim from
+ * @size:  the amount of memory to reclaim, in bytes
+ *
+ * Trigger one proactive reclaim pass on @memcg, similar to a write to
+ * the memory.reclaim cgroup file: pages are reclaimed according to the
+ * cgroup's own swappiness setting and swap is allowed. Note that,
+ * unlike memory.reclaim, this does not retry until @size is reached;
+ * callers can invoke it again if needed.
+ *
+ * The reclaim runs with GFP_KERNEL, so this function must not be called
+ * from a context that holds a filesystem lock (e.g. an LSM hook invoked
+ * with inode_lock held): the reclaim path may enter filesystem shrinkers
+ * and deadlock trying to reacquire the lock. Contexts that set
+ * PF_MEMALLOC_NOFS/NOIO are handled by the gfp context inheritance.
+ *
+ * Return:
+ *   The amount of memory actually reclaimed, in bytes (rounded to full
+ *   pages), or 0 if @size is smaller than a page or the calling task is
+ *   already in a reclaim/freeing context (PF_MEMALLOC).
+ */
+__bpf_kfunc unsigned long bpf_proactive_reclaim(struct mem_cgroup *memcg,
+						unsigned long size)
+{
+	return bpf_proactive_reclaim_pages(memcg, size / PAGE_SIZE, NULL);
+}
+
+/**
+ * bpf_proactive_reclaim_swappiness - proactively reclaim memory from a
+ *                                    memory cgroup with an explicit
+ *                                    swappiness
+ * @memcg:      the target memory cgroup to reclaim from
+ * @size:       the amount of memory to reclaim, in bytes
+ * @swappiness: swappiness override for this reclaim pass
+ *
+ * Same as bpf_proactive_reclaim(), except that the anon/file reclaim
+ * balance is controlled by @swappiness instead of the cgroup's
+ * swappiness setting. Valid values are [MIN_SWAPPINESS, MAX_SWAPPINESS]
+ * and SWAPPINESS_ANON_ONLY, which restricts reclaim to anon folios.
+ *
+ * Return:
+ *   The amount of memory actually reclaimed, in bytes (rounded to full
+ *   pages), (unsigned long)-1 if @swappiness is out of range, or 0 if
+ *   @size is smaller than a page or the calling task is already in a
+ *   reclaim/freeing context (PF_MEMALLOC).
+ */
+__bpf_kfunc unsigned long
+bpf_proactive_reclaim_swappiness(struct mem_cgroup *memcg, unsigned long size,
+				 int swappiness)
+{
+	if (swappiness < MIN_SWAPPINESS || swappiness > SWAPPINESS_ANON_ONLY)
+		return (unsigned long)-1;
+
+	return bpf_proactive_reclaim_pages(memcg, size / PAGE_SIZE,
+					   &swappiness);
+}
+
 __bpf_kfunc_end_defs();
 
 BTF_KFUNCS_START(bpf_memcontrol_kfuncs)
@@ -172,6 +287,9 @@ BTF_ID_FLAGS(func, bpf_mem_cgroup_usage)
 BTF_ID_FLAGS(func, bpf_mem_cgroup_page_state)
 BTF_ID_FLAGS(func, bpf_mem_cgroup_flush_stats, KF_SLEEPABLE)
 
+BTF_ID_FLAGS(func, bpf_proactive_reclaim, KF_SLEEPABLE)
+BTF_ID_FLAGS(func, bpf_proactive_reclaim_swappiness, KF_SLEEPABLE)
+
 BTF_KFUNCS_END(bpf_memcontrol_kfuncs)
 
 static const struct btf_kfunc_id_set bpf_memcontrol_kfunc_set = {
-- 
2.53.0

Re: [PATCH bpf-next v4 1/2] mm/bpf: Add bpf_proactive_reclaim kfuncs

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 4366 bytes --]

> diff --git a/mm/bpf_memcontrol.c b/mm/bpf_memcontrol.c
> index 716df49d7647..dc51868b3acf 100644
> --- a/mm/bpf_memcontrol.c
> +++ b/mm/bpf_memcontrol.c

[ ... ]

> @@ -159,6 +160,120 @@ __bpf_kfunc void bpf_mem_cgroup_flush_stats(struct mem_cgroup *memcg)
>  	mem_cgroup_flush_stats(memcg);
>  }
>
> +/*
> + * Shared implementation of the proactive reclaim kfuncs: performs one
> + * reclaim pass on @memcg with @nr_pages as the goal, allowing swap, and
> + * @swappiness as the anon/file balance override (NULL to follow the
> + * cgroup's own swappiness setting). Returns the reclaimed amount in
> + * bytes, keeping the byte-based unit of the kfuncs' @size argument.
> + */
> +static unsigned long
> +bpf_proactive_reclaim_pages(struct mem_cgroup *memcg, unsigned long nr_pages,
> +			    int *swappiness)

This isn't a bug, but would it be worth trimming the comment above
bpf_proactive_reclaim_pages() and de-duplicating the identical 'Return:'
paragraph that now appears in both kfunc kernel-docs, so the byte/page
unit conversion is documented in exactly one place?

> +{
> +	unsigned long nr_reclaimed;
> +
> +	if (!nr_pages || unlikely(bpf_in_reclaim_context()))
> +		return 0;
> +
> +	nr_reclaimed = try_to_free_mem_cgroup_pages(memcg, nr_pages, GFP_KERNEL,
> +						    MEMCG_RECLAIM_MAY_SWAP |
> +						    MEMCG_RECLAIM_PROACTIVE,
> +						    swappiness);
> +
> +	return nr_reclaimed * PAGE_SIZE;
> +}
> +
> +/**
> + * bpf_proactive_reclaim - proactively reclaim memory from a memory
> + *                         cgroup
> + * @memcg: the target memory cgroup to reclaim from
> + * @size:  the amount of memory to reclaim, in bytes
> + *
> + * Trigger one proactive reclaim pass on @memcg, similar to a write to
> + * the memory.reclaim cgroup file: pages are reclaimed according to the
> + * cgroup's own swappiness setting and swap is allowed. Note that,
> + * unlike memory.reclaim, this does not retry until @size is reached;
> + * callers can invoke it again if needed.
> + *
> + * The reclaim runs with GFP_KERNEL, so this function must not be called
> + * from a context that holds a filesystem lock (e.g. an LSM hook invoked
> + * with inode_lock held): the reclaim path may enter filesystem shrinkers
> + * and deadlock trying to reacquire the lock. Contexts that set
> + * PF_MEMALLOC_NOFS/NOIO are handled by the gfp context inheritance.
> + *
> + * Return:
> + *   The amount of memory actually reclaimed, in bytes (rounded to full
> + *   pages), or 0 if @size is smaller than a page or the calling task is
> + *   already in a reclaim/freeing context (PF_MEMALLOC).
> + */
> +__bpf_kfunc unsigned long bpf_proactive_reclaim(struct mem_cgroup *memcg,
> +						unsigned long size)
> +{
> +	return bpf_proactive_reclaim_pages(memcg, size / PAGE_SIZE, NULL);
> +}
> +
> +/**
> + * bpf_proactive_reclaim_swappiness - proactively reclaim memory from a
> + *                                    memory cgroup with an explicit
> + *                                    swappiness
> + * @memcg:      the target memory cgroup to reclaim from
> + * @size:       the amount of memory to reclaim, in bytes
> + * @swappiness: swappiness override for this reclaim pass
> + *
> + * Same as bpf_proactive_reclaim(), except that the anon/file reclaim
> + * balance is controlled by @swappiness instead of the cgroup's
> + * swappiness setting. Valid values are [MIN_SWAPPINESS, MAX_SWAPPINESS]
> + * and SWAPPINESS_ANON_ONLY, which restricts reclaim to anon folios.
> + *
> + * Return:
> + *   The amount of memory actually reclaimed, in bytes (rounded to full
> + *   pages), (unsigned long)-1 if @swappiness is out of range, or 0 if
> + *   @size is smaller than a page or the calling task is already in a
> + *   reclaim/freeing context (PF_MEMALLOC).
> + */
> +__bpf_kfunc unsigned long
> +bpf_proactive_reclaim_swappiness(struct mem_cgroup *memcg, unsigned long size,
> +				 int swappiness)
> +{
> +	if (swappiness < MIN_SWAPPINESS || swappiness > SWAPPINESS_ANON_ONLY)
> +		return (unsigned long)-1;
> +
> +	return bpf_proactive_reclaim_pages(memcg, size / PAGE_SIZE,
> +					   &swappiness);
> +}

[ ... ]


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/32339450346

Re: [PATCH bpf-next v4 1/2] mm/bpf: Add bpf_proactive_reclaim kfuncs

[Andrii Nakryiko]

On Wed, Aug 19, 2026 at 11:12 PM Hui Zhu <hui.zhu@linux.dev> wrote:
>
> From: Hui Zhu <zhuhui@kylinos.cn>
>
> Expose memcg proactive reclaim to sleepable BPF programs:
> unsigned long bpf_proactive_reclaim(memcg, size);
> unsigned long bpf_proactive_reclaim_swappiness(memcg, size, swappiness);
>
> They perform one reclaim pass on @memcg, like a write to memory.reclaim:
> swap is allowed, and the anon/file balance follows the cgroup's
> swappiness or an explicit override in [MIN_SWAPPINESS, MAX_SWAPPINESS]
> plus SWAPPINESS_ANON_ONLY. Both go through a shared helper,
> bpf_proactive_reclaim_pages(), which guards against reclaim recursion
> and calls try_to_free_mem_cgroup_pages() with GFP_KERNEL and
> MEMCG_RECLAIM_MAY_SWAP | MEMCG_RECLAIM_PROACTIVE, the same parameters
> user_proactive_reclaim() uses, and unlike memory.reclaim they do not
> retry until @size is reached.
>
> Reclaim must not recurse: try_to_free_mem_cgroup_pages() overwrites
> current->reclaim_state on entry and NULLs it on exit, so a nested call
> from an in-flight reclaim would corrupt the outer reclaim state (e.g.
> MGLRU dereferences current->reclaim_state->mm_walk). Both kfuncs
> therefore refuse to reclaim when PF_MEMALLOC is set or
> current->reclaim_state is non-NULL. The latter check also closes the
> window in try_to_free_mem_cgroup_pages() where reclaim_state is already
> installed but PF_MEMALLOC is not: only a tracepoint call sits in
> between, and while a sleepable BPF program cannot attach to the
> tracepoint itself, it can attach to the generated trace iterator
> function (__traceiter_mm_vmscan_memcg_reclaim_begin) via fentry.
>
> The kfuncs take @size in bytes; the return value is normalized to bytes
> as well, matching the byte-based unit of bpf_mem_cgroup_usage() and
> bpf_mem_cgroup_page_state(), so callers can mix them without manual
> page/byte conversions.
>
> An out-of-range @swappiness is reported with (unsigned long)-1 instead
> of 0, following the convention of bpf_mem_cgroup_vm_events() and
> bpf_mem_cgroup_page_state(), as 0 cannot be told apart from a
> successful pass that reclaimed nothing.
>
> Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
> ---
>  mm/bpf_memcontrol.c | 118 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 118 insertions(+)
>
> diff --git a/mm/bpf_memcontrol.c b/mm/bpf_memcontrol.c
> index 716df49d7647..dc51868b3acf 100644
> --- a/mm/bpf_memcontrol.c
> +++ b/mm/bpf_memcontrol.c
> @@ -6,6 +6,7 @@
>   */
>
>  #include <linux/memcontrol.h>
> +#include <linux/swap.h>
>  #include <linux/bpf.h>
>
>  __bpf_kfunc_start_defs();
> @@ -159,6 +160,120 @@ __bpf_kfunc void bpf_mem_cgroup_flush_stats(struct mem_cgroup *memcg)
>         mem_cgroup_flush_stats(memcg);
>  }
>
> +/*
> + * Reclaim must not recurse. try_to_free_mem_cgroup_pages() unconditionally
> + * overwrites current->reclaim_state on entry and resets it to NULL on exit.
> + * So invoking it from an in-flight reclaim would clobber the outer reclaim
> + * state and corrupt its accounting.
> + *
> + * The guards are PF_MEMALLOC and current->reclaim_state. Every reclaim
> + * entry point marks the current task with PF_MEMALLOC for the whole
> + * reclaim window: try_to_free_mem_cgroup_pages() and __perform_reclaim()
> + * do so via memalloc_noreclaim_save(), and kswapd keeps it set for its
> + * entire lifetime. A hook inside the reclaim path (shrink_node,
> + * shrink_slab, ...) executes in the context of the reclaiming task, where
> + * current->flags already carries the flag. The page allocator, the memcg
> + * charging path and node_reclaim() rely on the same flag to avoid reclaim
> + * recursion.
> + *
> + * reclaim_state is checked in addition because it is set slightly before
> + * PF_MEMALLOC in try_to_free_mem_cgroup_pages(), with only a tracepoint
> + * call in between. A sleepable BPF program cannot attach to the tracepoint
> + * itself, but it can attach to the generated trace iterator function
> + * (__traceiter_mm_vmscan_memcg_reclaim_begin) via fentry, so PF_MEMALLOC
> + * alone would leave that window open.
> + *
> + * Also, PF_MEMALLOC is set in some non-reclaim contexts (e.g. direct compaction
> + * and vmalloc), where the kfunc conservatively refuses to reclaim as well.
> + */
> +static bool bpf_in_reclaim_context(void)
> +{
> +       return (current->flags & PF_MEMALLOC) || current->reclaim_state;
> +}
> +
> +/*
> + * Shared implementation of the proactive reclaim kfuncs: performs one
> + * reclaim pass on @memcg with @nr_pages as the goal, allowing swap, and
> + * @swappiness as the anon/file balance override (NULL to follow the
> + * cgroup's own swappiness setting). Returns the reclaimed amount in
> + * bytes, keeping the byte-based unit of the kfuncs' @size argument.
> + */
> +static unsigned long
> +bpf_proactive_reclaim_pages(struct mem_cgroup *memcg, unsigned long nr_pages,
> +                           int *swappiness)
> +{
> +       unsigned long nr_reclaimed;
> +
> +       if (!nr_pages || unlikely(bpf_in_reclaim_context()))
> +               return 0;
> +
> +       nr_reclaimed = try_to_free_mem_cgroup_pages(memcg, nr_pages, GFP_KERNEL,
> +                                                   MEMCG_RECLAIM_MAY_SWAP |
> +                                                   MEMCG_RECLAIM_PROACTIVE,
> +                                                   swappiness);
> +
> +       return nr_reclaimed * PAGE_SIZE;
> +}
> +
> +/**
> + * bpf_proactive_reclaim - proactively reclaim memory from a memory
> + *                         cgroup
> + * @memcg: the target memory cgroup to reclaim from
> + * @size:  the amount of memory to reclaim, in bytes
> + *
> + * Trigger one proactive reclaim pass on @memcg, similar to a write to
> + * the memory.reclaim cgroup file: pages are reclaimed according to the
> + * cgroup's own swappiness setting and swap is allowed. Note that,
> + * unlike memory.reclaim, this does not retry until @size is reached;
> + * callers can invoke it again if needed.
> + *
> + * The reclaim runs with GFP_KERNEL, so this function must not be called
> + * from a context that holds a filesystem lock (e.g. an LSM hook invoked
> + * with inode_lock held): the reclaim path may enter filesystem shrinkers
> + * and deadlock trying to reacquire the lock. Contexts that set
> + * PF_MEMALLOC_NOFS/NOIO are handled by the gfp context inheritance.
> + *
> + * Return:
> + *   The amount of memory actually reclaimed, in bytes (rounded to full
> + *   pages), or 0 if @size is smaller than a page or the calling task is
> + *   already in a reclaim/freeing context (PF_MEMALLOC).
> + */
> +__bpf_kfunc unsigned long bpf_proactive_reclaim(struct mem_cgroup *memcg,
> +                                               unsigned long size)
> +{
> +       return bpf_proactive_reclaim_pages(memcg, size / PAGE_SIZE, NULL);
> +}
> +
> +/**
> + * bpf_proactive_reclaim_swappiness - proactively reclaim memory from a
> + *                                    memory cgroup with an explicit
> + *                                    swappiness
> + * @memcg:      the target memory cgroup to reclaim from
> + * @size:       the amount of memory to reclaim, in bytes
> + * @swappiness: swappiness override for this reclaim pass
> + *
> + * Same as bpf_proactive_reclaim(), except that the anon/file reclaim
> + * balance is controlled by @swappiness instead of the cgroup's
> + * swappiness setting. Valid values are [MIN_SWAPPINESS, MAX_SWAPPINESS]
> + * and SWAPPINESS_ANON_ONLY, which restricts reclaim to anon folios.
> + *
> + * Return:
> + *   The amount of memory actually reclaimed, in bytes (rounded to full
> + *   pages), (unsigned long)-1 if @swappiness is out of range, or 0 if
> + *   @size is smaller than a page or the calling task is already in a
> + *   reclaim/freeing context (PF_MEMALLOC).
> + */
> +__bpf_kfunc unsigned long
> +bpf_proactive_reclaim_swappiness(struct mem_cgroup *memcg, unsigned long size,
> +                                int swappiness)
> +{
> +       if (swappiness < MIN_SWAPPINESS || swappiness > SWAPPINESS_ANON_ONLY)
> +               return (unsigned long)-1;
> +
> +       return bpf_proactive_reclaim_pages(memcg, size / PAGE_SIZE,
> +                                          &swappiness);
> +}

I haven't followed previous discussion, so I apologize if this was
discussed, but if not, isn't it a bit an overkill to have second
variant just to provide optional swappiness? Valid range of swappinees
seems to be non-negative [0, 200], that special ANON is 201, so why
can't we defined that <0 swappiness just means no swappiness was
provided and get away with just one kfunc?

> +
>  __bpf_kfunc_end_defs();
>
>  BTF_KFUNCS_START(bpf_memcontrol_kfuncs)
> @@ -172,6 +287,9 @@ BTF_ID_FLAGS(func, bpf_mem_cgroup_usage)
>  BTF_ID_FLAGS(func, bpf_mem_cgroup_page_state)
>  BTF_ID_FLAGS(func, bpf_mem_cgroup_flush_stats, KF_SLEEPABLE)
>
> +BTF_ID_FLAGS(func, bpf_proactive_reclaim, KF_SLEEPABLE)
> +BTF_ID_FLAGS(func, bpf_proactive_reclaim_swappiness, KF_SLEEPABLE)
> +
>  BTF_KFUNCS_END(bpf_memcontrol_kfuncs)
>
>  static const struct btf_kfunc_id_set bpf_memcontrol_kfunc_set = {
> --
> 2.53.0
>

Re: [PATCH bpf-next v4 1/2] mm/bpf: Add bpf_proactive_reclaim kfuncs

[Kumar Kartikeya Dwivedi]

On Fri Aug 21, 2026 at 9:12 PM CEST, Andrii Nakryiko wrote:
> On Wed, Aug 19, 2026 at 11:12 PM Hui Zhu <hui.zhu@linux.dev> wrote:
>>
>> From: Hui Zhu <zhuhui@kylinos.cn>
>>
>> Expose memcg proactive reclaim to sleepable BPF programs:
>> unsigned long bpf_proactive_reclaim(memcg, size);
>> unsigned long bpf_proactive_reclaim_swappiness(memcg, size, swappiness);
>>
>> They perform one reclaim pass on @memcg, like a write to memory.reclaim:
>> swap is allowed, and the anon/file balance follows the cgroup's
>> swappiness or an explicit override in [MIN_SWAPPINESS, MAX_SWAPPINESS]
>> plus SWAPPINESS_ANON_ONLY. Both go through a shared helper,
>> bpf_proactive_reclaim_pages(), which guards against reclaim recursion
>> and calls try_to_free_mem_cgroup_pages() with GFP_KERNEL and
>> MEMCG_RECLAIM_MAY_SWAP | MEMCG_RECLAIM_PROACTIVE, the same parameters
>> user_proactive_reclaim() uses, and unlike memory.reclaim they do not
>> retry until @size is reached.
>>
>> Reclaim must not recurse: try_to_free_mem_cgroup_pages() overwrites
>> current->reclaim_state on entry and NULLs it on exit, so a nested call
>> from an in-flight reclaim would corrupt the outer reclaim state (e.g.
>> MGLRU dereferences current->reclaim_state->mm_walk). Both kfuncs
>> therefore refuse to reclaim when PF_MEMALLOC is set or
>> current->reclaim_state is non-NULL. The latter check also closes the
>> window in try_to_free_mem_cgroup_pages() where reclaim_state is already
>> installed but PF_MEMALLOC is not: only a tracepoint call sits in
>> between, and while a sleepable BPF program cannot attach to the
>> tracepoint itself, it can attach to the generated trace iterator
>> function (__traceiter_mm_vmscan_memcg_reclaim_begin) via fentry.
>>
>> The kfuncs take @size in bytes; the return value is normalized to bytes
>> as well, matching the byte-based unit of bpf_mem_cgroup_usage() and
>> bpf_mem_cgroup_page_state(), so callers can mix them without manual
>> page/byte conversions.
>>
>> An out-of-range @swappiness is reported with (unsigned long)-1 instead
>> of 0, following the convention of bpf_mem_cgroup_vm_events() and
>> bpf_mem_cgroup_page_state(), as 0 cannot be told apart from a
>> successful pass that reclaimed nothing.
>>
>> Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
>> ---
>>  mm/bpf_memcontrol.c | 118 ++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 118 insertions(+)
>>
>> diff --git a/mm/bpf_memcontrol.c b/mm/bpf_memcontrol.c
>> index 716df49d7647..dc51868b3acf 100644
>> --- a/mm/bpf_memcontrol.c
>> +++ b/mm/bpf_memcontrol.c
>> @@ -6,6 +6,7 @@
>>   */
>>
>>  #include <linux/memcontrol.h>
>> +#include <linux/swap.h>
>>  #include <linux/bpf.h>
>>
>>  __bpf_kfunc_start_defs();
>> @@ -159,6 +160,120 @@ __bpf_kfunc void bpf_mem_cgroup_flush_stats(struct mem_cgroup *memcg)
>>         mem_cgroup_flush_stats(memcg);
>>  }
>>
>> +/*
>> + * Reclaim must not recurse. try_to_free_mem_cgroup_pages() unconditionally
>> + * overwrites current->reclaim_state on entry and resets it to NULL on exit.
>> + * So invoking it from an in-flight reclaim would clobber the outer reclaim
>> + * state and corrupt its accounting.
>> + *
>> + * The guards are PF_MEMALLOC and current->reclaim_state. Every reclaim
>> + * entry point marks the current task with PF_MEMALLOC for the whole
>> + * reclaim window: try_to_free_mem_cgroup_pages() and __perform_reclaim()
>> + * do so via memalloc_noreclaim_save(), and kswapd keeps it set for its
>> + * entire lifetime. A hook inside the reclaim path (shrink_node,
>> + * shrink_slab, ...) executes in the context of the reclaiming task, where
>> + * current->flags already carries the flag. The page allocator, the memcg
>> + * charging path and node_reclaim() rely on the same flag to avoid reclaim
>> + * recursion.
>> + *
>> + * reclaim_state is checked in addition because it is set slightly before
>> + * PF_MEMALLOC in try_to_free_mem_cgroup_pages(), with only a tracepoint
>> + * call in between. A sleepable BPF program cannot attach to the tracepoint
>> + * itself, but it can attach to the generated trace iterator function
>> + * (__traceiter_mm_vmscan_memcg_reclaim_begin) via fentry, so PF_MEMALLOC
>> + * alone would leave that window open.
>> + *
>> + * Also, PF_MEMALLOC is set in some non-reclaim contexts (e.g. direct compaction
>> + * and vmalloc), where the kfunc conservatively refuses to reclaim as well.
>> + */
>> +static bool bpf_in_reclaim_context(void)
>> +{
>> +       return (current->flags & PF_MEMALLOC) || current->reclaim_state;
>> +}
>> +
>> +/*
>> + * Shared implementation of the proactive reclaim kfuncs: performs one
>> + * reclaim pass on @memcg with @nr_pages as the goal, allowing swap, and
>> + * @swappiness as the anon/file balance override (NULL to follow the
>> + * cgroup's own swappiness setting). Returns the reclaimed amount in
>> + * bytes, keeping the byte-based unit of the kfuncs' @size argument.
>> + */
>> +static unsigned long
>> +bpf_proactive_reclaim_pages(struct mem_cgroup *memcg, unsigned long nr_pages,
>> +                           int *swappiness)
>> +{
>> +       unsigned long nr_reclaimed;
>> +
>> +       if (!nr_pages || unlikely(bpf_in_reclaim_context()))
>> +               return 0;
>> +
>> +       nr_reclaimed = try_to_free_mem_cgroup_pages(memcg, nr_pages, GFP_KERNEL,
>> +                                                   MEMCG_RECLAIM_MAY_SWAP |
>> +                                                   MEMCG_RECLAIM_PROACTIVE,
>> +                                                   swappiness);
>> +
>> +       return nr_reclaimed * PAGE_SIZE;
>> +}
>> +
>> +/**
>> + * bpf_proactive_reclaim - proactively reclaim memory from a memory
>> + *                         cgroup
>> + * @memcg: the target memory cgroup to reclaim from
>> + * @size:  the amount of memory to reclaim, in bytes
>> + *
>> + * Trigger one proactive reclaim pass on @memcg, similar to a write to
>> + * the memory.reclaim cgroup file: pages are reclaimed according to the
>> + * cgroup's own swappiness setting and swap is allowed. Note that,
>> + * unlike memory.reclaim, this does not retry until @size is reached;
>> + * callers can invoke it again if needed.
>> + *
>> + * The reclaim runs with GFP_KERNEL, so this function must not be called
>> + * from a context that holds a filesystem lock (e.g. an LSM hook invoked
>> + * with inode_lock held): the reclaim path may enter filesystem shrinkers
>> + * and deadlock trying to reacquire the lock. Contexts that set
>> + * PF_MEMALLOC_NOFS/NOIO are handled by the gfp context inheritance.
>> + *
>> + * Return:
>> + *   The amount of memory actually reclaimed, in bytes (rounded to full
>> + *   pages), or 0 if @size is smaller than a page or the calling task is
>> + *   already in a reclaim/freeing context (PF_MEMALLOC).
>> + */
>> +__bpf_kfunc unsigned long bpf_proactive_reclaim(struct mem_cgroup *memcg,
>> +                                               unsigned long size)
>> +{
>> +       return bpf_proactive_reclaim_pages(memcg, size / PAGE_SIZE, NULL);
>> +}
>> +
>> +/**
>> + * bpf_proactive_reclaim_swappiness - proactively reclaim memory from a
>> + *                                    memory cgroup with an explicit
>> + *                                    swappiness
>> + * @memcg:      the target memory cgroup to reclaim from
>> + * @size:       the amount of memory to reclaim, in bytes
>> + * @swappiness: swappiness override for this reclaim pass
>> + *
>> + * Same as bpf_proactive_reclaim(), except that the anon/file reclaim
>> + * balance is controlled by @swappiness instead of the cgroup's
>> + * swappiness setting. Valid values are [MIN_SWAPPINESS, MAX_SWAPPINESS]
>> + * and SWAPPINESS_ANON_ONLY, which restricts reclaim to anon folios.
>> + *
>> + * Return:
>> + *   The amount of memory actually reclaimed, in bytes (rounded to full
>> + *   pages), (unsigned long)-1 if @swappiness is out of range, or 0 if
>> + *   @size is smaller than a page or the calling task is already in a
>> + *   reclaim/freeing context (PF_MEMALLOC).
>> + */
>> +__bpf_kfunc unsigned long
>> +bpf_proactive_reclaim_swappiness(struct mem_cgroup *memcg, unsigned long size,
>> +                                int swappiness)
>> +{
>> +       if (swappiness < MIN_SWAPPINESS || swappiness > SWAPPINESS_ANON_ONLY)
>> +               return (unsigned long)-1;
>> +
>> +       return bpf_proactive_reclaim_pages(memcg, size / PAGE_SIZE,
>> +                                          &swappiness);
>> +}
>
> I haven't followed previous discussion, so I apologize if this was
> discussed, but if not, isn't it a bit an overkill to have second
> variant just to provide optional swappiness? Valid range of swappinees
> seems to be non-negative [0, 200], that special ANON is 201, so why
> can't we defined that <0 swappiness just means no swappiness was
> provided and get away with just one kfunc?
>

I think my understanding of Shakeel's suggestion was that we only add one
bpf_proactive_reclaim() for now, if swappiness parameter is necessary we can
introduce the second API later. But given it seems Hui wants to add both, I
think it would make sense to do what Andrii said and just introduce one kfunc
with the swappiness parameter now.

But Hui, please wait for Shakeel to comment before respinning again.

>> +
>>  __bpf_kfunc_end_defs();
>>
>>  BTF_KFUNCS_START(bpf_memcontrol_kfuncs)
>> @@ -172,6 +287,9 @@ BTF_ID_FLAGS(func, bpf_mem_cgroup_usage)
>>  BTF_ID_FLAGS(func, bpf_mem_cgroup_page_state)
>>  BTF_ID_FLAGS(func, bpf_mem_cgroup_flush_stats, KF_SLEEPABLE)
>>
>> +BTF_ID_FLAGS(func, bpf_proactive_reclaim, KF_SLEEPABLE)
>> +BTF_ID_FLAGS(func, bpf_proactive_reclaim_swappiness, KF_SLEEPABLE)
>> +
>>  BTF_KFUNCS_END(bpf_memcontrol_kfuncs)
>>
>>  static const struct btf_kfunc_id_set bpf_memcontrol_kfunc_set = {
>> --
>> 2.53.0
>>

Re: [PATCH bpf-next v4 1/2] mm/bpf: Add bpf_proactive_reclaim kfuncs

[Shakeel Butt]

On Fri, Aug 21, 2026 at 09:38:31PM +0200, Kumar Kartikeya Dwivedi wrote:
> On Fri Aug 21, 2026 at 9:12 PM CEST, Andrii Nakryiko wrote:
> > On Wed, Aug 19, 2026 at 11:12 PM Hui Zhu <hui.zhu@linux.dev> wrote:
> >>
> >> From: Hui Zhu <zhuhui@kylinos.cn>
> >>
> >> Expose memcg proactive reclaim to sleepable BPF programs:
> >> unsigned long bpf_proactive_reclaim(memcg, size);
> >> unsigned long bpf_proactive_reclaim_swappiness(memcg, size, swappiness);
> >>
> >> They perform one reclaim pass on @memcg, like a write to memory.reclaim:
> >> swap is allowed, and the anon/file balance follows the cgroup's
> >> swappiness or an explicit override in [MIN_SWAPPINESS, MAX_SWAPPINESS]
> >> plus SWAPPINESS_ANON_ONLY. Both go through a shared helper,
> >> bpf_proactive_reclaim_pages(), which guards against reclaim recursion
> >> and calls try_to_free_mem_cgroup_pages() with GFP_KERNEL and
> >> MEMCG_RECLAIM_MAY_SWAP | MEMCG_RECLAIM_PROACTIVE, the same parameters
> >> user_proactive_reclaim() uses, and unlike memory.reclaim they do not
> >> retry until @size is reached.
> >>
> >> Reclaim must not recurse: try_to_free_mem_cgroup_pages() overwrites
> >> current->reclaim_state on entry and NULLs it on exit, so a nested call
> >> from an in-flight reclaim would corrupt the outer reclaim state (e.g.
> >> MGLRU dereferences current->reclaim_state->mm_walk). Both kfuncs
> >> therefore refuse to reclaim when PF_MEMALLOC is set or
> >> current->reclaim_state is non-NULL. The latter check also closes the
> >> window in try_to_free_mem_cgroup_pages() where reclaim_state is already
> >> installed but PF_MEMALLOC is not: only a tracepoint call sits in
> >> between, and while a sleepable BPF program cannot attach to the
> >> tracepoint itself, it can attach to the generated trace iterator
> >> function (__traceiter_mm_vmscan_memcg_reclaim_begin) via fentry.
> >>
> >> The kfuncs take @size in bytes; the return value is normalized to bytes
> >> as well, matching the byte-based unit of bpf_mem_cgroup_usage() and
> >> bpf_mem_cgroup_page_state(), so callers can mix them without manual
> >> page/byte conversions.
> >>
> >> An out-of-range @swappiness is reported with (unsigned long)-1 instead
> >> of 0, following the convention of bpf_mem_cgroup_vm_events() and
> >> bpf_mem_cgroup_page_state(), as 0 cannot be told apart from a
> >> successful pass that reclaimed nothing.
> >>
> >> Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
> >> ---
> >>  mm/bpf_memcontrol.c | 118 ++++++++++++++++++++++++++++++++++++++++++++
> >>  1 file changed, 118 insertions(+)
> >>
> >> diff --git a/mm/bpf_memcontrol.c b/mm/bpf_memcontrol.c
> >> index 716df49d7647..dc51868b3acf 100644
> >> --- a/mm/bpf_memcontrol.c
> >> +++ b/mm/bpf_memcontrol.c
> >> @@ -6,6 +6,7 @@
> >>   */
> >>
> >>  #include <linux/memcontrol.h>
> >> +#include <linux/swap.h>
> >>  #include <linux/bpf.h>
> >>
> >>  __bpf_kfunc_start_defs();
> >> @@ -159,6 +160,120 @@ __bpf_kfunc void bpf_mem_cgroup_flush_stats(struct mem_cgroup *memcg)
> >>         mem_cgroup_flush_stats(memcg);
> >>  }
> >>
> >> +/*
> >> + * Reclaim must not recurse. try_to_free_mem_cgroup_pages() unconditionally
> >> + * overwrites current->reclaim_state on entry and resets it to NULL on exit.
> >> + * So invoking it from an in-flight reclaim would clobber the outer reclaim
> >> + * state and corrupt its accounting.
> >> + *
> >> + * The guards are PF_MEMALLOC and current->reclaim_state. Every reclaim
> >> + * entry point marks the current task with PF_MEMALLOC for the whole
> >> + * reclaim window: try_to_free_mem_cgroup_pages() and __perform_reclaim()
> >> + * do so via memalloc_noreclaim_save(), and kswapd keeps it set for its
> >> + * entire lifetime. A hook inside the reclaim path (shrink_node,
> >> + * shrink_slab, ...) executes in the context of the reclaiming task, where
> >> + * current->flags already carries the flag. The page allocator, the memcg
> >> + * charging path and node_reclaim() rely on the same flag to avoid reclaim
> >> + * recursion.
> >> + *
> >> + * reclaim_state is checked in addition because it is set slightly before
> >> + * PF_MEMALLOC in try_to_free_mem_cgroup_pages(), with only a tracepoint
> >> + * call in between. A sleepable BPF program cannot attach to the tracepoint
> >> + * itself, but it can attach to the generated trace iterator function
> >> + * (__traceiter_mm_vmscan_memcg_reclaim_begin) via fentry, so PF_MEMALLOC
> >> + * alone would leave that window open.
> >> + *
> >> + * Also, PF_MEMALLOC is set in some non-reclaim contexts (e.g. direct compaction
> >> + * and vmalloc), where the kfunc conservatively refuses to reclaim as well.
> >> + */
> >> +static bool bpf_in_reclaim_context(void)
> >> +{
> >> +       return (current->flags & PF_MEMALLOC) || current->reclaim_state;
> >> +}
> >> +
> >> +/*
> >> + * Shared implementation of the proactive reclaim kfuncs: performs one
> >> + * reclaim pass on @memcg with @nr_pages as the goal, allowing swap, and
> >> + * @swappiness as the anon/file balance override (NULL to follow the
> >> + * cgroup's own swappiness setting). Returns the reclaimed amount in
> >> + * bytes, keeping the byte-based unit of the kfuncs' @size argument.
> >> + */
> >> +static unsigned long
> >> +bpf_proactive_reclaim_pages(struct mem_cgroup *memcg, unsigned long nr_pages,
> >> +                           int *swappiness)
> >> +{
> >> +       unsigned long nr_reclaimed;
> >> +
> >> +       if (!nr_pages || unlikely(bpf_in_reclaim_context()))
> >> +               return 0;
> >> +
> >> +       nr_reclaimed = try_to_free_mem_cgroup_pages(memcg, nr_pages, GFP_KERNEL,
> >> +                                                   MEMCG_RECLAIM_MAY_SWAP |
> >> +                                                   MEMCG_RECLAIM_PROACTIVE,
> >> +                                                   swappiness);
> >> +
> >> +       return nr_reclaimed * PAGE_SIZE;
> >> +}
> >> +
> >> +/**
> >> + * bpf_proactive_reclaim - proactively reclaim memory from a memory
> >> + *                         cgroup
> >> + * @memcg: the target memory cgroup to reclaim from
> >> + * @size:  the amount of memory to reclaim, in bytes
> >> + *
> >> + * Trigger one proactive reclaim pass on @memcg, similar to a write to
> >> + * the memory.reclaim cgroup file: pages are reclaimed according to the
> >> + * cgroup's own swappiness setting and swap is allowed. Note that,
> >> + * unlike memory.reclaim, this does not retry until @size is reached;
> >> + * callers can invoke it again if needed.
> >> + *
> >> + * The reclaim runs with GFP_KERNEL, so this function must not be called
> >> + * from a context that holds a filesystem lock (e.g. an LSM hook invoked
> >> + * with inode_lock held): the reclaim path may enter filesystem shrinkers
> >> + * and deadlock trying to reacquire the lock. Contexts that set
> >> + * PF_MEMALLOC_NOFS/NOIO are handled by the gfp context inheritance.
> >> + *
> >> + * Return:
> >> + *   The amount of memory actually reclaimed, in bytes (rounded to full
> >> + *   pages), or 0 if @size is smaller than a page or the calling task is
> >> + *   already in a reclaim/freeing context (PF_MEMALLOC).
> >> + */
> >> +__bpf_kfunc unsigned long bpf_proactive_reclaim(struct mem_cgroup *memcg,
> >> +                                               unsigned long size)
> >> +{
> >> +       return bpf_proactive_reclaim_pages(memcg, size / PAGE_SIZE, NULL);
> >> +}
> >> +
> >> +/**
> >> + * bpf_proactive_reclaim_swappiness - proactively reclaim memory from a
> >> + *                                    memory cgroup with an explicit
> >> + *                                    swappiness
> >> + * @memcg:      the target memory cgroup to reclaim from
> >> + * @size:       the amount of memory to reclaim, in bytes
> >> + * @swappiness: swappiness override for this reclaim pass
> >> + *
> >> + * Same as bpf_proactive_reclaim(), except that the anon/file reclaim
> >> + * balance is controlled by @swappiness instead of the cgroup's
> >> + * swappiness setting. Valid values are [MIN_SWAPPINESS, MAX_SWAPPINESS]
> >> + * and SWAPPINESS_ANON_ONLY, which restricts reclaim to anon folios.
> >> + *
> >> + * Return:
> >> + *   The amount of memory actually reclaimed, in bytes (rounded to full
> >> + *   pages), (unsigned long)-1 if @swappiness is out of range, or 0 if
> >> + *   @size is smaller than a page or the calling task is already in a
> >> + *   reclaim/freeing context (PF_MEMALLOC).
> >> + */
> >> +__bpf_kfunc unsigned long
> >> +bpf_proactive_reclaim_swappiness(struct mem_cgroup *memcg, unsigned long size,
> >> +                                int swappiness)
> >> +{
> >> +       if (swappiness < MIN_SWAPPINESS || swappiness > SWAPPINESS_ANON_ONLY)
> >> +               return (unsigned long)-1;
> >> +
> >> +       return bpf_proactive_reclaim_pages(memcg, size / PAGE_SIZE,
> >> +                                          &swappiness);
> >> +}
> >
> > I haven't followed previous discussion, so I apologize if this was
> > discussed, but if not, isn't it a bit an overkill to have second
> > variant just to provide optional swappiness? Valid range of swappinees
> > seems to be non-negative [0, 200], that special ANON is 201, so why
> > can't we defined that <0 swappiness just means no swappiness was
> > provided and get away with just one kfunc?
> >
> 
> I think my understanding of Shakeel's suggestion was that we only add one
> bpf_proactive_reclaim() for now, if swappiness parameter is necessary we can
> introduce the second API later. But given it seems Hui wants to add both, I
> think it would make sense to do what Andrii said and just introduce one kfunc
> with the swappiness parameter now.
> 
> But Hui, please wait for Shakeel to comment before respinning again.

Yes let's go with Andri's suggestion.

However Hui, don't respin the series. Let me go through it first. I am not happy
with the amount of text and comments added to the patches.

> 
> >> +
> >>  __bpf_kfunc_end_defs();
> >>
> >>  BTF_KFUNCS_START(bpf_memcontrol_kfuncs)
> >> @@ -172,6 +287,9 @@ BTF_ID_FLAGS(func, bpf_mem_cgroup_usage)
> >>  BTF_ID_FLAGS(func, bpf_mem_cgroup_page_state)
> >>  BTF_ID_FLAGS(func, bpf_mem_cgroup_flush_stats, KF_SLEEPABLE)
> >>
> >> +BTF_ID_FLAGS(func, bpf_proactive_reclaim, KF_SLEEPABLE)
> >> +BTF_ID_FLAGS(func, bpf_proactive_reclaim_swappiness, KF_SLEEPABLE)
> >> +
> >>  BTF_KFUNCS_END(bpf_memcontrol_kfuncs)
> >>
> >>  static const struct btf_kfunc_id_set bpf_memcontrol_kfunc_set = {
> >> --
> >> 2.53.0
> >>
> 

[PATCH bpf-next v4 2/2] selftests/bpf: add memcg async reclaim test

[Hui Zhu]

From: Hui Zhu <zhuhui@kylinos.cn>

Add memcg_async_reclaim selftest that verifies BPF-driven async
proactive reclaim can mitigate refault-induced slowdown under memory
pressure.

The test creates a parent cgroup with a fixed memory.max, and two
child cgroups (high/low) under it. Both children concurrently write
and repeatedly read-fault a file larger than the shared limit. A BPF
program monitors the "high" cgroup's WORKINGSET_REFAULT_FILE stat via
a periodic timer, and when it detects refault growth beyond a
threshold, triggers async reclaim on the "low" cgroup using
bpf_proactive_reclaim(), expecting the "high" cgroup's workload to
finish faster than without such reclaim. The reclaim work is queued
asynchronously via bpf_wq.

Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
---
 .../bpf/prog_tests/memcg_async_reclaim.c      | 480 ++++++++++++++++++
 .../selftests/bpf/progs/memcg_async_reclaim.c | 181 +++++++
 2 files changed, 661 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/memcg_async_reclaim.c
 create mode 100644 tools/testing/selftests/bpf/progs/memcg_async_reclaim.c

diff --git a/tools/testing/selftests/bpf/prog_tests/memcg_async_reclaim.c b/tools/testing/selftests/bpf/prog_tests/memcg_async_reclaim.c
new file mode 100644
index 000000000000..ece7bceaab3c
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/memcg_async_reclaim.c
@@ -0,0 +1,480 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Memory controller eBPF async reclaim test
+ */
+
+#include <test_progs.h>
+#include <sys/mman.h>
+#include <sys/stat.h>
+#include <sys/vfs.h>
+#include <sys/wait.h>
+#include <fcntl.h>
+#include <signal.h>
+#include <time.h>
+#include <unistd.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <limits.h>
+#include <linux/magic.h>
+
+#include "cgroup_helpers.h"
+
+struct bpf_args_s {
+	u64 high_cgroup_id;
+	u64 low_cgroup_id;
+	u64 event_delta_threshold;
+	u64 check_ns;
+};
+
+#include "memcg_async_reclaim.skel.h"
+
+#define FILE_SIZE (32 * 1024 * 1024ul)
+#define BUFFER_SIZE (4096)
+#define CG_LIMIT (32 * 1024 * 1024ul)
+#define READ_TIMES 50
+
+#define CG_DIR "/memcg_async_reclaim"
+#define CG_HIGH_DIR CG_DIR "/high"
+#define CG_LOW_DIR CG_DIR "/low"
+
+#define CHECK_PERIOD_NS (2 * 1000 * 1000ull)
+#define EVENT_DELTA_THRESHOLD 1
+
+/*
+ * The workload files must sit on a regular filesystem: with swap
+ * disabled for the cgroup, tmpfs/ramfs pages are unevictable and would
+ * OOM the cgroup instead of exercising reclaim; they are also charged
+ * as anonymous memory, so they never raise the WORKINGSET_REFAULT_FILE
+ * events the BPF program monitors. Fall back to the current directory
+ * when /tmp is backed by such a filesystem.
+ */
+static const char *workload_files_dir(void)
+{
+	struct statfs st;
+
+	if (!statfs("/tmp", &st) &&
+	    (st.f_type == TMPFS_MAGIC || st.f_type == RAMFS_MAGIC))
+		return ".";
+	return "/tmp";
+}
+
+/*
+ * The workload children run after test_progs hijacked stdio, so
+ * anything they print is lost with their private copy of the hijacked
+ * buffer. The exit status is the only diagnostics channel that reaches
+ * the parent, so each failing step gets its own code.
+ */
+enum child_exit_code {
+	CHILD_EXIT_OK = 0,
+	CHILD_EXIT_JOIN_CGROUP,
+	CHILD_EXIT_WRITE_FILE,
+	CHILD_EXIT_READ_FILE,
+	CHILD_EXIT_TIME_FILE,
+};
+
+static const char *child_exit_str(int code)
+{
+	switch (code) {
+	case CHILD_EXIT_OK:
+		return "success";
+	case CHILD_EXIT_JOIN_CGROUP:
+		return "join cgroup";
+	case CHILD_EXIT_WRITE_FILE:
+		return "write data file";
+	case CHILD_EXIT_READ_FILE:
+		return "read data file";
+	case CHILD_EXIT_TIME_FILE:
+		return "write time file";
+	default:
+		return "unknown";
+	}
+}
+
+static int setup_high_low_cgroups(u64 *high_cgroup_id, u64 *low_cgroup_id)
+{
+	int ret;
+	char limit_buf[20];
+
+	ret = setup_cgroup_environment();
+	if (!ASSERT_OK(ret, "setup_cgroup_environment"))
+		goto cleanup;
+
+	ret = create_and_get_cgroup(CG_DIR);
+	if (!ASSERT_GE(ret, 0, "create_and_get_cgroup " CG_DIR))
+		goto cleanup;
+	close(ret);
+
+	ret = enable_controllers(CG_DIR, "memory");
+	if (!ASSERT_OK(ret, "enable_controllers"))
+		goto cleanup;
+
+	snprintf(limit_buf, sizeof(limit_buf), "%lu", CG_LIMIT);
+	ret = write_cgroup_file(CG_DIR, "memory.max", limit_buf);
+	if (!ASSERT_OK(ret, "write_cgroup_file memory.max"))
+		goto cleanup;
+
+	/*
+	 * Keep the workloads from swapping out. With CONFIG_SWAP=n the
+	 * memory.swap.max file does not exist, and no swap can happen
+	 * anyway, so skip the write.
+	 */
+	if (!access("/proc/swaps", F_OK)) {
+		ret = write_cgroup_file(CG_DIR, "memory.swap.max", "0");
+		if (!ASSERT_OK(ret, "write_cgroup_file memory.swap.max"))
+			goto cleanup;
+	}
+
+	ret = create_and_get_cgroup(CG_HIGH_DIR);
+	if (!ASSERT_GE(ret, 0, "create_and_get_cgroup " CG_HIGH_DIR))
+		goto cleanup;
+	close(ret);
+
+	*high_cgroup_id = get_cgroup_id(CG_HIGH_DIR);
+	if (!ASSERT_GT(*high_cgroup_id, 0, "get_cgroup_id"))
+		goto cleanup;
+
+	ret = create_and_get_cgroup(CG_LOW_DIR);
+	if (!ASSERT_GE(ret, 0, "create_and_get_cgroup " CG_LOW_DIR))
+		goto cleanup;
+	close(ret);
+
+	*low_cgroup_id = get_cgroup_id(CG_LOW_DIR);
+	if (!ASSERT_GT(*low_cgroup_id, 0, "get_cgroup_id"))
+		goto cleanup;
+
+	return 0;
+
+cleanup:
+	cleanup_cgroup_environment();
+	return -1;
+}
+
+static int write_file(const char *filename)
+{
+	int ret = -1;
+	size_t written = 0;
+	char *buffer;
+	FILE *fp;
+
+	fp = fopen(filename, "wb");
+	if (!fp)
+		goto out;
+
+	buffer = malloc(BUFFER_SIZE);
+	if (!buffer)
+		goto cleanup_fp;
+
+	memset(buffer, 'A', BUFFER_SIZE);
+
+	while (written < FILE_SIZE) {
+		size_t to_write = FILE_SIZE - written < BUFFER_SIZE ?
+				  FILE_SIZE - written : BUFFER_SIZE;
+
+		if (fwrite(buffer, 1, to_write, fp) != to_write)
+			goto cleanup;
+		written += to_write;
+	}
+
+	ret = 0;
+cleanup:
+	free(buffer);
+cleanup_fp:
+	fclose(fp);
+out:
+	return ret;
+}
+
+static int read_file(const char *filename, int iterations)
+{
+	int ret = -1;
+	long page_size = sysconf(_SC_PAGESIZE);
+	char *map;
+	size_t i;
+	int fd;
+	struct stat sb;
+
+	fd = open(filename, O_RDONLY);
+	if (fd == -1)
+		goto out;
+
+	if (fstat(fd, &sb) == -1)
+		goto cleanup_fd;
+
+	if (sb.st_size != FILE_SIZE) {
+		fprintf(stderr, "File size mismatch: expected %lu, got %lu\n",
+			(unsigned long)FILE_SIZE, (unsigned long)sb.st_size);
+		goto cleanup_fd;
+	}
+
+	map = mmap(NULL, FILE_SIZE, PROT_READ, MAP_PRIVATE, fd, 0);
+	if (map == MAP_FAILED)
+		goto cleanup_fd;
+
+	for (int iter = 0; iter < iterations; iter++) {
+		for (i = 0; i < FILE_SIZE; i += page_size) {
+			/* access a byte to trigger page fault */
+			volatile char v = map[i];
+			(void)v;
+		}
+	}
+
+	if (munmap(map, FILE_SIZE) == -1)
+		goto cleanup_fd;
+
+	ret = 0;
+
+cleanup_fd:
+	close(fd);
+out:
+	return ret;
+}
+
+static int real_test_child_work(const char *cgroup_path, char *data_filename,
+				char *time_filename, int read_times)
+{
+	struct timespec start, end;
+	double elapsed;
+	FILE *fp;
+
+	if (join_parent_cgroup(cgroup_path))
+		return CHILD_EXIT_JOIN_CGROUP;
+
+	clock_gettime(CLOCK_MONOTONIC, &start);
+
+	if (write_file(data_filename))
+		return CHILD_EXIT_WRITE_FILE;
+
+	if (read_file(data_filename, read_times))
+		return CHILD_EXIT_READ_FILE;
+
+	clock_gettime(CLOCK_MONOTONIC, &end);
+
+	if (!time_filename)
+		return CHILD_EXIT_OK;
+
+	elapsed = (end.tv_sec - start.tv_sec) +
+		  (end.tv_nsec - start.tv_nsec) / 1000000000.0;
+	printf("%.6f\n", elapsed);
+
+	fp = fopen(time_filename, "w");
+	if (!fp)
+		return CHILD_EXIT_TIME_FILE;
+	fprintf(fp, "%.6f", elapsed);
+	fclose(fp);
+
+	return CHILD_EXIT_OK;
+}
+
+static int get_time(char *time_filename, double *time)
+{
+	int ret = -1;
+	FILE *fp;
+	char buf[64];
+
+	fp = fopen(time_filename, "r");
+	if (!ASSERT_OK_PTR(fp, "fopen"))
+		goto out;
+
+	if (!ASSERT_OK_PTR(fgets(buf, sizeof(buf), fp), "fgets"))
+		goto cleanup;
+
+	if (sscanf(buf, "%lf", time) != 1) {
+		PRINT_FAIL("sscanf %s", buf);
+		goto cleanup;
+	}
+
+	ret = 0;
+cleanup:
+	fclose(fp);
+out:
+	return ret;
+}
+
+static int
+run_high_low_workload(double *high_elapsed, double *low_elapsed, int read_times)
+{
+	char high_data_file[PATH_MAX];
+	char low_data_file[PATH_MAX];
+	char high_time_file[PATH_MAX];
+	char low_time_file[PATH_MAX];
+	const char *dir = workload_files_dir();
+	pid_t high_pid = -1, low_pid = -1;
+	pid_t wait_ret;
+	int fd, status;
+	int ret = -1;
+
+	snprintf(high_data_file, sizeof(high_data_file),
+		 "%s/memcg_async_high_data_XXXXXX", dir);
+	snprintf(low_data_file, sizeof(low_data_file),
+		 "%s/memcg_async_low_data_XXXXXX", dir);
+	snprintf(high_time_file, sizeof(high_time_file),
+		 "%s/memcg_async_high_time_XXXXXX", dir);
+	snprintf(low_time_file, sizeof(low_time_file),
+		 "%s/memcg_async_low_time_XXXXXX", dir);
+
+	fd = mkstemp(high_data_file);
+	if (!ASSERT_GE(fd, 0, "mkstemp"))
+		goto cleanup;
+	close(fd);
+
+	fd = mkstemp(low_data_file);
+	if (!ASSERT_GE(fd, 0, "mkstemp"))
+		goto cleanup;
+	close(fd);
+
+	fd = mkstemp(high_time_file);
+	if (!ASSERT_GE(fd, 0, "mkstemp"))
+		goto cleanup;
+	close(fd);
+
+	fd = mkstemp(low_time_file);
+	if (!ASSERT_GE(fd, 0, "mkstemp"))
+		goto cleanup;
+	close(fd);
+
+	low_pid = fork();
+	if (!ASSERT_GE(low_pid, 0, "fork low"))
+		goto cleanup;
+	if (low_pid == 0)
+		_exit(real_test_child_work(CG_LOW_DIR, low_data_file,
+					  low_time_file, read_times));
+
+	high_pid = fork();
+	if (!ASSERT_GE(high_pid, 0, "fork high"))
+		goto cleanup;
+	if (high_pid == 0)
+		_exit(real_test_child_work(CG_HIGH_DIR, high_data_file,
+					  high_time_file, read_times));
+
+	wait_ret = waitpid(low_pid, &status, 0);
+	if (!ASSERT_GT(wait_ret, 0, "low waitpid"))
+		goto cleanup;
+	/*
+	 * The child has been reaped and its PID can already be reused,
+	 * so mark it to keep cleanup from signaling an unrelated process.
+	 */
+	low_pid = -1;
+	if (!ASSERT_TRUE(WIFEXITED(status), "low exited"))
+		goto cleanup;
+	if (WEXITSTATUS(status) != CHILD_EXIT_OK) {
+		PRINT_FAIL("low child failed at: %s (exit status %d)",
+			   child_exit_str(WEXITSTATUS(status)),
+			   WEXITSTATUS(status));
+		goto cleanup;
+	}
+
+	wait_ret = waitpid(high_pid, &status, 0);
+	if (!ASSERT_GT(wait_ret, 0, "high waitpid"))
+		goto cleanup;
+	/* Same as above: the reaped PID must not be signaled again. */
+	high_pid = -1;
+	if (!ASSERT_TRUE(WIFEXITED(status), "high exited"))
+		goto cleanup;
+	if (WEXITSTATUS(status) != CHILD_EXIT_OK) {
+		PRINT_FAIL("high child failed at: %s (exit status %d)",
+			   child_exit_str(WEXITSTATUS(status)),
+			   WEXITSTATUS(status));
+		goto cleanup;
+	}
+
+	if (get_time(high_time_file, high_elapsed))
+		goto cleanup;
+	if (get_time(low_time_file, low_elapsed))
+		goto cleanup;
+
+	ret = 0;
+
+cleanup:
+	/* On failure, make sure no child process is left behind */
+	if (ret) {
+		if (high_pid > 0) {
+			kill(high_pid, SIGKILL);
+			(void)waitpid(high_pid, NULL, 0);
+		}
+		if (low_pid > 0) {
+			kill(low_pid, SIGKILL);
+			(void)waitpid(low_pid, NULL, 0);
+		}
+	}
+	unlink(low_time_file);
+	unlink(high_time_file);
+	unlink(low_data_file);
+	unlink(high_data_file);
+	return ret;
+}
+
+static int
+setup_bpf(u64 high_cgroup_id, u64 low_cgroup_id,
+	  struct memcg_async_reclaim **skel_ptr)
+{
+	struct memcg_async_reclaim *skel;
+	struct bpf_args_s bpf_args = {
+		.high_cgroup_id = high_cgroup_id,
+		.low_cgroup_id = low_cgroup_id,
+		.event_delta_threshold = EVENT_DELTA_THRESHOLD,
+		.check_ns = CHECK_PERIOD_NS,
+	};
+	LIBBPF_OPTS(bpf_test_run_opts, run_opts,
+		.ctx_in = &bpf_args,
+		.ctx_size_in = sizeof(bpf_args));
+	int prog_init_fd, err;
+
+	skel = memcg_async_reclaim__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "memcg_async_reclaim__open_and_load"))
+		return -1;
+
+	prog_init_fd = bpf_program__fd(skel->progs.wq_prog_init);
+
+	err = bpf_prog_test_run_opts(prog_init_fd, &run_opts);
+	if (!ASSERT_OK(err, "bpf_prog_test_run_opts"))
+		goto error_out;
+	if (!ASSERT_EQ(run_opts.retval, 0, "prog_init retval"))
+		goto error_out;
+
+	*skel_ptr = skel;
+	return 0;
+
+error_out:
+	memcg_async_reclaim__destroy(skel);
+	return -1;
+}
+
+void test_memcg_wq_async_reclaim(void)
+{
+	u64 high_cgroup_id, low_cgroup_id;
+	int err;
+	double high_time = 0.0, low_time = 0.0;
+	struct memcg_async_reclaim *skel = NULL;
+
+	err = setup_high_low_cgroups(&high_cgroup_id, &low_cgroup_id);
+	if (!ASSERT_OK(err, "setup_high_low_cgroups reclaim"))
+		return;
+
+	err = setup_bpf(high_cgroup_id, low_cgroup_id, &skel);
+	if (!ASSERT_OK(err, "setup_bpf"))
+		goto out;
+
+	err = run_high_low_workload(&high_time, &low_time, READ_TIMES);
+	if (!ASSERT_OK(err, "run_high_low_workload reclaim"))
+		goto out;
+
+	/*
+	 * The timing comparison below alone cannot distinguish a working
+	 * reclaim from a no-op one, so require that the BPF program
+	 * actually reclaimed memory from the low cgroup.
+	 */
+	if (!ASSERT_GT(skel->bss->reclaim_calls, 0, "reclaim_calls"))
+		goto out;
+	if (!ASSERT_GT(skel->bss->reclaimed_bytes, 0, "reclaimed_bytes"))
+		goto out;
+
+	if (high_time >= low_time)
+		PRINT_FAIL("high cgroup not improved with async reclaim: high_time=%f low_time=%f",
+			   high_time, low_time);
+
+out:
+	if (skel)
+		memcg_async_reclaim__destroy(skel);
+	cleanup_cgroup_environment();
+}
diff --git a/tools/testing/selftests/bpf/progs/memcg_async_reclaim.c b/tools/testing/selftests/bpf/progs/memcg_async_reclaim.c
new file mode 100644
index 000000000000..f69bfac62939
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/memcg_async_reclaim.c
@@ -0,0 +1,181 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include "vmlinux.h"
+#include "bpf_experimental.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include <bpf/bpf_core_read.h>
+
+#define CLOCK_MONOTONIC_ID	1
+#define PAGE_SIZE		4096UL
+#define RECLAIM_SIZE		(32 * PAGE_SIZE)
+#define RECLAIM_MAX_ITER	32
+
+struct bpf_args_s {
+	u64 high_cgroup_id;
+	u64 low_cgroup_id;
+	u64 event_delta_threshold;
+	u64 check_ns;
+};
+
+struct cgroup_memcg {
+	struct cgroup *cgrp;
+	struct mem_cgroup *memcg;
+};
+
+static u64 wq_high_cgroup_id;
+static u64 wq_low_cgroup_id;
+
+/*
+ * Statistics exposed to userspace through .bss, so the test can verify
+ * that reclaim actually happened instead of relying on timing alone.
+ */
+u64 reclaim_calls;
+u64 reclaimed_bytes;
+
+static int get_cgroup_memcg_from_id(u64 cgroup_id, struct cgroup_memcg *cm)
+{
+	cm->cgrp = bpf_cgroup_from_id(cgroup_id);
+	if (!cm->cgrp)
+		return -1;
+
+	cm->memcg = bpf_get_mem_cgroup(&cm->cgrp->self);
+	if (!cm->memcg) {
+		bpf_cgroup_release(cm->cgrp);
+		return -1;
+	}
+
+	return 0;
+}
+
+static void put_cgroup_memcg(struct cgroup_memcg *cm)
+{
+	bpf_put_mem_cgroup(cm->memcg);
+	bpf_cgroup_release(cm->cgrp);
+}
+
+static int get_cgroup_event(u64 cgroup_id, u64 *val)
+{
+	struct cgroup_memcg cm;
+
+	if (get_cgroup_memcg_from_id(cgroup_id, &cm))
+		return -1;
+	bpf_mem_cgroup_flush_stats(cm.memcg);
+	*val = bpf_mem_cgroup_page_state(cm.memcg,
+		bpf_core_enum_value(enum node_stat_item,
+				    WORKINGSET_REFAULT_FILE));
+	put_cgroup_memcg(&cm);
+
+	return 0;
+}
+
+static bool
+should_reclaim_cgroup(u64 cgroup_id, u64 *prev_event, u64 event_delta_threshold)
+{
+	u64 cur, delta;
+
+	if (get_cgroup_event(cgroup_id, &cur))
+		return false;
+
+	delta = cur - *prev_event;
+	*prev_event = cur;
+
+	return delta >= event_delta_threshold;
+}
+
+static int reclaim_cgroup(u64 cgroup_id)
+{
+	struct cgroup_memcg cm;
+	int i;
+
+	if (get_cgroup_memcg_from_id(cgroup_id, &cm))
+		return 0;
+
+	reclaim_calls++;
+	for (i = 0; i < RECLAIM_MAX_ITER; i++) {
+		u64 nr = bpf_proactive_reclaim(cm.memcg, RECLAIM_SIZE);
+
+		if (!nr)
+			break;
+		reclaimed_bytes += nr;
+	}
+
+	put_cgroup_memcg(&cm);
+
+	return 0;
+}
+
+struct wq_elem {
+	struct bpf_timer timer;
+	struct bpf_wq work;
+	u64 prev_event;
+	u64 event_delta_threshold;
+	u64 check_ns;
+};
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, __u32);
+	__type(value, struct wq_elem);
+} wq_map SEC(".maps");
+
+static int async_free(void *map, int *key, void *value)
+{
+	struct wq_elem *elem = value;
+
+	if (should_reclaim_cgroup(wq_high_cgroup_id, &elem->prev_event,
+		elem->event_delta_threshold)) {
+		reclaim_cgroup(wq_low_cgroup_id);
+		bpf_wq_start(&elem->work, 0);
+	}
+
+	return 0;
+}
+
+static int wq_timer_cb(void *map, int *key, struct wq_elem *elem)
+{
+	bpf_wq_start(&elem->work, 0);
+	bpf_timer_start(&elem->timer, elem->check_ns, 0);
+
+	return 0;
+}
+
+SEC("syscall")
+int wq_prog_init(struct bpf_args_s *ctx)
+{
+	struct wq_elem *elem;
+	__u32 key = 0;
+	int ret;
+
+	elem = bpf_map_lookup_elem(&wq_map, &key);
+	if (!elem)
+		return -1;
+
+	ret = bpf_wq_init(&elem->work, &wq_map, 0);
+	if (ret)
+		return ret;
+
+	ret = bpf_wq_set_callback(&elem->work, async_free, 0);
+	if (ret)
+		return ret;
+
+	ret = bpf_timer_init(&elem->timer, &wq_map, CLOCK_MONOTONIC_ID);
+	if (ret)
+		return ret;
+
+	ret = bpf_timer_set_callback(&elem->timer, wq_timer_cb);
+	if (ret)
+		return ret;
+
+	elem->prev_event = 0;
+	elem->event_delta_threshold = ctx->event_delta_threshold;
+	elem->check_ns = ctx->check_ns;
+
+	wq_high_cgroup_id = ctx->high_cgroup_id;
+	wq_low_cgroup_id = ctx->low_cgroup_id;
+
+	return bpf_timer_start(&elem->timer, elem->check_ns, 0);
+}
+
+char LICENSE[] SEC("license") = "GPL";
-- 
2.53.0

Re: [PATCH bpf-next v4 2/2] selftests/bpf: add memcg async reclaim test

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 6366 bytes --]

> diff --git tools/testing/selftests/bpf/prog_tests/memcg_async_reclaim.c tools/testing/selftests/bpf/prog_tests/memcg_async_reclaim.c
> new file mode 100644
> index 000000000000..ece7bceaab3c
> --- /dev/null
> +++ tools/testing/selftests/bpf/prog_tests/memcg_async_reclaim.c
> @@ -0,0 +1,480 @@

[ ... ]

> +/*
> + * The workload files must sit on a regular filesystem: with swap
> + * disabled for the cgroup, tmpfs/ramfs pages are unevictable and would
> + * OOM the cgroup instead of exercising reclaim; they are also charged
> + * as anonymous memory, so they never raise the WORKINGSET_REFAULT_FILE
> + * events the BPF program monitors. Fall back to the current directory
> + * when /tmp is backed by such a filesystem.
> + */

The comment states that tmpfs/ramfs pages "are also charged as anonymous
memory." Is this accurate?

Looking at the page cache charge path (mm/filemap.c __filemap_add_folio()),
tmpfs/shmem folios go through folio_memcg_charge() and land in
NR_FILE_PAGES/NR_SHMEM, which shows up in memory.stat's file/shmem
counters, not as anonymous memory.

The reason they don't trigger WORKINGSET_REFAULT_FILE is different:
shmem folios are swap-backed, so folio_is_file_lru() returns false
(include/linux/mm_inline.h), and workingset_refault() accounts them at
WORKINGSET_REFAULT_ANON instead (mm/workingset.c).

The conclusion the comment draws is correct, but the stated reason is
not. For a patch aimed at the mm/memcg audience, could the wording be
adjusted to reflect that they sit on the anon LRU because they are
swap-backed?

> +static const char *workload_files_dir(void)
> +{
> +	struct statfs st;
> +
> +	if (!statfs("/tmp", &st) &&
> +	    (st.f_type == TMPFS_MAGIC || st.f_type == RAMFS_MAGIC))
> +		return ".";
> +	return "/tmp";
> +}

[ ... ]

> +static int setup_high_low_cgroups(u64 *high_cgroup_id, u64 *low_cgroup_id)
> +{
> +	int ret;
> +	char limit_buf[20];
> +
> +	ret = setup_cgroup_environment();
> +	if (!ASSERT_OK(ret, "setup_cgroup_environment"))
> +		goto cleanup;
> +
> +	ret = create_and_get_cgroup(CG_DIR);
> +	if (!ASSERT_GE(ret, 0, "create_and_get_cgroup " CG_DIR))
> +		goto cleanup;
> +	close(ret);
> +
> +	ret = enable_controllers(CG_DIR, "memory");
> +	if (!ASSERT_OK(ret, "enable_controllers"))
> +		goto cleanup;

A subsystem pattern flags this as potentially concerning: the test
treats an absent memory controller as a test failure rather than a skip.

enable_controllers() ends up in __enable_controllers() which writes to
cgroup.subtree_control and returns 1 when that write fails. On a kernel
built with CONFIG_MEMCG=n or booted with cgroup_disable=memory, the
memory controller does not exist, the write is rejected, and the test
reports FAIL.

The feature being tested is CONFIG_MEMCG-gated: mm/Makefile builds
bpf_memcontrol.o as obj-$(CONFIG_MEMCG), so on such a kernel there is
nothing to test. Note that 123 other prog_tests use test__skip() with
a reason string for missing capabilities.

However, CONFIG_MEMCG=y is present in every per-arch BPF selftest config
(config.x86_64, config.aarch64, config.s390x, etc.), so BPF CI will
never hit this path. It is only reachable with a hand-rolled kernel
config that omits it. Given this context, is the ASSERT_OK() treatment
acceptable?

[ ... ]

> +void test_memcg_wq_async_reclaim(void)
> +{
> +	u64 high_cgroup_id, low_cgroup_id;
> +	int err;
> +	double high_time = 0.0, low_time = 0.0;
> +	struct memcg_async_reclaim *skel = NULL;
> +
> +	err = setup_high_low_cgroups(&high_cgroup_id, &low_cgroup_id);
> +	if (!ASSERT_OK(err, "setup_high_low_cgroups reclaim"))
> +		return;
> +
> +	err = setup_bpf(high_cgroup_id, low_cgroup_id, &skel);
> +	if (!ASSERT_OK(err, "setup_bpf"))
> +		goto out;
> +
> +	err = run_high_low_workload(&high_time, &low_time, READ_TIMES);
> +	if (!ASSERT_OK(err, "run_high_low_workload reclaim"))
> +		goto out;
> +
> +	/*
> +	 * The timing comparison below alone cannot distinguish a working
> +	 * reclaim from a no-op one, so require that the BPF program
> +	 * actually reclaimed memory from the low cgroup.
> +	 */
> +	if (!ASSERT_GT(skel->bss->reclaim_calls, 0, "reclaim_calls"))
> +		goto out;
> +	if (!ASSERT_GT(skel->bss->reclaimed_bytes, 0, "reclaimed_bytes"))
> +		goto out;
> +
> +	if (high_time >= low_time)
> +		PRINT_FAIL("high cgroup not improved with async reclaim: high_time=%f low_time=%f",
> +			   high_time, low_time);

Does the zero-margin wall-clock comparison make the test brittle?

The test's final pass/fail criterion is a single-run timing comparison
between two symmetric workloads with no baseline. PRINT_FAIL() expands to
test__fail(), so this marks the whole test FAILED whenever high_time >=
low_time, including on a tie or a 1-microsecond inversion.

The two children are symmetric by construction: run_high_low_workload()
forks them within microseconds of each other and both run the identical
real_test_child_work() body (write a 32 MB file, then read-fault it 50
times) inside sibling cgroups under one shared memory.max of 32 MB.

Which one finishes first is decided by page-reclaim ordering, disk service
order, CPU scheduling, and host load. The test pins only memory.max and
memory.swap.max but does not pin the reclaim implementation (classic LRU
vs MGLRU, selected at runtime), the dirty-ratio sysctls, the CPU count, or
the backing device.

On a machine where the BPF-driven reclaim of the low cgroup does not
translate into a measurable ordering advantage, a fully correct kernel
reports FAIL.

The comment just above acknowledges that the timing signal is weak. The two
ASSERT_GT checks on reclaim_calls and reclaimed_bytes are deterministic and
verify that bpf_proactive_reclaim() actually ran. The timing comparison
adds no deterministic coverage on top of them, only a nondeterministic
failure mode.

For comparison, prog_tests/test_overhead.c is the only other prog_test that
uses clock_gettime(CLOCK_MONOTONIC, ...), and it only prints its
measurements. Could the comparison either require a substantial margin, be
bounded with retry logic, or report the ratio informationally with
fprintf(stdout, ...) instead of PRINT_FAIL()?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/32339450346